#vDM30in30 - 11


I had some time yesterday, so I finally had the chance to start watching some of the puppet-conf presentations that I did not get a chance to attend personally.. As a reminder, you can find the slides and videos here.

I may morph this into a more 'static' page on my site, and dive in with some thoughts about certain presentations..


Attended -

A Roadmap for a Platform: Mixing Metaphors for Fun and Profit – Eric Sorenson, Puppet (videos) (slides)
Proof of Concept to 30K+ Hosts with Puppet - Petersen Allen, Salesforce (videos) (slides)
Cloud, Containers & the Impact on IT - Jeffrey Snover, Microsoft (videos) (slides)

Puppet Troubleshooting - Thomas Uphill (videos) (slides)
Turning Pain Into Gain - A Unit Testing Story - Nadeem Ahmad & Jordan Moldow (videos)(slides)
Watching the Puppet Show – Sean Porter, Heavy Water Operations (videos) (slides)
Avoiding Toxic Technical Debt Derivatives – R. Tyler Croy, CloudBees, Inc. (videos) (slides)

Using HashiCorp's Vault With Puppet – Seth Vargo, HashiCorp (videos) (slides)
Running Puppet Software in Docker Containers – Gareth Rushgrove, Puppet (videos) (slides)
DevOps Where You Wouldn't Have Expected – Thomas Limoncelli, StackOverflow.com (videos) (slides)
The Future of Testing Puppet Code – Gareth Rushgrove, Puppet (videos) (slides)

Watched -

Service Discovery and Puppet – Marc Cluet, Ukon Cherry (video) (slides)
Enjoying the Journey from Puppet 3.x to 4.x – Rob Nelson, AT&T (videos) (slides)
Collaboration and Empowerment: Driving Change in Infrastructure with Culture – Martin Jackson, Walmart (videos) (slides)
A Look at Looking in the Mirror Actionable Retrospectives - J. Paul Reed (videos) (slides)
External Data in Puppet 4 – R.I. Pienaar (videos) (slides)
Scaling Puppet on AWS ECS with Terraform and Docker – Maxime Visonneau, Trainline (videos) (slides)
Case Study: Puppets in the Government – Kathy Lee (co-author: Glenn Bailey) (videos) (slides)

Want to Watch -

The Truth, Nothing but the Truth: Why Type Systems are Important to Configuration Management – Henrik Lindberg, Puppet (videos) (slides)

Scaling Puppet and Puppet Culture at GitHub - Kevin Paulisse (videos) (slides)
Implementing Puppet within a Complex Enterprise – Jerry Caupain, KPN (videos) (slides)
Moving from Exec to Types and Providers – Martin Alfke, example42 GmbH (videos) (slides)
A Year in Open Source: Automated Compliance With Puppet – Trevor Vaughan, Onyx Point, Inc. (videos) (slides)
Automating Datastore Fleets with Puppet – Joseph Lynch, Yelp (videos) (slides)
Closing the Loop: Direct Change Control with Puppet – Nick Lewis, Puppet (videos) (slides)
Continuous Delivery and DevOps with Jenkins and Puppet Enterprise – Carl Caum, Puppet & Brian Dawson, CloudBees (videos) (slides)
Debugging Diversity – Anjuan Simmons, Assemble Systems (videos) (slides)
Direct Puppet and Application Management for the Puppet Platform – Ryan Coleman, Puppet (videos) (slides)
Puppet 4.x: The Low WAT-tage Edition – Nick Fagerlund, Puppet (videos) (slides)
Puppet Best Practices: Roles & Profiles – Gary Larizza, Puppet (videos) (slides)
Puppet Design Patterns - Lessons From the Gang of Four - David Danzilio (videos) (slides)
Device-Based Modules: Making Them as Simple as a Light Switch – TP Honey, Puppet (videos) (slides)
There is No “I” in DevOps – Bart Driscoll, Dell EMC (videos) (slides)
Writing Custom Types to Manage Web-Based Applications – Tim Cinel, Atlassian (videos) (slides)
Successful Puppet Implementation in Large Organizations – James Sweeny, Puppet (videos) (slides)
Best Practices for Puppet in the Cloud – Randall Hunt, Amazon & Andrew Popp, Servicechannel.com (videos) (slides)
Pulling the Strings to Containerize Your Life - Scott Coulton (videos) (slides)

docker-flow-swarm

#vDM30in30 - 9

I was reading through this week's Docker Weekly, and I saw something that caught my eye... docker-flow.  There was a great video demo about using it.

It serves as a similar role to Registrator, tied to HAProxy, tied with consul template, but directly compatible with docker swarm, and it can bridge to the docker network that you have created to run your swarm containers on.

This looks pretty interesting, and I think I will start using it for my configuration at home, as well as facilitating any demos that I give for our local DevOps Meetup group!

youtube-dl followup - with more docker!

#vDM30in30 - 10

Following up about youtube-dl, I saw in this week's Docker Weekly that someone had a post about running youtube-dl in a container to avoid the messy setup.

https://blog.rphl.io/selfhosted-youtube-downloader-with-docker/

I didn't have to much trouble installing it on my mac, with homebrew, but keeping different applications running in docker (possibly with swarm and docker network bridging/proxying)

youtube-dl to watch youtube offline

#vDM30in30 - 8

youtube-dl -

As you may know, the puppetconf videos, slides, and photos have been released.  I have a friend who is on slow satellite internet with a data-cap, but he is excited about watching the videos.  I have a useful tool installed on my laptop called youtube-dl.  What this allows me to do is point it at the 'playlist' url for all of the videos from puppetconf, and it will download (in parts) all of the videos.

The below commands (and time and bandwidth) are all that I need to pull down all of the videos.  I can resume it if the connection gets interrupted (happened at about 32/94 for me the first day).  

$ mkdir -p ~/puppetconf2016
$ cd ~/puppetconf2016
$ brew install youtube-dl
$ youtube-dl --yes-playlist https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa

After everything is finished downloading, I can transfer them to a USB drive for sharing with friends, and I can watch them while offline.

Hello from the otherside.... (inside the container)

#vDM30in30 - 7

Hello!

I have used ec2 tags to gather information into facter, and I had wanted to duplicate that same kind of logic using docker containers and labels... Unfortunately it seems like labels are 'external' only to the container.
I found several issues which all end up pointing at this 'introspection' issue.
https://github.com/docker/docker/issues/8427

It looks like the only way currently is to have some sort of external service manage that information for you (etcd/consul/mesos/pass it in via environment variables to your container / kubernettes downward-api) OR to mount the docker socket inside your container and go through the json output (not very secure)

Reasons for getting that information seem to range from enabling service discovery (knowing what actual host/IP you are bound to) to knowing what size JVM heap args to set inside a container based on actual container run environment.

It seems like memory information was added with https://github.com/docker/docker/pull/13312/files which allows a readonly mount of the /sys/fs/cgroup location specific to the container...

$ docker run --rm -it --entrypoint /bin/bash puppet/puppet-agent-ubuntu

root@96acc60738a7:/# cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
19025920
root@96acc60738a7:/# exit
exit
$ docker run -m 2048m --rm -it --entrypoint /bin/bash puppet/puppet-agent-ubuntu
WARNING: Your kernel does not support swap limit capabilities, memory limited without swap.
root@dab851113412:/#  cat /sys/fs/cgroup/memory/memory.max_usage_in_bytes
2654208
root@dab851113412:/#

Find non-utf8 characters in your puppet code!

#vDM30in30 - 6

I was checking out the #puppet channel on irc this morning, and someone came in asking for help with the following error..

err: Could not retrieve catalog from remote server: Error 400 on SERVER: (<unknown>): control characters are not allowed at line 1 column 1

I couldn't find anything puppet specific, but I found several posts referencing sidekiq and having that error, and it seemed to be due to the presence of non UTF-8 characters in their files/code.

I found a few answers on stackoverflow for detecting non UTF-8 chars..

http://stackoverflow.com/a/9395552 or http://stackoverflow.com/a/13702856

And to add in the 'search hiera and puppet and erb' files find option.  Here is the 'exclude chars we thing are not in range' command

find $CODEBASE/ -type f \( -name '*.pp' -o -name '*.yaml' -o -name '*.erb' \) | xargs grep --color='auto' -P -n "[\x80-\xFF]"

Here is the 'exclude chars that do not match the expected range' command

find $CODEBASE/ -type f \( -name '*.pp' -o -name '*.yaml' -o -name '*.erb' \) | xargs grep --color='auto' -P -n "[^\x00-\x7F]"

This is something that you could even add as a validation check to your CI system to prevent you from checking in invalid characters to your puppet codebase.

Roles and Profiles - an official example

#vDM30in30 #5

After first learning about the roles and profiles pattern, I spent quite a while searching long and hard for some good examples of the pattern.

Today, I stumbled onto some official docs in the PE 2016.4 pages that give a well thought out example of setting up jenkins masters, and the refactoring process that led them to the profile that they use now.



https://docs.puppet.com/pe/2016.4/r_n_p_intro.html
https://docs.puppet.com/pe/2016.4/r_n_p_full_example.html
https://docs.puppet.com/pe/2016.4/r_n_p_profiles.html