Installing gems in PDK on airgapped/offline system without functional rubygems.org

 I ran into a problem recently installing gems in an offline system... I am not understanding something about how the `pdk bundle` setup creates it's cache directories.

If you try to do a `pdk bundle exec gem install ~/path/to/module.gem`, it will place it's cache files in ~/.gem/ruby, but the PDK expects it's files in ~/.pdk/cache/ruby/

I followed these basic steps to get it 'working' enough...

1. Copy over .gem file and it's dependencies to the offline system to be able to do a local `gem install`
2. Install it using `pdk bundle gem install ~/gems/*`
3. Copy over the installed gems from ~/.gem to ~/.pdk/cache - `cp --no-clobber ~/.gem/ruby ~/.pdk/cache/`
   
4. Update your Gemfile (in your PDK base directory) to include the gems you want to local install
5. `pdk bundle install --local` to install the gems from the gemfile, but source them from the cache that you just manually populated.  

I'd love to know if there is a better way to do this!

rspec-puppet test with array parameter

Today I stumbled onto something that was much harder than I expected it to be while trying to write an rspec-puppet test.

Say you have a resource with a parameter that is an array, and you only want to validate that the array contains an element.

In our case, say that we have the following code somewhere in our puppet manifests

        host { 'foo.local':
          ip => '1.2.3.4',
          host_aliases => [ 'foo1.local', 'foo2.local' ],
        }

It is pretty direct to validate the exact array inside of an rspec-puppet test

    it { should contain_host('foo.local').with_host_aliases( ['foo1.local', 'foo2.local'] ) }

But that requires a very specific 'expected' value to be passed into the matcher.  What if we just wanted to see if the parameter array includes a given value?  

In a conversation about his problem in the #voxpupuli channel on irc@freenode, dev_el_ops mentioned "one of the design problems of rspec-pupet is that it  doesn't expose property values to the regular rspec matchers"

Since we couldn't get the values directly from the matchers, I had to come up with a different way to approach the problem, which is not quite as readable as I would have wanted.

    it {

        host_aliases = catalogue.resource('host', 'foo.local').send(:parameters)[:host_aliases]

        expect(host_aliases).to include('foo1.local')

    }

Lets break down what is happening.  catalogue.resource('host', 'foo.local') will fetch us the catalog's resource object of type 'host' with the title 'foo.local'.  we can then use 'send' to call the private parameters method on the object to get back it's parameter hash, which we then reference the :host_aliases key to retrieve our array.  

Once we have the array (stored in host_aliases), we can then use a simple 'expect' call with an 'include' matcher to check that the given array includes the value we expect.


Here is a gist of the entire example.

#vDM30in30 - 11


I had some time yesterday, so I finally had the chance to start watching some of the puppet-conf presentations that I did not get a chance to attend personally.. As a reminder, you can find the slides and videos here.

I may morph this into a more 'static' page on my site, and dive in with some thoughts about certain presentations..


Attended -

A Roadmap for a Platform: Mixing Metaphors for Fun and Profit – Eric Sorenson, Puppet (videos) (slides)
Proof of Concept to 30K+ Hosts with Puppet - Petersen Allen, Salesforce (videos) (slides)
Cloud, Containers & the Impact on IT - Jeffrey Snover, Microsoft (videos) (slides)

Puppet Troubleshooting - Thomas Uphill (videos) (slides)
Turning Pain Into Gain - A Unit Testing Story - Nadeem Ahmad & Jordan Moldow (videos)(slides)
Watching the Puppet Show – Sean Porter, Heavy Water Operations (videos) (slides)
Avoiding Toxic Technical Debt Derivatives – R. Tyler Croy, CloudBees, Inc. (videos) (slides)

Using HashiCorp's Vault With Puppet – Seth Vargo, HashiCorp (videos) (slides)
Running Puppet Software in Docker Containers – Gareth Rushgrove, Puppet (videos) (slides)
DevOps Where You Wouldn't Have Expected – Thomas Limoncelli, StackOverflow.com (videos) (slides)
The Future of Testing Puppet Code – Gareth Rushgrove, Puppet (videos) (slides)

Watched -

Service Discovery and Puppet – Marc Cluet, Ukon Cherry (video) (slides)
Enjoying the Journey from Puppet 3.x to 4.x – Rob Nelson, AT&T (videos) (slides)
Collaboration and Empowerment: Driving Change in Infrastructure with Culture – Martin Jackson, Walmart (videos) (slides)
A Look at Looking in the Mirror Actionable Retrospectives - J. Paul Reed (videos) (slides)
External Data in Puppet 4 – R.I. Pienaar (videos) (slides)
Scaling Puppet on AWS ECS with Terraform and Docker – Maxime Visonneau, Trainline (videos) (slides)
Case Study: Puppets in the Government – Kathy Lee (co-author: Glenn Bailey) (videos) (slides)

Want to Watch -

The Truth, Nothing but the Truth: Why Type Systems are Important to Configuration Management – Henrik Lindberg, Puppet (videos) (slides)

Scaling Puppet and Puppet Culture at GitHub - Kevin Paulisse (videos) (slides)
Implementing Puppet within a Complex Enterprise – Jerry Caupain, KPN (videos) (slides)
Moving from Exec to Types and Providers – Martin Alfke, example42 GmbH (videos) (slides)
A Year in Open Source: Automated Compliance With Puppet – Trevor Vaughan, Onyx Point, Inc. (videos) (slides)
Automating Datastore Fleets with Puppet – Joseph Lynch, Yelp (videos) (slides)
Closing the Loop: Direct Change Control with Puppet – Nick Lewis, Puppet (videos) (slides)
Continuous Delivery and DevOps with Jenkins and Puppet Enterprise – Carl Caum, Puppet & Brian Dawson, CloudBees (videos) (slides)
Debugging Diversity – Anjuan Simmons, Assemble Systems (videos) (slides)
Direct Puppet and Application Management for the Puppet Platform – Ryan Coleman, Puppet (videos) (slides)
Puppet 4.x: The Low WAT-tage Edition – Nick Fagerlund, Puppet (videos) (slides)
Puppet Best Practices: Roles & Profiles – Gary Larizza, Puppet (videos) (slides)
Puppet Design Patterns - Lessons From the Gang of Four - David Danzilio (videos) (slides)
Device-Based Modules: Making Them as Simple as a Light Switch – TP Honey, Puppet (videos) (slides)
There is No “I” in DevOps – Bart Driscoll, Dell EMC (videos) (slides)
Writing Custom Types to Manage Web-Based Applications – Tim Cinel, Atlassian (videos) (slides)
Successful Puppet Implementation in Large Organizations – James Sweeny, Puppet (videos) (slides)
Best Practices for Puppet in the Cloud – Randall Hunt, Amazon & Andrew Popp, Servicechannel.com (videos) (slides)
Pulling the Strings to Containerize Your Life - Scott Coulton (videos) (slides)

Find non-utf8 characters in your puppet code!

#vDM30in30 - 6

I was checking out the #puppet channel on irc this morning, and someone came in asking for help with the following error..

err: Could not retrieve catalog from remote server: Error 400 on SERVER: (<unknown>): control characters are not allowed at line 1 column 1

I couldn't find anything puppet specific, but I found several posts referencing sidekiq and having that error, and it seemed to be due to the presence of non UTF-8 characters in their files/code.

I found a few answers on stackoverflow for detecting non UTF-8 chars..

http://stackoverflow.com/a/9395552 or http://stackoverflow.com/a/13702856

And to add in the 'search hiera and puppet and erb' files find option.  Here is the 'exclude chars we thing are not in range' command

find $CODEBASE/ -type f \( -name '*.pp' -o -name '*.yaml' -o -name '*.erb' \) | xargs grep --color='auto' -P -n "[\x80-\xFF]"

Here is the 'exclude chars that do not match the expected range' command

find $CODEBASE/ -type f \( -name '*.pp' -o -name '*.yaml' -o -name '*.erb' \) | xargs grep --color='auto' -P -n "[^\x00-\x7F]"

This is something that you could even add as a validation check to your CI system to prevent you from checking in invalid characters to your puppet codebase.

Roles and Profiles - an official example

#vDM30in30 #5

After first learning about the roles and profiles pattern, I spent quite a while searching long and hard for some good examples of the pattern.

Today, I stumbled onto some official docs in the PE 2016.4 pages that give a well thought out example of setting up jenkins masters, and the refactoring process that led them to the profile that they use now.



https://docs.puppet.com/pe/2016.4/r_n_p_intro.html
https://docs.puppet.com/pe/2016.4/r_n_p_full_example.html
https://docs.puppet.com/pe/2016.4/r_n_p_profiles.html

Cyclomatic Complexity - A puppet lint plugin by Danzilio

#vDM30in30 Post #4!

One of the awesome things to come out of the Contributor's summit was David Danzilio and Henrik Lindberg collobrating on an Assignment Branch Prediction algorithm for Puppet code.  David recently released it as a plugin for puppet-lint!

Just shipped a puppet-lint plugin to check code quality metrics. Only implements ABC size right now, more to come! https://t.co/LUbUcbXR06

— David Danzilio (@djdanzilio) October 26, 2016

I recently gave a little micro-demo of the plugin at our local DevOps Meetup.

There are two ways to run the plugin:  gem install it and run the command, or add it to your Gemfile in your module, and it will automatically be added to your rspec lint check.

The problem that I had with demoing the plugin was that the warning/error threshold level could only be set in your Rakefile.  So I had two options... find a sufficintly complex module, or lower the complexity warning level in the Rakefile for an existing module.

Fortunatly, there exists a complex module in the voxpupuli ecosystem, puppet-staging!

$ git clone https://github.com/voxpupuli/puppet-staging

$ cd puppet-staging

$ find ./ -name '*.pp' | xargs puppet-lint | grep 'branch condition'

./manifests/file.pp - WARNING: assignment branch condition size is 76.69 on line 0

./manifests/extract.pp - WARNING: assignment branch condition size is 47.37 on line 0

If you wanted to modify the module to include those checks, and set a lower complexity threshold, you can make changes similar to the below.

$ git diff

diff --git a/Gemfile b/Gemfile

index 0571378..31f4ffa 100644

--- a/Gemfile

+++ b/Gemfile

@@ -22,6 +22,7 @@ group :test do

   gem 'puppet-lint-classes_and_types_beginning_with_digits-check',  :require => false

   gem 'puppet-lint-unquoted_string-check',                          :require => false

   gem 'puppet-lint-variable_contains_upcase',                       :require => false

+  gem 'puppet-lint-metrics-check',                                  :require => false

   gem 'metadata-json-lint',                                         :require => false

   gem 'puppet-blacksmith',                                          :require => false

   gem 'voxpupuli-release',                                          :require => false, :git => 'https://github.com/voxpupuli/voxpupuli-release-gem.git'

diff --git a/Rakefile b/Rakefile

index d00f247..0a00e25 100644

--- a/Rakefile

+++ b/Rakefile

@@ -18,6 +18,7 @@ exclude_paths = %w(

   spec/**/*

 )

 PuppetLint.configuration.ignore_paths = exclude_paths

+PuppetLint.configuration.metrics_abc_warning = 1

 PuppetSyntax.exclude_paths = exclude_paths

 desc 'Run acceptance tests'

Which will result in the following output when you run rake lint

$ bundle exec rake lint

examples/deploy.pp:0:abc_size:WARNING:assignment branch condition size is 2.24

examples/extract.pp:0:abc_size:WARNING:assignment branch condition size is 11.66

examples/file.pp:0:abc_size:WARNING:assignment branch condition size is 7.07

examples/init.pp:0:abc_size:WARNING:assignment branch condition size is 1.0

examples/scope_defaults.pp:0:abc_size:WARNING:assignment branch condition size is 3.32

examples/staging_parse.pp:0:abc_size:WARNING:assignment branch condition size is 12.53

manifests/deploy.pp:0:abc_size:WARNING:assignment branch condition size is 24.37

manifests/extract.pp:0:abc_size:WARNING:assignment branch condition size is 47.37

manifests/file.pp:0:abc_size:WARNING:assignment branch condition size is 76.69

manifests/init.pp:0:abc_size:WARNING:assignment branch condition size is 6.71

manifests/params.pp:0:abc_size:WARNING:assignment branch condition size is 25.73

Overall, it is a good idea to start applying different ideas that have been historically present in software development to Infrastructure As Code.  Thinking about the complexity of our code gives you insight into the testability and maintainability of your codebase.

DevOps Nov3rd Followup links

#vDM30in30 number 3!

Here are some links of interest from my DevOps Meetup talk tonight!

Slides and Videos are now posted at https://puppet.com/blog/puppetconf-2016-videos-slides-and-photos-are-here

https://puppetconf2016.sched.org/ - Schedule
Videos from puppetconf 2015
http://www.slideshare.net/DavidDanzilio/puppet-design-patterns-puppetconf
https://speakerdeck.com/garethr/running-puppet-in-docker - Gareth's presentation with examples of running a full version of puppetserver/puppetdb/puppetdashboard via docker.  https://github.com/puppetlabs/puppet-in-docker
https://speakerdeck.com/garethr/the-future-of-testing-puppet-code
Troubleshooting Puppet - https://twitter.com/uphillian/status/789218304619458560 - https://docs.google.com/presentation/d/1AT2j97HV_y2QNH_HFKwZ6ExQDKvfC_lt6qBZaai4ATo/edit#slide=id.g1839748d2e_0_11
Hashicorp Vault with Kubernettes - https://twitter.com/HashiCorp/status/793479714195243008
Seth Vargo didn't really have any actual slides, everything was based on examples that he has in this git repo.. https://github.com/sethvargo/vault-puppet


Here is a link to my slides.. you can clone the repo and open the html file in your browser.
https://github.com/Cinderhaze/DevOpsMeetupTalks/blob/master/2016_11_03_PuppetConf.html

Getting double use out of your Serverspec tests with Sensu

Hey Everyone! Second post for #vDM30in30!

I wanted to mention something that I learned at the Watching the Puppet Show presentation at #PuppetConf 2016, given by Sean Porter.

I had heard of Sensu before, but didn't know most of the details about it's operation.  You can find out about sensu at https://sensuapp.org/.

The most interesting part of the talk was that there exists a Sensu Serverspec integration... This allows you to use your Acceptance tests written in Serverspec as monitoring rules for your agents/nodes.  This helps keep you from rewriting the same 'logic' in both your acceptance tests and in your monitoring scripts.  If it is important enough to test for, you probably want to know if your operational app is behaving in the correct manner as well!

http://www.slideshare.net/m_richardson/serverspec-and-sensu-testing-and-monitoring-collide

https://github.com/sensu-plugins/sensu-plugins-serverspec

Hey everyone!

I just stumbled across this github repo.

https://github.com/hashicorp/puppet-bootstrap

It's a collection of distro native scripts for bootstrapping puppet.  If you need to get puppet installed and running on your server/vm, it's a good place to start from!

s3 ETag and puppet

Hello faithful readers!

Today, I ponder a problem... how to know if I have the latest version of a file from an s3 bucket, and how to get that logic into puppet?

Amazon s3 buckets can contain objects, which contain an ETag... which is sometimes the md5, and sometimes not..

In trying to figure out how to know if I should download an object or not, I was inspired by the s3file module out on the puppet forge.

https://github.com/branan/puppet-module-s3file/blob/master/manifests/init.pp

Notably, this slick shell command to pull out the ETag and compare it to the md5sum of the file..
"[ -e ${name} ] && curl -I ${real_source} | grep ETag | grep `md5sum ${name} | cut -c1-32`"
However, I saw that in my use case, we used aws s3 cp /path/to/file s3://bucketname/key/path/ and our uploads happen as multipart uploads at the low level.  this makes our ETag break into multiple parts for files much smaller than the rest of the internet (I think at either 5mb or 15mb boundaries), which makes it not so keen for the above situation.

also, we don't have a publicly available bucket to curl, so we have to replace the curl -I call with a aws s3api head-object --bucket bucketname --key key/path/file.

While playing with the s3api command, I looked at a file I had uploaded previously.  When I uploaded it, I had used s3cmd's sync option.  I saw the following output.

$ aws s3api head-object --bucket bucketname --key noarch/epel-release-6-8.noarch.rpm {    "AcceptRanges": "bytes",     "ContentType": "binary/octet-stream",     "LastModified": "Sat, 19 Sep 2015 03:27:25 GMT",     "ContentLength": 14540,     "ETag": "\"2cd0ae668a585a14e07c2ea4f264d79b\"",     "Metadata": {        "s3cmd-attrs": "uid:502/gname:staff/uname:~~~~/gid:20/mode:33188/mtime:1352129496/atime:1441758431/md5:2cd0ae668a585a14e07c2ea4f264d79b/ctime:1441385182"    }}

When I manually uploaded the file using aws s3 cp, I saw the following header info on the s3 object

$ aws s3api head-object --bucket bucketname --key epel-release-6-8.noarch.rpm {    "AcceptRanges": "bytes",     "ContentType": "binary/octet-stream",     "LastModified": "Sat, 19 Sep 2015 03:39:13 GMT",     "ContentLength": 14540,     "ETag": "\"2cd0ae668a585a14e07c2ea4f264d79b\"",     "Metadata": {}}

So, the MD5 IS in there... IF I use a program/command that sets it in the metadata.  

Back to my problem at hand... how would I know if I already downloaded the file?  What are my options?

1. Figure out some sort of script to calculate the md5 sum of the file.  This would require me assuming the number of multi-part chunks
2. Always make our upload process either not use multi-part uploads (keeping the ETag equal to the MD5sum) or use some tool or manually set the metadata to contain an md5sum
3. Pull out the ETag value when I download the file, and store an additional file with my downloaded s3 file (e.g. if I have s3://bucket/foo.tar.gz, pull out it's ETag and write it to foo.tar.gz.etag after a successful download)

All three have disadvantages.

#1 - This script has to assume or calculate multi-part upload chunk size.  It is explored in some of the answers at this StackOverflow post. 

#2 - This requires our uploads to have happened a certain way.  If someone circumvents the standard process for uploading (uses a multipart upload, or uploads with the aws s3 cp instead of aws s3api put-object), then we would always download the item. 

# 3 - This requires us saving a separate metadata file alongside the file download.  If the application we are downloading the file for is sensitive to additional files hanging around, this may interfere.  there IS a big plus in this case though.  

aws s3api get-object has a helpgul argument.... 

--if-none-match (string) Return the  object  only  if  its  entity  tag

       (ETag) is different from the one specified, otherwise return a 304 (not

       modified).

If I use that, the s3 call would skip downloading the file if the ETag matches.  

Also, with option 3... how do I do it in puppet?  Do I have some clever chaining of exec resources?  Do I create a resource type that utilizes the aws-ruby sdk?

Hrm, another post with ideas and options but no solutions.... I'll have to ponder how I should move forward with this, and fuel a future blog post!

Thanks for reading!

AWS Tags as Facter facts

Hello everyone!

I hate it when I am looking for a solution, find other people online asking for the same solution... figure it out myself, and then the comments section is closed on the page that I found, so I can't even help anyone else in the future that stumbles onto that page....

I was browsing around, trying to figure out how to adopt the Roles and Profiles pattern to our puppet configuration on AWS, when I stumbled across this post.

http://sorcery.smugmug.com/2013/01/14/scaling-puppet-in-ec2/

After spending a while poking around at the AWS CLI and looking for solutions, I found a few examples similar to the following stack overflow example around the web.
http://stackoverflow.com/a/7122649

ec2-describe-tags \
  --filter "resource-type=instance" \
  --filter "resource-id=$(ec2-metadata -i | cut -d ' ' -f2)" \
  --filter "key=Name" | cut -f5

Pair a command similar to this with the fact that you can make a simple facter fact, either by sticking the command you want to run in an exec call similar to this snippet from the facter docs

Facter.add(:powerstates) do confine :kernel => 'Linux' setcode do Facter::Core::Execution.exec('cat /sys/power/states') end end

... or by directly passing your command as a string to the setcode block.

One thing of note... I am still very new to ruby, but I occasionally had issues with errors happening with my AWS CLI call, and that would cause ALL of my facts to fail to load.  wrapping the entire thing with a rescue StandardError clause fixed it for now.

You can also reference other facts from within facter, so instead of pulling the value from ec2-metadata, I could get it from the built in ec2_instance_id fact that was already on my box (which I presume gets it from the same source)

Wrapping that all together, It would look something like the following...

begin
Facter.add(:tag_name) do confine :kernel => 'Linux'
 tag = 'Name'
cmd = 'ec2-describe-tags --filter "resource-type=instance" --filter "resource-id=' + Facter.value(:ec2_instance_id) + ' --filter "key=' + tag + ' | cut -f5'
setcode do Facter::Core::Execution.exec(cmd)
end end
rescue StandardError
end

You can replace the tag 'Name' with any other tag that you have specified (Role, Env, Tier, etc...) and pull information from the tags assigned to your instance.

You'll just have to make sure you put your custom fact in a module that you'll deploy to the boxes that need to know that information.  If you have some sort of base profile you are using, you could put it in it's lib/facter/ directory.

One thing of note... every time facter gathers facts, it's making a call to the AWS api... if you are pulling down information for multiple tags or multiple pieces of info for separate facts, it may be better to either cache the results of the call as unfiltered json and manipulate it from it's raw json inside some ruby code, or cache the value you get back from the call.  I've also heard of people putting a proxy between themselves and the amazon API to mitigate the same calls happening over and over.

Another useful thing to point out.  You don't really HAVE to make any calls to the API if you don't want to.  You can manually create your own facter fact that is static to the box at creation time (in the case of AWS, in your user-data script).  Since you won't typically design your individual VMs to gracefully change roles, and the role changes would probably also require security group changes, you may be able to just create a text file in /etc/facter/facts.d/ that would contain puppet_role=myrole, and then you could reference the fact $::puppet_role from facter.

I hope this is useful for someone, as I had a hard time tracking down this specific information when I was trying to solve the problem myself!


P.S.

Another note from the earlier linked Stackoverflow post...

In case you use IAM instead of explicit credentials, use these IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {    
      "Effect": "Allow",
      "Action": [ "ec2:DescribeTags"],
      "Resource": ["*"]
    }
  ]
}

Hello everyone!

This is mostly notes to myself after watching this talk (after DevOps Weekly told me about it in one of their emails).  I found the following bits of info pretty interesting and wanted to begin to apply this to some of the things I've been working on.

https://puppetlabs.com/presentations/building-hyper-secure-vpc-aws-puppet


16:41 -> Created an puppet module to apply IS benchmarking

20:56 -> rsyslog => graylog2 to roll all of their logs into one place

23:35 -> Network traffic logging... AWS Security Groups and Network ACL's don't log anything, Needed to log all traffic going in and out on any level of the VPC
      -> Puppet + IPTables +Rsyslog +Graylog2
28:26 -> Facter fact for determining zone with ugly regex... tag zone by IP address... same for 'tier'

31:00 -> greylog2 was really good, millions of events in, really fast.

32:40 -> Performance of large catalogs was bad with puppet 2.7, Hiera-Gpg is cumbersome
  recursion to remove checksums on big directories
   file { "/etc/somedir":
     recurse => true,
     ignore => ['work', 'temp', 'log'],
     checksum -> none,
   }
    (you don't want checksums on tomcat work directories)
 
    Hiera-GBG is cumbersome, they switched to a mysql hiera backend

34:45 -> cloudformation json is ugly...
         CFNDSL = ruby DSL for CloudFormation templates https://github.com/howech/cfndsl
         use for cloudformation template generatoin

'Ugly'
35:22 -> unified state and lifecycle management -> Doesnot exist...
37:13 -> One single truth  source for
         1. audit trail/logging
         2. Instance status
         3. App status
         4. CRUD actions on the whole infrastructure

39:40 -> puppetlabs aquired cloudsmith... is that heading toward some unified state and lifecycle management?

40:50 -> CIS, tmp should be on a different disk, did some trickery to shuffle it around?

42:00 -> Switched from CIS application and snapshot of AMI to applying the CIS benchmark once a day and using all default AMI
 -> That prevented people from taking a hardened base image and fudging something setup by CIS .  Once every halfhour got to cumbersome

Who zookeeps the zookeepers?

Hello everyone!

I am wondering if anyone in the DevOps community knows of a good way to manage Zookeeper instances in an autoscaling group on Amazon AWS... here's some background.

I am using puppet to manage the instances, with a custom fact based on a tag to set a 'role', using the roles and profiles pattern.

The zookeeper configuration file needs to know about all of the other instances available.  Should I use a script run on the master (with the generate function) to pull down a list of EC2s in that scaling group using the aws command?  Custom fact on the box (using a similar command)?

I did see a serverfault post on using puppetDB.  We haven't used puppetdb yet in our environment, and due to security policies, it takes a while to add new software to our environment.


Does anyone have any other suggestions for managing the zookeeper config's list of servers?

How to pass a variable to a one line command.

Hi everyone!

Today I learned (while trying to create a Facter fact ) that if you need to run a one line command that requires an Environment variable set, it's not as straight forward as you would expect.


I started with the example from the Puppet Docs

Facter.add(:rubypath) do
  setcode 'which ruby'
end

This gave me the following output

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $rubypath":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact: /usr/bin/ruby
Notice: /Stage[main]/Main/Notify[Fact: /usr/bin/ruby]/message: defined 'message' as 'Fact: /usr/bin/ruby'
Notice: Finished catalog run in 0.07 seconds

 replaced the command with an example command to cover my case.

Facter.add(:example_fact) do
  setcode 'VAR=hello; echo $VAR'
end

And here is my output, what went wrong!?

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.11 seconds
Notice: Fact:
Notice: /Stage[main]/Main/Notify[Fact: ]/message: defined 'message' as 'Fact: '
Notice: Finished catalog run in 0.06 seconds

I figured that the variable declaration was being lost somewhere... how about I run it in a sub-shell?

Facter.add(:example_fact) do
  setcode 'sh -c "VAR=hello; echo $VAR" '
end

I ran that, and got the same output as last time!  But wait!  if I reverse the order of the quotes...

Facter.add(:example_fact) do
  setcode "sh -c 'VAR=hello; echo $VAR' "
end

Suddenly, it works!

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact: hello
Notice: /Stage[main]/Main/Notify[Fact: hello]/message: defined 'message' as 'Fact: hello'
Notice: Finished catalog run in 0.06 seconds

One small problem... When I ran my actual command, it didn't work!  It wasn't reading my variable!  Now to model this...

dawiest@old-laptop:~$ echo 'echo $VAR' > /tmp/example.sh && chmod u+x /tmp/example.sh

And here is my updated fact.

Facter.add(:example_fact) do
  setcode "sh -c 'export VAR=hello; /tmp/example.sh' "
end

When I run it again, I get the following output

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact:
Notice: /Stage[main]/Main/Notify[Fact: ]/message: defined 'message' as 'Fact: '
Notice: Finished catalog run in 0.06 seconds

So... the problem is that the variable was in scope for the echo call above, but it wasn't making it into my script.....  time to export our variable!

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact: hello
Notice: /Stage[main]/Main/Notify[Fact: hello]/message: defined 'message' as 'Fact: hello'
Notice: Finished catalog run in 0.06 seconds

That looks a lot better!


TL:DR.  If you are trying to run a command inside of a fact, you need to make sure you export your variable so it passes to the scope of your script, and you probably have to wrap it in a subshell.

Managing puppet modules with vagrant

A little while ago, I started my journey learning about DevOps, starting out by diving into some of the tooling (Puppet), and adding a few DevOps podcasts into my rotation.

One problem that plagued the process from the beginning... how to handle puppet modules?   Here's the evolution of my thought process...


Stage 1... ignorance

I simple used GIT to control my /etc/puppet/modules/ directory!  Lots of redundant code in here that was already saved in other git repos I could/should just fetch, which gives me an idea...

Stage 2..... frustration
Git Submodules!  I was just manually pulling down most of my modules directly from their git repos instead of though puppet forge anyway, so how about adding them as submodules to my repo, and having git fetch them for me!

Stage 3...... starting automation
Now that I was trying to build easy to run Vagrantfiles that had my puppet configs, I need some way to set it all up....  Initial trick.... adding a 'setup' script in my vagrant directory that pre-downloads all the modules that I run before the first setup! (either by directly calling git clone on the repo, or by calling git submodule init).  This was still a 'prestep' to running vagrant-up, so that didn't quite feel right

Stage 4...... feels right!
One of the first things I did after joining twitter was to ask #puppetize what the best way to manage modules was, and they did not disappoint!  I was redirected to this blog, which shows how you can use puppet librarian to manage your modules, but as to not require everyone to have ruby-gems installed on their main machine, they pull it down inside of their vagrant instance (since most vagrant images already have ruby-gems installed since ruby is a core component of puppet and chef.



Utilizing Configuration management with Vagrant is a powerful tool to help build easily repeatable environments for  both Dev and Deploy!  I hope this helps someone!  I know it was quite a struggle initially!