DevOps Nov3rd Followup links

#vDM30in30 number 3!

Here are some links of interest from my DevOps Meetup talk tonight!

Slides and Videos are now posted at https://puppet.com/blog/puppetconf-2016-videos-slides-and-photos-are-here

https://puppetconf2016.sched.org/ - Schedule
Videos from puppetconf 2015
http://www.slideshare.net/DavidDanzilio/puppet-design-patterns-puppetconf
https://speakerdeck.com/garethr/running-puppet-in-docker - Gareth's presentation with examples of running a full version of puppetserver/puppetdb/puppetdashboard via docker.  https://github.com/puppetlabs/puppet-in-docker
https://speakerdeck.com/garethr/the-future-of-testing-puppet-code
Troubleshooting Puppet - https://twitter.com/uphillian/status/789218304619458560 - https://docs.google.com/presentation/d/1AT2j97HV_y2QNH_HFKwZ6ExQDKvfC_lt6qBZaai4ATo/edit#slide=id.g1839748d2e_0_11
Hashicorp Vault with Kubernettes - https://twitter.com/HashiCorp/status/793479714195243008
Seth Vargo didn't really have any actual slides, everything was based on examples that he has in this git repo.. https://github.com/sethvargo/vault-puppet


Here is a link to my slides.. you can clone the repo and open the html file in your browser.
https://github.com/Cinderhaze/DevOpsMeetupTalks/blob/master/2016_11_03_PuppetConf.html

Hey everyone!

I just stumbled across this github repo.

https://github.com/hashicorp/puppet-bootstrap

It's a collection of distro native scripts for bootstrapping puppet.  If you need to get puppet installed and running on your server/vm, it's a good place to start from!

s3 ETag and puppet

Hello faithful readers!

Today, I ponder a problem... how to know if I have the latest version of a file from an s3 bucket, and how to get that logic into puppet?

Amazon s3 buckets can contain objects, which contain an ETag... which is sometimes the md5, and sometimes not..

In trying to figure out how to know if I should download an object or not, I was inspired by the s3file module out on the puppet forge.

https://github.com/branan/puppet-module-s3file/blob/master/manifests/init.pp

Notably, this slick shell command to pull out the ETag and compare it to the md5sum of the file..
"[ -e ${name} ] && curl -I ${real_source} | grep ETag | grep `md5sum ${name} | cut -c1-32`"
However, I saw that in my use case, we used aws s3 cp /path/to/file s3://bucketname/key/path/ and our uploads happen as multipart uploads at the low level.  this makes our ETag break into multiple parts for files much smaller than the rest of the internet (I think at either 5mb or 15mb boundaries), which makes it not so keen for the above situation.

also, we don't have a publicly available bucket to curl, so we have to replace the curl -I call with a aws s3api head-object --bucket bucketname --key key/path/file.

While playing with the s3api command, I looked at a file I had uploaded previously.  When I uploaded it, I had used s3cmd's sync option.  I saw the following output.

$ aws s3api head-object --bucket bucketname --key noarch/epel-release-6-8.noarch.rpm {    "AcceptRanges": "bytes",     "ContentType": "binary/octet-stream",     "LastModified": "Sat, 19 Sep 2015 03:27:25 GMT",     "ContentLength": 14540,     "ETag": "\"2cd0ae668a585a14e07c2ea4f264d79b\"",     "Metadata": {        "s3cmd-attrs": "uid:502/gname:staff/uname:~~~~/gid:20/mode:33188/mtime:1352129496/atime:1441758431/md5:2cd0ae668a585a14e07c2ea4f264d79b/ctime:1441385182"    }}

When I manually uploaded the file using aws s3 cp, I saw the following header info on the s3 object

$ aws s3api head-object --bucket bucketname --key epel-release-6-8.noarch.rpm {    "AcceptRanges": "bytes",     "ContentType": "binary/octet-stream",     "LastModified": "Sat, 19 Sep 2015 03:39:13 GMT",     "ContentLength": 14540,     "ETag": "\"2cd0ae668a585a14e07c2ea4f264d79b\"",     "Metadata": {}}

So, the MD5 IS in there... IF I use a program/command that sets it in the metadata.  

Back to my problem at hand... how would I know if I already downloaded the file?  What are my options?

1. Figure out some sort of script to calculate the md5 sum of the file.  This would require me assuming the number of multi-part chunks
2. Always make our upload process either not use multi-part uploads (keeping the ETag equal to the MD5sum) or use some tool or manually set the metadata to contain an md5sum
3. Pull out the ETag value when I download the file, and store an additional file with my downloaded s3 file (e.g. if I have s3://bucket/foo.tar.gz, pull out it's ETag and write it to foo.tar.gz.etag after a successful download)

All three have disadvantages.

#1 - This script has to assume or calculate multi-part upload chunk size.  It is explored in some of the answers at this StackOverflow post. 

#2 - This requires our uploads to have happened a certain way.  If someone circumvents the standard process for uploading (uses a multipart upload, or uploads with the aws s3 cp instead of aws s3api put-object), then we would always download the item. 

# 3 - This requires us saving a separate metadata file alongside the file download.  If the application we are downloading the file for is sensitive to additional files hanging around, this may interfere.  there IS a big plus in this case though.  

aws s3api get-object has a helpgul argument.... 

--if-none-match (string) Return the  object  only  if  its  entity  tag

       (ETag) is different from the one specified, otherwise return a 304 (not

       modified).

If I use that, the s3 call would skip downloading the file if the ETag matches.  

Also, with option 3... how do I do it in puppet?  Do I have some clever chaining of exec resources?  Do I create a resource type that utilizes the aws-ruby sdk?

Hrm, another post with ideas and options but no solutions.... I'll have to ponder how I should move forward with this, and fuel a future blog post!

Thanks for reading!

Hello everyone!

This is mostly notes to myself after watching this talk (after DevOps Weekly told me about it in one of their emails).  I found the following bits of info pretty interesting and wanted to begin to apply this to some of the things I've been working on.

https://puppetlabs.com/presentations/building-hyper-secure-vpc-aws-puppet


16:41 -> Created an puppet module to apply IS benchmarking

20:56 -> rsyslog => graylog2 to roll all of their logs into one place

23:35 -> Network traffic logging... AWS Security Groups and Network ACL's don't log anything, Needed to log all traffic going in and out on any level of the VPC
      -> Puppet + IPTables +Rsyslog +Graylog2
28:26 -> Facter fact for determining zone with ugly regex... tag zone by IP address... same for 'tier'

31:00 -> greylog2 was really good, millions of events in, really fast.

32:40 -> Performance of large catalogs was bad with puppet 2.7, Hiera-Gpg is cumbersome
  recursion to remove checksums on big directories
   file { "/etc/somedir":
     recurse => true,
     ignore => ['work', 'temp', 'log'],
     checksum -> none,
   }
    (you don't want checksums on tomcat work directories)
 
    Hiera-GBG is cumbersome, they switched to a mysql hiera backend

34:45 -> cloudformation json is ugly...
         CFNDSL = ruby DSL for CloudFormation templates https://github.com/howech/cfndsl
         use for cloudformation template generatoin

'Ugly'
35:22 -> unified state and lifecycle management -> Doesnot exist...
37:13 -> One single truth  source for
         1. audit trail/logging
         2. Instance status
         3. App status
         4. CRUD actions on the whole infrastructure

39:40 -> puppetlabs aquired cloudsmith... is that heading toward some unified state and lifecycle management?

40:50 -> CIS, tmp should be on a different disk, did some trickery to shuffle it around?

42:00 -> Switched from CIS application and snapshot of AMI to applying the CIS benchmark once a day and using all default AMI
 -> That prevented people from taking a hardened base image and fudging something setup by CIS .  Once every halfhour got to cumbersome

Who zookeeps the zookeepers?

Hello everyone!

I am wondering if anyone in the DevOps community knows of a good way to manage Zookeeper instances in an autoscaling group on Amazon AWS... here's some background.

I am using puppet to manage the instances, with a custom fact based on a tag to set a 'role', using the roles and profiles pattern.

The zookeeper configuration file needs to know about all of the other instances available.  Should I use a script run on the master (with the generate function) to pull down a list of EC2s in that scaling group using the aws command?  Custom fact on the box (using a similar command)?

I did see a serverfault post on using puppetDB.  We haven't used puppetdb yet in our environment, and due to security policies, it takes a while to add new software to our environment.


Does anyone have any other suggestions for managing the zookeeper config's list of servers?

It's been a while....

Hello everyone!

I haven't posted in a while... I just wanted to keep you updated with what's been going on.

I accepted a different position where I work which deals with bringing a lot more devops practices where we are usually caught behind the times.

Trying to help spearhead a significant culture change to an entrenched industry though a pathfinding project working with automation, cloud technologies, and adapting a large legacy codebase has been a lot of fun and a lot of work!  

Ontop of all of my extra time/work devoted to that, I've had some other issues come up in my life.  I'll try to give more information about that in a future post, but with the above mentioned issues and the business of summer upon me... well, we have put other things higher on our priority list, and I haven't had as much time for the blog.

While i'm on that, here is my list of excuses!

• My Macbook from 2008 stopped booting!
• I visited my brother who sailed onboard Le' hermione from France to Yorktown, VA.
• I drove out to MI to visit my mother!
• I spent a week leaving work at a normal time (instead of putting in extra time working on my project) to help out with our local 
• I'm a lazy bum who isn't blogging as much as I should!

Well, that about sums it up for now.  Have a good one, interwebs!

How to pass a variable to a one line command.

Hi everyone!

Today I learned (while trying to create a Facter fact ) that if you need to run a one line command that requires an Environment variable set, it's not as straight forward as you would expect.


I started with the example from the Puppet Docs

Facter.add(:rubypath) do
  setcode 'which ruby'
end

This gave me the following output

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $rubypath":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact: /usr/bin/ruby
Notice: /Stage[main]/Main/Notify[Fact: /usr/bin/ruby]/message: defined 'message' as 'Fact: /usr/bin/ruby'
Notice: Finished catalog run in 0.07 seconds

 replaced the command with an example command to cover my case.

Facter.add(:example_fact) do
  setcode 'VAR=hello; echo $VAR'
end

And here is my output, what went wrong!?

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.11 seconds
Notice: Fact:
Notice: /Stage[main]/Main/Notify[Fact: ]/message: defined 'message' as 'Fact: '
Notice: Finished catalog run in 0.06 seconds

I figured that the variable declaration was being lost somewhere... how about I run it in a sub-shell?

Facter.add(:example_fact) do
  setcode 'sh -c "VAR=hello; echo $VAR" '
end

I ran that, and got the same output as last time!  But wait!  if I reverse the order of the quotes...

Facter.add(:example_fact) do
  setcode "sh -c 'VAR=hello; echo $VAR' "
end

Suddenly, it works!

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact: hello
Notice: /Stage[main]/Main/Notify[Fact: hello]/message: defined 'message' as 'Fact: hello'
Notice: Finished catalog run in 0.06 seconds

One small problem... When I ran my actual command, it didn't work!  It wasn't reading my variable!  Now to model this...

dawiest@old-laptop:~$ echo 'echo $VAR' > /tmp/example.sh && chmod u+x /tmp/example.sh

And here is my updated fact.

Facter.add(:example_fact) do
  setcode "sh -c 'export VAR=hello; /tmp/example.sh' "
end

When I run it again, I get the following output

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact:
Notice: /Stage[main]/Main/Notify[Fact: ]/message: defined 'message' as 'Fact: '
Notice: Finished catalog run in 0.06 seconds

So... the problem is that the variable was in scope for the echo call above, but it wasn't making it into my script.....  time to export our variable!

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact: hello
Notice: /Stage[main]/Main/Notify[Fact: hello]/message: defined 'message' as 'Fact: hello'
Notice: Finished catalog run in 0.06 seconds

That looks a lot better!


TL:DR.  If you are trying to run a command inside of a fact, you need to make sure you export your variable so it passes to the scope of your script, and you probably have to wrap it in a subshell.

Expecting more - gpg and expect

Hi everyone!

I recently needed to ssh to a machine, with a hard to remember password, and was unable to use ssh keys.

I had seen lots of examples for connecting though ssh that looked something like this...

#!/usr/bin/expect -f
set timeout 60
spawn ssh username@machine1.domain.com

expect "Password: "
send "Password\r"

send "\r" # This is successful. I am able to login successfully to the first machine

# at this point, carry on scripting the first ssh session:
send "ssh username@machine2.domain.com\r"
expect ...

Can you see something terribly wrong there?  The password is stored in plain text in the file! Even setting restrictive file access permissions, that's not a very secure way of handling passwords.

I had read about gpg for storing/extracting secrets when I was learning puppet with the puppet 3 cookbook.

The steps to use gpg are as follows.

#generate gpg key
temp@old-laptop:~$ gpg --gen-key
gpg (GnuPG) 1.4.16; Copyright (C) 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: directory `/home/temp/.gnupg' created
gpg: new configuration file `/home/temp/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/temp/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/temp/.gnupg/secring.gpg' created
gpg: keyring `/home/temp/.gnupg/pubring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: John Doe
Email address: jd@JohnDoe.com
Comment: example
You selected this USER-ID:
    "John Doe (example) <jd@JohnDoe.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key. 

gpg: /home/temp/.gnupg/trustdb.gpg: trustdb created
gpg: key CFE3FAA2 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/CFE3FAA2 2015-04-04
      Key fingerprint = 5851 BF85 33A2 01E5 895F  0BD2 34B0 F74F CFE3 FAA2
uid                  John Doe (example) <jd@JohnDoe.com>
sub   2048R/D7422C09 2015-04-04

#make file containing secret - please do it with an editor!
temp@old-laptop:~$ echo "1234" > ~/passfile

#encrypt file
temp@old-laptop:~$ gpg -e -r jd@JohnDoe.com ~/passfile

#remove plaintext file
temp@old-laptop:~$ ls
examples.desktop  passfile  passfile.gpg
temp@old-laptop:~$ rm passfile

#decrypt file
temp@old-laptop:~$ gpg -d ~/passfile.gpg

You need a passphrase to unlock the secret key for
user: "John Doe (example) <jd@JohnDoe.com>"
2048-bit RSA key, ID D7422C09, created 2015-04-04 (main key ID CFE3FAA2)

can't connect to `/run/user/1000/keyring-WFzMA0/gpg': Permission denied
gpg: can't connect to `/run/user/1000/keyring-WFzMA0/gpg': connect failed
gpg: encrypted with 2048-bit RSA key, ID D7422C09, created 2015-04-04
      "John Doe (example) <jd@JohnDoe.com>"
1234

But, how do I tie that into an expect script for remotely logging in?

I initially found the exec statement for expect.

...
set password [exec gpg -d passfile.gpg]
...

It just didn't work properly for me.  GPG would prompt me for my passphrase, but it would output some notification text and the key to my display, and while it would be stored in the variable, it confused expect and didn't properly spawn my ssh connection. 

If I faked out getting the value directly, it would function as expected

...
set password [exec echo 1234]
...

Hrm... so what was I doing wrong? If I ran the same command and grabbed the output and redirected it to a file, it only contained my password.

gpg -d passfile.gpg > passfile.out
cat passfile.out

After reading about 50 different stack overflow posts that mostly said 'Why don't you just use ssh keys' or just had examples in plain text, I FINALLY found a clue!

http://ubuntuforums.org/showthread.php?t=1790094

This poor guy got some bad advice, where they aren't using the exec statement (he's just assigning the command string to the password variable), but he had the part that I missed.... the --quiet and --no-tty arguments!

once I added those to my gpg command, everything finally worked!

Here is my final example script -

#!/usr/bin/expect
 set pass [exec gpg --decrypt --quiet --no-tty passfile.gpg]

spawn ssh temp@localhost
expect "password:"
send "$pass\r"
interact

Expect the unexpected!

I have been working with linux since Highschool (which isn't as long ago for me as it is for some), and I have just found a new command-line tool!

Expect

I was attempting to automate the 'store user config' process, and the weblogic.Admin way of doing it wasn't behaving(complaining about invalid pad in the key), so I was defaulting back to the WLST command.  I ran into the same problem described here.  Since in my specific scenario, I needed some environment configuration that didn't translate properly, I needed a way to execute it directly in my bash script.

Using the following WLST script (here's the one from StackOverflow above)

#storeUserConfig.py
connect(user1,p@ss,'t3://myhost:9999')
storeUserConfig(userConfigFile='userconfig.secure',userKeyFile='userkey.secure') 
disconnect() 
exit()

I tried piping output directly into the command.... which didn't work!

yes | java -cp $PATH_TO_WEBLOGIC_JAR weblogic.WLST storeUserConfig.py"

and

java -cp $PATH_TO_WEBLOGIC_JAR weblogic.WLST storeUserConfig.py" << EOF
y
EOF

After a coworker suggested "expect", I came up with the following solution...

#createKeyAndConfig.sh

## Do some stuff to set up the env similar to weblogic's included stopWebLogic.sh
...

CMD="java -cp $PATH_TO_WEBLOGIC_JAR weblogic.WLST storeUserConfig.py"

expect -c "
  set timeout -1
  spawn \"$CMD\"
  expect \"y or n\"
  send \"y\r\"
  expect eof
"

The expect command will 'spawn' the java process.  It will wait until it sees the "y or n" portion of output from the script, when it will then 'send' a y followed by a newline to the program.

The first line, 'set timeout -1' was because creating the key file takes longer then the default timeout in expect, so it would terminate early (leaving behind a 0 length userkey.secure file)

Managing puppet modules with vagrant

A little while ago, I started my journey learning about DevOps, starting out by diving into some of the tooling (Puppet), and adding a few DevOps podcasts into my rotation.

One problem that plagued the process from the beginning... how to handle puppet modules?   Here's the evolution of my thought process...


Stage 1... ignorance

I simple used GIT to control my /etc/puppet/modules/ directory!  Lots of redundant code in here that was already saved in other git repos I could/should just fetch, which gives me an idea...

Stage 2..... frustration
Git Submodules!  I was just manually pulling down most of my modules directly from their git repos instead of though puppet forge anyway, so how about adding them as submodules to my repo, and having git fetch them for me!

Stage 3...... starting automation
Now that I was trying to build easy to run Vagrantfiles that had my puppet configs, I need some way to set it all up....  Initial trick.... adding a 'setup' script in my vagrant directory that pre-downloads all the modules that I run before the first setup! (either by directly calling git clone on the repo, or by calling git submodule init).  This was still a 'prestep' to running vagrant-up, so that didn't quite feel right

Stage 4...... feels right!
One of the first things I did after joining twitter was to ask #puppetize what the best way to manage modules was, and they did not disappoint!  I was redirected to this blog, which shows how you can use puppet librarian to manage your modules, but as to not require everyone to have ruby-gems installed on their main machine, they pull it down inside of their vagrant instance (since most vagrant images already have ruby-gems installed since ruby is a core component of puppet and chef.



Utilizing Configuration management with Vagrant is a powerful tool to help build easily repeatable environments for  both Dev and Deploy!  I hope this helps someone!  I know it was quite a struggle initially!