Friday, July 31, 2015

AWS Tags as Facter facts

Hello everyone!

I hate it when I am looking for a solution, find other people online asking for the same solution... figure it out myself, and then the comments section is closed on the page that I found, so I can't even help anyone else in the future that stumbles onto that page....

I was browsing around, trying to figure out how to adopt the Roles and Profiles pattern to our puppet configuration on AWS, when I stumbled across this post.

After spending a while poking around at the AWS CLI and looking for solutions, I found a few examples similar to the following stack overflow example around the web.

ec2-describe-tags \
  --filter "resource-type=instance" \
  --filter "resource-id=$(ec2-metadata -i | cut -d ' ' -f2)" \
  --filter "key=Name" | cut -f5
Pair a command similar to this with the fact that you can make a simple facter fact, either by sticking the command you want to run in an exec call similar to this snippet from the facter docs

Facter.add(:powerstates) do confine :kernel => 'Linux' setcode do Facter::Core::Execution.exec('cat /sys/power/states') end end

... or by directly passing your command as a string to the setcode block.

One thing of note... I am still very new to ruby, but I occasionally had issues with errors happening with my AWS CLI call, and that would cause ALL of my facts to fail to load.  wrapping the entire thing with a rescue StandardError clause fixed it for now.

You can also reference other facts from within facter, so instead of pulling the value from ec2-metadata, I could get it from the built in ec2_instance_id fact that was already on my box (which I presume gets it from the same source)

Wrapping that all together, It would look something like the following...

Facter.add(:tag_name) do confine :kernel => 'Linux'
 tag = 'Name'
cmd = 'ec2-describe-tags --filter "resource-type=instance" --filter "resource-id=' + Facter.value(:ec2_instance_id) + ' --filter "key=' + tag + ' | cut -f5'
setcode do Facter::Core::Execution.exec(cmd)
end end
rescue StandardError

You can replace the tag 'Name' with any other tag that you have specified (Role, Env, Tier, etc...) and pull information from the tags assigned to your instance.

You'll just have to make sure you put your custom fact in a module that you'll deploy to the boxes that need to know that information.  If you have some sort of base profile you are using, you could put it in it's lib/facter/ directory.

One thing of note... every time facter gathers facts, it's making a call to the AWS api... if you are pulling down information for multiple tags or multiple pieces of info for separate facts, it may be better to either cache the results of the call as unfiltered json and manipulate it from it's raw json inside some ruby code, or cache the value you get back from the call.  I've also heard of people putting a proxy between themselves and the amazon API to mitigate the same calls happening over and over.

Another useful thing to point out.  You don't really HAVE to make any calls to the API if you don't want to.  You can manually create your own facter fact that is static to the box at creation time (in the case of AWS, in your user-data script).  Since you won't typically design your individual VMs to gracefully change roles, and the role changes would probably also require security group changes, you may be able to just create a text file in /etc/facter/facts.d/ that would contain puppet_role=myrole, and then you could reference the fact $::puppet_role from facter.

I hope this is useful for someone, as I had a hard time tracking down this specific information when I was trying to solve the problem myself!


Another note from the earlier linked Stackoverflow post...

In case you use IAM instead of explicit credentials, use these IAM permissions:
  "Version": "2012-10-17",
  "Statement": [
      "Effect": "Allow",
      "Action": [ "ec2:DescribeTags"],
      "Resource": ["*"]

Saturday, July 25, 2015

Hello everyone!

This is mostly notes to myself after watching this talk (after DevOps Weekly told me about it in one of their emails).  I found the following bits of info pretty interesting and wanted to begin to apply this to some of the things I've been working on.

16:41 -> Created an puppet module to apply IS benchmarking

20:56 -> rsyslog => graylog2 to roll all of their logs into one place

23:35 -> Network traffic logging... AWS Security Groups and Network ACL's don't log anything, Needed to log all traffic going in and out on any level of the VPC
      -> Puppet + IPTables +Rsyslog +Graylog2
28:26 -> Facter fact for determining zone with ugly regex... tag zone by IP address... same for 'tier'

31:00 -> greylog2 was really good, millions of events in, really fast.

32:40 -> Performance of large catalogs was bad with puppet 2.7, Hiera-Gpg is cumbersome
  recursion to remove checksums on big directories
   file { "/etc/somedir":
     recurse => true,
     ignore => ['work', 'temp', 'log'],
     checksum -> none,
    (you don't want checksums on tomcat work directories)
    Hiera-GBG is cumbersome, they switched to a mysql hiera backend

34:45 -> cloudformation json is ugly...
         CFNDSL = ruby DSL for CloudFormation templates
         use for cloudformation template generatoin

35:22 -> unified state and lifecycle management -> Doesnot exist...
37:13 -> One single truth  source for
         1. audit trail/logging
         2. Instance status
         3. App status
         4. CRUD actions on the whole infrastructure

39:40 -> puppetlabs aquired cloudsmith... is that heading toward some unified state and lifecycle management?

40:50 -> CIS, tmp should be on a different disk, did some trickery to shuffle it around?

42:00 -> Switched from CIS application and snapshot of AMI to applying the CIS benchmark once a day and using all default AMI
 -> That prevented people from taking a hardened base image and fudging something setup by CIS .  Once every halfhour got to cumbersome

Tuesday, July 7, 2015

Who zookeeps the zookeepers?

Hello everyone!

I am wondering if anyone in the DevOps community knows of a good way to manage Zookeeper instances in an autoscaling group on Amazon AWS... here's some background.

I am using puppet to manage the instances, with a custom fact based on a tag to set a 'role', using the roles and profiles pattern.

The zookeeper configuration file needs to know about all of the other instances available.  Should I use a script run on the master (with the generate function) to pull down a list of EC2s in that scaling group using the aws command?  Custom fact on the box (using a similar command)?

I did see a serverfault post on using puppetDB.  We haven't used puppetdb yet in our environment, and due to security policies, it takes a while to add new software to our environment.

Does anyone have any other suggestions for managing the zookeeper config's list of servers?