s3 ETag and puppet

Hello faithful readers!

Today, I ponder a problem... how to know if I have the latest version of a file from an s3 bucket, and how to get that logic into puppet?

Amazon s3 buckets can contain objects, which contain an ETag... which is sometimes the md5, and sometimes not..

In trying to figure out how to know if I should download an object or not, I was inspired by the s3file module out on the puppet forge.

https://github.com/branan/puppet-module-s3file/blob/master/manifests/init.pp

Notably, this slick shell command to pull out the ETag and compare it to the md5sum of the file..
"[ -e ${name} ] && curl -I ${real_source} | grep ETag | grep `md5sum ${name} | cut -c1-32`"
However, I saw that in my use case, we used aws s3 cp /path/to/file s3://bucketname/key/path/ and our uploads happen as multipart uploads at the low level.  this makes our ETag break into multiple parts for files much smaller than the rest of the internet (I think at either 5mb or 15mb boundaries), which makes it not so keen for the above situation.

also, we don't have a publicly available bucket to curl, so we have to replace the curl -I call with a aws s3api head-object --bucket bucketname --key key/path/file.

While playing with the s3api command, I looked at a file I had uploaded previously.  When I uploaded it, I had used s3cmd's sync option.  I saw the following output.

$ aws s3api head-object --bucket bucketname --key noarch/epel-release-6-8.noarch.rpm {    "AcceptRanges": "bytes",     "ContentType": "binary/octet-stream",     "LastModified": "Sat, 19 Sep 2015 03:27:25 GMT",     "ContentLength": 14540,     "ETag": "\"2cd0ae668a585a14e07c2ea4f264d79b\"",     "Metadata": {        "s3cmd-attrs": "uid:502/gname:staff/uname:~~~~/gid:20/mode:33188/mtime:1352129496/atime:1441758431/md5:2cd0ae668a585a14e07c2ea4f264d79b/ctime:1441385182"    }}

When I manually uploaded the file using aws s3 cp, I saw the following header info on the s3 object

$ aws s3api head-object --bucket bucketname --key epel-release-6-8.noarch.rpm {    "AcceptRanges": "bytes",     "ContentType": "binary/octet-stream",     "LastModified": "Sat, 19 Sep 2015 03:39:13 GMT",     "ContentLength": 14540,     "ETag": "\"2cd0ae668a585a14e07c2ea4f264d79b\"",     "Metadata": {}}

So, the MD5 IS in there... IF I use a program/command that sets it in the metadata.  

Back to my problem at hand... how would I know if I already downloaded the file?  What are my options?

1. Figure out some sort of script to calculate the md5 sum of the file.  This would require me assuming the number of multi-part chunks
2. Always make our upload process either not use multi-part uploads (keeping the ETag equal to the MD5sum) or use some tool or manually set the metadata to contain an md5sum
3. Pull out the ETag value when I download the file, and store an additional file with my downloaded s3 file (e.g. if I have s3://bucket/foo.tar.gz, pull out it's ETag and write it to foo.tar.gz.etag after a successful download)

All three have disadvantages.

#1 - This script has to assume or calculate multi-part upload chunk size.  It is explored in some of the answers at this StackOverflow post. 

#2 - This requires our uploads to have happened a certain way.  If someone circumvents the standard process for uploading (uses a multipart upload, or uploads with the aws s3 cp instead of aws s3api put-object), then we would always download the item. 

# 3 - This requires us saving a separate metadata file alongside the file download.  If the application we are downloading the file for is sensitive to additional files hanging around, this may interfere.  there IS a big plus in this case though.  

aws s3api get-object has a helpgul argument.... 

--if-none-match (string) Return the  object  only  if  its  entity  tag

       (ETag) is different from the one specified, otherwise return a 304 (not

       modified).

If I use that, the s3 call would skip downloading the file if the ETag matches.  

Also, with option 3... how do I do it in puppet?  Do I have some clever chaining of exec resources?  Do I create a resource type that utilizes the aws-ruby sdk?

Hrm, another post with ideas and options but no solutions.... I'll have to ponder how I should move forward with this, and fuel a future blog post!

Thanks for reading!

that's yum mmmy!

Hello everyone!

I've been pretty sparse lately, and today's topic is pretty light, but Today I Learned (TIL) that puppet has a yum repository resource!  While I was trying to figure out if they had a augeas lens for the type of ini file that the format is stored in, I was very happy to find a native type in base puppet!

One feature that caught my eye is that there is a value of s3_enabled...

s3_enabled

(Property: This attribute represents concrete state on the target system.)

Access the repository via S3. Valid values are: False/0/No or True/1/Yes. Set this toabsent to remove it from the file completely.

Valid values are absent. Values can match /^(True|False|0|1|No|Yes)$/i.

(↑ Back to yumrepo attributes)

This has me intrigued, since we are working in AWS.  Does this mean that we can create a repository as flat flies in an s3 bucket?

I found this amazing article on setting up s3 based yum repos using IAM authorization...

So, does it work!?

Yes and no.... According to this Pull Request, The plugin only supports IAM Signature version 2, which is only in place for older AWS regions.  Newer regions only support IAM signature version 4.. Regions like China, Frankfurt, etc..   So only certain areas will be able to use the plugin as is.

I have tried duplicating the steps of the article on both a N. Virginia and a Frankfurt instance, and everything worked correctly in N. VA, and I had the same error as the pull request (400) when trying to use both yum-s3-iam and yum-s3-plugin.    I have a really rough github project here which I used to aid my setup.  I had to 'yum install -y git puppet3', clone my repository, and run my init script, then run sudo puppet apply ~/s3-repo-sandbox/s3_plugin.pp to apply my changes.  I had pre-setup a bucket named dawiest-repo, in which I placed a noarch repository created similar to the above article.


Assuming you can get the above plugin to work for your region..... how can you make sure your repos are set up before any packages are installed?  You could directly add require parameters to each package that needs it, but it is probably better to use the 'spaceship operator' to collect all the references and create the require entries for you!

http://serverfault.com/a/461869

Although stages can handle this and so can specific yum repo dependencies, better is to declare the relationship generically.

Just put Yumrepo <| |> -> Package <| provider != 'rpm' |> in your puppet manifest.

node default {
  Yumrepo <| |> -> Package <| provider != 'rpm' |>
}

This makes it so that all yumrepo types will get processed before any packages that don't have 'rpm' as their provider. That latter exclusion is so that I can use the (for example) epel-release RPM package to help install the yum repo.

Also, this article give a good description of the process needed for setting up gpg keys for your repo!

AWS Tags as Facter facts

Hello everyone!

I hate it when I am looking for a solution, find other people online asking for the same solution... figure it out myself, and then the comments section is closed on the page that I found, so I can't even help anyone else in the future that stumbles onto that page....

I was browsing around, trying to figure out how to adopt the Roles and Profiles pattern to our puppet configuration on AWS, when I stumbled across this post.

http://sorcery.smugmug.com/2013/01/14/scaling-puppet-in-ec2/

After spending a while poking around at the AWS CLI and looking for solutions, I found a few examples similar to the following stack overflow example around the web.
http://stackoverflow.com/a/7122649

ec2-describe-tags \
  --filter "resource-type=instance" \
  --filter "resource-id=$(ec2-metadata -i | cut -d ' ' -f2)" \
  --filter "key=Name" | cut -f5

Pair a command similar to this with the fact that you can make a simple facter fact, either by sticking the command you want to run in an exec call similar to this snippet from the facter docs

Facter.add(:powerstates) do confine :kernel => 'Linux' setcode do Facter::Core::Execution.exec('cat /sys/power/states') end end

... or by directly passing your command as a string to the setcode block.

One thing of note... I am still very new to ruby, but I occasionally had issues with errors happening with my AWS CLI call, and that would cause ALL of my facts to fail to load.  wrapping the entire thing with a rescue StandardError clause fixed it for now.

You can also reference other facts from within facter, so instead of pulling the value from ec2-metadata, I could get it from the built in ec2_instance_id fact that was already on my box (which I presume gets it from the same source)

Wrapping that all together, It would look something like the following...

begin
Facter.add(:tag_name) do confine :kernel => 'Linux'
 tag = 'Name'
cmd = 'ec2-describe-tags --filter "resource-type=instance" --filter "resource-id=' + Facter.value(:ec2_instance_id) + ' --filter "key=' + tag + ' | cut -f5'
setcode do Facter::Core::Execution.exec(cmd)
end end
rescue StandardError
end

You can replace the tag 'Name' with any other tag that you have specified (Role, Env, Tier, etc...) and pull information from the tags assigned to your instance.

You'll just have to make sure you put your custom fact in a module that you'll deploy to the boxes that need to know that information.  If you have some sort of base profile you are using, you could put it in it's lib/facter/ directory.

One thing of note... every time facter gathers facts, it's making a call to the AWS api... if you are pulling down information for multiple tags or multiple pieces of info for separate facts, it may be better to either cache the results of the call as unfiltered json and manipulate it from it's raw json inside some ruby code, or cache the value you get back from the call.  I've also heard of people putting a proxy between themselves and the amazon API to mitigate the same calls happening over and over.

Another useful thing to point out.  You don't really HAVE to make any calls to the API if you don't want to.  You can manually create your own facter fact that is static to the box at creation time (in the case of AWS, in your user-data script).  Since you won't typically design your individual VMs to gracefully change roles, and the role changes would probably also require security group changes, you may be able to just create a text file in /etc/facter/facts.d/ that would contain puppet_role=myrole, and then you could reference the fact $::puppet_role from facter.

I hope this is useful for someone, as I had a hard time tracking down this specific information when I was trying to solve the problem myself!


P.S.

Another note from the earlier linked Stackoverflow post...

In case you use IAM instead of explicit credentials, use these IAM permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {    
      "Effect": "Allow",
      "Action": [ "ec2:DescribeTags"],
      "Resource": ["*"]
    }
  ]
}

Hello everyone!

This is mostly notes to myself after watching this talk (after DevOps Weekly told me about it in one of their emails).  I found the following bits of info pretty interesting and wanted to begin to apply this to some of the things I've been working on.

https://puppetlabs.com/presentations/building-hyper-secure-vpc-aws-puppet


16:41 -> Created an puppet module to apply IS benchmarking

20:56 -> rsyslog => graylog2 to roll all of their logs into one place

23:35 -> Network traffic logging... AWS Security Groups and Network ACL's don't log anything, Needed to log all traffic going in and out on any level of the VPC
      -> Puppet + IPTables +Rsyslog +Graylog2
28:26 -> Facter fact for determining zone with ugly regex... tag zone by IP address... same for 'tier'

31:00 -> greylog2 was really good, millions of events in, really fast.

32:40 -> Performance of large catalogs was bad with puppet 2.7, Hiera-Gpg is cumbersome
  recursion to remove checksums on big directories
   file { "/etc/somedir":
     recurse => true,
     ignore => ['work', 'temp', 'log'],
     checksum -> none,
   }
    (you don't want checksums on tomcat work directories)
 
    Hiera-GBG is cumbersome, they switched to a mysql hiera backend

34:45 -> cloudformation json is ugly...
         CFNDSL = ruby DSL for CloudFormation templates https://github.com/howech/cfndsl
         use for cloudformation template generatoin

'Ugly'
35:22 -> unified state and lifecycle management -> Doesnot exist...
37:13 -> One single truth  source for
         1. audit trail/logging
         2. Instance status
         3. App status
         4. CRUD actions on the whole infrastructure

39:40 -> puppetlabs aquired cloudsmith... is that heading toward some unified state and lifecycle management?

40:50 -> CIS, tmp should be on a different disk, did some trickery to shuffle it around?

42:00 -> Switched from CIS application and snapshot of AMI to applying the CIS benchmark once a day and using all default AMI
 -> That prevented people from taking a hardened base image and fudging something setup by CIS .  Once every halfhour got to cumbersome

Who zookeeps the zookeepers?

Hello everyone!

I am wondering if anyone in the DevOps community knows of a good way to manage Zookeeper instances in an autoscaling group on Amazon AWS... here's some background.

I am using puppet to manage the instances, with a custom fact based on a tag to set a 'role', using the roles and profiles pattern.

The zookeeper configuration file needs to know about all of the other instances available.  Should I use a script run on the master (with the generate function) to pull down a list of EC2s in that scaling group using the aws command?  Custom fact on the box (using a similar command)?

I did see a serverfault post on using puppetDB.  We haven't used puppetdb yet in our environment, and due to security policies, it takes a while to add new software to our environment.


Does anyone have any other suggestions for managing the zookeeper config's list of servers?