TiL - remember how to shell quote!

Just the other day, I was doing something over and over.... trying to kill a stubborn tomcat server that wouldn't gracefully die.  sudo service tomcat stop just wouldn't cut it.

I was getting tired of repeatedly typing 'ps aux | grep tomcat', copying out the Process id, and passing it to kill -9.   Time for some simple automation... an alias!

So, how to find the Process?  First, we use ps to display a list of running processes, and the grep utility to select only the lines that contain tomcat.

vagrant@client1:~$ ps aux | grep tomcat

tomcat7   6355  0.9 26.2 1050336 63972 ?       Sl   00:37   0:04 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start

vagrant   7504  0.0  0.3  10460   940 pts/0    S+   00:46   0:00 grep --color=autotomcat

I needed to make sure I was only grabbing the tomcat process.  Since I only have the tomcat server being run by the tomcat user, and the output of PS starts with the user who is running the process, I could use a regular expression that matches from the beginning of the line to the word tomcat.  To make sure I can use the beginning of line char (^) in my regex, I use the '-E, --extended-regexp' argument.

My command looks something like this

vagrant@client1:~$ ps aux | grep -E '^tomcat'

tomcat7   6355  0.7 25.6 1050336 62424 ?       Sl   00:37   0:05 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start

I then need to pull out the second record, which contains the process ID.  For that, I use awk.   It simply takes the output and breaks it into 'records', by default it uses whitespace.  Using 'print $2' prints out the second record.

Here is the updated command.

vagrant@client1:~$ ps aux | grep -E '^tomcat' | awk '{ print $2 }'

6355

Most important note about the above command.... I initially forgot to use the single quote ('), and I accidentally used the double quote (").  What happens in that case is that your shell will try to do a variable substation, and replace $2 with the variable 2 ( which is usually used when you are passing arguments to a shell script, $0 is the name of the script, $1 is the first argument, $2 is the second, etc...).  Since there was nothing in $2, it turned the awk command into 'print', which simply prints the entire line.

Here is the command without output using the wrong quotes.

vagrant@client1:~$ ps aux | grep -E '^tomcat' | awk "{ print $2 }"

tomcat7   6355  0.6 25.6 1050336 62424 ?       Sl   00:37   0:05 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start

Now, that will return only the PID... but what to do to it?  I need to pass it as an argument to the kill command, so I'll run it in a subshell by putting it between two backpacks (`).  I use the -9 argument to force kill to kill it totally.  Remember to use sudo if you aren't running as the tomcat user!

My full command now looks like this

vagrant@client1:~$ sudo kill -9 `ps aux | grep -E '^tomcat' | awk '{ print $2 }'`

When successful, this command has no output.

Remember to properly quote your shell variables, happy command line fu everyone!

that's yum mmmy!

Hello everyone!

I've been pretty sparse lately, and today's topic is pretty light, but Today I Learned (TIL) that puppet has a yum repository resource!  While I was trying to figure out if they had a augeas lens for the type of ini file that the format is stored in, I was very happy to find a native type in base puppet!

One feature that caught my eye is that there is a value of s3_enabled...

s3_enabled

(Property: This attribute represents concrete state on the target system.)

Access the repository via S3. Valid values are: False/0/No or True/1/Yes. Set this toabsent to remove it from the file completely.

Valid values are absent. Values can match /^(True|False|0|1|No|Yes)$/i.

(↑ Back to yumrepo attributes)

This has me intrigued, since we are working in AWS.  Does this mean that we can create a repository as flat flies in an s3 bucket?

I found this amazing article on setting up s3 based yum repos using IAM authorization...

So, does it work!?

Yes and no.... According to this Pull Request, The plugin only supports IAM Signature version 2, which is only in place for older AWS regions.  Newer regions only support IAM signature version 4.. Regions like China, Frankfurt, etc..   So only certain areas will be able to use the plugin as is.

I have tried duplicating the steps of the article on both a N. Virginia and a Frankfurt instance, and everything worked correctly in N. VA, and I had the same error as the pull request (400) when trying to use both yum-s3-iam and yum-s3-plugin.    I have a really rough github project here which I used to aid my setup.  I had to 'yum install -y git puppet3', clone my repository, and run my init script, then run sudo puppet apply ~/s3-repo-sandbox/s3_plugin.pp to apply my changes.  I had pre-setup a bucket named dawiest-repo, in which I placed a noarch repository created similar to the above article.


Assuming you can get the above plugin to work for your region..... how can you make sure your repos are set up before any packages are installed?  You could directly add require parameters to each package that needs it, but it is probably better to use the 'spaceship operator' to collect all the references and create the require entries for you!

http://serverfault.com/a/461869

Although stages can handle this and so can specific yum repo dependencies, better is to declare the relationship generically.

Just put Yumrepo <| |> -> Package <| provider != 'rpm' |> in your puppet manifest.

node default {
  Yumrepo <| |> -> Package <| provider != 'rpm' |>
}

This makes it so that all yumrepo types will get processed before any packages that don't have 'rpm' as their provider. That latter exclusion is so that I can use the (for example) epel-release RPM package to help install the yum repo.

Also, this article give a good description of the process needed for setting up gpg keys for your repo!

TIL - paste!

Hi everyone!

Today, I was helping a co-worker with a script.  He needed to find some set of directories, add /*.jar to the end of each, and put them together in a single line as coma-separated values.

Enter paste!   (Funny thought, typing 'man paste' into google turned out much better than I thought it would!)

Say we had a list of all the files on my RaspberryPi's home directory, and we wanted to put them together as a comma separated list...   We would simply pipe them through paste with the --delimiters=, and --serial flags set.

$ ls ~/Desktop/ | paste -d, -s
debian-reference-common.desktop,idle3.desktop,idle.desktop,lxterminal.desktop,midori.desktop,ocr_resources.desktop,paste.out,pistore.desktop,python-games.desktop,scratch.desktop,shutdown.desktop,sonic-pi.desktop,wolfram-language.desktop,wolfram-mathematica.desktop,wpa_gui.desktop

Even though I've been using linux since about 2003, I feel like I stumble onto little nuggets like this pretty often!