Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Tuesday, November 1, 2016

TIL - xargs tricks!

Hello everyone!  Today is the first day of  #vDM30in30 (For details, check out http://discoposse.com/vdm30in30/).  I was inspired to participate by rnelson0 in the #voxpupuli channel on freenode.

I have been using xargs for a while, but I learned something new!  Today I Learned (TIL) about the --replace-string or -I argument.

What IS xargs?  Straight from the xargs man page

 xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a back‐slash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial-arguments followed by items read from standard input.   

What can xargs do for us?  If you have the result of a command that you wish to pass in as arguments to another command, xargs is a more natural way to flow from one command to the other using a pipe operator ( | ) instead of embedding the results in a for loop.


A good set of xargs examples can be found here.  The feature that I didn't know about was the -I, or --replace-str argument. From the man page...

       -I replace-str
              Replace occurrences of replace-str in the initial-arguments with names read from standard input.  Also, unquoted blanks do not terminate input items; instead  the  separator is the newline character.  Implies -x and -L 1.


What that allows you to do is to use the value from stdin in an arbitrary (and possibly repeated) position in your command.

If you wanted to create a backup file of all files in a directory except for a few that you wish to exclude, you can do something like the following..

$ ls
a  b  c  dime  penny  quarter
$ mkdir archive
$ ls | grep -v -e dime -e penny -e archive # This will print the list of files, excluding dime, penny, and the archive directory
a
b
c
quarter
$ ls | grep -v -e dime -e penny -e archive | xargs -I{} cp {} archive/{}.bak
$ ls archive/
a.bak  b.bak  c.bak  quarter.bak

You specify the symbol for the -I argument, {} in this case, which will be replaced by the results from the previous command.

Where else have I used xargs lately?  If you want to find all of the files of a particular type that contain a particular string, and edit them one by one:

$ find ./ -name '*.pp' | grep -v params | xargs grep  params -l | xargs -n 1 vi

What that will do is find all files that match the pattern of *.pp, remove any files that contain params in the name (grep -v is an inverse match), xargs will pass all of the listed files into the grep command,  -l prints the name of any matching files, then the final xargs will send each file, one at a time, into vi for editing.


Before someone learns about xargs, they may end up embedding a bash for loop into their commands, something like

$ for i in $(ls | grep -v archive); do cp $i archive/$i.bak; done

With a short example like that, it may not be to hard to follow, but as your examples get much longer, it can get hard to tell which parts of the command belong where.  It is usually much easier to read left to right.

From left to right using xargs..
$ ls | grep -v archive | xargs -I{} cp {} archive/{}.bak

Thursday, October 15, 2015

TiL - remember how to shell quote!

Just the other day, I was doing something over and over.... trying to kill a stubborn tomcat server that wouldn't gracefully die.  sudo service tomcat stop just wouldn't cut it.

I was getting tired of repeatedly typing 'ps aux | grep tomcat', copying out the Process id, and passing it to kill -9.   Time for some simple automation... an alias!

So, how to find the Process?  First, we use ps to display a list of running processes, and the grep utility to select only the lines that contain tomcat.

vagrant@client1:~$ ps aux | grep tomcat
tomcat7   6355  0.9 26.2 1050336 63972 ?       Sl   00:37   0:04 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start

vagrant   7504  0.0  0.3  10460   940 pts/0    S+   00:46   0:00 grep --color=autotomcat

I needed to make sure I was only grabbing the tomcat process.  Since I only have the tomcat server being run by the tomcat user, and the output of PS starts with the user who is running the process, I could use a regular expression that matches from the beginning of the line to the word tomcat.  To make sure I can use the beginning of line char (^) in my regex, I use the '-E, --extended-regexp' argument.

My command looks something like this
vagrant@client1:~$ ps aux | grep -E '^tomcat'
tomcat7   6355  0.7 25.6 1050336 62424 ?       Sl   00:37   0:05 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start
I then need to pull out the second record, which contains the process ID.  For that, I use awk.   It simply takes the output and breaks it into 'records', by default it uses whitespace.  Using 'print $2' prints out the second record.

Here is the updated command.
vagrant@client1:~$ ps aux | grep -E '^tomcat' | awk '{ print $2 }'
6355

Most important note about the above command.... I initially forgot to use the single quote ('), and I accidentally used the double quote (").  What happens in that case is that your shell will try to do a variable substation, and replace $2 with the variable 2 ( which is usually used when you are passing arguments to a shell script, $0 is the name of the script, $1 is the first argument, $2 is the second, etc...).  Since there was nothing in $2, it turned the awk command into 'print', which simply prints the entire line.

Here is the command without output using the wrong quotes.
vagrant@client1:~$ ps aux | grep -E '^tomcat' | awk "{ print $2 }"
tomcat7   6355  0.6 25.6 1050336 62424 ?       Sl   00:37   0:05 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start

Now, that will return only the PID... but what to do to it?  I need to pass it as an argument to the kill command, so I'll run it in a subshell by putting it between two backpacks (`).  I use the -9 argument to force kill to kill it totally.  Remember to use sudo if you aren't running as the tomcat user!

My full command now looks like this

vagrant@client1:~$ sudo kill -9 `ps aux | grep -E '^tomcat' | awk '{ print $2 }'`
When successful, this command has no output.

Remember to properly quote your shell variables, happy command line fu everyone!

Monday, August 3, 2015

TIL - paste!

Hi everyone!

Today, I was helping a co-worker with a script.  He needed to find some set of directories, add /*.jar to the end of each, and put them together in a single line as coma-separated values.

Enter paste!   (Funny thought, typing 'man paste' into google turned out much better than I thought it would!)

Say we had a list of all the files on my RaspberryPi's home directory, and we wanted to put them together as a comma separated list...   We would simply pipe them through paste with the --delimiters=, and --serial flags set.

$ ls ~/Desktop/ | paste -d, -s

debian-reference-common.desktop,idle3.desktop,idle.desktop,lxterminal.desktop,midori.desktop,ocr_resources.desktop,paste.out,pistore.desktop,python-games.desktop,scratch.desktop,shutdown.desktop,sonic-pi.desktop,wolfram-language.desktop,wolfram-mathematica.desktop,wpa_gui.desktop

Even though I've been using linux since about 2003, I feel like I stumble onto little nuggets like this pretty often!