How to pass a variable to a one line command.

Hi everyone!

Today I learned (while trying to create a Facter fact ) that if you need to run a one line command that requires an Environment variable set, it's not as straight forward as you would expect.


I started with the example from the Puppet Docs

Facter.add(:rubypath) do
  setcode 'which ruby'
end

This gave me the following output

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $rubypath":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact: /usr/bin/ruby
Notice: /Stage[main]/Main/Notify[Fact: /usr/bin/ruby]/message: defined 'message' as 'Fact: /usr/bin/ruby'
Notice: Finished catalog run in 0.07 seconds

 replaced the command with an example command to cover my case.

Facter.add(:example_fact) do
  setcode 'VAR=hello; echo $VAR'
end

And here is my output, what went wrong!?

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.11 seconds
Notice: Fact:
Notice: /Stage[main]/Main/Notify[Fact: ]/message: defined 'message' as 'Fact: '
Notice: Finished catalog run in 0.06 seconds

I figured that the variable declaration was being lost somewhere... how about I run it in a sub-shell?

Facter.add(:example_fact) do
  setcode 'sh -c "VAR=hello; echo $VAR" '
end

I ran that, and got the same output as last time!  But wait!  if I reverse the order of the quotes...

Facter.add(:example_fact) do
  setcode "sh -c 'VAR=hello; echo $VAR' "
end

Suddenly, it works!

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact: hello
Notice: /Stage[main]/Main/Notify[Fact: hello]/message: defined 'message' as 'Fact: hello'
Notice: Finished catalog run in 0.06 seconds

One small problem... When I ran my actual command, it didn't work!  It wasn't reading my variable!  Now to model this...

dawiest@old-laptop:~$ echo 'echo $VAR' > /tmp/example.sh && chmod u+x /tmp/example.sh

And here is my updated fact.

Facter.add(:example_fact) do
  setcode "sh -c 'export VAR=hello; /tmp/example.sh' "
end

When I run it again, I get the following output

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact:
Notice: /Stage[main]/Main/Notify[Fact: ]/message: defined 'message' as 'Fact: '
Notice: Finished catalog run in 0.06 seconds

So... the problem is that the variable was in scope for the echo call above, but it wasn't making it into my script.....  time to export our variable!

dawiest@old-laptop:~$ puppet apply -e 'notify{"Fact: $example":} '
Notice: Compiled catalog for old-laptop in environment production in 0.10 seconds
Notice: Fact: hello
Notice: /Stage[main]/Main/Notify[Fact: hello]/message: defined 'message' as 'Fact: hello'
Notice: Finished catalog run in 0.06 seconds

That looks a lot better!


TL:DR.  If you are trying to run a command inside of a fact, you need to make sure you export your variable so it passes to the scope of your script, and you probably have to wrap it in a subshell.

Expecting more - gpg and expect

Hi everyone!

I recently needed to ssh to a machine, with a hard to remember password, and was unable to use ssh keys.

I had seen lots of examples for connecting though ssh that looked something like this...

#!/usr/bin/expect -f
set timeout 60
spawn ssh username@machine1.domain.com

expect "Password: "
send "Password\r"

send "\r" # This is successful. I am able to login successfully to the first machine

# at this point, carry on scripting the first ssh session:
send "ssh username@machine2.domain.com\r"
expect ...

Can you see something terribly wrong there?  The password is stored in plain text in the file! Even setting restrictive file access permissions, that's not a very secure way of handling passwords.

I had read about gpg for storing/extracting secrets when I was learning puppet with the puppet 3 cookbook.

The steps to use gpg are as follows.

#generate gpg key
temp@old-laptop:~$ gpg --gen-key
gpg (GnuPG) 1.4.16; Copyright (C) 2013 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: directory `/home/temp/.gnupg' created
gpg: new configuration file `/home/temp/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/temp/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/temp/.gnupg/secring.gpg' created
gpg: keyring `/home/temp/.gnupg/pubring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: John Doe
Email address: jd@JohnDoe.com
Comment: example
You selected this USER-ID:
    "John Doe (example) <jd@JohnDoe.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key. 

gpg: /home/temp/.gnupg/trustdb.gpg: trustdb created
gpg: key CFE3FAA2 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/CFE3FAA2 2015-04-04
      Key fingerprint = 5851 BF85 33A2 01E5 895F  0BD2 34B0 F74F CFE3 FAA2
uid                  John Doe (example) <jd@JohnDoe.com>
sub   2048R/D7422C09 2015-04-04

#make file containing secret - please do it with an editor!
temp@old-laptop:~$ echo "1234" > ~/passfile

#encrypt file
temp@old-laptop:~$ gpg -e -r jd@JohnDoe.com ~/passfile

#remove plaintext file
temp@old-laptop:~$ ls
examples.desktop  passfile  passfile.gpg
temp@old-laptop:~$ rm passfile

#decrypt file
temp@old-laptop:~$ gpg -d ~/passfile.gpg

You need a passphrase to unlock the secret key for
user: "John Doe (example) <jd@JohnDoe.com>"
2048-bit RSA key, ID D7422C09, created 2015-04-04 (main key ID CFE3FAA2)

can't connect to `/run/user/1000/keyring-WFzMA0/gpg': Permission denied
gpg: can't connect to `/run/user/1000/keyring-WFzMA0/gpg': connect failed
gpg: encrypted with 2048-bit RSA key, ID D7422C09, created 2015-04-04
      "John Doe (example) <jd@JohnDoe.com>"
1234

But, how do I tie that into an expect script for remotely logging in?

I initially found the exec statement for expect.

...
set password [exec gpg -d passfile.gpg]
...

It just didn't work properly for me.  GPG would prompt me for my passphrase, but it would output some notification text and the key to my display, and while it would be stored in the variable, it confused expect and didn't properly spawn my ssh connection. 

If I faked out getting the value directly, it would function as expected

...
set password [exec echo 1234]
...

Hrm... so what was I doing wrong? If I ran the same command and grabbed the output and redirected it to a file, it only contained my password.

gpg -d passfile.gpg > passfile.out
cat passfile.out

After reading about 50 different stack overflow posts that mostly said 'Why don't you just use ssh keys' or just had examples in plain text, I FINALLY found a clue!

http://ubuntuforums.org/showthread.php?t=1790094

This poor guy got some bad advice, where they aren't using the exec statement (he's just assigning the command string to the password variable), but he had the part that I missed.... the --quiet and --no-tty arguments!

once I added those to my gpg command, everything finally worked!

Here is my final example script -

#!/usr/bin/expect
 set pass [exec gpg --decrypt --quiet --no-tty passfile.gpg]

spawn ssh temp@localhost
expect "password:"
send "$pass\r"
interact

What's the diff?

Hello everyone!


Have you ever had to compare two files?  It's pretty easy on linux, there is a nice standard tool called 'diff' which can show you the differences between two text files.  you also can use vimdiff to get a nicer way of seeing the differences, word by word instead of just line by line...

But what do you do if you need to diff two jar files?  or if you want to diff two binary files while also displaying any strings that happen to be present?

Behold, a nice script that will let you use almost any tool on your path to output a representation of, well, ANYTHING, and then diff it with the tool of your choice.  I may end up developing a version that could take different command line arguments to change things up, but right now I just edit the file and save it with different names, like jardiff.pl, binarydiff.pl, etc...

jardiff.pl

#!/usr/bin/perl -w
use strict;
use warnings;
use diagnostics;

my $left = shift();
my $right = shift();

die unless ((-e $left) and (-e $right));

my $vis_tool = "jar -tf";
my $diff_tool = "vimdiff";

my $cmd="bash -c \"$diff_tool <($vis_tool $left) <$vis_tool $right)\"";
exit system($cmd);

Lets go over what is going on with each line...

#!/usr/bin/perl -w
use strict;
use warnings;
use diagnostics;

A good way to start out perl scripts, first line tells your shell what program to execute, and the rest of the lines are good ways of checking yourself if you make a mistake.. giving useful parsing error messages, enforcing stricter syntax, etc..

my $left = shift();
my $right = shift();

This brings the 1st and 2nd arguments off the command line and stores them in the $left and $right values.

die unless ((-e $left) and (-e $right));

This line exits the script if both the left and right arguments do not exist as files on your system.  I should add a usage statement to make things easier.

my $vis_tool = "jar -tf";
my $diff_tool = "vimdiff";

Here is where our choices come in.  In this instance, our visualization tool is 'jar -tf', which lists the contents of a jar specified by the file you pass to the command.  If you also care about the dates and timestamps, you can add the -v command to also display that additional information.

The $diff_tool is pretty self explanatory... you can use diff, vimdiff, or whatever diff program you have available.

my $cmd="bash -c \"$diff_tool <($vis_tool $left) <$(vis_tool $right)\"";
exit system($cmd);

And this simply puts all the parts together, and runs it on the shell. Lets work left to right.

To make sure we execute this with bash, we use bash -c \" $CMD_HERE \" to execute the command.  We need to escape the quotes because we quoted the string to assign it to my $cmd.

The $diff_tool is pretty simple, we just envoke the command we specified earlier.  In most cases, your diff tool expects to get two files to compare (you can use a tool like diff3 to do a three-way diff).

This brings us to the <( ... )operator.  This executes the given command in a subshell, and output gets passed in as a file handle back to the original command.  If we didn't do this, we would have to dump the output of each of the individual $vis_tool commands to a temp file, and then pass each file to the diff tool, and then clean up our temp files.

Finally, we actually execute the command with

exit system($cmd);

The other really useful way that I make this script is my 'binary' diff version.  Most people would use hexdump to  get a more diff-able version of a binary file, but I actually prefer to use xxd to get output that is both in hex and it attempts to 'stringify' as much of the binary as it can, and it shows it off to the side.

daryl$ xxd xpp3_min-1.1.4c.jar | head 
0000000: 504b 0304 0a00 0000 0000 12a5 6a35 0000  PK..........j5..
0000010: 0000 0000 0000 0000 0000 0900 0000 4d45  ..............ME
0000020: 5441 2d49 4e46 2f50 4b03 040a 0000 0008  TA-INF/PK.......
0000030: 0011 a56a 35e5 823f 7b5e 0000 006a 0000  ...j5..?{^...j..
0000040: 0014 0000 004d 4554 412d 494e 462f 4d41  .....META-INF/MA
0000050: 4e49 4645 5354 2e4d 46f3 4dcc cb4c 4b2d  NIFEST.MF.M..LK-
0000060: 2ed1 0d4b 2d2a cecc cfb3 5230 d433 e0e5  ...K-*....R0.3..
0000070: 72cc 4312 712c 484c ce48 5500 8a01 25cd  r.C.q,HL.HU...%.
0000080: f48c 78b9 9c8b 5213 4b52 5374 9d2a 41ea  ..x...R.KRSt.*A.
0000090: 4df5 0ce2 0d8c 7593 0ccc 1534 824b f314  M.....u....4.K..

This diff technique has been very helpful for me comparing all kinds of files, and I hope it can help you too!

Expect the unexpected!

I have been working with linux since Highschool (which isn't as long ago for me as it is for some), and I have just found a new command-line tool!

Expect

I was attempting to automate the 'store user config' process, and the weblogic.Admin way of doing it wasn't behaving(complaining about invalid pad in the key), so I was defaulting back to the WLST command.  I ran into the same problem described here.  Since in my specific scenario, I needed some environment configuration that didn't translate properly, I needed a way to execute it directly in my bash script.

Using the following WLST script (here's the one from StackOverflow above)

#storeUserConfig.py
connect(user1,p@ss,'t3://myhost:9999')
storeUserConfig(userConfigFile='userconfig.secure',userKeyFile='userkey.secure') 
disconnect() 
exit()

I tried piping output directly into the command.... which didn't work!

yes | java -cp $PATH_TO_WEBLOGIC_JAR weblogic.WLST storeUserConfig.py"

and

java -cp $PATH_TO_WEBLOGIC_JAR weblogic.WLST storeUserConfig.py" << EOF
y
EOF

After a coworker suggested "expect", I came up with the following solution...

#createKeyAndConfig.sh

## Do some stuff to set up the env similar to weblogic's included stopWebLogic.sh
...

CMD="java -cp $PATH_TO_WEBLOGIC_JAR weblogic.WLST storeUserConfig.py"

expect -c "
  set timeout -1
  spawn \"$CMD\"
  expect \"y or n\"
  send \"y\r\"
  expect eof
"

The expect command will 'spawn' the java process.  It will wait until it sees the "y or n" portion of output from the script, when it will then 'send' a y followed by a newline to the program.

The first line, 'set timeout -1' was because creating the key file takes longer then the default timeout in expect, so it would terminate early (leaving behind a 0 length userkey.secure file)

Managing puppet modules with vagrant

A little while ago, I started my journey learning about DevOps, starting out by diving into some of the tooling (Puppet), and adding a few DevOps podcasts into my rotation.

One problem that plagued the process from the beginning... how to handle puppet modules?   Here's the evolution of my thought process...


Stage 1... ignorance

I simple used GIT to control my /etc/puppet/modules/ directory!  Lots of redundant code in here that was already saved in other git repos I could/should just fetch, which gives me an idea...

Stage 2..... frustration
Git Submodules!  I was just manually pulling down most of my modules directly from their git repos instead of though puppet forge anyway, so how about adding them as submodules to my repo, and having git fetch them for me!

Stage 3...... starting automation
Now that I was trying to build easy to run Vagrantfiles that had my puppet configs, I need some way to set it all up....  Initial trick.... adding a 'setup' script in my vagrant directory that pre-downloads all the modules that I run before the first setup! (either by directly calling git clone on the repo, or by calling git submodule init).  This was still a 'prestep' to running vagrant-up, so that didn't quite feel right

Stage 4...... feels right!
One of the first things I did after joining twitter was to ask #puppetize what the best way to manage modules was, and they did not disappoint!  I was redirected to this blog, which shows how you can use puppet librarian to manage your modules, but as to not require everyone to have ruby-gems installed on their main machine, they pull it down inside of their vagrant instance (since most vagrant images already have ruby-gems installed since ruby is a core component of puppet and chef.



Utilizing Configuration management with Vagrant is a powerful tool to help build easily repeatable environments for  both Dev and Deploy!  I hope this helps someone!  I know it was quite a struggle initially!

Weblogic Identity Asserter and Athorization Provider in one!

When trying to create a Custom Identity Asserter AND Authorization Provider MBean for weblogic, I ran into some trouble..  After following all of the examples, I was unable to get my MBean to behave as both!

While trying to find solutions, my googleFu turned up a few lead that left me feeling like it could not be done.

http://www.ateam-oracle.com/why-do-i-need-an-authenticator-when-i-have-an-identity-asserter/

https://community.oracle.com/thread/796040


 The problem I had was if my MDF file extended IdentityAsserter, it would not contain a method to 'getControlFlag' to pass to the LoginModule.

I found this great example at http://danielveselka.blogspot.com/2011/10/weblogic-custom-identity-asserter.html. The only problem?  He skirts around the same issue by manually assigning a constant value to the controlFlag that gets passed to the AppConfigurationEntry in the getConfiguration() method.  

 public void initialize(ProviderMBean mbean, SecurityServices services)
  {
    System.out.println("SimpleSampleIdentityAsserterProviderImpl.initialize");
    SimpleSampleIdentityAsserterMBean myMBean = (SimpleSampleIdentityAsserterMBean)mbean;
    description  = myMBean.getDescription() + "\n" + myMBean.getVersion();
    
    controlFlag = LoginModuleControlFlag.SUFFICIENT;
...
} 

  private AppConfigurationEntry getConfiguration(HashMap options)
  {
      System.out.println("SimpleSampleIdentityAsserterProviderImpl: getConfiguration");
    // make sure to specify the simple sample authenticator's login module
    // and to use the control flag from the simple sample authenticator's mbean.
    return new
      AppConfigurationEntry(
        "examples.security.providers.authentication.simple.SimpleSampleLoginModuleImpl",
        controlFlag,
        options
      );
  }

In that example, he has the following in the MDF.  If I had this, My MBean would not build with the getControlFlag() method.

 Extends       = "weblogic.management.security.authentication.IdentityAsserter"

If I extended the Authenticator, my assertIdentity() methods from the IdentityAsserter would never get called, but I WOULD have the getControlFlag() method

 Extends       = "weblogic.management.security.authentication.Authenticator"

Turns out this is an easy fix.  There is an attribute value for Implements in the MDF Element Syntax. Having an MBeanType similar to the following solved my problem!

<MBeanType
 Name          = "SimpleSampleIdentityAsserter"
 DisplayName   = "SimpleSampleIdentityAsserter"
 Package       = "examples.security.providers.identityassertion.simple"
 Extends       = "weblogic.management.security.authentication.IdentityAsserter"
 Implements    = "weblogic.management.security.authentication.Authenticator"
 PersistPolicy = "OnUpdate"
>

This allows it to implement all the features of both, but be a single entry that you can add to your weblogic security realm.

Kill-a-watt for 240V appliances?

I've previously used a standard kill-a-watt device to check on power usage around my house.  I wanted to be able to measure more then my standard three prong, 120V receptacles however. 

If you want to measure 240V appliances, I have stumbled across someone who built something to do just that!

If you want to measure things which are hard-wired to your panel, or your whole panel itself, You can look to openenergymonitor.org for information and their shop where you can order any of the parts you would need.  I haven't ordered anything from them yet, but they are based in the UK.