Monday, February 12, 2024

Installing gems in PDK on airgapped/offline system without functional rubygems.org

 I ran into a problem recently installing gems in an offline system... I am not understanding something about how the `pdk bundle` setup creates it's cache directories.


If you try to do a `pdk bundle exec gem install ~/path/to/module.gem`, it will place it's cache files in ~/.gem/ruby, but the PDK expects it's files in ~/.pdk/cache/ruby/


I followed these basic steps to get it 'working' enough...

  1. Copy over .gem file and it's dependencies to the offline system to be able to do a local `gem install`
  2. Install it using `pdk bundle gem install ~/gems/*`
  3. Copy over the installed gems from ~/.gem to ~/.pdk/cache - `cp --no-clobber ~/.gem/ruby ~/.pdk/cache/`
  4. Update your Gemfile (in your PDK base directory) to include the gems you want to local install
  5. `pdk bundle install --local` to install the gems from the gemfile, but source them from the cache that you just manually populated.  
I'd love to know if there is a better way to do this!

Compare a whole file content with rspec-puppet

 I was attempting to use the entire body of a file for a quick and dirty 'golden master' style test while I was refactoring some things.

I ran into a problem with having a multi-line string representing the full body of my output - I would run into this problem after putting in something like the following( https://github.com/puppetlabs/rspec-puppet/issues/100 ):

Failure/Error: it { is expected.to contain_file('/mock/file.sh').with_content(multiline_content) }
  expected that the catalogue would contain File[/mock/file.sh] with content set to supplied string
  Diff: 
    <The diff is empty, are your objects producing identical '#inspect' output?>

And my test looked something like this ...

my_content=<<-CONTENT.chomp
file line one
file line two
file last line
CONTENT

on_supported_os.each do |os, os_facts|
  context "on #{os}" do
  it { is_expected.to contain_file('/mock/file.sh').with_content(my_content) }
  end
end

Since I was unable to figure out a 'proper' solution (with and without the chomp) for this problem, I did the next best thing - I simply looped over each line of the file input, and made sure all the lines were in the content - this could give a false 'passing' as it doesn't prevent extra lines from being in there, or the line order isn't tracked, but it was 'good enough' for my short purpose

my_content=<<-CONTENT.chomp
file line one
file line two
file last line
CONTENT

on_supported_os.each do |os, os_facts|
  context "on #{os}" do
  it {
    my_content.each_line do |line|
      is_expected.to contain_file('/mock/file.sh').with_content(/#{Regexp.quote(line)}/) 
  }
  end
end


Hope this helps someone, or perhaps someone see's this and can tell me what I'm doing wrong!

Thursday, December 1, 2022

Into the world of 3d printing

I have long been interested in 3d printing, but I have mostly had other people print for me.  I got myself a Prusa MK3s+, and have been doing the SD card shuffle to get my prints to the printer, which has been working flawlessly.  


I have a raspberry pi lying around, but for reasons I cannot figure out, my previous install had docker, docker-compose, and k3s installed on it.  Whenever I accessed the pi, I would always get a 404, and I didn't see any processes listening to 80/443 on my system.  There was probably something that the k3s install had done, but I wasn't able to track it down.  

To verify that I didn't have some other phantom issue on my network, I pulled down the latest copy of raspbian, cloned https://github.com/OctoPrint/octoprint-docker and went to install docker-compose, using the way I knew (which was apparently the 'old way'), through pip.

I tried to `sudo pip3 install docker-compose` and ran into an error that I hadn't seen before when installing python modules.... `error: can't find Rust compiler`...  One of the dependencies of docker-compose is the cryptography package, and the newest version that was being pulled in now requires rust to build it.  The error message itself and various resources online said they've fixed it by updating pip... `sudo pip3 install --upgrade pip`, but that didn't solve my problem.  

I installed the rust tools `sudo pip3 install setuptools_rust` but it still failed the same way.  I installed the packages for my raspberry pi using apt( `sudo apt-get install cargo`, but they had to old of a compiler... The dependency required 1.56, and the package installed for raspbian was 1.42.

I followed the instructions from the error installing the crypto module, and that wanted me to use `rustup` to manage my rust version.  I pulled down the rustup installer, and that went fine, but when I tried to install docker-compose with sudo (a mistake, but oh well, I'll be reinstalling the OS onto my harddrive instead of an SD card later), the install of docker-compose still failed, due to my user having rustc on their path, but the root user not having it.  So I switched to a root shell, installed rustup, and was able to run my pip install of docker-compose.


As a side path, while I was trying to figure out how to install docker-compose, I saw some mention that the python version was v1, and v2 was a statically linked go binary to run as a docker plugin.... which led me to having to do a little  bit of manual fighting with https://github.com/docker/compose ... The install script was detecting me as arm64, but I had installed the 32 bit OS, and my system reported as arm7.  So I had to manually download the correct build, and update the binary to use that instead of what it pulled down.


the Octoprint-docker guides are all using the old docker-compose syntax, so I needed to use  utility called compose-switch to convert my syntax for me..... one problem, they didn't build the arm7 binary, only amd64 and arm64 (I had installed the 32 bit os..) 

So no problem right?  Someone even had a snippet in a github issue to cross-compile it using the go toolchain in a docker run on my laptop then copy it over to my pi...   But when I was following the advice in the last comment in the issue thread, they mentioned that the initial poster had the path wrong, but I was getting an error during the `docker cp` step... After a bit of digging, it turned out the initial suggestion of copying from /go/bin/linux_arm/compose-switch was correct, and /go/bin/compose-switch was wrong.  Once I fixed that up, I could copy over the compose-swich binary and manually follow the setup steps to set compose-switch as the etc-alternatives for docker-compose, which would remap my commands to `docker compose`.  


Once I fixe all the above, I was able to run `docker-compose up -d` and see octoprint running on my pi!


I didn't have my printer plugged in yet, but I was seeing the interface and not getting a 404 when trying to contact my pi.  

I did want to verify that the webcam worked properly, and I had to remember to go into `raspi-config` and enable 3. Interface Options and I1 Legacy Camera Enable.  Once I did that (and rebooted), and updated my paths for my camera according to the octoprint docker image readme, and added the right env vars and devices to my docker compose file (uncommented from the octoprint-docker provided docker-compose.yml file, I was able to see my camera stills/stream from within the Octoprint UI.


Lots of jumping around for something that I thought was going to be about 3 commands from a fresh raspbian install!

Tuesday, October 25, 2022

Shaking off the dust

 I've resolved to start keeping track of helpful information again, so reconfigured access to the ol' blog.  


Working on copying down some videos to an old phone for our daughter to watch in airplane mode so the main phones aren't used/abused/reconfigured...   There are a few songs that are more bearable at different playback speeds.. Stumbled upon this comment on a thread that gave a good example of modifying both the audio and video playback speeds at the same time with a complex filter in ffmpeg.


https://davidwalsh.name/video-speed#comment-509672

Relevant code snippet, to use the same 'multiplier' even though in one case it's a divisor.  

ffmpeg.exe -i input.mp4 -filter_complex "[0:v]setpts=PTS/1.3[v];[0:a]atempo=1.3[a]" -map "[v]" -map "[a]" output.mp4

Friday, December 15, 2017

Today, I was trying to experiment a bit with the go programming language.  I wanted to get started, but the 'install' section of the golang site seemed a bit daunting, and I figured I should use a tool similar to 'RVM' or 'rbenv' from the ruby space for go.  I discovered https://github.com/moovweb/gvm/, but ran into a small hiccup.  The install instructions were to pipe some commands into a bash shell, but after running them, the `gvm` command appeared to be invoking something else!  

From the output, it appeared to be sdkman, which previously was called gvm for 'groovy version manager'. It appears that the install of sdk man places it's files into ~/.sdkman, in addition to calling thing in all three of ~/.profile, ~/.bashrc, and ~/.bash_profile.  I had to explicitly clear out the statements sourcing the sdkman binaries to allow gvm to function properly.  

After clearing out sdkman, I was able to successfully get the latest go installed (after installing go 1.4 from a binary, due to the source compile failing).  `which gvm` now pointted at the go gvm, instead of sdkman!

Tuesday, January 10, 2017

rspec-puppet test with array parameter

Today I stumbled onto something that was much harder than I expected it to be while trying to write an rspec-puppet test.

Say you have a resource with a parameter that is an array, and you only want to validate that the array contains an element.

In our case, say that we have the following code somewhere in our puppet manifests

        host { 'foo.local':
          ip => '1.2.3.4',
          host_aliases => [ 'foo1.local', 'foo2.local' ],
        }

It is pretty direct to validate the exact array inside of an rspec-puppet test

    it { should contain_host('foo.local').with_host_aliases( ['foo1.local', 'foo2.local'] ) }

But that requires a very specific 'expected' value to be passed into the matcher.  What if we just wanted to see if the parameter array includes a given value?  

In a conversation about his problem in the #voxpupuli channel on irc@freenode, dev_el_ops mentioned "one of the design problems of rspec-pupet is that it  doesn't expose property values to the regular rspec matchers"

Since we couldn't get the values directly from the matchers, I had to come up with a different way to approach the problem, which is not quite as readable as I would have wanted.

    it {
        host_aliases = catalogue.resource('host', 'foo.local').send(:parameters)[:host_aliases]
        expect(host_aliases).to include('foo1.local')
    }

Lets break down what is happening.  catalogue.resource('host', 'foo.local') will fetch us the catalog's resource object of type 'host' with the title 'foo.local'.  we can then use 'send' to call the private parameters method on the object to get back it's parameter hash, which we then reference the :host_aliases key to retrieve our array.  

Once we have the array (stored in host_aliases), we can then use a simple 'expect' call with an 'include' matcher to check that the given array includes the value we expect.


Here is a gist of the entire example.

Saturday, November 12, 2016



#vDM30in30 - 11


I had some time yesterday, so I finally had the chance to start watching some of the puppet-conf presentations that I did not get a chance to attend personally.. As a reminder, you can find the slides and videos here.

I may morph this into a more 'static' page on my site, and dive in with some thoughts about certain presentations..


Attended -

A Roadmap for a Platform: Mixing Metaphors for Fun and Profit – Eric Sorenson, Puppet (videos) (slides)
Proof of Concept to 30K+ Hosts with Puppet - Petersen Allen, Salesforce (videos) (slides)
Cloud, Containers & the Impact on IT - Jeffrey Snover, Microsoft (videos) (slides)

Puppet Troubleshooting - Thomas Uphill (videos) (slides)
Turning Pain Into Gain - A Unit Testing Story - Nadeem Ahmad & Jordan Moldow (videos)(slides)
Watching the Puppet Show – Sean Porter, Heavy Water Operations (videos) (slides)
Avoiding Toxic Technical Debt Derivatives – R. Tyler Croy, CloudBees, Inc. (videos) (slides)
Using HashiCorp's Vault With Puppet – Seth Vargo, HashiCorp (videos) (slides)
Running Puppet Software in Docker Containers – Gareth Rushgrove, Puppet (videos) (slides)
DevOps Where You Wouldn't Have Expected – Thomas Limoncelli, StackOverflow.com (videos) (slides)
The Future of Testing Puppet Code – Gareth Rushgrove, Puppet (videos) (slides)

Watched -

Service Discovery and Puppet – Marc Cluet, Ukon Cherry (video) (slides)
Enjoying the Journey from Puppet 3.x to 4.x – Rob Nelson, AT&T (videos) (slides)
Collaboration and Empowerment: Driving Change in Infrastructure with Culture – Martin Jackson, Walmart (videos) (slides)
A Look at Looking in the Mirror Actionable Retrospectives - J. Paul Reed (videos) (slides)
External Data in Puppet 4 – R.I. Pienaar (videos) (slides)
Scaling Puppet on AWS ECS with Terraform and Docker – Maxime Visonneau, Trainline (videos) (slides)
Case Study: Puppets in the Government – Kathy Lee (co-author: Glenn Bailey) (videos) (slides)

Want to Watch -

The Truth, Nothing but the Truth: Why Type Systems are Important to Configuration Management – Henrik Lindberg, Puppet (videos) (slides)
Scaling Puppet and Puppet Culture at GitHub - Kevin Paulisse (videos) (slides)
Implementing Puppet within a Complex Enterprise – Jerry Caupain, KPN (videos) (slides)
Moving from Exec to Types and Providers – Martin Alfke, example42 GmbH (videos) (slides)
A Year in Open Source: Automated Compliance With Puppet – Trevor Vaughan, Onyx Point, Inc. (videos) (slides)
Automating Datastore Fleets with Puppet – Joseph Lynch, Yelp (videos) (slides)
Closing the Loop: Direct Change Control with Puppet – Nick Lewis, Puppet (videos) (slides)
Continuous Delivery and DevOps with Jenkins and Puppet Enterprise – Carl Caum, Puppet & Brian Dawson, CloudBees (videos) (slides)
Debugging Diversity – Anjuan Simmons, Assemble Systems (videos) (slides)
Direct Puppet and Application Management for the Puppet Platform – Ryan Coleman, Puppet (videos) (slides)
Puppet 4.x: The Low WAT-tage Edition – Nick Fagerlund, Puppet (videos) (slides)
Puppet Best Practices: Roles & Profiles – Gary Larizza, Puppet (videos) (slides)
Puppet Design Patterns - Lessons From the Gang of Four - David Danzilio (videos) (slides)
Device-Based Modules: Making Them as Simple as a Light Switch – TP Honey, Puppet (videos) (slides)
There is No “I” in DevOps – Bart Driscoll, Dell EMC (videos) (slides)
Writing Custom Types to Manage Web-Based Applications – Tim Cinel, Atlassian (videos) (slides)
Successful Puppet Implementation in Large Organizations – James Sweeny, Puppet (videos) (slides)
Best Practices for Puppet in the Cloud – Randall Hunt, Amazon & Andrew Popp, Servicechannel.com (videos) (slides)
Pulling the Strings to Containerize Your Life - Scott Coulton (videos) (slides)