Find non-utf8 characters in your puppet code!

#vDM30in30 - 6

I was checking out the #puppet channel on irc this morning, and someone came in asking for help with the following error..

err: Could not retrieve catalog from remote server: Error 400 on SERVER: (<unknown>): control characters are not allowed at line 1 column 1

I couldn't find anything puppet specific, but I found several posts referencing sidekiq and having that error, and it seemed to be due to the presence of non UTF-8 characters in their files/code.

I found a few answers on stackoverflow for detecting non UTF-8 chars..

http://stackoverflow.com/a/9395552 or http://stackoverflow.com/a/13702856

And to add in the 'search hiera and puppet and erb' files find option.  Here is the 'exclude chars we thing are not in range' command

find $CODEBASE/ -type f \( -name '*.pp' -o -name '*.yaml' -o -name '*.erb' \) | xargs grep --color='auto' -P -n "[\x80-\xFF]"

Here is the 'exclude chars that do not match the expected range' command

find $CODEBASE/ -type f \( -name '*.pp' -o -name '*.yaml' -o -name '*.erb' \) | xargs grep --color='auto' -P -n "[^\x00-\x7F]"

This is something that you could even add as a validation check to your CI system to prevent you from checking in invalid characters to your puppet codebase.