Sam Stelfox

Thoughts from a systems hacker and developer.

Cli

Fail Fast in Bash Scripts

I found myself writing another bash script that should exit should any of the few commands within it fail to run. As I began writing some error handling after each command, and isolating the sections into bash functions I figured there had to be a better way. After a little Googling and a trip through the bash manpages sure enough:

#!/bin/bash

function error_handler() {
  echo "Error occurred in script at line: ${1}."
  echo "Line exited with status: ${2}"
}

trap 'error_handler ${LINENO} $?' ERR

set -o errexit
set -o errtrace
set -o nounset

echo "Everything is running fine..."

# A command outside of a conditional that will always return a exit code of 1
test 1 -eq 0

echo "This will never run, as a command has failed"
echo "Using unset variable ${TEST} will also cause this script to exit"

The first piece of that is setting up an error handler that will get run whenever an error condition occurs with the script. You can use this section to roll back any changes or cleanup your environment as well as give you some debug information about the failure.

I'm then setting a few bash options, The following is a description taken more or less directly from the bash man pages:

-o errexit: Exit immediately if a pipeline (which may consist of a single simple command), a subshell command enclosed in parentheses, or one of the commands executed as part of a command list enclosed by braces exits with a non-zero status.

-o errtrace: If set, any trap on ERR is inherited by shell functions, command substitutions, and commands execute in a subshell environment.

-o nounset: Treat unset variables and parameters other than the special parameters "@" and "*" as an error when performing parameter expansion.

If anything goes wrong in the script it will fail once, fail fast, and let you know where it died.

Ruby's Option Parser - a More Complete Example

Recently while writing a Ruby program I needed to parse some command line options. Helpfully Ruby provides a module named OptionParser to make this easy. I found a few parts of the documentation ambiguous and a few others down right confusing.

The catch I hit was the required field. In my mind the definition of a required argument is something that needs to be passed on the commandline to continue. What OptionParser actually means is that a value isn't required when the argument is passed. OptionParser already provides boolean switches, so when someone would use an optional switch is beyond me.

To make it a little more clear and to have something to work from in the future I created the following chunk of code that includes a Configuration singleton that can be used anywhere within your codebase to access the run-time configuration, a sample parser with a wide range of different types of options, and it will load configuration from a file named config.yml in the same directory.

I feel like the following is a much more complete explanation of how OptionParser is supposed to be used with supporting code.

#!/usr/bin/env ruby

# This file provides an example of creating a command line application with a
# wide variety of command line options, parsing and the like as well as global
# configuration singleton that can be relied on throughout a program.
#
# This entire setup lives within the "Example" module. These are really common
# names and it would be a shame to override required functionality in other code
# that wasn't properly namespaced.

require 'optparse'
require 'singleton'
require 'yaml'

module Example
  # Defines the available configuration options for the configuration
  ConfigurationStruct = Struct.new(:enum, :list, :required, :optional, :verbose, :float)

  class Configuration
    include Singleton

    # Initialize the configuration and set defaults:
    @@config = ConfigurationStruct.new

    # This is where the defaults are being set
    @@config.enum = :one
    @@config.list = []
    @@config.optional = nil
    @@config.verbose = false

    def self.config
      yield(@@config) if block_given?
      @@config
    end

    # Loads a YAML configuration file and sets each of the configuration values to
    # whats in the file.
    def self.load(file)
      YAML::load_file(file).each do |key, value|
        self.send("#{key}=", value)
      end
    end

    # This provides an easy way to dump the configuration as a hash
    def self.to_hash
      Hash[@@config.each_pair.to_a]
    end

    # Pass any other calls (most likely attribute setters/getters on to the
    # configuration as a way to easily set/get attribute values 
    def self.method_missing(method, *args, &block)
      if @@config.respond_to?(method)
        @@config.send(method, *args, &block)
      else
        raise NoMethodError
      end
    end

    # Handles validating the configuration that has been loaded/configured
    def self.validate!
      valid = true

      valid = false if Configuration.required.nil?

      raise ArgumentError unless valid
    end
  end

  class ConfigurationParser
    def self.parse(args)
      opts = OptionParser.new do |parser|

        parser.separator ""
        parser.separator "Specific options:"

        parser.on("--enum ENUM", [:one, :two, :three], "This field requires one of a set of predefined values be", "set. If wrapped in brackets this option can be set to nil.") do |setting|
          Configuration.enum = setting
        end

        parser.on("-l", "--list x,y", Array, "This command flag takes a comma separated list (without", "spaces) of values and turns it into an array. This requires", "at least one argument.") do |setting|
          Configuration.list = setting
        end

        parser.on("--[no-]verbose", "This is a common boolean flag, setting verbosity to either", "true or false.") do |setting|
          Configuration.verbose = setting
        end

        parser.on("--optional [STR]", "This command doesn't require a string to be passed to it, if", "nothing is passed it will be nil. No error will be raised if", "nothing is passed to it that logic needs to be handled", "yourself.") do |setting|
          Configuration.optional = setting
        end

        parser.on("-r", "--required STR", "This command requires a string to be passed to it.") do |setting|
          Configuration.required = setting
        end

        parser.on("--float NUM", Float, "This command will only accept an integer or a float.") do |setting|
          Configuration.float = setting
        end

        parser.on_tail("-h", "--help", "--usage", "Show this usage message and quit.") do |setting|
          puts parser.help
          exit
        end

        parser.on_tail("-v", "--version", "Show version information about this program and quit.") do
          puts "Option Parser Example v1.0.0"
          exit
        end
      end

      opts.parse!(args)
    end
  end
end

if File.exists?("config.yml")
  Example::Configuration.load("config.yml")
end

Example::ConfigurationParser.parse(ARGV)
Example::Configuration.validate!

require "json"
puts JSON.pretty_generate(Example::Configuration.to_hash)

Using VIM as Your Password Manager

There are all kinds of password managers out there. Everything from web services that are quite solid and respectible, to native desktop apps.

A lot of these are simply too heavy for me, involve installing software on a computer to access in addition to sharing the file around, or required you to remember multiple account details before you could get access to any individual password.

Due too the various complexities and lack of matching use cases a couple years ago I set out to develop my own open-source version of Passpack. In the interim though I needed a solution for keeping track of my hundreds of various accounts and their passwords.

Around this time I was ramping up my usage of vim and happened to come across a very fortunate command entirely by accident. Enter vimcrypt.

For any plaintext file you, while in command mode you can type the command :X and it will ask you for a password to encrypt your file with. By default this uses a remarkably weak algorithm called pkzip which isn't secure enough for me to trust it with my keys.

Since vim 7.3 and later, :X has also supported an additional cipher; The much stronger blowfish algorithm. You can enable this by running the command :set cryptmethod=blowfish. I chose to add the following lines to my ~/.vimrc file:

" When encrypting any file, use the much stronger blowfish algorithm
set cryptmethod=blowfish

This was a fantastic interim solution as I have yet to find a development or production linux system that hasn't been excessively locked down (and probably not somewhere I'd put my password file anyway) that didn't already have vim installed.

Using this personally required me coming up with a pseudo-file format that would allow me to quickly and easily find the credentials I needed. I settled on the simple format shown off below:

Oneline Account Description
  Site: <URL of Site's login page>
  Username: <username for the site>
  Password: <password for the site>
  Email: <email I used to register>

  Login with: <email|username> # Only necessary when I have both

  ** Address on file **
  ** Phone on file **

You'll notice I also used this to keep track of whether an account had physical information tied to it. When I moved this made it very quick for me to search for accounts that I needed to update with my new mailing address.

As with many solutions this "temporary" one became more and more permanent as my motivation to build the Passpack competitor dwindled. My problem had been solved and I was no longer compelled to put any effort into a solution.

If this still isn't strong enough for your tastes, the vim wiki has some additional ways you can encrypt your files. These all require additional setup and failed my requirements in that they generally require additional files or setup before I can access my passwords.

Hope this helps some other weary CLI warrior some trouble. Cheers!

Update: I received a wonderful recommendation from a user named sigzero over on Reddit. For additional security they added the following line to their ~/.vimrc file.

autocmd BufReadPost * if &key != "" | set noswapfile nowritebackup viminfo= nobackup noshelltemp history=0 secure | endif

It disables additional files that vim may write copies to such as swap files and backups, prevents dangerous shell commands, and prevents vim from storing a history of commands.

Update 2: I received another fantastic recommendation from another reddit user, this time from NinlyOne. At their recommendation, I've prepended the following modeline to my password. It automatically folds each password entry to prevent potential shoulder surfing. You can open up an entry using the command zo and close it back up with zc. It's worth noting that this is tied to my indented file format.

# vim: fdm=indent fdn=1 sw=2:

Backing up Gmail with fetchmail

This morning I found myself in need of a large set of emails to test a particular set of code. Ideally these emails would be broken out into easily digestable pieces, and it was strictly for my own personal testing so I wasn't concerned with using my own live data for this test (There will probably be another post on this project later on).

Having used fetchmail with good results in the past I decided it was a good idea to take this opportunity to also backup my Gmail account into the common Maildir format (which essentially breaks out emails into individual files meeting my requirements).

The first step was to enable POP access to my account through Gmail's interface. You can accomplish this with the following steps.

  1. Login to Gmail
  2. Click on the gear icon
  3. Choose settings
  4. Forwarding and POP/IMAP
  5. Enable POP for all mail
  6. When messages are accessed with POP... Keep"
  7. Save Changes.

Ensure you have fetchmail and procmail installed. For me on Fedora this can be accomplished using yum by running the following commands:

$ sudo yum install fetchmail procmail -y

We need to configure fetchmail to let it know where to retrieve our mail from. This configuration file lives at $HOME/.fetchmailrc. By default fetchmail will send all retrieved mail to the local SMTP server over a normal TCP connection. This isn't necessary or ideal, rather we'll additionally supply a local mail delivery agent (procmail) to handle processing the mail into the Maildir format.

poll pop.gmail.com
protocol pop3
timeout 300
port 995
username "[email protected]" password "yourpassword"
keep
ssl
sslcertck
sslproto TLS1
mda "/usr/bin/procmail -m '/home/<username>/.procmailrc'"

Be sure to set the permissions on the .fetchmailrc file to 0600:

$ chmod 0600 $HOME/.fetchmailrc

We'll now need to configure procmail to properly deliver our mail to the local Maildir folder. Procmail's configuration by default lives in $HOME/.procmailrc

LOGFILE=$HOME/.procmail.log
MAILDIR=$HOME
VERBOSE=on

:0
Maildir/

With that done, simply run the fetchmail command. In my experience this can take a while process and it seems like Google limits the number of emails you can download at a time, so you may need to run the command a couple of times to get all your emails.