Sam Stelfox

Thoughts from a software engineer, systems hacker and Linux gubernāre.

Better Practices With Sudo

I work with a lot of different linux machines from embedded devices, to cloud servers and open stack hosts. For many of them I'm either the sole administrator or one of three or less with administrative access. Where there are multiple administrative users, we all are generally working as backups to each other. We use sudo whenever we need to execute a task with privileges on any of these machines with no direct root login permitted remotely.

I must confess I have established two habits over time that are against best practices with regard to sudo; Using it to execute a root shell only, and not restricting which commands can be run with sudo.

I'm sure many other administrators commit these sins as well. I've always gotten sudo to the 'good enough' point without ever learning how to configure it properly countless times, which mostly meant leaving the distribution's defaults.

At face value, executing a shell this way doen't seem to pose a problem. We use auditd to record administrative changes, and the kernel can track our original login UID and record that in addition to our effective user ID. Permission to use sudo is still restricted to a subset of trusted administrators.

Using this default configuration is forming bad habits and after working through it it's not particularily hard to make a drastic improvement on the granularity of control.

I'm going to work through the changes I've made slowly building up my final config.

These changes, if made incorrectly or with the wrong paths to binaries may effect your ability to get privileged access to the system. I strongly encourage you to maintain a root shell independent of the shell you are using to test just in case you need to revert a breaking change.

Minimal Configuration

Rather than looking at what needs to be changed, or removed I prefer to start with a minimal effective configuration.

Most distribution's default sudo configuration pass through environment variables related to locale and a few others. I have left these out since the way I see sudo executed most commonly (sudo su -), removes any environment variables passed through anyway. If you work on multi-lingual systems or otherwise your administrators make use of multiple system locales, you will want to re-introduce the locale values used.

My entire starting sudo config is the following:

Defaults env_reset
Defaults !visiblepw

root      ALL=(ALL)     ALL
%wheel    ALL=(root)    ALL

This is very similar to most distribution's configurations if you ignore the environment variables and comments. The root user and members of the wheel group can all execute anything as sudo as long as the user can authenticate through PAM and the mechanism won't display their password.

There is also a small restriction in place that ensures members of the wheel group will only be executing commands as the root user. Executing as other user's directly should be a special case and added separately.

Usually distributions also include additional sudo configuration by including all files in /etc/sudoers.d. This config isn't going to be terribly long so we may as well KISS it and not allow the inclusion of other files.

No Need for the su

The first habit I wanted to break was executing sudo su - instead of sudo -s. Generally when sudo is configured correctly, administrators are supposed to minimize the number of times dropping to a root shell. There are always going to be times when a root shell is necessary.

The differences between the two methods of executing a root shell are subtle. They are creating to different types of shells. Executing sudo su - creates a login shell, while sudo -s doesn't. Both can be subtly changed to provide the other type (Adding the -i flag to sudo, or removing the - from su).

A login shell resets your environment, spawns the new user's default shell (in this case root's default shell) and executes the user's profile scripts in addition to the shell's rc files.

By not using a login shell, administrators can keep their preferred shells while allowing selective bits of their configuration (whitelisted environment variables) through to the new session.

By removing su from the process, administrators can enforce permitted root shells just like whitelisting or blacklisting any other binary on the system. The only way to enforce this transition is to blacklist su directly.

A blacklist is added by creating a command alias that includes the commands to be blacklisted, then adjusting ACLs to make use of them. These need to be defined before they're used. Generally this means all command aliases are at the top of the config file. The following command alias will be used for our blacklist. The path to su is valid for CentOS 7, other distributions do vary.

Cmnd_Alias BLACKLIST = /bin/su

To enforce the blacklist the wheel group ACL needs to be adjusted to the following:

%wheel  ALL=(root)  ALL,!BLACKLIST

Now when you try to execute sudo su - you'll instead get this warning after authenticating:

Sorry, user <username> is not allowed to execute '/bin/su -' as root on <hostname>.

This warning will enforce not using the less ideal mechanism.

Brief Interlude on Blacklists

I'm going to be adding several more things to different forms of blacklists inside sudo. Some of these may be unacceptably inconvenient for some environments. If you find the explained reason insufficient to justify the inconvenience and are willing to accept the risk, remove the offender from the blacklist.

There is also always a risk that programs allowed through the blacklist have the ability to execute blacklisted applications as root. The blacklist applies only to direct execution through sudo.

Preventing 'commonly used' escalation vectors does make it that much harder on potential attackers and may allow you see an attack in progress through the logs. This should not be considered perfect though. A good example of these vectors is the utility awk. If allowed to be executed through sudo an unrestricted root shell can be acquired with the following command:

sudo awk 'BEGIN {system("/bin/sh")}'

Editing Files as Root

Commonly when I wanted to edit a particular sensitive configuration file, I would drop to a root shell, then open the file in my preferred editor, possibly saving along the way until I was done. Less commonly I would open my editor directly using sudo skipping the shell entirely.

The partially complete saves as part of that workflow, have caused issues though they're temporary. Sudo provides a utility, sudoedit, that covers this use case. It make a copy of the file to be edited into a temporary directory, and allows you to edit and save as you like. When you're done save the file and it will replace the real file with the temporary one you've been editing.

Editing the sudoers file itself should be done using the visudo command. And can be invoked by:

sudo visudo

It's a good idea to restrict the list of editors that can be used by visudo (this doesn't affect sudoedit at all) by adding the following line (replace this with your preferred, colon separated list of editors):

Defaults editor = /bin/vim:/bin/nano

User Writable Directories

Since the blacklist functionality is based on full paths to binaries, there is a quick way for a user with sudo permissions to bypass the blacklist for a specific program, copy it somewhere else.

When an attacker gets into a system and downloads a binary off their site they want to run with privileges. They'll have to put it somewhere they have permission to write to.

This is less of a threat if you always require authentication to use sudo, trust all your administrators, and are confident their credentials will never be stolen.

A salve to both problems is simply to prevent sudo from executing files in user writable directories, and ensuring it has a sane path to lookup trusted binaries. The following three lines need to be added to the sudoers file:

Cmnd_Alias USER_WRITEABLE = /home/*, /tmp/*, /var/tmp/*
Defaults ignore_dot
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

We also need to modify our wheel ACL to prevent the execution in the aliases locations. Replace your previous line with the following one:

%wheel  ALL=(root)  ALL,!BLACKLIST,!USER_WRITEABLE

Preventing Breakouts

I've already shown that there is a way to abuse individual commands to expose a root shell. There are a few additional common applications that can regularly shell out, advanced text editors, pagers, several unix utilities and any interactive programming shell are easy candidates.

These utilities likely still need to be available for general administrative purposes, but we don't want them in turn executing other programs. Sudo has a trick up it's sleeve for this, the noexec flag.

There are two ways to effectively apply this, a whitelist and a blacklist. I encourage you to try the whitelist approach first as it does offer substantially better protection against this potential abuse.

Before applying this it is useful to know how this works and what it's limitations are. Sudo disables the exec call using LD_PRELOAD and defining an alternate version of the system call. This is largely effective, but will only work with dynamically linked programs (most coming from a distribution are going to be dynamically linked).

Whitelisting Programs w/ Exec Privileges

This is very strict but also very effective. We need to ensure that things we expect and want to be able to execute other programs (like shells) still can. Additionally visudo in turn executes your editor, so it to needs to be able to spawn programs.

Be very sure of the paths in the following change. If you have no shells, or editors that can be executed as root through sudo you may lock yourself out of your system privileges.

Cmnd_Alias SHELLS = /bin/sh, /bin/bash
Cmnd_Alias ALLOWED_EXEC = /usr/sbin/visudo
Defaults noexec
Defaults!ALLOWED_EXEC,SHELLS !noexec

As you use this in your environment you will probably find programs that behave incorrectly and will need to be added to the whitelist. This whitelist (assuming your paths are correct) will at least be enough to allow future modifications of the sudoers file.

Blacklisting Programs w/ Exec Privileges

This is a much milder version of the exec restrictions, and won't catch unknown abuses. This will also have the least impact on normal operations to apply and is better than nothing.

Cmnd_Alias EDITORS     = /bin/vim
Cmnd_Alias PAGERS      = /bin/less, /bin/more
Cmnd_Alias BREAKOUT    = /bin/awk, /bin/find
Cmnd_Alias DEVEL_SHELL = /bin/perl, /bin/python, /bin/ruby

Defaults!EDITORS,PAGERS,BREAKOUT,DEVEL_SHELL noexec

TTYs!

Enforcing the use of TTYs generally prevents non-interactive processes from executing anything as sudo either remotely or locally. Examples of this might be from cron, apache, or from a remote Jenkins server. In almost all cases prevention of this type of execution is the ideal behavior.

There are a couple of very visible search results on this topic that indicate there isn't any security benefit to this, but their are exceptions as well. The argument that seems to have the most merit, is that no special privileges are required to create a PTY. This in turn means an attacking process could spawn the PTY required, and continue it's attack.

The same argument could be used in favor of the option. An attacker would have learn they need to make this adjustment and actively work around it. As the administrator you know the option is set and should be able to work around it more easily than the attacker.

The most common form of pain seems to be remotely executing privileged commands through ssh. By providing the SSH command being executed the '-t' flag twice, the client will force a PTY allocation even when there is no local tty. Other more stubborn use cases can be individually exempted.

When the user already has a local TTY, the sudoers man page calls out to an additional potential attack vector around TTYs under the 'use_pty' option:

A malicious program run under sudo could conceivably fork a background process that retains to the user's terminal device after the main program has finished executing. Use of this option will make that impossible.

I haven't been able to find any attacks that exploit this possibility, but I have yet to be impacted by turning that feature on within sudo. Making both changes can be done by adding the following line to the sudoers config.

Defaults requiretty, use_pty

Notification of Violation

Receiving immediate notification when privilege gain has been attempted can be invaluable to stopping an attacker before they can do any damage. If the linux system has a properly configured MTA forwarding root's email to relevant parties it is recommended to have failure mailed to them directly to take action.

Defaults mail_badpass, mail_no_perms
Defaults mailfrom = root
Defaults mailsub = "Sudo Policy Violation on %H by %u"

The overridden subject provides everything but the command itself (which isn't available through the expanded variables) needed to quickly judge a threat at a glance.

Auditing Interactive Shells

With all the protections put in place so far, we still have no visibility or restrictions on what administrators do with the root shells when they use them. These should hopefully be relatively few and far between.

Built into sudo is an option to record execution of commands. This has proven to be valuable to narrow down things that have gone wrong, or see how something was done before. This may not prove useful as much for an audit tool as a user with root privileges can purge the recordings and logs.

If auditing is the goal, use of the kernel audit subsystem may be a better choice, but will only give you the command and arguments executed. This shows what was displayed to the privileged shell directly. There will be a future article covering the use of the audit subsystem and centralizing the information in a future post.

If you didn't go the whitelist exec route, to enable this you will need to pull in the 'SHELLS' command alias from there to make use of this.

Defaults!SHELLS log_output

Once this is in place you can get a list of recorded sessions using the command:

$ sudo sudoreplay -l
Feb 26 17:56:18 2016 : jdoe : TTY=/dev/pts/7 ; CWD=/home/jdoe ; USER=root ; TSID=000001 ; COMMAND=/bin/bash

To view an individual session provide sudoreplay with the TSID value of the session like so:

$ sudo sudoreplay 000001

Refer to the man page of sudoreplay for additional tricks such as speeding up playback.

Final Config

Some of the options from above I have combined into a single config line. This uses the stricter whitelist policy for exec privileges.

# /etc/sudoers

Cmnd_Alias ALLOWED_EXEC = /usr/sbin/visudo
Cmnd_Alias BLACKLIST = /usr/bin/su
Cmnd_Alias SHELLS = /usr/bin/sh, /usr/bin/bash
Cmnd_Alias USER_WRITEABLE = /home/*, /tmp/*, /var/tmp/*

Defaults env_reset, mail_badpass, mail_no_perms, noexec, requiretty, use_pty
Defaults !visiblepw

Defaults editor = /usr/bin/vim
Defaults mailfrom = root
Defaults mailsub = "Sudo Policy Violation on %H by %u"
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin

Defaults!ALLOWED_EXEC,SHELLS !noexec
Defaults!SHELLS log_output

root    ALL=(ALL)   ALL
%wheel  ALL=(root)  ALL,!BLACKLIST,!USER_WRITEABLE

Sharing Context Between Dependent Rake Tasks

I use Rakefiles quite a bit like traditional Makefiles, in that I specify immediate dependencies for an individual task and Rake will execute all of them. If a file or directory is the dependency and it exists, the task that creates it will be skipped. A contrieved Rakefile example might look like:

file 'sample' do |t|
  puts 'Creating sample directory'
  Dir.mkdir(t.name)
end

file 'sample/population.txt' => ['sample'] do |t|
  puts 'Creating sample population file...'
  # Perhaps download a dataset? Lets just create the file
  File.write(t.name, "---> Very important data <---\n")
end

task :process_population => ['sample/population.txt'] do
  puts 'Check out our data!'
  # Do some processing... whatever you need to...
  puts File.read('sample/population.txt')
end

The first time you run it you'll the following output:

$ rake process_population
Creating sample directory
Creating sample population file...
Check out our data!
---> Very important data <---

And subsequent runs will skip the creation since they're already present:

$ rake process_population
Check out our data!
---> Very important data <---

This is fine for statically implementing file contents, but what if you need additional information to generate the file? With a normal rake task you can provide bracketed arguments to access additional information like so:

task :args_example, :word do |t, args|
  puts "The word is: #{args.word}"
end

You'd use it like so:

$ rake args_example[data]
The word is: data

That information isn't made available to the dependent tasks though so we need to broaden our scope a little bit. There is another way to provide arguments to Rake using key value pairs. This has a bonus that was kind of an obvious solution once I found it. Rake provides the values of key/value pairs to a task via environment variables. Another contrived example of how to use this (specifically with a file dependency example):

file 'passed_state' do |t|
  puts 'Creating state file'
  File.write(t.name, ENV['state'])
end

task :read_state => ['passed_state'] do
  puts File.read('passed_state')
end
$ rake read_state state=something
Creating state file
something

State has been transferred! There is a gotcha, that is handling expiration of data yourself. Passing in state again with a different value you'll see the problem:

$ rake read_state state=notsomething
something

It won't recreate that file again until it's removed which you'll need to handle on your own.

Ruby Code Quality Metrics

I like getting unopionated feedback on the quality of the code I write. Sometimes I can get this from other developers but they tend to get annoyed being asked after every commit whether they consider it an approvement.

There are a few utilities for Ruby codebases such as flay, flog, and rubocop as well as hosted services such as Code Climate that can help you identify chunks of code that can use some work.

While not directly connected to the quality of the code, I also make use of yard and simplecov to assess documentation and test coverage of the codebases I work on.

Using the tools means very little without some reference or understanding doesn't get you very far. For a while I've been using flog and only comparing the numbers against other codebases I control. I finally googled around and found a blog post by a developer named Jake Scruggs from a while ago (2008).

The blog post includes a rough table for assessing scores on individual methods reported from the flog utility. From what I can tell the ranges are still pretty accurate. I've tweaked the descriptions a bit to fit my mental understanding a bit but the table is here:

Method Score Description
0 - 10 Awesome
10 - 20 Decent
20 - 40 Might need refactoring
40 - 60 Should probably review
60 - 100 Danger
100 - 200 Raise the alarm
200+ Seriously what are you doing!?

I wanted to extend this with a second table providing a scale for the overall method average with a more aggressive scale (an individual couple of methods can be justifiably complex but the overall code base shouldn't be riddled with them) but had a hard time working it out.

I've seen some awesome code bases with a score of 6.4 on average, some bad larger ones with 7.8. Even some mediocre ones around a score of 10.6.

I guess I'll have to think more on it...

Creating an Empty Git Branch

Every now and then I find myself wanting to create a new empty branch in an existing repository. It's useful for things such as [Github Pages][1] so you're able to keep your content source in the master branch while only keeping the output in the gh-pages branch. I've also used it for testing a complete rewrite of a code base without the overhead of creating a new repo and copying access permissions.

This is a pretty straight forward trick to do. You create the brach by indicating you want the new branch to be an orphan by passing the '--orphan' flag like so:

git checkout --orphan NEW_BRANCH_NAME

This leaves all the files in place but effectively uncommitted like you just initialized a new repository. Add and commit any files you'd like to keep then delete the rest, everything will still be preserved in the original branches.

With that done you should be able to easily switch just using a normal 'checkout' between your normal branches and this new tree.

[1]:

Unbuffered Pipe Filters

I need to filter a live logstream for only relevant events and quickly hit an issue that I wasn't expecting. The grep in my pipe chain was waiting until it received all the output from the prior command before it began to attempt to filter it.

Reading through the grep man page I came across the --line-buffered flag which provides exactly what I needed. I wasn't using the tail command but it serves really well in this situation to demonstrate the use:

tail -f /var/log/maillog | grep --line-buffered -i error

Hope this saves someone a headache in the future!

Dependency Prelink Issues

While running an aide check on one of my servers after updating it, I started seeing a large number of very concerning warning messages:

/usr/sbin/prelink: /bin/mailx: at least one of file's dependencies has changed since prelinking
Error on exit of prelink child process
/usr/sbin/prelink: /bin/rpm: at least one of file's dependencies has changed since prelinking
Error on exit of prelink child process
/usr/sbin/prelink: /sbin/readahead: at least one of file's dependencies has changed since prelinking
Error on exit of prelink child process
/usr/sbin/prelink: /lib64/libkrb5.so.3.3: at least one of file's dependencies has changed since prelinking
Error on exit of prelink child process
/usr/sbin/prelink: /lib64/libgssapi_krb5.so.2.2: at least one of file's dependencies has changed since prelinking

The list went on with maybe a total of forty packages and libraries. My initial reaction was 'Did I get hacked?'. Before running the updates I ran an aide verification check which returned no issues and the files that were now displaying the issue were in the packages that got updated.

What was the next worse scenario? The packages had been tampered with and I just installed malicious files. This didn't seem likely as the packages are all signed with GPG and an aide check would have caught tampering with my trust database, the gpg binary, or the aide binary. Still a key could have been comprimised.

After some Googling I came across people with similar issues, (including one annoyingly paywalled RedHat article on the issue). Several people simply ended the conversation on the assumption the user with the issue had been hacked. Finally I came across one helpful individual with the fix. The binaries just need to have their prelink cache updated again. This can be accomplished with the following command on CentOS 6.5 (probably the same on others).

/usr/sbin/prelink -av -mR

Update: Ultimately I decided to follow my own advice (search for prelink) and just simply disabled prelinking too prevent it from interferring with aide checks and causing other weird issues. The memory trade-off isn't valuable enough for me.

Fast Hex to Decimal in Bash

I needed too turn some hexidecimal values into decimal in a bash script and found a real easy way too do it. The following is a very short bash script demostrating how too turn the hexidecimal string "deadbeefcafe" into it's equivalent decimal value of "244837814094590".

#!/bin/bash

INPUT="deadbeefcafe"
OUTPUT=$((0x${INPUT}))

echo $OUTPUT

SPF & DKIM Records in Route 53

I'm going to do a more detailed post on emailing from Amazon's infrastructure soon, but in the meantime I wanted to quickly throw out solutions too a couple of problems I encountered. These are all specific too Amazon's Route 53, and most are user error (myself).

SPF Invalid Characters or Format

After generating my SPF record, I jumped into Route 53, created a new record pasted in my record, attempted to save and received the following message:

The record set could not be saved because:

  • The Value field contains invalid characters or is in an invalid format.

I was using the SPF record type at a time (see the next section) and assumed that I had messed up the format of my record in some way. I banged my head against the wall and through RFCs thoroughly before I found the solution...

Solution: Wrap your SPF records in quotation characters.

No SPF Record Found / Validation Failure

Since I have my DMARC policy in place (I'll cover this in my email follow up), I receive daily domain reports from Google whenever something fails validation about my domain. After switching to Route 53 for DNS the authresult component started showing up as fail for SPF.

Testing around a few online SPF validators indicated that none of them were able to see my new SPF record, and there had been plenty of time for it too propagate.

The SPF resource record type (RRTYPE 99) is available in Route 53 even though the record type has been deprecated. Not being familiar with this particular decision, I assumed I should be using it instead of the TXT record I've used for every other domain, and it would be handled correctly or more intelligently.

Solution: Either switch the SPF record too a TXT record. or my preference duplicate it into a TXT record so you have both.

Invalid DKIM record

This one had me scratching my head for a while. This was my first time deploying DKIM on a domain that I was not running a Bind name server for. OpenDKIM is nice enough too generate a Bind record for you which works perfectly. It's output looks like the following:

default._domainkey.example.tld.   IN      TXT     ( "v=DKIM1; k=rsa; t=y; s=email; "
          "p=MIHfMA0GCSqGSIb3DQEBAQUAA4HNADCByQKBwQC2Cwpa/+Xhfkzn0QnyQoxRwoJPb+s51dIt9UtFLMlMFuYa/k3GBwZ7UWeyAaQJ3RibSzKV/YwgFuMrzyISrLNSuL2k1bQlQQG8nl23Mu9Mowcb+mV2/3G7roshK6kOLNA0IV2SBl8/0UoNZR/x7c1lzVtVqdj0vW1SsJzgGfbt4LGRvCPyjdg+SLpYtOd/Li4Y1pvHgSRKQRrklpKeJo"
          "nJQ4+lXWqzYtuX9xdNH46ck2HUl56Ob4cy3/gYCJBWrAsCAwEAAQ==" )  ; ----- DKIM key default for example.tld

Copying and pasting everything between the parens in the value field and pasting them into Route 53 works flawlessly. The catch? This won't be treated as a single record, but three individual responses. None of which are complete and valid DKIM records.

This happens because Route 53's value field treats newlines as separate records.

Solution: Turn it into one long string so it isn't covering multiple lines right? Not quite...

TXTRDATATooLong

Combining the DKIM key into one string like so:

"v=DKIM1; k=rsa; t=y; s=email; p=MIHfMA0GCSqGSIb3DQEBAQUAA4HNADCByQKBwQC2Cwpa/+Xhfkzn0QnyQoxRwoJPb+s51dIt9UtFLMlMFuYa/k3GBwZ7UWeyAaQJ3RibSzKV/YwgFuMrzyISrLNSuL2k1bQlQQG8nl23Mu9Mowcb+mV2/3G7roshK6kOLNA0IV2SBl8/0UoNZR/x7c1lzVtVqdj0vW1SsJzgGfbt4LGRvCPyjdg+SLpYtOd/Li4Y1pvHgSRKQRrklpKeJonJQ4+lXWqzYtuX9xdNH46ck2HUl56Ob4cy3/gYCJBWrAsCAwEAAQ=="

And attempting to save results in the following error message:

Invalid Resource Record: FATAL problem: TXTRDATATooLong encountered at ...

Now we're left in a tricky spot. After some research the reason behind this is clear, and makes sense. Though it is another poor usability bug in the way Amazon's Route 53 behaves. Individual DNS UDP packets are limited too 255 characters for their response.

Too properly deliver records longer than that DNS servers are supposed to break up the response into chunks. Properly implemented clients combine these chunks together (with no spaces, newlines or other characters added). What this means is that the record can be broken up transparently behind the scenes anywhere in the message and the client will put it back together correctly.

The Route 53 entry form won't handle this for you though, and in hindsight it looks like Bind might not do it for you though I suspected that was more for readability of zone files rather than a technical limitation (and I haven't tested whether Bind is intelligent enough too handle just a long string).

Solution: Take the original output of Bind between the parens and just remove the newline characters, leave the quotation marks and spaces between the sections like the following sample and you'll be golden:

"v=DKIM1; k=rsa; t=y; s=email; " "p=MIHfMA0GCSqGSIb3DQEBAQUAA4HNADCByQKBwQC2Cwpa/+Xhfkzn0QnyQoxRwoJPb+s51dIt9UtFLMlMFuYa/k3GBwZ7UWeyAaQJ3RibSzKV/YwgFuMrzyISrLNSuL2k1bQlQQG8nl23Mu9Mowcb+mV2/3G7roshK6kOLNA0IV2SBl8/0UoNZR/x7c1lzVtVqdj0vW1SsJzgGfbt4LGRvCPyjdg+SLpYtOd/Li4Y1pvHgSRKQRrklpKeJo" "nJQ4+lXWqzYtuX9xdNH46ck2HUl56Ob4cy3/gYCJBWrAsCAwEAAQ=="

Hope this helps someone else!

Unregistering From WhisperPush After Flashing a New ROM

I've been playing around with my Nexus 5 lately. It was quickly rooted and I began playing with various ROMs that had been pre-built for the Nexus 5. My first stop was the CyanogenMod. Since I'd last used CyanogenMod they added a built-in framework that provides transparent text message encryption called WhisperPush.

WhisperPush is an implementation of Moxie Marlinspike's highly respected TextSecure and I was very excited at the possibility of using it. I immediately signed up for the service.

After a day of use I found CyanogenMod far too unstable too use on my primary device. It locked up multiple times the first day and mobile data simply wouldn't work all day. I promptly formatted and flashed my phone, I haven't settled on a new ROM but that's not what this post is about.

It occurred to me after flashing the phone I was still subscribed to WhisperPush. If anyone that texts me was signed up as well. I'd never receive even an encrypted blob, it would just silently fail.

Searching around I found there is very little information on it, and no official way to unregister, especially after you've wiped your device and no longer have your credentials. Ultimately I found a fairly easy solution, just re-register and perform the backdoor steps too de-register.

I wiped my phone again and installed a new copy of the CyanogenMod nightly. Booted it up and re-enabled WhisperPush. It didn't even note that my number was registered in the past.

I found the solution somewhere in the CyanogenMod forums (though I lost the link, and I'm now too lazy to go find it again). You can unregister by performing the following steps:

  1. Connect your computer with ADB too the phone and pair the computer with the phone.
  2. Enable developer options by opening the system settings, choosing 'About phone' and clicking on the 'Build number' about 7 times (it will start counting down).
  3. Open up the developer options found in the root of the system settings menu and enable root access for 'Apps and ADB'.
  4. On the computer use adb shell to get a shell on the device.
  5. Switch to root using the su command.
  6. Run the following command too view the WhisperPush internal settings:

    cat /data/user/0/org.whispersystems.whisperpush/shared_prefs/org.whispersystems.whisperpush_preferences.xml`
    
  7. Note down the value for pref_registered_number (this should be your phone number with a preceeding '+') and pre_push_password.

  8. Exit the shell.

Finally too unregister we need too make a DELETE request against the WhisperPush API. The classic HTTP swiss army knife curl is going to help us on this front. Run the following command on any linux computer with curl installed, replacing the registered number and registered password with the value you recorded earlier.

curl -v -k -X DELETE --basic --user ${pref_registered_number}:${pre_push_password} https://whisperpush.cyanogenmod.org/v1/accounts/gcm

Be sure too include the '+' in your pref_registered_number. You should end up with a status code of 204. The output will look something like the following (credentials removed).

* About to connect() to whisperpush.cyanogenmod.org port 443 (#0)
*   Trying 54.201.5.27...
* Connected to whisperpush.cyanogenmod.org (54.201.5.27) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
*   subject: OU=Operations,O="Cyanogen, Inc.",[email protected],C=US,ST=Washington,L=Seattle,CN=whisperpush.cyanogenmod.org
*   start date: Nov 26 05:39:18 2013 GMT
*   expire date: Nov 24 05:39:18 2023 GMT
*   common name: whisperpush.cyanogenmod.org
*   issuer: [email protected],CN=Authority,OU=Operations,O="Cyanogen, Inc.",L=Seattle,ST=Washington,C=US
* Server auth using Basic with user '${pref_registered_number}'
> DELETE /v1/accounts/gcm HTTP/1.1
> Authorization: Basic ${encoded credentials}
> User-Agent: curl/7.29.0
> Host: whisperpush.cyanogenmod.org
> Accept: */*
> 
< HTTP/1.1 204 No Content
< Server: nginx/1.1.19
< Date: Wed, 23 Jul 2014 01:45:25 GMT
< Connection: keep-alive
< 
* Connection #0 to host whisperpush.cyanogenmod.org left intact

I don't have any way too check that I'm unregistered but it seems too have worked. Here is hoping this helps some else out in the future.

Using OpenWRT's Dnsmasq as a TFTP Server

I recently reflashed my primary router to a newer version of OpenWRT and attempted to follow my own directions written in an earlier blog post to add PXE booting to my local network using the dnsmasq service built in. After following my advice I found that the dnsmasq service wasn't starting.

Looking into the logread output I finally saw that this was due too a permission issue. Combining this with the output of ps too identify the user that dnsmasq was running on I was able to both modify my instructions and use OpenWRT's own config system to perform the configuration instead of modifying the dnsmasq configuration.

First was solving the permissions issue. I created a dedicated directory at /var/tftp and changed the ownership to 'nobody' and 'nogroup' and mode too '0755'.

Previously I used /var/lib/tftp, however, the default permissions on the /var/lib directory is too restrictive and I didn't want to reduce the rest of that directories security posture simply too allow directory traversal.

Next up was getting the TFTP portion of dnsmasq configured and running. Open up /etc/config/dhcp and under the 'dnsmasq' section add the following lines (or if these lines already exist adjust the values to match).

  option enable_tftp '1'
  option tftp_root '/var/tftp'
  option dhcp_boot 'pxelinux.0'

Run uci commit dhcp too commit the changes and finally /etc/init.d/dnsmasq restart To apply the changes. You'll want too put the 'pxelinux.0' and associated configuration files into the /var/tftp directory too complete the PXE booting configuration.

I'll probably write a blog post covering my PXE setup and configuration if I don't get distracted by other projects.

Fixing Erratic BMC Controller on PowerEdge C6100

I randomly started experiencing an issue with one blade in one of my PowerEdge C6100 blades. It wouldn't obey all commands issued too it via IPMI or through the BMC's web interface. Additionally the blade would randomly power on when off, and the front light would consistently blink as if a hardware fault was detected.

This has been bothering me for a while, but it was my spare blade and wasn't affecting my lab in anyway so I've ignored it. I finally needed it for a project and looked into what may be causing the issue.

A thread on the Serve the Home forums lead to me too a solution, even though my symptoms didn't quite match up with what I was experiencing.

I downloaded the PowerEdge C6100 Owner's Manual for the jumper information, and found it too be redudant. The board itself has each of the jumpers clearly labeled.

After pulling the affected chassis out of the server I connected the pins for the CMOS reset, CMOS password reset, and system reset for about 15 seconds. I pulled the jumpers, reinstalled the blade and it's happy once again. Problem solved.

Update: After performing a few commands via the web interface the issue returned. I'm still looking for a solution too the problem.

Update 2: I'm now suspecting that the issue may be related too me not updating the FSB, which is responsible for handling power management of individual nodes as well as reporting temperature and command response from the BMC.

AWS Reserved Instance Pricing

The current large project I'm working on is going to be hosted on AWS and I was requested to do a cost estimate. Looking into it, it quickly became clear that reserved instances could potentially save quite a bit of cash but there was a catch (isn't there alway).

There is an upfront cost for reserving the instance and in exchange you get a reduced hourly rate. After running the numbers one thing wasn't clear too me, is the upfront cost credit towards running machines or a fee you never see again?

I immediately assumed the latter based on the numbers for one simple reason. If you use the 'Light Reserved Instance' with a 1 year reservation, have your machine running 24/7 the whole year it will cost your more than running the same instance as 'on demand'. This was true for their m1.small, m3.medium, and m3.large which was the only ones I ran the numbers for.

I searched the internet and wasn't able to find a solid answer to the question until I asked Amazon's customer service directly.

Ultimately there probably is a price point where 1 year light reserved instances make sense, and if you're looking too run 24/7 for the whole year you'll want to do a heavy anyway for the most savings but it still surprised me.

I'll probably do a project later using d3.js to get some direct hours run vs total cost for various instances. It'll probably be a fun project.

Modifying the Hosts File in a Docker Container

Before I describe the issue that I encountered, let me be very clear. This hack is potentially dangerous and should absolutely only be done in development environments. This won't affect your host system, only the docker container so the most damage you'll do is prevent hostname and possibly user/group lookups within the container itself.

Alright with that out of the way, I was actively working on a codebase that uses subdomains as part of the identifier. Rather than setup a full DNS server, point my local system at it and load in the domains I wanted to simply modify the /etc/hosts file inside the environment.

Docker mounts an /etc/hosts file inside it's containers, read-only, and the container's 'root' user has had it's mount permissions revoked so it's not able to be modified. Other users have encountered this issue, and a novel workaround was put forward. The solution however makes use of perl, and is specific too ubuntu base systems.

I'll explain the solution after showing a more general way to accomplish the same thing. Different linux systems will store their libraries in different directory structures. CentOS is different from Fedora, which is different from Ubuntu and Debian. All of them name their libraries, in this case we're looking for 'libnss_files.so.2'.

You can find where your copy of this library lives with the following command. This should be run inside the docker container that you want to modify the /etc/hosts file in.

find / -name libnss_files.so.2 -print 2> /dev/null

Pay attention to the path, multiple files may show up and you want the one that matches your system's running kernel (generally x86_64 systems will have their libraries in a lib64 directory).

Once you've found this add the following lines to your Dockerfile. Make sure you modify the path in the copy in the first line to the path of your copy of the library. Once done you'll use the /var/hosts file to modify your hosts file instead.

RUN mkdir -p /override_lib && cp /etc/hosts /var/ && cp /usr/lib64/libnss_files.so.2 /override_lib
RUN sed -ie 's:/etc/hosts:/var/hosts:g' /override_lib/libnss_files.so.2
ENV LD_LIBRARY_PATH /override_lib

So what is this actually doing? On linux systems, name configurations such as DNS, username, and group lookups are generally handled by the 'nss' or name service switch configuration tools including the hosts file. The library that we're copying and modifying is a very specific to reading from files on the system and includes the default paths to these files.

Generally you have to be very careful when you're manipulating strings within compiled libraries. The length of the string is encoded along with it, so at a minimum it's important that the string is the same length or less. You can get away with less but it requires additionally writing an end of string character as well.

Too make this hack simple, we're simply replacing the 'etc' with 'var', both systems directories that regular users generally should have read access but not write access too.

Finally we need to tell all programs that need to perform lookups using hostnames in the hosts file to make use of our modified library instead of the system one. Linux will look for shared libraries at runtime in any paths set in in the LD_LIBRARY_PATH (colon delimited just like PATH) and this doesn't require any privileges too set.

And the result? An editable hosts file, with no extra services. I can't stress enough though, there could be bad ramifications from modifying libraries this way. This is definitely not a 'production ready' hack.

Extracting Content From Markdown

Recently I've been playing around with building a pure javascript full text search engine for static content sites like this one. One of the challenges with doing this has been working around the Markdown markup embedded in the written content.

Most of the markdown syntax can be stripped out simply by removing all non-alphanumeric characters from the document and move on. This doesn't solve one of the bigger challenges I've experienced... code blocks. Code blocks have plenty of regular english-ish words and can easily skew keyword detection within it.

I didn't want to write my own Markdown parser, so I started with the one already in use by this site's renderer (redcarpet). Another Github user, Markus Koller or toupeira on Github provided the basis for the code that became the redcarpet "StripDown" formatter, which was designed to essentially render a Markdown document without the markup.

It does almost exactly what I want, except it still outputs raw code inside the content. The following code sample includes a modified version that excludes any code blocks. My content is also formatted inside the markdown documents to never be longer than 80 lines, this also turns individual paragraphs and list items into individual lines for paragraph detection.

require 'redcarpet'
require 'redcarpet/render_strip'

class ContentRenderer < Redcarpet::Render::StripDown
  def block_code(*args)
    nil
  end

  def list_item(content, list_type)
    content.gsub("\n", " ") + "\n\n"
  end

  def paragraph(text)
    text.gsub("\n", " ") + "\n\n"
  end
end

markdown = Redcarpet::Markdown.new(ContentRenderer, fenced_code_blocks: true)
puts markdown.render(File.read('sample_markdown_article.md'))

The above code will print out just the content of the markdown formatted file 'sample_markdown_article.md'.

PG::Error: ERROR: Type 'Hstore' Does Not Exist

I've been using the PostgreSQL's hstore extension in a Rails application lately and kept encountering the error that is this post's namesake. It would specifically happen when a database had been dropped, recreated and I freshly ran the migrations.

It seems that while Rails 4 supports the HStore datatype, it doesn't enable the extension itself. I've found two ways too solve this issue in wildly different ways.

First Solution: Enable HStore by Default

This is the common solution that is recommended too solve this issue. It enables the HStore extension by default on all newly created databases. Too understand this you need to know a bit about PostgreSQL's behavior.

When a new database is created, PostgreSQL creates a copy of a special pre-existing database named 'template1' by default. Anything done too this database will be reflected in all new databases, including enabling extensions.

Too enable the HStore extension on the template1 database you can execute the following command (generally as the postgres user or with your authentication of choice).

psql -d template1 -c 'CREATE EXTENSION hstore;'

Second Solution: Rails Migration

The above solution doesn't sit well with me. While it's uncommon for any individual PostgreSQL server to be shared among different applications with different databases, the possibility is there. Perhaps the application will get de-commisioned and the DBA will simply drop the associated database and roles instead of setting up a new one.

Disconnecting the requirements of the application from the application itself always seems to lead too trouble.

Rails already has a mechanism too handle modifications too the database overtime, migrations. They're solid, well tested, and encapsulate not only how to get the database to a particular state but also how to return it back to it's prior state (generally).

We can also do this without using raw SQL which now also seems a bit... off to me. The following is a sample Rails migration that will both enable and disable the extension:

class ManageHstore < ActiveRecord::Migration
  def change
    reversible do |op|
      op.up { enable_extension 'hstore' }
      op.down { disable_extension 'hstore' }
    end
  end
end

Now the biggest problem with this migration is that too use it, you need too plan ahead of time too use the extension or not worry about freshly running all the migrations (generally because you dropped and created the database). This migration needs to be named so it alphabetically comes before any migration in your application that makes use of the HStore datatype.

ActiveRecord uses timestamps at the beginning of the migration names to handle this alphabetic sorting, and such you'll want to fake this in before you used the HStore datatype.

Chain Loading Kernels

I've found several places where I needed to be able to update my kernels but for one reason or another can't update the kernel that gets booted initially. A couple of these situations were:

  • Running Custom or Updated Kernels on DigitalOcean (this is one of their biggest failings IMHO)
  • Allowing updating of kernels on embedded linux devices that require their kernel flashed into NVRAM.
  • Running an embedded system that used an active/backup partition scheme for updating.

In all cases the process was pretty much the same, though there were some custom changes to the preliminary init system depending on what I needed to get done, especially with the last one which I may cover in a different article.

In all cases these were done on a RedHat based distribution like CentOS, Scientific Linux, RHEL, or even Fedora. For those users of Debian based systems you'll need to adjust the scripts too your system though I can't imagine anything other than the package names changing.

This assumes you already have the kernel and initramfs you want to boot installed on your local filesystem at /boot/vmlinuz-custom and /boot/initramfs.img.

A quick background on how this works, when the linux kernel is compiled an init program is configured to be the first thing triggered, by default and in most situations this will be the executable /sbin/init. This init process is then responsible for starting the rest of the daemons and processes that make up the systems we regularly interact with.

There are tools that allow you too effectively execute another kernel to run in place of the kernel that is already running. There are some catches though as the new kernel won't always re-initialize all devices (since they've already been initialized) and that can lead too some weird behaviors with processes that already have hooks on those devices.

Too prevent any issues you need to load the new kernel as early in the boot process as possible. Doing this in the init program is pretty much as early as you can get and makes for a pretty stable system (I've yet to experience any issues with machines running this way).

There are several different init systems and they all behave a little differently, as far as I know only systemd supports a means of automatically executing a different kernel but I am personally not a systemd fan and it would be too late in the boot process already for me too trust the chain load. You can reliably chain load kernels regardless of what your normal init system is though very easily and that's what I'm going to cover here.

You'll need to have the kexec tools installed on your system. This is pretty straight-forward:

yum install kexec-tools -y

Next we're going to shift the standard init process off to the side, someplace still accessible so we can call it later (this will need to be done as root).

mv /sbin/init /sbin/init.original

Now we need to create our own init script that will handle detecting if it's the new or old kernel, replacing the kernel if it is indeed an old one, and starting up the normal init process if it's the new kernel.

Now there is a very important catch here, whatever process starts up first is given PID 1 which is very important in kernel land. Whatever process is PID 1 will inherit all zombie processes on the system and will need to handle them. Since our shell script is the first thing started up it will get PID 1 for both the old and new kernel and getting the process handling code correct is not a trivial issue.

What we really need is to hand over PID 1 to the init process so it can do it's job normally as if the shell script never existed. There is a native function to do exactly this in these shell scripts: exec.

Our simple shell script to do the chain load looks like this:

#!/bin/bash

# Detect if this is the old kernel (not booted with the otherwise meaningless
# 'kexeced' parameter.
if [ $(grep -q ' kexeced$' /proc/cmdline) ]; then
  kexec --load /boot/vmlinuz-custom --initrd=/boot/initramfs.img \
    --reuse-cmdline --append=' kexeced'
  kexec --exec
fi

# If we made it this far we're running on the new kernel, trigger the original
# init binary with all the options passed too this as well as having it take
# over this process's PID.
exec /sbin/init.original "[email protected]"

After rebooting you should be in your new kernel which you can verify with uname -a and also by examining the /proc/cmdline file for the existence of the 'kexeced' flag.

If you modify the script above, be very careful as any execution error will cause your system to die and recovery will only be possible by mounting the filesystem on another linux system and fixing it.

In a future article I'll cover how to use this trick to build an active / backup system allowing you to fall back to a known good system when booting fails which is incredibly useful for embedded devices in the field that need updates but are not easy to get too or replace when an update bricks the system.

Calculating RSA Key Fingerprints in Ruby

I regularily find myself working on projects that involve the manipulation and storage of RSA keys. In the past I've never had to worry about identification or presentation of these keys. Normally I've only got one too three pairs at most that I'm manipulating (server, certificate authority, client).

I've not found myself working on a project that involves presenting the certificates to users for selection and comparison. The obvious way too do this is take a page out of other developer's books and present the key's fingerprint.

For those unfamiliar with key fingerprints, they are a condensed way to compare differing RSA with a high probability that if the fingerprints match, so do the keys. These are generally based on a cryptographic digest function such as SHA1 and MD5, and you'll see them most commonly when connecting to a new SSH host and will look like the following:.

The authenticity of host 'some.fakedomain.tld (127.0.0.1)' can't be established.
RSA key fingerprint is 0c:6c:dd:32:b5:59:40:1d:ac:05:24:4f:04:bc:e0:f3.
Are you sure you want to continue connecting (yes/no)?

The string of 32 hex characters presented there can be compared with another known value to make sure you're connecting to the correct SSH server and will always be the same length regardless of the bit-strength of the keys used. Without the fingerprint, users would have to compare 256 hex characters for a 1024 bit key, which is a very low security key.

You can calculate the SSH fingerprint for your SSH key or a SSH host key using the ssh-keygen command like so:

ssh-keygen -lf ~/.ssh/id_rsa
ssh-keygen -lf /etc/ssh/ssh_host_key.pub

It will work when the path is either a private RSA key or a public key formatted for SSH authorizied key files.

X509 certificates also use a key fingerprint to help identify a certificate's signing authority. What I rapidly learned through this investigation was that they are calculated slightly differently from SSH fingerprints even if they're in the same format.

I couldn't find any good Ruby code that calculated either, and the alternatives were some dense C++. Luckily SSH fingerprints are pretty documented in RFC4253 and RFC4716. Fingerprints on RSA keys for use with OpenSSL are less clear, and there is a different method for calculating the fingerprints of certificates.

Slowly working through the undocumented bits of Ruby's OpenSSL wrapper, the RFCs and a couple of C++ implementations I finally got a set of working implementations that calculate the following fingerprints in Ruby:

  • MD5 & SHA1 fingerprints for RSA SSH keys
  • Fingerprints of RSA keys for use with x509 certificates
  • Fingerprints of x509 certificates

The easiest being a regular x509 certificate:

require 'openssl'

path_to_cert = '/tmp/sample.crt'
cert = OpenSSL::X509::Certificate.new(File.read(path_to_cert))
puts OpenSSL::Digest::SHA1.hexdigest(cert.to_der).scan(/../).join(':')

You can compare the output of the above code with OpenSSL's implementation with the following command:

openssl x509 -in /tmp/sample.crt -noout -fingerprint

Please note that case sensitivity doesn't matter here (OpenSSL will return upper case hex codes).

The next one I got working was the SSH fingerprints thanks to the RFCs metioned earlier.

require 'openssl'

path_to_key = '/tmp/ssh_key'

key = OpenSSL::PKey::RSA.new(File.read(path_to_key))
data_string = [7].pack('N') + 'ssh-rsa' + key.public_key.e.to_s(0) + key.public_key.n.to_s(0)
puts OpenSSL::Digest::MD5.hexdigest(data_string).scan(/../).join(':')

Please note: The above only works for RSA SSH keys.

Calculating a SHA1 fingerprint for SSH hosts is as simple as replacing the 'MD5' class with 'SHA1' or any of the other support digest algorithms.

The last one was the hardest to track down and implement, eventually I found the answer in RFC3279 under section 2.3.1 for the format of the public key I would need to generate before performing a digest calculation on it.

require 'openssl'

path_to_key = '/tmp/x509_key.pem'

key = OpenSSL::PKey::RSA.new(File.read(path_to_key))
data_string = OpenSSL::ASN1::Sequence([
  OpenSSL::ASN1::Integer.new(key.public_key.n),
  OpenSSL::ASN1::Integer.new(key.public_key.e)
])
puts OpenSSL::Digest::SHA1.hexdigest(data_string.to_der).scan(/../).join(':')

Disabling Gnome's Keyring in Fedora 19

An update too Fedora a while ago started causing some unexpected behavior with my dotfiles. Specifically the way I was handling my SSH agent. My SSH keys when added to my agent automatically expire after a couple of hours.

After the update, when that expiration came I started receiving errors in my shell that looked similar to the following (Since I fixed it I am not able to get the exact working again):

Warning: Unable to connect to SSH agent

I also noticed that periodically I got a Gnome keyring popup asking for my SSH agent rather than my command-line client. I'm personally not a big fan of Gnome, but I deal with because it's the default for Fedora, tends to stay out of your way, and switching to something else is just not a project I've had time for.

Now Gnome was very much getting in my way. I dealt with it for several months now and finally got sick of it.

I tracked this down too the gnome-keyring-daemon which was starting up and clobbering the contents of my SSH_AUTH_SOCK variable along with my GPG_AGENT_INFO environment. Not very friendly.

There were a couple paths that I could've gone for for solving this situation. The first, and easiest way to probably have dealt with this was too put some logic into my ~/.bashrc file that detected when the gnome-keying-agent was running, kill it and clean up after it. It might look something like this:

if [ -n "${GNOME_KEYRING_PID}" ]; then
  if $(kill -0 ${GNOME_KEYRING_PID}); then
    kill ${GNOME_KEYRING_PID}
  fi
fi

unset GNOME_KEYRING_CONTROL SSH_AUTH_SOCK GPG_AGENT_INFO GNOME_KEYRING_PID

I share my dotfiles along a lot of different systems and don't like system-specific behaivior getting in there. Instead I choose to find what was starting up the keyring daemon and preventing it from doing so. Without a good place to start and stubbornly refusing to Google this particular problem I took the brute force approach of grep for the binary name in the /etc directory.

Sure enough in /etc/xdg/autostart I found a series of background daemons that I definitely did not want nor need running. As root I ran the following command to purge them from my system:

cd /etc/xdg/autostart
rm -f gnome-keyring-{gpg,pkcs11,secrets,ssh}.desktop \
  gnome-welcome-tour.desktop imsettings-start.desktop \
  evolution-alarm-notify.desktop caribou-autostart.desktop

Make sure you read through that command as I'm deleting more services than just the Gnome keyring related daemons. The first solution will keep your system in a default state, but this will permanently prevent the obnoxious behaivior on your system for all users and prevents you from adding hacks to your bashrc to work around mis-behaving software.

I hope this helps someone else!

One-Liner SSL Certificate Generation

I regularily find myself in need of generating a quick SSL key and certificate pair. I've been using a one-liner for a while to generate these certificates. No annoying user prompts just a quick fast certificate pair.

echo -e "XX\n\n \n \n\n*\n\n" | openssl req -new -x509 -newkey rsa:2048 \
  -keyout service.key -nodes -days 90 -out service.crt &> /dev/null

A few notes about this, it is an ultimate wildcard matching any and all hostnames with no location specific information, it should under no circumstances be used for a production service. It's a 2048 bit key and only valid for for roughly three months.

Preventing Tmux Lockups

Anyone that has used SSH, Tmux or Screen for a while will have inevitably dumped excessive output to their terminal. Depending on the size of the output you may have experienced the dreaded lockup. That horrible realization seconds after you hit the command where signals just stop working and you just have to sit there and wait for your terminal to catch up.a

There is a piece of remote connection software called Mosh that I've been told handles this pretty well, but I don't yet trust its security model and it doesn't prevent the same thing from happening locally.

This is especially bad if you're working in a multi-pane tmux window as it's locks up all the terminals in the same window, and prevents you from changing to the other windows.

I've had this issue happen to me one too many times but never thought of looking for a solution until a friend of mine, Gabe Koss, made a passing comment along the lines of "Too bad tmux can't rate limit the output of a terminal".

A quick search through the doc and two relatively recent configuration options popped out doing exactly what I was looking for (c0-change-internal, and c0-change-trigger). Googling around for good values, left me wanting. A lot of people were recommending setting the values to 100 and 250 respectively; These are the defaults and since I still experience the issue are clearly not working for me.

To set the variables to something more reasonable I had to understand what they were doing. A 'C0' sequence is one that modifies the screen beyond a normal character sequence, think newlines, carriage returns, backspaces. According to the tmux man page, the trigger will catch if the number of c0 sequences per millisecond exceeds the number in the configuration file, at which point it will start displaying an update once every interval number of milliseconds.

I can't see faster than my eye's refresh rate so that seems like a decent starting point. According to wikipedia the human eye/brain interface can process 10-12 images per second but we can notice 'choppiness' below 48 FPS. Since I won't be reading anything flying by that fast I settled on a maximum rate of 10 FPS updated in my shell, or an interval of '100ms'.

For the trigger I was signficantly less scientific, I dropped the trigger by 50, reloaded my tmux configuration, cat'd a large file and tested whether I could immediately kill the process and move between panes. I finally settled on a value of '75' for the trigger rate. It does make the output seem a little choppy but it is signficantly nicer to not kill my terminal.

TL;DR Add the following lines to your ~/.tmux.conf file and you'll be in a much better shape:

setw -g c0-change-interval 50
setw -g c0-change-trigger  75

Finding Ruby Subclasses

While working through a problem I found it would be immensely useful to be able to enumerate all of the current subclasses of a particular class. After thinking about this for a while I settled on a good old friend of mine, ObjectSpace.

For those not familiar with the ObjectSpace module, it is a means to inspect and access the items being tracked by Ruby's garbage collector. This means it has a hook into every living object, and more dangerously, every near-death object.

ObjectSpace provides a method for enumerating instances of a specific class, specifcally named each_object which takes a class. With Ruby all classes are in fact instances of the Class class. This allows us to enumerate every available class by passing it to the enumerator like so:

ObjectSpace.each_object(Class).to_a

Alright so we now have an array of every single class that could possibly be instantiated, how do we narrow it down to just the ones we're interested in? Once again Ruby provides with the ancestors method, combine that with a select and we can quickly narrow it down. You can see it in the following example:

[1] pry(main)> TargetSubclass = Class.new(String)
=> TargetSubclass
[2] pry(main)> ObjectSpace.each_object(Class).select { |k| k.ancestors.include?(String) }
=> [String, TargetSubclass]

Hmm, that's not quite right though. We have found all the subclasses but we've also grabbed the parent class. With one small modification we eliminate that as well.

[1] pry(main)> TargetSubclass = Class.new(String)
=> TargetSubclass
[2] pry(main)> ObjectSpace.each_object(Class).select { |k| k.ancestors.include?(String) && k != String }
=> [TargetSubclass]

That line is rather long though, and I generally like to avoid multiple tests in a select block. There is a tad bit of syntactic sugar provided by Ruby allowing us to accomplish the same thing, our final example is ultimately the solution I went with:

[1] pry(main)> TargetSubclass = Class.new(String)
=> TargetSubclass
[2] pry(main)> ObjectSpace.each_object(Class).select { |k| k < String }
=> [TargetSubclass]

Putting this into a method:

def subclasses(klass)
  ObjectSpace.each_object(Class).select { |k| k < klass }
end

If you were so inclined you could extend the Class class with a method to make this available anywhere like so:

class Class
  def self.subclasses
    ObjectSpace.each_object(Class).select { |k| k < self }
  end
end

I'm personally not a fan of extending any of the core classes unless absolutely necessary, but too each there own.

Creating Crypt Style SHA512 Passwords With Ruby

I needed to generate crypt-style SHA512 passwords in ruby for an /etc/shadow file. After a bunch of Googling and messing around with the OpenSSL library I finally found a very simple built-in way to handle this.

require 'securerandom'

'password'.crypt('$6$' + SecureRandom.random_number(36 ** 8).to_s(36))

You'll get a string that looks like:

$6$4dksjo1b$Lt194Dwy7r/7WbM8MezYZysmGcxjaiisgTrTBbHkyBZFXeqQTG0J5hep4wLM/AmYxlGNLRy0OWATLDZCqjwCk.

If you don't want to use the SecureRandom module you can replace the random call with simply rand(36 ** 8) though this isn't recommended.

Enjoy!

Setting Linux System Timezone

I change the timezone on the linux systems so rarely that I almost always have to look it up. I'm writing it up here for my own personal reference. With any luck it'll also help others.

The system timezone is controlled by the /etc/localtime file and is generally symlinked to locale files stored in /usr/share/zoneinfo. Generally I like to keep my systems on UTC as I my machines are in several timezones and it makes all the logs have consistent times.

To set the system time to UTC you'd run the following command as root:

ln -sf /usr/share/zoneinfo/UTC /etc/localtime

Other timezones can be found in the /usr/share/zoneinfo and are generally broken up by continent with a few exceptions.

As a user it's obviously more useful to see the time in my local timezone and this can be overridden on a per-user basis using the TZ environment variable. I stick this in my ~/.bashrc file and it just works transparently:

export TZ="America/Los_Angeles"

Starting Puppetmaster on Fedora 19

I was trying to get puppet running out of the box on Fedora 19 and found a bug exists in their systemd service file. After installing puppet and puppet-server, whenever I tried to start the server with the following command:

systemctl start puppetmaster.service

It would hang for a long time and the following error message would show up in the log:

Jan 19 03:42:18 puppet-01 puppet-master[1166]: Starting Puppet master version 3.3.1
Jan 19 03:42:18 puppet-01 systemd[1]: PID file /run/puppet/master.pid not readable (yet?) after start.
Jan 19 03:43:07 puppet-01 systemd[1]: puppetmaster.service operation timed out. Terminating.
Jan 19 03:43:07 puppet-01 puppet-master[1166]: Could not run: can't be called from trap context

Starting puppet directly from the command line using the same command specified in the service file would work fine, but that wasn't really a solution. Turns out puppet, additionally I would briefly see the puppetmaster service open up port 8140 before systemd would kill it.

Turns out the systemd service script is looking in the wrong location for the pidfile. All of the pids are stored in /var/run/puppet/ with a filename of either agent.pid or master.pid depending on the mode it was run as. The systemd script, as the log indicates is looking for the pid files in /run/puppet.

The real solution would be to bring this too the attention of the script maintainers, but I haven't had a lot of luck going through those processes. Instead you can work around the issue without any beauracracy by changing the rundir configuration option in /etc/puppet/puppet.conf to /run/puppet, and creating /run/puppet (with puppet as the user and group owning the directory).

After that, voila! The service starts up. You'd think a QA process would catch that the service script doesn't work...

Updating BMC on Dell PowerEdge C6100

I just received my Dell PowerEdge C6100 and found it's software quite a bit outdated. After searching around quite a bit I found the resources lacking for explaining how to perform these updates. So in this post I'm going to quickly cover updating the BMC firmware on each blade.

The system I received had four different versions of the BMC software installed, additionally Two were branded as MegaRAC and the others branded as Dell. This update didn't fix the branding (and I'd love to remove the Dell branding as it's kind of annoying) it did, however, fix a number of other issues that I was experiencing such as:

  1. Console Redirection failing to connect
  2. BMC losing it's network connection after a couple of minutes
  3. Slow responses, with occasional failures to load pages
  4. Remote IPMI tools being unable to read sensors status

The first step is too download the latest version of of the BMC software from Dell's support site (Or a direct link, I've also taken the liberty of hosting a copy myself). I recommend you go through the process of entering the service tag of each of the blades and make sure that Dell recognizes them as existing even if they're out of support.

There has been mention of versions of these blades that had custom modifications for DCS and any attempts to modify the BIOS or BMC will likely cause you to end up bricking the remote management board or the motherboard.

Even with the regular board there is always a risk of bricking it, though firmware updates have gotten a lot more reliable and I haven't experienced a mis-flashed motherboard in years. You've been warned.

The BMC was fairly straight-foward. I installed the 64-bit version of Fedora 19 on a thumbdrive, downloaded version 1.30 of the BMC software (get the file named PEC6100BMC130.exe). The file itself is a self-extracting zip archive which can be extracted using the regular unzip utility.

unzip PEC6100BMC130.exe

Inside you'll find two folders, KCSFlash and SOCFlash should both be put on the live drive within the KCSFlash. You'll need to set the execute bit on the contents of the linux directory and the linux.sh file. You'll also need to install the glibc.i686 package. Afterwards it's as simple as booting each chassis off the drive and as root run the linux.sh script.

If the KCSFlash fails, the SOCFlash will more likely than not work but it is slightly more dangerous. If you need it mark the linux/flash8.sh, linux/socflash, and linux/socflash_x64 as executable in the SOCFlash folder and run the flash8.sh script.

After that you're going to want to reboot into the BIOS and ensure the IPMI ethernet port is set to dedicated, as this switched it back to "Shared" on me.

Using Dnsmasq as a Standalone TFTP Server

If you've come across this blog post with the intention of setting up TFTP on an modern version of OpenWRT I have a more recent blog post detailing how too configure your system.

I found myself in need of a TFTP server but wanted to avoid having all of the xinet.d packages and services on my system (even if they were disabled). While looking for alternatives I found out that dnsmasq has a built-in read-only TFTP server.

I already have a DNS and DHCP server on my network and didn't want dnsmasq to take on either of those roles so my first challenge was finding a way to prevent dnsmasq from running those bits of it's code, or failing that I would just firewall off the service. Luckily it's quite easy to disable both bits of funtionality.

For DHCP you simply have to leave out any of the dhcp option in the configuation file, DNS you just tell it to operate on port 0 and it will be disabled.

So my whole config starting out looks like this:

# Disable DNS
port=0

Now I need to configure the TFTP bits of dnsmasq. This too was rather simple only requiring me to add the following to my already terse config file:

# Enable the TFTP server
enable-tftp
tftp-root=/var/lib/tftp

I created the root directory for my TFTP server and started it up with the following commands:

mkdir /var/lib/tftp
systemctl enable dnsmasq.service
systemctl start dnsmasq.service

Voila, TFTP running and happy. If you have a firewall running you'll also want to open ports 69/tcp and 69/udp (though I suspect only the UDP one is needed).

Configuring PXE Booting on OpenWRT

I needed to support PXE booting on my home network. I use OpenWRT as my main router and DHCP server and it took me a bit of searching how to configure the BOOTP next server to redirect local clients to my Arch TFTP/NFS server for booting, so I'm placing the config here to help others who might be looking to do the same thing.

It's worth noting that this isn't a guide on setting up PXE booting completely on an OpenWRT, you'll need another system that is running a configured TFTP server. I'll write up how I setup my Arch box as a TFTP server at a later date.

The config itself was very simple; You just need to add a couple lines to /etc/config/dhcp. You'll want to replace 10.0.0.45 with whatever your local TFTP server is.

config boot linux
  option filename      'pxelinux.0'
  option serveraddress '10.0.0.45'
  option servername    'Arch-Pixie'

The filename pxelinux.0 is from syslinux, and the servername has no technical meaning, but it provides nice information to the clients. In this case I've used the name of my Arch linux server that I'll be booting off of.

Hope this helps someone out. Cheers!

Running Emails Through Ruby

Following up on my earlier post where I covered how to backup your Gmail account using fetchmail and procmail; I wanted to cover how I was additionally processing received mail through ruby.

This was part of a larger project where I was doing statistical analysis on my email while evaluating various data stores. To get the emails into the various data stores, I used the ruby script to parse, process and store the emails as they came in.

If you're going to be doing any form of mail manipulation or statistics I highly recommend the mail gem. It did almost everything I needed out of the box, though it didn't correctly enumerate any of the additional headers.

Procmail is a highly flexible mail filtering and local delivery agent. Without much effort you can pass the mail it is handling through a series of filters which can manipulate and reject mail before eventually delivering it to your inbox. In light of this, we're going to make a filter that simply counts the total number of emails the script has processed, and add a header to the message that indicates this count.

#!/usr/bin/env ruby

require 'mail'

# Get the email message from STDIN or a passed filename
message = ""
while input = ARGF.gets
  message += input
end

# Parse the email into a ruby object
msg = Mail.new(message)

# Location of our count file
count_file = "#{ENV['HOME']}/.mail_counter.txt"

# Load or initialize our count value and increment it
count = File.exists?(count_file) ? File.read(count_file).to_i : 0
count += 1

# Update our count on disk
File.write(count_file, count.to_s)

# Add our header with the count
msg.header.fields << Mail::Field.new("X-Mail-Counter: #{count}")

# Output the now modified message back out to $stdout
begin
  $stdout.puts msg.to_s
rescue Errno::EPIPE
  exit(74)
end

Make sure you mark the script executable after saving it.

If you followed along with my earlier post the only change we need to make is to add our ruby mail processor as a procmail filter. I've stored the script in ~/.bin/mail-counter.rb, if you've stored it in a different location you'll want to update your path to reflect that.

Filters in procmail are handled by using the pipe helper. The following is a minimum working example of a procmailrc file to make use of our filter:

MAILDIR=$HOME
VERBOSE=on

:0fw
| /home/sstelfox/Documents/ruby/riak-mail-indexer/counter.rb

:0
Maildir/

Store the above file in ~/.procmailrc. The next time you run fetchmail those headers will be added to the messages before being delivered and you can watch the count increment by looking at the contents of ~/.mail_counter.txt.

The following are a few additional sources I made use of while writing this article:

Access GET Parameters With Coffeescript

I've been working on a pure javascript based search engine for this static website and needed to access a get parameter within the URL.

I found a few solutions online but they usually made use of jQuery or weren't in coffeescript. A few others would only extract an individual named parameter at a time. The following will return all of them in Javascript's equilvalent of a hash (or dictionary if you prefer) in the form of an object.

getParams = ->
  query = window.location.search.substring(1)
  raw_vars = query.split("&")

  params = {}

  for v in raw_vars
    [key, val] = v.split("=")
    params[key] = decodeURIComponent(val)

  params

console.log(getParams())

If compiled and included in a page the above will print out the parameters as a hash object to the console.

Downloading Google Mail and Calendar Data

I recently posted a guide on backing up your Gmail with fetchmail. This unfortunately doesn't include your calendar data. It seems like backing up was a hot enough topic that the Google Gmail team are releasing an official backup method. It's not completely in the wild yet but I definitely look forward to poking around in it.

Now if only Google let you download everything they know about you as well... Would definitely make for an interesting read.

Taking Back the Sky

During my daily review of various new sources I came across one particular article that was both concerning and very amusing. Drones have been getting more and more popular, and more accessible. They've been getting used by the military, law enforcement, recently Amazon (though they've abandoned that for now), you can even purchase one for your iPhone at airports.

The security of these systems hasn't been thoroughly tested publicly, though there is at least one report of a military drone being stolen already. With the beginnings of various commercial uses of drones and not just hobbiest seeing what they can do with them.

Someone had to start the testing eventually and Samy Kamkar took the lead this time with his new project Skyjack. The article that tipped me off to this project can be found over on threatpost

The gist of the project is a drone that can forcibly disconnect other drone's controller and take it's place to "steal" the drone.

I can imagine a whole cyber-punk thriller action scene where corporate anarchists hijack a drone and use it's trusted status within the drone's network to hack in and take control of the entire CNC drone system to further their goals.

Fail Fast in Bash Scripts

I found myself writing another bash script that should exit should any of the few commands within it fail to run. As I began writing some error handling after each command, and isolating the sections into bash functions I figured there had to be a better way. After a little Googling and a trip through the bash manpages sure enough:

#!/bin/bash

function error_handler() {
  echo "Error occurred in script at line: ${1}."
  echo "Line exited with status: ${2}"
}

trap 'error_handler ${LINENO} $?' ERR

set -o errexit
set -o errtrace
set -o nounset

echo "Everything is running fine..."

# A command outside of a conditional that will always return a exit code of 1
test 1 -eq 0

echo "This will never run, as a command has failed"
echo "Using unset variable ${TEST} will also cause this script to exit"

The first piece of that is setting up an error handler that will get run whenever an error condition occurs with the script. You can use this section to roll back any changes or cleanup your environment as well as give you some debug information about the failure.

I'm then setting a few bash options, The following is a description taken more or less directly from the bash man pages:

-o errexit: Exit immediately if a pipeline (which may consist of a single simple command), a subshell command enclosed in parentheses, or one of the commands executed as part of a command list enclosed by braces exits with a non-zero status.

-o errtrace: If set, any trap on ERR is inherited by shell functions, command substitutions, and commands execute in a subshell environment.

-o nounset: Treat unset variables and parameters other than the special parameters "@" and "*" as an error when performing parameter expansion.

If anything goes wrong in the script it will fail once, fail fast, and let you know where it died.

Using VIM as Your Password Manager

There are all kinds of password managers out there. Everything from web services that are quite solid and respectible, to native desktop apps.

A lot of these are simply too heavy for me, involve installing software on a computer to access in addition to sharing the file around, or required you to remember multiple account details before you could get access to any individual password.

Due too the various complexities and lack of matching use cases a couple years ago I set out to develop my own open-source version of Passpack. In the interim though I needed a solution for keeping track of my hundreds of various accounts and their passwords.

Around this time I was ramping up my usage of vim and happened to come across a very fortunate command entirely by accident. Enter vimcrypt.

For any plaintext file you, while in command mode you can type the command :X and it will ask you for a password to encrypt your file with. By default this uses a remarkably weak algorithm called pkzip which isn't secure enough for me to trust it with my keys.

Since vim 7.3 and later, :X has also supported an additional cipher; The much stronger blowfish algorithm. You can enable this by running the command :set cryptmethod=blowfish. I chose to add the following lines to my ~/.vimrc file:

" When encrypting any file, use the much stronger blowfish algorithm
set cryptmethod=blowfish

This was a fantastic interim solution as I have yet to find a development or production linux system that hasn't been excessively locked down (and probably not somewhere I'd put my password file anyway) that didn't already have vim installed.

Using this personally required me coming up with a pseudo-file format that would allow me to quickly and easily find the credentials I needed. I settled on the simple format shown off below:

Oneline Account Description
  Site: <URL of Site's login page>
  Username: <username for the site>
  Password: <password for the site>
  Email: <email I used to register>

  Login with: <email|username> # Only necessary when I have both

  ** Address on file **
  ** Phone on file **

You'll notice I also used this to keep track of whether an account had physical information tied to it. When I moved this made it very quick for me to search for accounts that I needed to update with my new mailing address.

As with many solutions this "temporary" one became more and more permanent as my motivation to build the Passpack competitor dwindled. My problem had been solved and I was no longer compelled to put any effort into a solution.

If this still isn't strong enough for your tastes, the vim wiki has some additional ways you can encrypt your files. These all require additional setup and failed my requirements in that they generally require additional files or setup before I can access my passwords.

Hope this helps some other weary CLI warrior some trouble. Cheers!

Update: I received a wonderful recommendation from a user named sigzero over on Reddit. For additional security they added the following line to their ~/.vimrc file.

autocmd BufReadPost * if &key != "" | set noswapfile nowritebackup viminfo= nobackup noshelltemp history=0 secure | endif

It disables additional files that vim may write copies to such as swap files and backups, prevents dangerous shell commands, and prevents vim from storing a history of commands.

Update 2: I received another fantastic recommendation from another reddit user, this time from NinlyOne. At their recommendation, I've prepended the following modeline to my password. It automatically folds each password entry to prevent potential shoulder surfing. You can open up an entry using the command zo and close it back up with zc. It's worth noting that this is tied to my indented file format.

# vim: fdm=indent fdn=1 sw=2:

Riak Talk Summary

This evening Gabe Koss and myself gave a lightning talk on Riak at the Burlington Web Applications Group. Due too time constraints there was a lot of material that we simply weren't able to cover, but with any luck I'll be covering in future blog posts.

There was a great turn out as usual, and it was lots of fun. If you're in the Burlington, VT area and are interested in web applications written in any language you should show up. You'll learn something and meet a lot of cool people. You might even get a book out of the deal.

You can take a look at the slides if you missed the talk.

Update: Gabe has taken the time to write up our presentation. You can get closer to the full experience of our talk through his solid writeup. Go check it out!

Backing up Gmail with fetchmail

This morning I found myself in need of a large set of emails to test a particular set of code. Ideally these emails would be broken out into easily digestable pieces, and it was strictly for my own personal testing so I wasn't concerned with using my own live data for this test (There will probably be another post on this project later on).

Having used fetchmail with good results in the past I decided it was a good idea to take this opportunity to also backup my Gmail account into the common Maildir format (which essentially breaks out emails into individual files meeting my requirements).

The first step was to enable POP access to my account through Gmail's interface. You can accomplish this with the following steps.

  1. Login to Gmail
  2. Click on the gear icon
  3. Choose settings
  4. Forwarding and POP/IMAP
  5. Enable POP for all mail
  6. When messages are accessed with POP... Keep"
  7. Save Changes.

Ensure you have fetchmail and procmail installed. For me on Fedora this can be accomplished using yum by running the following commands:

$ sudo yum install fetchmail procmail -y

We need to configure fetchmail to let it know where to retrieve our mail from. This configuration file lives at $HOME/.fetchmailrc. By default fetchmail will send all retrieved mail to the local SMTP server over a normal TCP connection. This isn't necessary or ideal, rather we'll additionally supply a local mail delivery agent (procmail) to handle processing the mail into the Maildir format.

poll pop.gmail.com
protocol pop3
timeout 300
port 995
username "[email protected]" password "yourpassword"
keep
ssl
sslcertck
sslproto TLS1
mda "/usr/bin/procmail -m '/home/<username>/.procmailrc'"

Be sure to set the permissions on the .fetchmailrc file to 0600:

$ chmod 0600 $HOME/.fetchmailrc

We'll now need to configure procmail to properly deliver our mail to the local Maildir folder. Procmail's configuration by default lives in $HOME/.procmailrc

LOGFILE=$HOME/.procmail.log
MAILDIR=$HOME
VERBOSE=on

:0
Maildir/

With that done, simply run the fetchmail command. In my experience this can take a while process and it seems like Google limits the number of emails you can download at a time, so you may need to run the command a couple of times to get all your emails.

Talking about Laboratory B

Recently Justin England and myself were interviewed on a local television channel about the local hackerspace which we are both founding members of.

We covered several general topics about hackerspaces, how the organization is run, and some of the stuff we do at the Lab. Take a look if you're at all interested in any of those!

Ruby's Option Parser - a More Complete Example

Recently while writing a Ruby program I needed to parse some command line options. Helpfully Ruby provides a module named OptionParser to make this easy. I found a few parts of the documentation ambiguous and a few others down right confusing.

The catch I hit was the required field. In my mind the definition of a required argument is something that needs to be passed on the commandline to continue. What OptionParser actually means is that a value isn't required when the argument is passed. OptionParser already provides boolean switches, so when someone would use an optional switch is beyond me.

To make it a little more clear and to have something to work from in the future I created the following chunk of code that includes a Configuration singleton that can be used anywhere within your codebase to access the run-time configuration, a sample parser with a wide range of different types of options, and it will load configuration from a file named config.yml in the same directory.

I feel like the following is a much more complete explanation of how OptionParser is supposed to be used with supporting code.

#!/usr/bin/env ruby

# This file provides an example of creating a command line application with a
# wide variety of command line options, parsing and the like as well as global
# configuration singleton that can be relied on throughout a program.
#
# This entire setup lives within the "Example" module. These are really common
# names and it would be a shame to override required functionality in other code
# that wasn't properly namespaced.

require 'optparse'
require 'singleton'
require 'yaml'

module Example
  # Defines the available configuration options for the configuration
  ConfigurationStruct = Struct.new(:enum, :list, :required, :optional, :verbose, :float)

  class Configuration
    include Singleton

    # Initialize the configuration and set defaults:
    @@config = ConfigurationStruct.new

    # This is where the defaults are being set
    @@config.enum = :one
    @@config.list = []
    @@config.optional = nil
    @@config.verbose = false

    def self.config
      yield(@@config) if block_given?
      @@config
    end

    # Loads a YAML configuration file and sets each of the configuration values to
    # whats in the file.
    def self.load(file)
      YAML::load_file(file).each do |key, value|
        self.send("#{key}=", value)
      end
    end

    # This provides an easy way to dump the configuration as a hash
    def self.to_hash
      Hash[@@config.each_pair.to_a]
    end

    # Pass any other calls (most likely attribute setters/getters on to the
    # configuration as a way to easily set/get attribute values 
    def self.method_missing(method, *args, &block)
      if @@config.respond_to?(method)
        @@config.send(method, *args, &block)
      else
        raise NoMethodError
      end
    end

    # Handles validating the configuration that has been loaded/configured
    def self.validate!
      valid = true

      valid = false if Configuration.required.nil?

      raise ArgumentError unless valid
    end
  end

  class ConfigurationParser
    def self.parse(args)
      opts = OptionParser.new do |parser|

        parser.separator ""
        parser.separator "Specific options:"

        parser.on("--enum ENUM", [:one, :two, :three], "This field requires one of a set of predefined values be", "set. If wrapped in brackets this option can be set to nil.") do |setting|
          Configuration.enum = setting
        end

        parser.on("-l", "--list x,y", Array, "This command flag takes a comma separated list (without", "spaces) of values and turns it into an array. This requires", "at least one argument.") do |setting|
          Configuration.list = setting
        end

        parser.on("--[no-]verbose", "This is a common boolean flag, setting verbosity to either", "true or false.") do |setting|
          Configuration.verbose = setting
        end

        parser.on("--optional [STR]", "This command doesn't require a string to be passed to it, if", "nothing is passed it will be nil. No error will be raised if", "nothing is passed to it that logic needs to be handled", "yourself.") do |setting|
          Configuration.optional = setting
        end

        parser.on("-r", "--required STR", "This command requires a string to be passed to it.") do |setting|
          Configuration.required = setting
        end

        parser.on("--float NUM", Float, "This command will only accept an integer or a float.") do |setting|
          Configuration.float = setting
        end

        parser.on_tail("-h", "--help", "--usage", "Show this usage message and quit.") do |setting|
          puts parser.help
          exit
        end

        parser.on_tail("-v", "--version", "Show version information about this program and quit.") do
          puts "Option Parser Example v1.0.0"
          exit
        end
      end

      opts.parse!(args)
    end
  end
end

if File.exists?("config.yml")
  Example::Configuration.load("config.yml")
end

Example::ConfigurationParser.parse(ARGV)
Example::Configuration.validate!

require "json"
puts JSON.pretty_generate(Example::Configuration.to_hash)

Keep Your Gems Updated

I recently went back through my backups recently and found quite a few old abandoned projects. Looking back on the code I see some things I'm impressed with, but the majority of the code I wouldn't write today. Thats not to say the code is bad, or doesn't function. It did exactly what I wanted to accomplish at the time, just not necessarily in the most efficient way.

This archive of old code made me start wondering how much old code I'm using in the projects that I'm currently writing. Not code that I've written but code that I'm depending on, specifically gems. As of this writing I have 26 active ruby projects in various states of development all of which make use of RVM and bundler.

Conveniently enough, bundler provides an easy way to update all the gems installed in a project unless specific version information was provided in the Gemfile. None of my projects have had a version directly specified in the Gemfile with the exception of Rails. Each project also has solid test coverage (though I must admit it's usually not complete).

For each project I went through and ran bundle update and kept track of the results. I did not keep track of unique gems so the four Rails projects probably had a lot of duplicate gems each one more or less likely to have different versions of different gems installed depending on when I started the project.

Across all of the different projects I had 2214 gems installed. Of those 813 had updates. My initial plan was to go through the updates and see how many of those updates were security or bugfixes, how many were added features, or performance improvements, but I wasn't counting on the shear number of gems that my projects were depending on.

The big question for myself after I updated the Gems was how much will this be now? Running through the thousands of tests in all of the projects I had exactly 7 tests that were now failing and they were all due too projects that removed or renamed a piece of functionality that I was making use of. In one case I had to extend the core Hash method to replace the functionality. All in all it took me about a quarter of an hour to fix all the tests after updating my Gems.

Since I didn't actually go through all of the gems I don't know for sure that my projects are in anyway more secure, faster, or more stable but I can't imagine they're in a worse state. If you have test coverage on your projects you should try and update the gems and see for yourself.

Auditing Heroku SSH Keys

A good friend of mine recently left the organization I work for and the task of resetting our passwords and auditing credentials fell on me. Since we use Heroku for our development platform I needed to not only reset the credentials for the web portion (which conveniently also handles resetting the API key) but also revoke any SSH keys he may have added to access it.

Sadly Heroku does not seem to provide any web interface that I could find for examining what keys were associated with the account. Searching for this information also didn't turn up very valuable results; most people were looking to add keys or resolve issues with missing keys rather than revoking them. I suspect not many people think of SSH keys when it comes time to revoke access which is a dire mistake.

I took to the command line to solve my issue as I knew you could list and add keys that way, so it was a minor leap of logic to assume they could revoke keys as well. I ran heroku help keys to get the syntax for the commands and was pleasantly surprised to see an additional option listed in there:

keys:clear       #  remove all authentication keys from the current user

As a now two person web-shop it's not a terrible amount of work to add our keys back in and looking through there were already some keys in there that should have been revoked long ago. One command and our applications were safe from mischief, though I know my former associate wouldn't abuse that privilege beyond perhaps pointing out the security flaw I'd allowed.

CarrierWave, S3 and Filenames

This is going to be a real quick post. I'm using the "carrier_wave" gem with "fog" for one of my projects and found that when a file is stored on S3 the "identifier", and "filename" methods return nil. I got around this issue in two separate ways neither of which I'm particularly happy about.

Outside of the uploader, you can use the File utility and the URL of the object to get the base filename like so:

File.basename(Model.asset.url)

If you try and do this within the uploader itself like this:

File.basename(self.url)

It will work, but not when creating additional versions such as thumbnails as the file hasn't actually been created yet so a URL can't be built and you'll get an error trying to perform File.basename(nil). You'd need to go back up to the model and get the normal version's URL like so:

File.basename(self.model.asset.url)

Now if you're trying to get the file name to build part of the store_dir, you've just created an infinite loop! Ruby will be happy to tell you that the stack level too deep (SystemStackError). So ultimately how did I end up getting it into my store_dir?

self.model.attributes["asset"]

The file name gets stored raw directly in the database, and thus you can pull it out by accessing the value directly without going through the accessor that get overridden by CarrierWave. I'm pretty sure this is a bug, and will report it with example code and a test (as is appropriate for any bug report hint) as soon as my dead line has passed.

Security Through Obesity

Jeremy Spilman recently proposed changes to how user's hashes are stored in website's and companies databases.  This post was originally going to look at some of the issues involved in the scheme he envisioned, however, he rather quickly posted a followup article with a well thought out solution that countered all of the issues that other people and myself were able to come up with. I'd strongly recommend reading both if you haven't done so. Instead of announcing flaws, I'm turning this into a post with a simple functional implementation of the described scheme in Ruby using DataMapper.

At first I'd like to point out that this is one of those few examples where a form of security through obscurity is actually increasing not only the perceived security but the cost to attack a system as well.

Please note this code is a minimal, functional, example and should not be used in production. It is missing a lot of things that I personally would add before attempting to use this but that is an exercise for the reader. It is licensed under the MIT license. I'll walk through the code briefly afterwards going over some bits.

# encoding: utf-8

require "rubygems"           # You only need this if you use bundler
require "dm-core"
require "dm-migrations"
require "dm-sqlite-adapter"
require "dm-validations"
require "scrypt"

DataMapper.setup :default, "sqlite:hash.db"

class User
  include DataMapper::Resource 

  property :id,             Serial
  property :username,       String, :required => true,
                                    :unique => true 
  property :crypt_hash,     String, :required => true,
                                    :length => 64
  property :salt,           String, :required => true,
                                    :length => 25 

  def check_password(plaintext_password)
    encrypted_hash = scrypt_helper(plaintext_password, self.salt)
    hash_obj = SiteHash.first(:crypt_hash => encrypted_hash)

    if hash_obj.nil?
      puts "Invalid password"
      return false
    end

    verification_hash = scrypt_helper(plaintext_password, hash_obj.salt)

    if self.crypt_hash == verification_hash
      return true
    else
      puts "WARNING: Found matching hash, but verification failed."
      return false
    end
  end

  def password=(plaintext_password)
    generate_salt

    encrypted_password = SiteHash.new
    encrypted_password.crypt_hash = scrypt_helper(plaintext_password,
                                                  self.salt)
    encrypted_password.save

    self.crypt_hash = scrypt_helper(plaintext_password,
                                    encrypted_password.salt)
  end

  private

  def generate_salt
    self.salt = SCrypt::Engine.generate_salt(:max_time => 1.0)
  end

  def scrypt_helper(plaintext_password, salt)
    SCrypt::Engine.scrypt(plaintext_password, salt,
                          SCrypt::Engine.autodetect_cost(salt),
                          32).unpack('H*').first
  end
end

class SiteHash
  include DataMapper::Resource

  property :id,             Serial
  property :crypt_hash,     String,   :required => true,
                                      :length => 64
  property :salt,           String,   :required => true,
                                      :length => 25

  def initialize(*args)
    super
    generate_salt
  end

  private

  def generate_salt
    self.salt = SCrypt::Engine.generate_salt(:max_time => 1.0)
  end
end

DataMapper.finalize
DataMapper.auto_upgrade!

I tried to keep this as a simple minimum implementation without playing golf. Strictly speaking the validations on the data_mapper models aren't necessary and could have been removed, in this case, however, the length fields do actually indicate a bit more of what you might expect to see in the database, while the requires are just good habits living on.

Both of the two models are required to have both a salt and a hash, the name 'crypt_hash' was chosen do too a conflict with one of data_mapper's reserved words 'hash', the same goes for the model name, however, that class comes from elsewhere. Raw scrypt'd hashes are 256 bits long or 64 hex characters long, while the salts are 64 bits (16 hex characters) plus some meta-data totaling 25 hex characters in this example.

Salts are hashes are computed by the 'scrypt' gem. In this example I've bumped up the max time option to create a hash from the default of 0.2 seconds up to 1 second. This is one of those things that I could have left out as the default is fine for an example, but it also couldn't hurt slightly increasing it in case someone did copy-paste this into production.

The one thing that I'd like to point out is a couple of 'puts' statements I dropped in the check_password method on the User model. The first one simply announces an invalid password. A lot of these could indicate a brute force attack. The second one is more serious, it indicates that there is either a bug in the code, a hash collision has occurred, or an attacker has been able to drop in hash of their choosing into the site_hashes table, but haven't updated the verification hash on the user model yet. I'd strongly recommend reading through both of Jeremy's posts if you want to understand how this threat works and specifically the second post to see how the verification hash protects what it does.

So how would you use this code? Well you'd want to create a user with a password and then check if their password is valid or not later on like so:

User.create(:username => 'admin', :password => 'admin')
User.first(:username => 'admin').check_password('admin')

One of the key ways this separation increases the security of real users's hashes is by having a large number of fake hashes in the hash table that the attackers will have to crack at the same time. As a bonus I've written a module to handle just that for the code I've already provided. Once again this is licensed under the MIT license and should not be considered production ready.

# This is the code above, you can also include everything below
# this in the same file if you're into that sort of thing
require "user_hash_example"

module HashFaker
  def self.fast_hash
    SiteHash.create(:crypt_hash => get_bytes(32))
  end

  def self.hash
    SiteHash.create(:crypt_hash => scrypt_helper(get_bytes(24),
                                                 generate_salt))
  end

  def self.generate_hashes(count = 5000, fast = false)
    count.times do
      fast ? fast_hash : hash
    end
  end

  private

  def self.generate_salt
    SCrypt::Engine.generate_salt(:max_time => 1.0)
  end

  def self.get_bytes(num)
    OpenSSL::Random.random_bytes(num).unpack('H*').first
  end

  def self.scrypt_helper(plaintext_password, salt)
    SCrypt::Engine.scrypt(plaintext_password, salt,
                          SCrypt::Engine.autodetect_cost(salt),
                          32).unpack('H*').first
  end
end

Adding a table prefix to DataMapper tables

So I recently encountered a situation where I needed to define a prefix on the tables used by the "data_mapper" gem. When I went searching I found quite a bit of information about similar projects in Python, and PHP named DataMapper but nothing about the ruby "data_mapper". The search continued eventually ending in my reading through the source of the data_mapper gem only to find that there was no feature for simply defining a prefix.

Reading through the source though did allow me to find any easy way to implement such functionality. The following snippet is a minimalistic data_mapper initialization and setup of one model with a table prefix of "source_" (chosen at random and of no significance).

# encoding: utf-8

# (1)
require "dm-core"
require "dm-migrations"

# (2)
module PrefixNamingConvention
  def self.call(model_name)
    # (3)
    prefix = "source_"
    # (4)
    table_name = DataMapper::NamingConventions::Resource::UnderscoredAndPluralized.call(model_name)

    "#{prefix}#{table_name}"
  end
end

# (5)
DataMapper::Logger.new($stdout, :debug)

# (6)
DataMapper.setup(:default, "sqlite:example.db")
DataMapper.repository(:default).adapter.resource_naming_convention = PrefixNamingConvention

# (7)
class Person
  include DataMapper::Resource

  property :id, Serial
  property :first_name, String
  property :last_name, String
  property :email, String
end

# (8)
DataMapper.finalize
DataMapper.auto_upgrade!

So here are some notes on what's going on in this snippet. Each area that I will be discussing has been annotated with a number like "# (1)" to make it easier to find a section you have questions about.

  1. Since this is an example I'm only including the bare minimum data mapper gems to accomplish the task. If you're using bundler you may need to also require "rubygems" to get this too work.
  2. This is where the real work happens, DataMapper uses external modules that receive the "call" method to handle the conversion of class names to table names. By default DataMapper uses the module "DataMapper::NamingConventions::Resource::UnderscoredAndPluralized", which I'll use later to maintain the same names.
  3. This is where I'm defining the table prefix. This could be defined in a global, call another method or class, whatever your heart desires to get a string that will be used as a prefix.
  4. Here I'm getting what DataMapper would have named the table if I wasn't interferring
  5. I'm logging to standard out so that I can see the queries called to verify that DataMapper is creating tables with the names that I want. This is used later on in this post to demonstrate this solution working, however, it could be left out without affecting anything.
  6. Initial setup of a sqlite database, and then the good stuff. Once a database has been setup with a specific adapter you can change the naming convention DataMapper will use to generate table names. This is accomplished by passing the module constant name through the repositories adapter and too "resource_naming_convention" as demonstrated in the code.
  7. Here I'm defining an example model of no importance. This is purely for demonstration, normally DataMapper would name this model "people".
  8. Inform DataMapper we're done setting it up and to run the migrations to create the model defined.

When you run this ruby file (assuming you have the "data_mapper" and "dm-sqlite-adapter" gem installed) you'll see output very similar too this:

~ (0.001402) PRAGMA table_info("source_people")
~ (0.000089) SELECT sqlite_version(*)
~ (0.077840) CREATE TABLE "source_people" ("id" INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, "first_name" VARCHAR(50), "last_name" VARCHAR(50), "email" VARCHAR(50))

Notice the third line? Specifically the name of the table? It's named exactly as it would have been except now it has a prefix of "source_".

Hope this saves someone else some trouble. Cheers!

Thoughts on IPv6 Security and Mitigation

I setup IPv6 on my home network with an OpenWRT router and Hurricane Electric and now I suddenly have an opinion on the state of IPv6 security. This is something that I've been meaning to do for some time and have been mulling over in the back of my mind. I'll go over the details from start to finish of setting up hurricane electric on the router in another post as the information to do so is very scattered and disjointed. It does appear to be very well documented on the OpenWRT wiki but I found that they leave out some very important steps, so stay tuned for that.

Lets start with the loss of NAT. NAT was never intended to be a security measure and lots of people will argue with me for saying that it was. However, the truth of the matter is that any machine that is behind a NAT is not directly addressable from the internet without someone on the inside intentionally poking holes (even if that someone is a bad-guy).

Anyone technically knowledgeable enough can usually use fingerprints of how a machine responds to different types of traffic both normal and unusual to identify what operating system, version, and specific details about the services and potential vulnerabilities of that machine. This information is invaluable to attackers.

One of IPv6's biggest selling points (and one that I quite enjoy, don't misunderstand this post) is that every device is addressable. This can potentially allow attackers to learn more about what they're attacking.

So how do you counter this? Well firewalls can help quite a bit in thwarting OS fingerprinting, but even the strongest firewall won't completely prevent this. Another, more protective layer that IPv6 gives you for free is it's sheer size.

Doing a pure enumeration of a single home subnet remotely (that is you are not on the link IPv6 local link) would take millennia by some estimations, as opposed the the IPv4 address space which could be done at home in a matter of weeks. One house vs the world. That is the scale we are now working at in IPv6.

The vast scale, however, while enough to defend against random scans, will not prevent your address showing up in server logs that you connect to. A single IPv6 address can be scanned just as easily as an IPv4 address.

What's more is that once an attacker has a presence on a subnet they can enumerate every single machine on that network in a matter of seconds to minutes. Since most home users are infected through drive by trojans, found in emails, and websites that the user chooses to go to, and attackers are already used to not having direct access to a machine from the internet, means the slowness and difficulty of the raw scans just simply won't be an issue.

Due too the ease of enumerating local networks and that they probably contain more vulnerable machines.... I'll leave that extrapolation as an exercise to my readers. I do predict that infrastructure will become a larger target due to this, just to collect their logs to attack home users.

The next thing that I want to bring up is IPv6, by default, uses the interface's MAC address to generate the last 12 characters / 48 bits of the IPv6 address. The rest of the local address consists of an identifier indicating that the address was generated using a MAC address and was not randomly generated.

So what does this actually tell us? Well if the OS and OS version can help up specify attacks, why not the brand and possibly the model of the MAC address? What attack vectors are waiting in the firmware of our ethernet and wireless cards? Firmware that almost never gets updated, and is known to have bugs and quirks?

That one is actually an easy one, RFC 3041 defines "Privacy Addresses". These are completely random local addresses that get generated once a day. Logs become increasingly useless on the server end, there is a lot of decoys on networks, and we're no longer exposing as much information to potential hostiles.

I use Fedora 17 and it took me a while to figure out how to enable privacy addresses the "Red Hat" way. You can easily do it generally on any Linux system with sysctl. Just add the following to your /etc/sysctl.conf and reload sysctl:

net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2

But that isn't the "Red Hat" way. Red Hat manages it's interfaces and network configuration through various interface configuration files living in /etc/sysconfig/network-scripts. For any IPv6 enabled interface you can turn on privacy addresses with the following line for example in "/etc/sysconfig/network-scripts/ifcfg-eth0":

IPV6_PRIVACY="rfc3041"

After restarting your network interfaces they will additionally have privacy addresses that will be change automatically. The interfaces still have the MAC based addresses as well but they will not longer be the default and thus will not show up in remote server logs.

Now, what would be a solid and strong step forward was a way to have a local machine register it's privacy address with a local IDS/IPS with an expiration, and to automatically trigger the IDS/IPS whenever a new connection is made to an expired privacy address. It would almost be like a free honey pot on your own network.

Updating to a Newer Rails on DreamHost

I've started getting heavily into Ruby on Rails development and wanted to use it in my DreamHost account. I've had a shared web account with them since 2004 and they've always treated me well. As a former PHP developer they've always had everything that I've needed (minus PostgreSQL but everyone has their flaws hahah) and I was sure they would have my back with Rails.

Unfortunately I found a very old version of Ruby, an old version of Rails and a sort of locked down way to install gems. Googling and DreamHost's wiki resulted in lots of questions but few real this-is-how-you-do-it answers. I finally managed to piece together a solid solution to get a newer Rails running on DreamHost through their Passenger installation. This solution doesn't fix the version of Ruby I have to use but it gives me enough control over the actual Rails environment that I'm willing to look past it.

For those out there, yes I am aware of Heroku and professionally I have sites on there. Yes I know it's free for a basic account but as soon as you start getting into real hosting the web worker dynos, database dynos, log drains all quickly add up to a monthly price that costs me more than two years of DreamHost hosting.

Please note that this guide assumes you are starting from a completely fresh DreamHost user. If you've made changes to your shell files, you'll need to adapt them for this. I also assume that you have not yet setup the site to be "hosted". What I mean by this is the domain shows up in your DreamHost account under a heading of "Registered domains without hosting". It isn't a big deal if you have already set it up but you'll probably need pay attention to the last part to ensure it is setup properly.

So what will this guide get you?

  • A version of Rails that is 8 minor revisions newer (3.0.11 vs 3.0.3). This includes several security fixes.
  • An isolated gemset from other applications, preventing nightmare gem dependency issues when hosting several rails app in the same account
  • A very well defined environment that will allow you to replicate a live server environment easily in a development environment

SSH into your DreamHost user, open up .bashrc file and add this:

export PATH=$HOME/.gems/bin:$HOME/opt/bin:$PATH
export GEM_HOME=$HOME/.gems
export GEM_PATH="$GEM_HOME"
export RUBYLIB="$HOME/opt/lib:$RUBYLIB"

alias gem="nice -n19 ~/opt/bin/gem"

And .bash_profile:

source ~/.bashrc

Run the following commands to setup the environment for the upcoming steps:

[dreamhost]$ mkdir ~/{src,opt}
[dreamhost]$ cd src
[dreamhost]$ wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz
[dreamhost]$ tar -xzf rubygems-1.3.7.tgz
[dreamhost]$ rm -f rubygems-1.3.7.tgz
[dreamhost]$ cd rubygems-1.3.7/
[dreamhost]$ ruby setup.rb --prefix=$HOME/opt
[dreamhost]$ cd ~/opt/bin/
[dreamhost]$ ln -s gem1.8 gem

At this point you're ready to start using the rubygems version you just installed.

[dreamhost]$ gem -v
1.3.6
[dreamhost]$ source ~/.bash_profile
[dreamhost]$ gem -v
1.3.7

You'll want to update to the latest version locally:

[dreamhost]$ gem update --system
Updating RubyGems
Updating rubygems-update
Successfully installed rubygems-update-1.8.17
Updating RubyGems to 1.8.17
Installing RubyGems 1.8.17
RubyGems 1.8.17 installed

RubyGems installed the following executables:
        /home/<username>/opt/bin/gem1.8
[dreamhost]$ gem -v
1.8.17

Please note that you might have a newer version if one has been released since I wrote this guide.

Because I don't need ruby documentation in my server environment I disable the installation and generation by default by creating a ~/.gemrc file and populating it like so:

install: --no-rdoc --no-ri
update: --no-rdoc --no-ri

Install bundler and rake locally.

[dreamhost]$ gem install bundler
Fetching: bundler-1.0.22.gem (100%)
Successfully installed bundler-1.0.22
1 gem installed
[dreamhost]$ gem install rake
Fetching: rake-0.9.2.2.gem (100%)
Successfully installed rake-0.9.2.2
1 gem installed

Installing more gems than this into the local system isn't recommended. If you define the gems for individual applications you want inside your project's Gemfile you can prevent dependency hell when you're updating gems.

Instead we can setup RVM to handle gemsets.

[dreamhost]$ bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)

You'll need to source your .bash_profile file again and make sure that RVM is up and working like so:

[dreamhost]$ source ~/.bash_profile
[dreamhost]$ rvm -v

rvm 1.10.2 by Wayne E. Seguin , Michal Papis  [https://rvm.beginrescueend.com/]

Before we can start using project specific gemsets we need to install a matching ruby version with the DreamHost one. This is where most people will want to differ from my instructions but I'll tell you now it won't work. DreamHost as of this tutorial was using ruby-1.8.7-p72 you can check this by running the following:

[dreamhost]$ ruby -v
ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]

We need to use this version specifically because this is what DreamHost's passenger (aka mod_rails) will be running under. Using another version will work while testing from your command line, but issues will crop up on your production site. There isn't anyway to work around that short of purchasing a VPS or dedicated server, since you're reading this I'm going to assume you've already considered that and passed on that option. Of course you're also welcome to completely disregard this warning as well.

So lets go ahead and install the appropriate ruby version through RVM:

[dreamhost]$ rvm install ruby-1.8.7-p72
[dreamhost]$ rvm use --default ruby-1.8.7-p72
[dreamhost]$ ruby -v
ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux]
[dreamhost]$ which ruby
/home//.rvm/rubies/ruby-1.8.7-p72/bin/ruby

And now we're using our own copy of the same version of ruby and it's being run out of our home directory. Excellent. This means we can now create unique gemsets for our projects without worrying about dependencies between them.

Lets setup a simple project for example.com using rails 3.0.11 just as a sample. Why not 3.1.x or 3.2.x? Well the catch is another old gem running in DreamHost's Passenger. They are running rack version 1.2.1 and as of 3.1.0 rails started requiring rack 1.3.2 or later, with rails 3.2.x it jumped all the way up to 1.4.5 or later. Of course I found this out the hard way... This could be worked around by running the application as a FastCGI application and I intend to put up a guide for that as soon as I can get that environment stable.

We'll start by creating a the domain folder for it and setting up an .rvmrc file with a unique gemset for the project.

[dreamhost]$ mkdir ~/example.com
[dreamhost]$ cd ~/example.com
[dreamhost]$ rvm --create --rvmrc [email protected]_com

Leave the directory and come back in. It should alert you about a "new or modified" .rvmrc file in the directory. This is the one you just created and yes you want to trust it. You trust yourself don't you?

Verify we're inside our gemset real quick:

[dreamhost]$ rvm gemset list

gemsets for ruby-1.8.7-p72 (found in /home//.rvm/gems/ruby-1.8.7-p72)
=> example_com
   global

Perfect the arrow is next to our gemset. Lets install rails 3.0.11 (DreamHost currently provides version 3.0.3).

[dreamhost]$ gem install rails -v 3.0.11

It will take a few moments for ruby gems to go out pull down the package lists, figure out dependencies, grab to source and install them so get a drink of water and come back.

Great! You're back lets make sure we're running the right version of rails now:

[dreamhost]$ rails -v
Rails 3.0.11

Voila! Lets get that project going with a mysql backend:

[dreamhost]$ rails new . -d mysql

And because it's good to do lets get this under version control.

[dreamhost]$ git init
Initialized empty Git repository in /home/<username>/example.com/.git/
[dreamhost]$ git add .
[dreamhost]$ git commit -m "Initial project commit"

Before we go any further there is a gem that needs to be added to our Gemfile. If you bundled and ran the console right now it would be seem happy but there is a subtle error that you wouldn't find out until the very end and it's better to get it out of the way now.

Add the following line to the Gemfile anywhere in the main part of the config, I prefer near the top:

gem 'rack', '1.2.1'

Then we'll re-bundle to make sure we have it:

[dreamhost]$ bundle install

You can make sure your project is happy by popping open the rails console.

[dreamhost]$ rails console
Loading development environment (Rails 3.0.11)
1.8.7 :001 >

Woo! That right there is Rails 3.0.11 working on DreamHost. Since it's working lets add another commit:

[dreamhost]$ git commit -a -m "Added rack 1.2.1 and tested for working rails 3.0.11"

I've already setup a database for this site named "example_com" with user "examplewebperson" and password "web_persons_password" on "mysql.example.com". DreamHost makes this process pretty intuitive through their panel and there are plenty of other tutorials out there to help you along if you get stuck so I'm going to skip over that part. I also personally prefer to use sqlite3 for development and testing as it means I don't have to run another service on my development machines. In that light this is what my config/database.yml looks like:

development:
  adapter: sqlite3
  encoding: utf8
  database: db/development.sqlite3

test:
  adapter: sqlite3
  encoding: utf8
  database: db/test.sqlite3

production:
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: example_com
  pool: 5
  username: examplewebperson
  password: web_persons_password
  host: mysql.example.com

We'll need to add the sqlite3 gem for the test and development environment, add this to the end of your Gemfile:

group :development, :test do
  gem 'sqlite3'
end

And once again run:

[dreamhost]$ bundle install

Lets make sure our development and then production databases are happy:

[dreamhost]$ rake db:migrate
[dreamhost]$ sqlite3 db/development.sqlite3
SQLite version 3.5.9
Enter ".help" for instructions
sqlite> .schema
CREATE TABLE "schema_migrations" ("version" varchar(255) NOT NULL);
CREATE UNIQUE INDEX "unique_schema_migrations" ON "schema_migrations" ("version");
sqlite> .quit
[dreamhost]$ RAILS_ENV=production rake db:migrate
[dreamhost]$ mysql -u examplewebperson -pweb_persons_password -h mysql.example.com example_com

mysql> show tables;
+----------------------------+
| Tables_in_example_com |
+----------------------------+
| schema_migrations          |
+----------------------------+
1 row in set (0.01 sec)
mysql>; quit

Lets commit one last time now that we have some schema:

[dreamhost]$ git add .
[dreamhost]$ git commit -m "Configured database"

We're almost there, open up config/environment.rb and add this line to the top of this file. Pay close attention after the @ sign on that as it should be the same as the gemset you created earlier.

if ENV["RACK_ENV"] == "production"
  ENV['GEM_PATH'] = File.expand_path('~/.rvm/gems/[email protected]_com') + ':/usr/lib/ruby/gems/1.8'
end

Last thing, since we're working around a lot of things we need to run bundle install in the production environment to make sure we have all the dependencies we need for the live site. This unfortunately will need to be done everytime you manipulate your gem sets:

[dreamhost]$ RAILS_ENV=production bundle install

Open up the DreamHost panel and find the domain your setting up under the "Manage Domains" setting. Add Hosting to it. Make sure you are running it under the user that we just set everything up in with a web directory set to/public. For example my testing domain would be example.com/public. Check the checkbox for "Passenger" and then hit the button "Fully host this domain".

As soon as you receive an email from the friendly DreamHost robot, load your site up in a web browser and you should be greeted with the stock Rails 3.0.11 site. Note that the "About your application's environment" link will not work in the production environment, this is intentional and expected for Rails.

This is where I leave you to develop your site. Good luck!

Ruby's XMLRPC::Client and SSL

For the past few days I've been working on a Ruby project that needed to interact with a remote XMLRPC API. This isn't particularly unusual but it was the first time from within a Ruby application. Luckily enough Ruby has a built in XMLRPC client that handles a lot of the messy bits.

The XMLRPC::Client class itself seems fairly simple. There are only a handful of methods, five of which are for opening a new connection in a few different ways, and at least two ways to open each type of connection.

As a starting point this was a simplified chunk of code that I was using to connect to the remote API:

require 'xmlrpc/client'

class APIConnection
  def initialize(username, password, host)
    # Build the arguments for the XMLRPC::Client object
    conn_args = {
      :user => username,
      :password => password,
      :host => host,
      :use_ssl => true,
      :path => "/api"
    }

    @connection = XMLRPC::Client.new_from_hash(conn_args)
  end

  def version
    @connection.call("version")
  end
end

The problem I ran into was when connecting to a server using HTTPS. I knew that this certificate was good however I continued to get the message:

warning: peer certificate won't be verified in this SSL session

Ruby has taken the approach of by default not including any trusted certificate authorities which I greatly appreciate especially considering that in 2010 and 2011 12 certificate authorities were known to have been hacked including major ones such as VeriSign, and DigiNotar. Some of which were proven to have issued false certificates.

Since XMLRPC::Client doesn't expose it's SSL trust settings through it's methods I went on a bit of a journey through Google to find an answer. What I found was overly disturbing, a lot of people don't seem to understand what SSL is actually for. The solutions I found from the most egregious to least:

  • Disabling OpenSSL certificate checking globally with  OpenSSL::SSL::VERIFY_NONE
  • Overriding the Net::HTTP certificate checking
  • Disabling OpenSSL certificate checking locally by extending XMLRPC::Client and over-riding how it was establishing connections
  • Using an SSL stripping proxy

I couldn't find a solution out there that didn't the security conscious voice in my head scream in despair. I asked on StackOverflow for a good solution. When I asked I didn't have a good grasp on how Ruby was handling SSL certificates at all. The thorough answer from emboss didn't quite answer my question but it gave me more than enough to really hunt down what I wanted.

First stop, I needed the certificates that I'll be using to verify the connection. Every single certificate authority that issues certificates for public websites makes the public portion of their certificates available and this is what we need to verify the connection. To find out which ones you specifically need you can go to the API server's address and look at it's certificate information by clicking on the site's lock icon. Every browser is a little different so you'll have to find this out on your own. With Chrome (and perhaps others) you can download each of the certificates in the chain that you'll need to verify the server's certificate.

The server I was connecting to was using a RapidSSL certificate, who has been verified by GeoTrust. You want to grab their certificates base64 encoded in PEM format. Stick them all in a "ca.crt" file. For these two CAs you're file will look a lot like this one:

-----BEGIN CERTIFICATE-----
MIID1TCCAr2gAwIBAgIDAjbRMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTAwMjE5MjI0NTA1WhcNMjAwMjE4MjI0NTA1WjA8MQswCQYDVQQG
EwJVUzEXMBUGA1UEChMOR2VvVHJ1c3QsIEluYy4xFDASBgNVBAMTC1JhcGlkU1NM
IENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAx3H4Vsce2cy1rfa0
l6P7oeYLUF9QqjraD/w9KSRDxhApwfxVQHLuverfn7ZB9EhLyG7+T1cSi1v6kt1e
6K3z8Buxe037z/3R5fjj3Of1c3/fAUnPjFbBvTfjW761T4uL8NpPx+PdVUdp3/Jb
ewdPPeWsIcHIHXro5/YPoar1b96oZU8QiZwD84l6pV4BcjPtqelaHnnzh8jfyMX8
N8iamte4dsywPuf95lTq319SQXhZV63xEtZ/vNWfcNMFbPqjfWdY3SZiHTGSDHl5
HI7PynvBZq+odEj7joLCniyZXHstXZu8W1eefDp6E63yoxhbK1kPzVw662gzxigd
gtFQiwIDAQABo4HZMIHWMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQUa2k9ahhC
St2PAmU5/TUkhniRFjAwHwYDVR0jBBgwFoAUwHqYaI2J+6sFZAwRfap9ZbjKzE4w
EgYDVR0TAQH/BAgwBgEB/wIBADA6BgNVHR8EMzAxMC+gLaArhilodHRwOi8vY3Js
Lmdlb3RydXN0LmNvbS9jcmxzL2d0Z2xvYmFsLmNybDA0BggrBgEFBQcBAQQoMCYw
JAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmdlb3RydXN0LmNvbTANBgkqhkiG9w0B
AQUFAAOCAQEAq7y8Cl0YlOPBscOoTFXWvrSY8e48HM3P8yQkXJYDJ1j8Nq6iL4/x
/torAsMzvcjdSCIrYA+lAxD9d/jQ7ZZnT/3qRyBwVNypDFV+4ZYlitm12ldKvo2O
SUNjpWxOJ4cl61tt/qJ/OCjgNqutOaWlYsS3XFgsql0BYKZiZ6PAx2Ij9OdsRu61
04BqIhPSLT90T+qvjF+0OJzbrs6vhB6m9jRRWXnT43XcvNfzc9+S7NIgWW+c+5X4
knYYCnwPLKbK3opie9jzzl9ovY8+wXS7FXI6FoOpC+ZNmZzYV+yoAVHHb1c0XqtK
LEL2TxyJeN4mTvVvk0wVaydWTQBUbHq3tw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDVDCCAjygAwIBAgIDAjRWMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMDIwNTIxMDQwMDAwWhcNMjIwNTIxMDQwMDAwWjBCMQswCQYDVQQG
EwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEbMBkGA1UEAxMSR2VvVHJ1c3Qg
R2xvYmFsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2swYYzD9
9BcjGlZ+W988bDjkcbd4kdS8odhM+KhDtgPpTSEHCIjaWC9mOSm9BXiLnTjoBbdq
fnGk5sRgprDvgOSJKA+eJdbtg/OtppHHmMlCGDUUna2YRpIuT8rxh0PBFpVXLVDv
iS2Aelet8u5fa9IAjbkU+BQVNdnARqN7csiRv8lVK83Qlz6cJmTM386DGXHKTubU
1XupGc1V3sjs0l44U+VcT4wt/lAjNvxm5suOpDkZALeVAjmRCw7+OC7RHQWa9k0+
bw8HHa8sHo9gOeL6NlMTOdReJivbPagUvTLrGAMoUgRx5aszPeE4uwc2hGKceeoW
MPRfwCvocWvk+QIDAQABo1MwUTAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTA
ephojYn7qwVkDBF9qn1luMrMTjAfBgNVHSMEGDAWgBTAephojYn7qwVkDBF9qn1l
uMrMTjANBgkqhkiG9w0BAQUFAAOCAQEANeMpauUvXVSOKVCUn5kaFOSPeCpilKIn
Z57QzxpeR+nBsqTP3UEaBU6bS+5Kb1VSsyShNwrrZHYqLizz/Tt1kL/6cdjHPTfS
tQWVYrmm3ok9Nns4d0iXrKYgjy6myQzCsplFAMfOEVEiIuCl6rYVSAlk6l5PdPcF
PseKUgzbFbS9bZvlxrFUaKnjaZC2mqUPuLk/IH2uSrW4nOQdtqvmlKXBx4Ot2/Un
hw4EbNX/3aBd7YdStysVAq45pmp06drE57xNNB6pXE0zX5IJL4hmXXeXxx12E6nV
5fEWCRE11azbJHFwLJhWC9kXtNHjUStedejV0NxPNO3CBWaAocvmMw==
-----END CERTIFICATE-----

Ugly right? That's what ruby needs though. But how do we get XMLRPC::Client to actually use that information without hacking it all to pieces? Net::HTTP has a few methods that allow you to set the appropriate connection settings and XMLRPC::Client uses Net::HTTP. If XMLRPC::Client allowed to you specify this directly somehow I would've been a lot happier.

Here's that code snippet again, this time forcing certificate verification with the ca.crt file. This code assumes that the ca.crt file lives in the same directory as the connection script:

require 'xmlrpc/client'

class APIConnection
  def initialize(username, password, host)
    # Build the arguments for the XMLRPC::Client object
    conn_args = {
      :user => username,
      :password => password,
      :host => host,
      :use_ssl => true,
      :path => "/api"
    }

    @connection = XMLRPC::Client.new_from_hash(conn_args)

    @connection.instance_variable_get("@http").verify_mode = OpenSSL::SSL::VERIFY_PEER
    @connection.instance_variable_get("@http").ca_file = File.join(File.dirname(__FILE__), "ca.crt")
  end

  def version
    @connection.call("version")
  end
end

Those last two lines in the initialize method first dive into the connection we've already setup (but before it's been called), grab the of Net::HTTP and tells it to force peer verification and to use the certificate file we created before. No more warning, and we're actually safe.

Exploration of an ACN Iris 3000

So I found a dirt cheap video SIP phone (ACN Iris 3000) at a local HAM fest. After looking around I found the vendor has locked in the phone with their specific service with an iron grip and had gone out of business. I guess I should expect that kind of anti-competitive behavior from a business that Donald Trump has a vested interest in.

I've come across one post on a forum that seems to have been crawled and copied out every where. The poster had cracked it and got it working with an Asterisk server which is what my ultimate goal for this phone is, however they claim to have done it by getting root through telnet. The problem being that port 23 (telnet) is not open so this was a dead end.

This is a running document of how I'm doing it, you'll notice that I'm writing this as I go.

Reconnaissance

First thing's first a little run down of what I've found. I can change the network address and the way it's handling networking (either bridged or NAT). I can not get into the Administrators menu which is where all the juicy bits seem to be. I've found that the factory reset code is 7517517, though that doesn't get me anything beyond cleaning up it's last known phone number.

Through a very thorough nmap scan I've found that ports TCP 21, 79, 113, 513, 514, 554, 5060, 7022, and 8080 are all open. There doesn't appear to be any UDP ports available which actually surprised me, since SIP over UDP is pretty common.

7022 and 8080 both immediately caught my eye. 7022 looks like someone moved SSH (port 22) to a non-standard port, and 8080 is a very common alternate port for HTTP. Connecting to 7022 via telnet confirmed my suspicions of SSH. I received this prompt:

Connected to 10.0.0.85. Escape character is ']'. SSH-2.0-dropbear_0.45

Bingo. SSH it is, and an old version of dropbear at that.  Unfortunately as the one poster I found said the password was neither blank nor 'root'. I suspect that they had an older firmware revision and these 'bugs' were ironed out in a later revision. That's ok though it'll just take a bit more work.

As for port 8080 it is definitely running a web configuration interface. All it asks for is a password (which we don't have). The extension for the login page (esp) makes me suspect that the Iris device is running a copy of AppWebServer or something similar and using embedded javascript as the server side processing. For now that doesn't provide much but it could be very useful later on.

Attack

So while looking for an exploit for DropBear 0.45 I started up a SSH dictionary attack and encountered by first real problem. The screen started blinking while running three or more threads trying to break in, at first I thought it was kind of funny but then it turned off completely.

Turns out the adapter I have for it is only rated for pushing out 500mA and the phone itself takes up to 1500mA, apparently I hit that limit and browned-out the phone. It still seems to work but if I want to take this route I'll need a more robust power supply. Looking around I found a 1500mA supply and after checking the boards for damage I gave it a shot and everything seems to be working OK.

Unfortunately I wasn't able to find any viable exploits for that particular DropBear version as the vulnerabilities that had been found were either DoS vulnerabilities or were only useful with valid credentials.

The basic dictionary attack failed and I started up a more comprehensive one. I could easily start brute forcing this but it would take a very long time, especially if the company realized that a weak password wasn't cutting it.

Linux N Issues & KDE Multi-Monitor Woes

So I recently did a fresh install of Fedora 14 with KDE installed (not the KDE spin mind you) on my ThinkPad. I'm pleasantly surprised with hows it's working everything seems to be working out the box very stably. I used it without issue for a solid month and a half without a single issue.

Earlier this week I started having issues with my wireless card on some networks, but not at all of them. The most prominent one being my home network. I've had issues with my access point dropping connections before on a wide array of machines and not actually dropping it (ie: my laptop would see it as connected but the AP wouldn't exchange traffic with it). So when I started seeing this behavior I expected that issue to have cropped up again.

The actual behavior that I was witnessing was this:

  1. Connect to wireless network
  2. Use the connection for 5-10 seconds
  3. Pages would start timing out even though the connection was still 'active'

Disconnecting and reconnecting to the wireless would start the situation all over again, which quickly became frustrating but I didn't have time to mess with it so I just plugged into an ethernet port and went about my business.

The next day I received my the logwatch from my laptop (Yes, my laptop sends it's logs to my email) and it mentioned more than 20,000 new entries of an error I've never seen before:

iwlagn 0000:03:00.0: BA scd_flow 0 does not match txq_id 10

After poking around a bit online I found that the issue is with a recent kernel update (I'm currently running 2.6.35.11-83) changing the behavior of some sanity checks to wireless connections that support 'N'. Turns out my wireless card and all the networks I was having issues with support 'N'. Good to know.

It was easy enough to solve that issue, the kernel module just needed an option passed to it that I believe just disables 'N'. This is all well and good but I haven't had to manually pass options to a kernel module for a couple of releases now (I think the last was Fedora 10). Since then the file /etc/modules.conf has been deprecated in favor of placing files in /etc/modules.d/. There are some files that come stock in there but none are passing parameters to modules and the naming scheme doesn't seem to conform to anything.

I was unsure if there was something specific I had to name the file or if it needed to be in one of the existing files. I ended up creating the file /etc/modprobe.d/iwlagn.conf and putting the following in it:

options iwlagn 11n_disable=1
# This one might be needed instead
#options iwlagn 11n_disable50=1

After I rebooted the problem vanished like it was never there. If you notice there is a second option in there that is commented out. I found some people where the first option didn't work but replacing it with that second option did, so if one doesn't work for you try the second option.

The second issue that I encountered was just this morning. I'm working at a remote site today that I'm not at very often, and am using my laptop as my desktop workhorse. Usually when I'm at a remote site, I steal an office that isn't in use and claim it as my own, and today was no different. There was a screen in this office that had its power plugged it but it's VGA cable was just sitting there. I figured 'why not?' so I plugged it in and KDE happily announced that it detected a new display and offered to take me to the settings interface that would allow me to configure it.

"Awesome!" I thought, multiple desktops has always been one of those things I had to tweak and search around for and it looks like KDE is making some serious strides in their support for it. When I turned it on I wasn't really paying attention to the settings and my laptop display ended up on the wrong side of the screen and my primary desktop on the LCD. Not exactly what I wanted but it's cool that it was that easy to setup.

I went back into the configuration options switched things around, but no matter what I did, the 'primary desktop' was always on the external monitor. What's more is that there isn't any option for selecting the primary display! That used to be there...

I hunted around before getting frustrated and searching around online. Sure enough other people were annoyed by this regression but the solution was very easy (though it appears you have to do it everytime).

To set the primary monitor to my laptop screen (LCDS1) I just opened a shell and put this in:

xrandr --output LCDS1 --primary

Poof! Everything is all set and I'm happy once again. I hope that the KDE developers put back the primary display selection in the settings but for now it's easy enough. Hopefully this will help other people on the net.

IPv6 Enabled WebServers

I've happily been using DreamHost for over 6 years now. They recently announced that they are IPv6 enabled. All of my webservers are happily on the IPv6 internet now thanks to them and it was free to boot. In my upcoming IPv6 trials I now know at the very least I will still be able to update this blog.

Thanks DreamHost for staying on top of technology like this! You never disappoint.

Exploration of IPv6 Part 0: Overview and Project Goals

Last week I finally picked up a project that I've been putting off for far too long. IPv6. With all of the Class A IPv4 subnets assigned and the total pool of available addresses rapidly dwindling it was time to make the move. When I started hunting down information about IPv6 and using it on my local network I was inundated with a lot of information about what IPv6 is, why we need it, and when everyone will need to be on it but very little about how to implement it.

To this end I'm writing these posts about the how and glossing over the what, why, and when. For those of you who are interested in that information I strongly recommend you read the Wikipedia article on IPv6. It covers everything that I've found in other blog posts and news releases and a lot more that isn't.

On to the specifics. My home network is closer to a small-business network crossed with an enterprise network. I have multiple subnets devoted to specific tasks (Public, DMZ, Server, Trusted LAN). I have two wireless networks, the first one is WPA2 802.1x using my own PKI for certificates, and the second is WEP secured with static addresses MAC restrictions and does not route to the internet. The latter is for some very old devices I have that don't support the secure wireless but still need to talk to a local service.

Ultimately I should not have to sacrifice any of my security to upgrade to IPv6 and I firmly believe in defence in depth. Off the top of my head the only thing I lose security wise is NAT which a lot of people argue is security through obscurity and not actually a layer of security but I personally disagree. I don't have to lose NAT as there are NAT66 options out there but to be honest, we need to start looking at the future and sticking with old and outdated security models will only end up causing a lot of trouble. I also don't want to deal with the thought of 216,386 potential addresses out there (That's assuming every IPv6 address is hiding a NAT shudder).

My servers are all running either Fedora 14 or CentOS 5.5 as I'm most comfortable with the Red Hat architecture. Anyone out there using a different system architecture will have to look elsewhere (sorry). I also have Mac OS X (10.3 and 10.4) and Windows 7 clients. Lucky for me they all support IPv6 natively, they just need to be configured.

So what do I want to accomplish with this project? I intend to break this down into stages, each stage will be followed up with a post on this blog with the details of what I did, how I accomplished it, and any pitfalls I ran into. These stages will be:

  1. Firewall, Initial Connection, and Security Plan
  2. Getting the clients online (Routing, FreeRADIUS, radvd)
  3. Getting the public servers online (Apache, Postfix, Bind)
  4. Getting file & database servers online (MySQL, OpenLDAP, Samba, NFSv4)
  5. Getting support services online (Kerberos, Syslog, Snort)
  6. Pulling the plug on IPv4 (NAT64, DNS64)
  7. One month IPv6 only report

Part 2 may also include ISC DHCPd configuration for stateful IPv6 client configuration. This will be something that I cover all the pros and cons about in the security plan and will be decided then. At the very end (Parts 6 & 7) I will have all of my home systems running IPv6 only for one month too see the state of IPv6 and see what clients still work. Thanks to the magic of NAT64 and DNS64 I should be able to access everything that is still IPv4 as if it was IPv6.

Stay tuned. This is going to be a bumpy ride.

The Home Network and NAT as a Security Layer

One of the hot-topics for IPv6 (which I have been thinking about a lot lately) is NAT. I normally wouldn't go into detail about specifics that are obvious to people in my field but for the sake of this post I will. NAT or Network Address Translation, is a way for a large number of computers to share a single public IP address.

The router that is handling the NAT will keep track of connections coming in and out of it and re-write the destination IP to an internal address to keep the traffic flowing.

NAT was necessary with IPv4 because IPv4 only had 4,294,967,296 addresses on the internet and quite a few those were unroutable or reserved. With a world population of 7 billion only half the population of the planet could have a single device online at a time. IPv6 solves this issue by increasing the number of public routable address to 3.4×1038. That means that each person alive could have 4.9x1028 addresses online at any given time.

So what does this mean for NAT? Clearly we don't need it any more right? There is no way I'll ever use 4.9x1028 addresses. I'm willing to bet Google doesn't own that many machines. Well this is where the debate starts. NAT was never designed to be used as a security tool and it has even had some security ramifications because of it.

The weakness of NAT is also it's primary strength. What do I mean by that? You can't attack a computer if you can't talk to it. In my opinion, this alone has protected innumerable regular home users from all kinds of terrible things online. A standard COTS router that comes with most internet connections will stop port-scans and automated attack tools at the door.

Sounds good right? So why would people be opposed to it? Simple. It complicates things. There are a limited number of simultaneous connections that can go through a single NAT device. This hard limit of 65,536 connections (in reality this number is an order of magnitude less - 32,768) isn't changed between IPv4 and IPv6 and there really isn't a good reason to change it. Sounds like a lot but trust me it gets used up quickly.

There is also identity reasons, behind a NAT could be 100 people or 1 and to the rest of the world it will all look the same. If someone breaks into a home network there isn't any way to differentiate that cracker from a normal user to the outside world. This privacy also gives home users plausible deniability for anything that happens on their network.

But those arguments against have very little to do with security. So what is all this hype about NAT being a form of security through obscurity? The argument I come across whenever I ask neigh-sayers about NAT, is that if a user gets infected then the network can still be enumerated behind the NAT as if all the computers were on the Internet. This argument has one fatal flaw. It is depending on a user to get infected. A firewall has this exact same "vulnerability" so would they argue that a firewall is not a layer of security? I thought not.

NAT has it's problems, but claiming it is not a security layer is just plain wrong. IPv6 is here to stay and we should really start looking at the security implications of everything involving it. New security models need to be created and lots of of research needs to be done in this area still. In the mean time I suspect a lot of malware and viruses will start making use of IPv6 and how relatively unknown it really is.

A Little Surprise Nestled Among Software

So yesterday my morning started out by my organization's WAN administrator pointing out some weird https traffic coming from one of our building's subnets. I looked and sure enough there was a steadily growing stream of https traffic coming from there. There was a few offices in that building and several labs that weren't being used for the summer. Definitely not enough people to be downloading the 10Mb/s and growing stream of information.

My first thought was virus outbreak. Using the net flows generated by the building's head switch I watched as dozens of computers created https connections to an IP address block owned by a German telecom. Another red flag. The entire lab seemed to have it.

There were no PTR records to indicate any kind of legitimate service running on the IPs they were connecting to. Great. That lab had just been ghosted with an image so the master must have it. Sure enough the master was making the same connections. I blocked all https traffic on that subnet and went about digging deeper into the infection.

This is where I'm going to go on a bit of a tangent rant. Windows UAC. The labs have all had Windows 7 Enterprise deployed to them and for security reasons we left UAC on. It's annoying but for the most part it's a good security addition to Windows, with the exception of use cases beyond how the developers use their machines...

I'm far more comfortable doing everything on a computer from a keyboard, even in Windows. It's quite rare for me to use the mouse unless I'm playing a game, doing image manipulation or browsing the web. So when I open a command prompt from and administrator account and run a command that gets stopped by UAC I expect the same prompt I would for any program that I tried to run using Window's GUI or a Windows equivalent of the venerable sudo utility.

You have to start the command prompt with Administrator privileges which involves navigating through a context menu in a context menu in a context menu (the right click menu, in the accessories folder menu, in the programs menu...). Alternatively you can make it so the command prompt always starts with Administrator privileges, but I don't always need them and don't want them when I don't need them.

If this was limited to the command prompt I wouldn't be as bothered, but it does the same damn thing from the run prompt. Rather than popping up a dialog box to confirm an action Windows will immediately deny the ability to run the program and you're left navigating menus (or in the case of the group policy editor, traversing Window's folder structure to find the executable so you can right click on the damn thing).

Moving on... a quick run of netstat showed the connection attempts to the foreign servers stuck in a SYN_SENT mode. Good, it wasn't talking to anyone any more and the subnet's traffic graph reflected that. I ran "netstat -b" to find out what program was creating the connection. "Akamai".

I know that sounded familiar. I vaguely remembered it having to do with DNS and connecting user's to the closest server to deliver content but that didn't involve a client and definitely wasn't as noisy or as obnoxious as this thing. So where was it? Well no one was logged into any of the computers while this was happening so it was running before login so probably snuck itself in as a service.

Usually I'd just jump straight to the registry to try and find this kind of thing but I already had the computer management window open so I took a quick peek at the service listing. Right nr the top "Akamai NetSession Interface" was set to start on boot. Bingo, but this looked some what legitimate. We didn't willingly install this, but it had all the signs of professionally made software and wasn't trying to hide itself beyond not being listed as an installed program.

I shutdown all the computers on the subnet and went looking for answers. Apparently Akamai has started a service that allows companies to cache content used by their programs on user's local machines.

We didn't want this software, it's using a lot of traffic, and nowhere were we told that it would also be installed or what the ramifications would be. Regardless of what the company wants to call this, it's malware as far as I'm concerned even if it the software wasn't designed with a malicious intent it was consuming large amounts of resources that we didn't willingly give it.

All the software on the lab machines was there for a reason, and it wasn't obvious which of the pieces of software it came with. Without knowing where it came from, I know that simply removing it from the computers wouldn't be enough to stop it from coming back. I couldn't block the traffic as it was using standards compliant https connections.

Disabling the service seemed to do the trick so I wrote a quick Administrative Template for GPO that I could deploy across our domain to permanently disable the service on any machine that was unfortunate enough to get this nasty thing. If your familiar with GPO's and Administrative Templates here is the source of mine that I used to manage this damn service:

CLASS MACHINE
CATEGORY !!SystemOnly
  POLICY !!PolicyName
    KEYNAME "SYSTEM\CurrentControlSet\Services\Akamai"
    PART !!PartName DROPDOWNLIST REQUIRED
      VALUENAME "Start"
      ITEMLIST
        NAME "Automatic" VALUE NUMERIC 1
        NAME "Manual" VALUE NUMERIC 3
        NAME "Disabled" VALUE NUMERIC 4 DEFAULT
      END ITEMLIST
    END PART
  END POLICY
END CATEGORY

[strings]
SystemOnly="System"
PolicyName="Akamai NetSession Interface"
PartName="Akamai NetSession Interface Service Status"</code>

After enabling the policy and a couple of reboots later and the traffic was gone. I personally despise any company that installs anything other than what you want. It's just bad practice. If you have to hide it, your users probably don't want it. Don't piss off your user base, that's just bad for business.

Inspeckt and User Input

I started working on a rather large PHP based project that needed to take input from user's of the website clients (Both form submission and URL based navigation). These are without question the largest attack surface and most easily exploited location for vulnerabilities to be introduced into a web application.

For reasons outside of the topic of this post I decided to not use a pre-made framework such as Zend to build the application. This left me in a tricky situation as it meant I'd have to come up with a solution to validate that input. I went searching for a standalone library to handle this validation for me. Something that had already been tested and examined by the wonderful open source community (Can you tell where my loyalties lie?).

I finally get pointed in the direction of Inspekt. From the Google Code page:

Inspekt acts as a sort of 'firewall' API between user input and the rest of the application. It takes PHP superglobal arrays, encapsulates their data in an "cage" object, and destroys the original superglobal. Data can then be retrieved from the input data object using a variety of accessor methods that apply filtering, or the data can be checked against validation methods. Raw data can only be accessed via a 'getRaw()' method, forcing the developer to show clear intent.

Inspekt conveniently comes with several predefined validators (called tests) including email, zip code, alphanumeric only, credit card numbers, IP addresses, hostnames, even custom regular expressions. Filter extend this into a whole new area, returning only the type of information you want and stripping everything out of the variable.

The best part is that it is written in a way that you don't have to use it exclusively with POST and GET requests. You can create a 'cage' around any variable or array or just put filters and tests on them. It never gets rid of the raw input either if you ever need that again. For anyone out there that needs to validate input from anywhere, I'd strongly recommend taking a look at Inspekt.

Nice Update

I know I don't update this blog as often as I'd like and I have had a lot going on in my project realms to put up here. That sadly is not what this short little post is about. Normally, I really don't care about updates to WordPress. I perform them diligently to prevent breaches into my sites, however they don't really come with any other benefits.

This time however I was greeted with a clean refreshing theme. I've got to give it to the WordPress UI designers on this one. It is simple and that's all that I need.

pfSense and Gaming

Previously while reviewing open source firewall distributions (which I never finished), I mentioned some gaming issues. I found a solution that seems to fix all the issues I had with gaming but unfortunately creates a few new issues that I didn't arise. The issue lies with how pfSense performs NAT'ing.

The ports that a particular client opens under some cases can be easily predicted and in turn exploited. To prevent this pfSense randomizes the external port that traffic is coming from. This is perfectly fine for any application that follows networking standards. However it seems that games such as Call of Duty and Company of Heroes passes the port they open inside their network protocol which is what the game servers then use to connect back to them rather than looking at the actual frames that arrive from the client. This means that the server will be trying to connect back on the wrong port as pfSense has changed it.

I can see some impossibly small security benefits for the game programmers to do this. I'm guessing more likely than not this was a failing of the network library they use where they don't have access to the information in the frames of the packets so they wrote a work around for it, and in turn violate basic network programming best practices.

So what's the solution? I strongly recommend that anyone reading this read the complete article as it is possible to prevent yourself from being able to use your network connection behind pfSense. If any setting doesn't look quite like I describe it your probably using a different version and should consult with the good people in the IRC channels or in their forums.

Here are the steps to get it to work:

  1. Open up Firewall -> NAT
  2. There should be three tabs, 'Port Forward', '1:1', and 'Outbound'
  3. Click on the 'Outbound' tab
  4. There should be two radio buttons, by default "Automatic outbound NAT rule generation (IPsec passthrough)" is selected. Choose "Manual Outbound NAT rule generation (Advanced Outbound NAT (AON))"
  5. Verify that there is a mapping with the following settings:
    • Interface: WAN
    • Source: Your internal (LAN) subnet
    • Source Port: *
    • Destination: *
    • Destination Port: *
    • NAT Address: *
    • NAT Port: *
    • Static Port: YES
  6. If you have more than one subnet you'll need to do this for each one.
  7. Save and Apply the settings

That should be it, there has only been two issues that I've found, one I'm not really sure if it's related. About the same time as when I made this change UPnP broke. I'm not sure why but it's something that seems to break quite a bit with pfSense and I can see this being related to NAT'ing.

The second issue is traffic in between subnets. While it seems to mostly work, there have been a few oddities such as a computer on one subnet is able to talk to a computer on a different subnet as long as it initiates the connection. If the other computer trys to initiate the connection then it will just time out. It definitely wasn't related to firewall rules (I set an allow all on both interfaces and turned off the firewall on both computers). Switching back to auto-nat'ing resolved the issue.

Project Hermes: Broad Overview

For the past few months, the little free time that I haven't spent with friends or playing with my personal servers has been put towards working out the details of a rather ambitious plan for my car. The goal? Set my car up as a mobile internet relay for emergency use. I've named this project Hermes. My naming scheme for all of my projects is based off mythological figures, folk lores and beasts. It's a nice broad name set and I can usually associate the goal of the project with a well known (or not) story. Hermes, messenger of the gods, patron of boundaries and those that cross them. Hermes often helped travelers have a safe and easy journey. Perfect.

Now on to some details, what do I want out of this project? What are my requirements? I want to be able to quickly establish reliable mobile short range and long range communication which could be brought into an area where other more common forms of communication have failed. This means having a broad range of mobile wireless transmitting and receiving stations built into my car and ready to be turned on at a moments notice.

Natural disasters cause a lot of chaos and disorganization. Priority should always be the welfare of those affected including getting water, food, medical supplies and shelter to those without. To accomplish this organization and communication needs to be in place to facilitate people communicating these needs. That is one reason why I'm doing this, the second being that it's just plain cool.

The most prevalent form of communication in use today is the Internet. Even our phones have started using the Internet as a backbone for their infrastructure. However, finding an internet connection around an event such as hurricane Katrina is impossible due to the sheer volume of damage. What's the solution? There is a means of communication which seems to be overlooked more than not by most people; HAM radio.

HAM or Amateur radio allows for extremely long range communication (in certain cases to the other side of the world). Amateur radio operates in many frequencies all of which you are required to have a license to transmit on. Licenses are fairly easy to acquire.

I learned the material the test covers in about three hours and paid $30 to get my Technician license. Amateur radio is usually used as the core means to communicate during national events due to the lack of infrastructure and long range that is provided by such a setup. Perfect for this project.

As a short range communication, two commercial access points with custom firmware should do the trick. One for broadcasting and one for connecting to additional access points. I want to be able to connect to additional access points to provide an internet connection to the car's future internal network, and through my car to become a hub for packet radio communications over Amateur radio.

This won't provide a lot of bandwidth (about 9600bps) which after over head and protocol communications works out to about 750 characters a second of communications. That may seem slow but thats a hell of a lot faster than trying to communicate the same message via voice.

In the event that cell service is available outside of the affected area but still within range of ham radio, a cell phone with a data connection and a bluetooth link can be the source of internet rather than leeching off a stranger's wifi.

To handle the routing of data, packet radio modem, bluetooth and wireless link, I'm going to need some kind of dedicated computer. Adding all of these devices to my car causes a problem. Power. Remote communications back to the hub can be accomplished using a handheld ham radio and a laptop computer.

Since I can't rely on the electrical grid during a natural disaster this means I'll need a way to charge their batteries from my base station, which is another major power drain. I could leave my car on to run the alternator and power the devices that way but I would quickly run out of gas and my car's battery wouldn't hold up for extended problems, leaving my stranded. My solution to this is a second electrical grid in my car.

A few deep-cycle marine batteries in the trunk (where I've decided all of this equipment is going to end up) and some high efficiency solar panels on the roof, combined with an AC charger for when I do have the luxury of plugging in should be enough to power a communications hub for an extended period of time.

So that's where I'm at. I've tentatively started pricing out solar panels and in-car ham radios for the project. Stay tuned! This is going to be an interesting ride...

Open Source Firewall Reviews: Virtualization

So I hit a snag while attempting to review the open source firewalls on my list. I still fully intend to get around it but the two spare boxes I had kicking around to do the testing on both decided that they didn't like their cd-rom drives. An excessive amount of required overtime at my job left me not very willing to get PXE installations of the various firewall distributions working. The only alternative I was left with was virtualization.

I've been playing around with virtualization quite a bit and have been having a lot of fun doing it. I've been focused on freely available server class virtualization. I trust linux. I have a lot of experience configuring it just to my liking and can harden a linux machine to an almost obsessive level.

While I have heard a lot of good things about VMWare and they do provide both a linux version and a free 'Viewer' the server level stuff is not available for free so I didn't consider them. This has the downside that the various firewall distributions tend to provide VMWare appliances pre-configured. I went with KVM. This choice is something that will need to be left for another blog post.

m0n0wall was not an option right off the bat. I couldn't even get the ISO to boot to do the installation. pfSense worked like a charm, no fuss at all. I still have yet to look at how the others are at dealing with being virtualized. Stay tuned.

Open Source Firewall Reviews: Intro & pfSense

Every now and then I like to refresh parts of my home network. New technology comes out, new versions of software come out and new exploits and attacks come out. This time around I felt it was about time that I look at the various firewall distributions that have come out in the past couple of years. I'm going to perform various reviews over the course of the next few weeks time (and memory) permitting.

I have a few slightly more obscure requirements for my firewall, some of which very few reviewers touch on. These being support for 802.1q VLAN tagging, a built in traffic shaper, a tftp server and options to broadcast it as a PXE boot server in the DHCP options, a captive portal on an arbitrary interface, and last but not least updates that don't require a full system rebuild.

The more common features on my requirements are port forwarding, stateful firewall with per interface rules, remote syslog logging, DHCP server, DMZ, and VPN access (perferably PPTP and OpenVPN).

For a long time I've been an avid supporter of pfSense. For those of you not interested in following the link, pfSense is a BSD firewall/router based on a very strong distribution called m0n0wall. pfSense is in a nutshell a frankenstein of some of the best pieces of the open source BSDs out there all wrapped in a bunch of PHP based scripts.

As much as I love pfSense there have been a few issues that I've grown to live with. The first being a large number of NAT issues. I'm not really sure how to classify this but it is very apparent if you play any multi-player online games (and exponentially more apparent if you have more than one person tying to play online at the same time). I'm going to use Company of Heroes and Call of Duty: Modern Warfare 2 as examples for this.

Right off the bat connecting online in Modern Warfare 2, it will display your "NAT Status" as either open or closed. When one person is connected it is usually "Open", when two people are connected one or both will display "Closed". Now this isn't a terribly big issue for MW2 as even with a closed NAT you can still play the game perfectly fine. The trick is that you won't be able to join any games (or even get into a lobby) with someone that is on the same network as you. So much for working together.

With Company of Heroes things get considerably worse. Even with only one person trying to play the game, whenever hosting or joining an online game about 50% of the individual players you try and connect to or that try to connect to you will fail with a "NAT negotiation error". This makes playing or hosting anything more than a 1v1 a royal pain, and more than a 2v2 nigh impossible.

Using a commercial off-the-shell router (in this case an old D-Link 524 I had laying around) these issues go away completely. At the time of this writing I am using version 1.2.3-RELEASE (built on Sun Dec 6 23:38:21 EST 2009). The NAT issues are known and you can find many people with the same issues complaining in their forums. The answer is almost always use static NAT mapping, but this only works for one player. It does seem to solve both issues though (as long as only one person being able to play is acceptable).

The next issue I've run across seems to have something to do with the state table. Now this is a tricky issue because it might be some dirty trick that Comcast is playing on me and I haven't tested any other firewall distribution yet to see where the problem lies.

Right off the bat, pfSense has a maximum state table size of 10,000. This is WAY more than enough for any home or small business networks. With five people behind it, I've only seen the state table jump as high as 1,500 and that was with all of us running torrents. The problem seems to be that anymore than 900 entries in the state table cause severe degredation in performance. How severe? With a state table of 958, it took 43 seconds for text to be echoed back to me over an SSH connection. That's impossibly bad latency. This issue is quickly resolved by blowing away the state table or rebooting the firewall, but will quickly crop back up when the computers start re-establishing the connections. The built in traffic shaper only seems to make this problem worse not better. (As an aside I'm running pfSense on 1.6Ghz box with 512 Mb of ram, this is far more than the minimum specs of 100Mhz and 128Mb of ram.)

Also while pfSense officially supports having a captive portal on a interface, I've only been able to get this working for a short period of time and that was back in version 1.21. So beware anyone wanting to use pfSense for a hotspot. The only reason I really wanted a captive portal was so that I could broadcast a second wireless AP that the public could connect to anytime and they would be able to see a kind notice asking them not to abuse the bandwidth I'm freely sharing.

So with these issues why have I stayed with pfSense for so long? It is a fantastic, stable system when you don't need to worry about torrents or gaming. There are quite a few packages that can be one click installed through the web interface. It provides good stats about the status of the system in a clean and easy to navigate interface.

pfSense has been doing a wonderful job of meeting all of my requirements with the exception of the captive portal but that wasn't really for me anyway, the gaming issues, and of course the state table. For anybody out there looking for a solid well rounded firewall don't let my issues deter you. There is a version 2 in the works and is currently available that might solve all of my issues, although it's alpha software right now and I didn't really want to have to troubleshoot my home network all the time due to buggy software when I'm trying to relax.

Image Crawler Meets rm -f *

So for giggles I wrote a simple web crawler that archived any image it found on a site that rapidly updated. My plan was to take all of the images resize them to a fraction of what they were and make a collage progression of the images posted over a 24 hours period. If that was successful I was going to try and sort the image by the highest peak on a gray scale histogram of the image.

I let my crawler go and stopped it after a 24 hour period. I ls'd the directory the images were being saved in to see how many I got and my ssh session locked up... Or so I thought. I hit Ctrl - C and nothing happened... So I closed my window and opened a new one. Did the same ls and the same thing happened. At this point I was really confused, did my system get compromised? Maybe a trojan ls was put on my system that was broken somehow?

The last thought put me into a panic, I raced to a system that held incremental backups of this entire system and ran an md5sum on /bin/ls on it and on every ls that I could find in the backup. There hadn't been any change. At this point I was fairly certain that wasn't the case so I moved on...

Maybe there was a filename that hit some weird glitch in ls's programming causing it too lock up. If this was the case how should I go about fixing it? I started thinking about how it would be a nice contribution to the community if I could figure out the error and report it, but I decided to take the lazy way out.

I cd'd to the directory and ran an rm -f * and was greeted with this:

/bin/rm: Argument list too long.

Wait what? All I get was this irritating and slightly elusive error message. So the * was being expanded, and making the list too long? To be sure I started hunting in the man pages. man rm recommended me to info coreutils 'rm invocation'. I read through and couldn't find any limitations or warnings that might relate to the problem. The only thing I could find in there was a line that said:

GNU rm', like every program that uses thegetopt' function to parse its arguments...

Alrighty moving on... so getopt is parsing it's options... man and info pages on getopt don't really reveal anything... So to Google!

After a bit of googling I found the answer, getopt's argument limit is 1024. So... how many files did I have? I wanted to give ls one more try... I typed it in and sure enough console froze... or did it? I walked away and did other things. When I came back I had a large list of files, longer than my console buffer... Ok-day, lets try this again 'ls -l | wc -l'. After about five minutes it came back again with just the number.

177654

Whoa... I wasn't expecting that... So how do I get around it... 'find . | xargs rm -f' and.... WOO Victory! The directory is clean and happy again... Now i'll just have to figure out how to cut down the number of files, or at the very least organize them into more managable chunks...

Impressive Setup...

So I joined a project named VOOM (Very Open Object Model). It's a group of PHP developers from around the world that all came together through the magic of the PHP General Mailing List.

The group got together to build a set of common classes such as data validators that developers could just drop into there code, much like PEAR. One of the goals is a common standard covering code testing, coverage, format, documentation and without the dependancies that are riddled throughout things like PEAR. The project is not intended to be a framework or anything else like that. Just high quality useful code available to the public.

The thing that is so impressive to me is first off how quickly the group not only got together and organized but how quickly a full development environment went up and became usable. We currently have an IRC channel, subversion, automated testing, wiki, coding templates.

Most of that was done by one person in the group, Nathan Rixham. I've never worked in any setup that... complete. Due too lack of time and creative inspiration I still haven't though I'm really looking forward to it.

The War Between the Text Editors and the IDEs

Long has been the battle of programming in a text editor and IDEs. Both sides have made good points and slung mud at the other. What all these zealots fail to realize is that there are merits to both sides. This is true with almost every war going on and while to each person the other side might seem ridiculous that doesn't make the other side wrong. But I digress.

This topic recently surfaced on a private mailing list that me and about thirty of my friends share. The topic didn't start as anything to do with development at all but rather around the cheap netbooks and there effect on the economy, moved into the time it takes to setup a computer after a full install. He upgraded his OS because the software that he runs needed new libraries that would break his old system. After upgrading he found weird compatibility issues with the new libraries and his system.

All the time he spent setting up his computer was lost development time on his various freelance projects. This is when it came around to development and development environments.

It is obvious to anyone who works with computers that a great deal of time can be spent on installing, configuring, and maintaining (upgrading) software. Since no real work is accomplished by this, such time can be thought of overhead time or as part of the "cost" of doing business in the computer field.

Back in the old days I programmed using a plain text editor and a command line compiler, perhaps together with a make-like tool. There was no fancy IDE with zillions of dependencies on complex graphical systems, no huge libraries, no complex web browsers with multiple plug-ins, etc, etc. Most of my time was spent actually writing programs. Huh.

I've used IDEs and I've used text editors. Personally I prefer text editors a lot more, but I definitely see the merits. Since most of my development is around PHP, a built in compiler doesn't do me much good. Subversion support between IDEs vary and with every one you have to learn a different user interface. On the other hand they allow you to quickly keep track of large number of files. There are other features that IDEs have but this is the only feature I can think of that is exclusive to IDEs that isn't in a plain text editor.

Those familiar with vim or emacs (yes another war and for the record I'm on the vim side), will be aware that they have syntax highlighting that would be the other feature that the IDE side tends to argue is exclusive to them. Which is sort of true since vim and emacs kind of blend both sides.

IDEs are bloated and complex with lots of features and buttons which takes a lot of time to learn. Text editors are simple all they do is edit text. vim and emacs both started out as simple text editors and they still behave exactly as they did but they have a lot of features that are common in IDEs. You can even compile from within them and at least in vim if the the compiler throws an error vim can jump straight to the line the error is on.

I haven't even begun to cover this war but I'm fairly tired of typing. What my verdict? Why not get the best of both worlds and just use vim or emacs?