Fortigate Firewall Logstash Grok filter

I’ve been playing with Logstash recently, just this week I was asked to import a Fortigate firewall log. I did this by putting up a logstash syslog interface on a specific port, tagging the inbound traffic as type=fortigate and then using a simple RE and the kv{} filter to parse the log.

The gist can be seen here, or embedded below:

Bash SSH known_hosts tab completion

This morning a discussion with a friend about various shells lead me to think it would be nice if my bash shell could tab complete hostnames from .ssh/known_hosts when I type ‘ssh <tab>’. I soon found this blog post which nicely documents how to do it. I made a directory in $HOME called .bash.completion and then added this to my .profile, which loops round any files in there, sourcing them individually:

if [ -d ${HOME}/.bash.completion ]; then
 for file in ${HOME}/.bash.completion/* ; do
   source $file
 done
fi

All sorted. However, it wasn’t long before I discovered that ‘ssh user@<tab>’ doesnt work, I tend to use this quite a lot so wanted to see if I could fix up the bash function to support that use case. Bit of hacking around and I’ve got it working, the replacement ssh-completion file is shown below:

# Add bash completion for ssh: it tries to complete the host to which you
# want to connect from the list of the ones contained in ~/.ssh/known_hosts
__ssh_known_hosts() {
    if [[ -f ~/.ssh/known_hosts ]]; then
        cut -d " " -f1 ~/.ssh/known_hosts | cut -d "," -f1
    fi
}
_ssh() {
    local cur known_hosts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    known_hosts="$(__ssh_known_hosts)"
    if [[ ! ${cur} == -* ]] ; then
        if [[ ${cur} == *@* ]] ; then
            COMPREPLY=( $(compgen -W "${known_hosts}" -P ${cur/@*/}@ -- ${cur/*@/}) )
        else
            COMPREPLY=( $(compgen -W "${known_hosts}" -- ${cur}) )
        fi
    fi
    return 0
}
complete -o bashdefault -o default -o nospace -F _ssh ssh 2>/dev/null \
    || complete -o default -o nospace -F _ssh ssh

Happy.

Monitoring beanstalkd with monit

A new project I’ve just deployed onto live uses the fast and lightweight beanstalkd work queue.  As part of putting it into live I wanted to get at least some basic monitoring on the beanstalkd daemon.  All my servers run monit for keeping an eye on processes I care about, so that seemed a good place to start.

I had a bit of a look around the interwebs but couldn’t find any examples, so I set about putting something together myself. As it turns out beanstalkd has a nice simple text protocol, detailed in protocol.txt which is included in the source, and monit has the ability to send and expect arbitrary strings to a given port.  For starters I’ve set it up to issue a stats command and check for the expected response which is:

OK <bytes>\r\n

Adding this to my monit config file for beanstalkd gives me:

check process beanstalkd with pidfile /var/run/beanstalkd.pid
 start "/etc/init.d/beanstalkd start"
 stop "/etc/init.d/beanstalkd stop"
 if failed port 11300
   send "stats\r\n"
   expect "OK [0-9]{1,}\r\n"
 then alert

This is a pretty simple check but at least will enable monit to alert if beanstalkd isn’t listening or is returning garbage.

I think in the long term I’d like nagios to alert on  the current-jobs-delayed and possibly the current-waiting variables reported by the stats-tube command for a given tube, but that is for another day!

Another thought is possibly to have a cacti graph of the total-jobs field, it looks like a always increasing counter, so should be pretty trivial to hook up to cacti.

Missing BNX2 firmware for Debian PXE/Netboot installations

I seem to have a inherent disklike of Debian and the feeling appears to be mutual. It never makes my life easy. Just this morning I needed to install it on a Dell R210 rather than our usual Centos builds. The server is 15 odd miles away, so I took my standard route of PXE installing. After downloading the netboot.tar.gz and dropping the right files in place on my netboot server, I booted the R210 and began the install. Only a couple of screens in I was presented with this most unhelpful message. So I have to drive 30 miles to plug a usb stick into this machine to continue? That isn’t acceptable imho.

In my case it was the non free firmware for the Broadcom ethernet cards in the machine, I needed this package.

Unhelpful Debian

Turns out there is a fix. You just need to download the missing .deb, cpio it and cat it into the end of the initrd:

% mkdir /tmp/firmware
% cd /tmp/firmware
% wget http://ftp.uk.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2_0.28+squeeze1_all.deb
% cd /tmp
% find firmware | cpio -o | gzip -c > firmware.cpio.gz

Now cat the resultant firmware.cpio.gz onto the end of the existing initrd.gz, which for me was in /tftpboot/debian-installer/amd64/initrd.gz. So I ran this:

% cat /tmp/firmware.cpio.gz >> /tfpboot/debian-installer/amd64/initrd.gz

That will overwrite the initrd.gz without warning, so might want to take a backup of it first.

Apple Aperture 3.4 update

So on the day of iOS6 Apple also released the 3.4 update to Aperture and osX 10.8.2. It seems if you apply all these updates, Aperture first updates your library, and then quits every time you load it. Brilliant, gee thanks Apple. why bother with actually testing software, you wouldn’t want to dent the 100 billion you have in the bank. Anyway, apparently it is something to do with the Facebook account info held within Aperture, you can zero this out by runnning the following at the command line:

defaults remove com.apple.Aperture AccountConfigurations

This worked for me but Aperture no longer knew about my Flickr, or Facebook accounts. At least I got my photos back. Between this and iOS6 maps, this isn’t a great week to own Apple products.

 

Local RPMs update

This is an update to the earlier post where I linked to some RPMs which I maintain for my own purposes. If you find these useful, please feel free to download them.

In the filenames, el6 is Red Hat Enterprise Linux 6 (Centos), el5 is Red Hat Enterprise Linux 5 (Centos), fc7 is Fedora Core 7.

If a link is broken, feel free to have a click around the SVN repository, the root of where I keep all the RPM stuff is here. Or please email me.

SVN to (BitBucket) Git migration

Notes from the migration of my personal SVN repo to Git and also onto the hosted Git platform Bitbucket.org.

First step was to tell my local git install who I was:

$ git config --global user.name "Robin Kearney"
$ git config --global user.email "robin@kearney.co.uk"

I always set the following too:

$ git config --global color.diff auto
$ git config --global color.status auto
$ git config --global color.branch auto

Then to import a section of my SVN repo:

$ git svn clone --authors-file=path/to/authors_file SVN_REPO_URL LOCAL_DIR
$ cd LOCAL_DIR
$ git svn show-ignore > .gitignore
$ git remote add origin git@bitbucket.org:rk295/GIT_REPO.git
$ git push origin master

Some people advise using -s on the ‘git svn’ command, which tells git to expect trunk/ tags/ etc. But I dont use those in my repo, so I omitted it.

SVN_REPO_URL is the full http://…. URL to the SVN repo to import, or in my case a sub-section of it.

The --authors-file tells git the location of a text file which maps SVN users to Git users, mine looked like this:

robin = Robin Kearney <robin@kearney.co.uk>

Using a non supported Timemachine volume

Always forget this bit:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Speficially needed if you are using a Linux box, shared by Apple Talk.

From here in this instance, although there are loads of howtos.

AirPrint for iOS on Linux

I quite often used to find myself wanting to print from my iPad, so when Apple announced AirPrint I thought things were looking good. According to the press release you would need either a HP printer or you could print to a shared printer on an existing mac. Sorted, I’ve got the latter of those, things were looking rosy.

Things took a turn for the worse when I read that they had pulled support for printing via shared printers and were only going to allow AirPrint to certain (at the moment only HP) printers.  I wanted the feature, but not enough to replace my trusty Lexmark.

Luckily there are people like this guy on the internet who took the time to figure out how it works and have since published a nice simple guide about how to setup a Linux box as a AirPrint server. Mainly thanks to Cups and some Avahi magic.

Sorted! And I’ve used it sufficiently to think spending an hour or so setting it up was worth while.

Upgrading through every version of windows

This guy takes a computer with MSDOS 5.0 installed and then upgrades through every major version of Windows all the way up to Windows 7.