Software updates by Troy D. Hanson

Starting processes at boot time, and keeping them running, is not glamorous but it’s important (if your computer gets power cycled, it’s nice to have your applications bounce right back… but I’m sure you already do this).

I put together a utility called pmtr (process monitor) to run my application daemons at boot time, and restart them as needed. I wanted to keep all of these jobs defined in one place– one configuration file. No rooting around in sysvinit files. Just my application’s jobs, all in one place. Like this,

/etc/pmtr.conf

  job {
    name weather-server
    cmd /usr/local/bin/weather -v
  }

  # this one keeps stderr and stdout
  job {
    name doppler-update
    cmd /usr/bin/dop-up
    dir /var/log
    err dop.err
    out dop.out
  }

That’s a simple pmtr.conf with just two jobs. No rocket science – just two jobs, in one place. There’s a handful of job configuration options it supports.

A friend asked me to support “deltas” if you change the configuration file- and this turned into one of the most useful features. It’s really two handy features that go together: pmtr monitors the file (using inotify) for changes so you don’t have to signal it when you make a change; then, when it rescans the configuration, it only fiddles with the jobs you’ve actually changed. You can disable or reconfigure jobs, or add new ones, and have that take effect without disrupting the other ones.

It’s not sysvinit, it’s not upstart, it’s not sophisticated. It’s just … handy.

You can read more about pmtr on it’s GitHub page.

I’ve posted my ØMQ zcontrol control port library to GitHub tonight. This makes it easy to embed an interactive control port in your ØMQ (v2 or v3)-based C programs.

Example of using a control port

% ./zcon -e tcp://127.0.0.1:3333
Connecting to tcp://127.0.0.1:3333.
zcon> help
help                 this text
version              version info
shutdown             shutdown server
quit                 close session

zcon> version
Current 0MQ version is 3.0.3

zcon> quit

You can add commands, for example — to control or monitor your program — using the zcontrol API. Use the zcon client shown above to run these commands interactively. Or, you can invoke them from any ØMQ req socket. I’ve used zcontrol both ways, e.g. manually and from a PHP script.

For further information please see,

• the zcontrol documentation: http://troydhanson.github.com/zcontrol/
• or git clone git://github.com/troydhanson/zcontrol.git

Feedback or suggestions are welcome- feel free to leave comments.

kvspool, coming soon

Pending some final approvals, I’m looking forward to releasing a new library this week for streaming data processing. It’s a small API for spooling off events (key-value sets, aka a dictionary, hash, etc). From there, other programs can consume them (possibly at a slower rate, hence you can couple fast and slow processes, while limiting the total resource consumption of the buffer). Or you can publish the stream to remote subscribers, and it will continuously receive the events written to the local stream. It’s a tiny API in C, Python, Perl and Java.

Zero MQ

Just a few words of appreciation for Zero MQ (sorry, “ØMQ”). These guys have taken a new conceptual approach to sockets (layering an API over standard TCP/IP sockets) that takes care of most of the headaches in network programs. I won’t go into detail but I immediately used pub-sub, went from there to using req-rep, to push-pull, and I love that the API is tiny, the docs are great (I love the ASCII diagrams rendered nicely as PNG, I discovered they are using ditaa for that, by the way). I ported my controlport library for use with ZeroMq-message-loop based programs. Hope to post it here soon.

Entropy

For those who appreciate a dose of theory with their morning coffee, my notes about measuring entropy of a byte stream are on on Scripts & Snippets. Since I’m more interested in practical applications, here’s a program that calculates the ultimate percentage to which an input file could be compressed.

uthash v1.9.4 has been released.

Version 1.9.4 (2011-06-05)

• uthash now supports MurmurHash v3
• utlist now includes concatenation macros (LL_CONCAT and DL_CONCAT)
• utarray now supports binary search (utarray_find)
• utstring now supports a new-or-clear-existing macro (utstring_renew)
• documented technique for a multi-level hash table
• clarified scope requirements for UT_icd in the utarray documentation
• fixed termination when utstring_clear is followed by utstring_body
• fixed utarray_inserta macro when used with complex arguments
• on Visual Studio define missing type uint8_t
• Debian/Ubuntu include uthash in the package uthash-dev.
• uthash has been downloaded 16,211 times.

Thanks to Yu Feng, Richard Cook, Dino Ciuffetti, Chris Groer, and Arun Cherian for feedback and fixes in this release!

Edited to add: nice example of an LRU cache with uthash at http://jehiah.cz/a/uthash

Here’s an example of software just working well: I went to write a “startup script” for my process (you know, one of those init.d startup scripts that gets your process going when the box is rebooted). I learned that, on Ubuntu, the init process has been replaced with the very friendly “upstart” package.

Here’s how a startup script looks. This is for the acpid daemon:

# acpid - ACPI daemon
#
# The ACPI daemon provides a socket for other daemons to multiplex kernel
# ACPI events from, and a framework for reacting to those events.

description   "ACPI daemon"

start on runlevel [2345]
stop on runlevel [!2345]

expect fork
respawn

exec acpid -c /etc/acpi/events -s /var/run/acpid.socket

I copied this file (renaming it for my own process and adjusting the exec line), put the file in /etc/init/pm.conf and voilà– my work was done. I just had to start up the process manually the first time using,

service pm start

A few pointers:

• If you want to see the upstart logs, look in /var/log/daemon.log.
• upstart itself is documented in init(8) (man -s 8 init)
• the conf file syntax is documented in init(5) (man -s 5 init)

uthash v1.9.3 has been released. This is a minor bug fix release.

Version 1.9.3 (2010-10-31)

• fix an ifdef for compatibility with Intel compiler (thanks, degski!)
• fix HASH_ITER macro to satisfy C++ casting rules (thanks, Erik Bai!)