uthash v1.9.8 has been released.

What’s in version 1.9.8 (2013-03-10)

• We now have a HASH_REPLACE macro (thanks, Nick Vatamaniuc!)
• fixed clang warnings (thanks wynnw!)
• fixed utarray_insert when inserting past array end (thanks Rob Willett!)
• you can now find uthash on GitHub
• there’s a uthash Google Group

On GitHub…
The official home of uthash is now the Github site. You can go there to clone it, grab a zipfile, or to contribute changes back as a pull request.

Google Group
The uthash Google Group is the new “go-to” place, to ask questions. Please ask them there first- and help me answer them there, if you can help!

Simplifying
Regarding versioned releases- it’s much easier to just merge changes, than to also bump version numbers and do announcements. Going forward I may only bump version numbers rarely and keep the master branch as a rolling current release.

I’ve also removed a few things– the PDF version of the documentation, and the tarball of the sources. Long live the tarball.

I’m bad at replying to people
An apology to people who have sent contributions and questions and didn’t hear back from me for weeks or months at a time: sorry! My life, and probably yours, is hectic. I hope the Google Group and the pull queue on Github help.

Twitter
You can follow me at @troydhanson for this kind of news too.

Thanks, again, to everyone who helps keep uthash going.

uthash v1.9.7 has been released.

A lot of people have sent me patches for uthash, utlist and utstring that have been building up for a while now. So- without further ado, here’s uthash 1.9.7:

Version 1.9.7 (2012-10-09)

• utstring now supports substring search using utstring_find (thanks, Joe Wei!)
• utlist can now prepend and replace elements (thanks, Zoltán Lajos Kis!)
• utlist prev/next fields can now have any names (thanks, Pawel S. Veselov!)
• uthash quiets a clang warning (thanks, Roman Divacky and Baptiste Daroussin!)
• uthash userguide example shows key uniqueness check (thanks, Richard Cook!)
• uthash HASH_MUR compiles under MSVC++ 10 in C mode (thanks, Arun Kirthi Cherian!)
• utstring now supports format checking in utstring_printf (thanks, Donald Carr!)

What is uthash?

If you’re new here, welcome. Uthash is a hash table for C structures. It’s implemented entirely as a single header file, uthash.h. In the same spirit, it includes a dynamic string implementation in utstring.h, a set of linked list tools in utlist.h and a dynamic array in utarray.h. They’re all in C, dependency-free, and open-source.

For docs and download links see the uthash website.

From the something-completely-different department:

What do you get when you mix a holiday weekend, with a cold and fever and a son who likes Minecraft? A lot of fun actually. As a dad and son activity, we set up a Minecraft server. To take it up a notch, we threw in a Mumble server for voice chat. It’s one thing to start them manually (or via GUI in the case of the Minecraft server). But I wanted to put them on a server without GUI access, and install them as services- so they’d restart if they die, or if the box reboots. There’s various ways to do this but I decided to run both servers under my pmtr process monitor.

Here’s my configuration from /etc/pmtr.conf:

job {
 name minecraft-server
 dir /home/mc/minecraft
 cmd /usr/bin/java -Xms32M -Xmx292M -jar minecraft_server.jar nogui
 user minecraft
}

job {
 name murmur-server
 dir /home/mc/murmur
 cmd /usr/bin/murmurd -fg -ini murmur.ini
 user minecraft
}

I grabbed the minecraft_server.jar, the Murmur binaries (the server for Mumble voice chat clients), a Linux box, put pmtr on it, and edited some config files. (Minecraft has server.properties, Murmur has murmur.ini, and pmtr has pmtr.conf). My son got to see under the hood of a Linux server, and then as the fruits of our labors we could enjoy some Minecraft with friends.

On a final note- I’m blown away with the artistry of Minecraft and the quality of Mumble. I even asked my son to turn his Minecraft guy so we could watch the Minecraft sunset- the gradients are just beautiful!, the falling snow or the rain. Mumble (which is free) makes our voice chats sound like we’re in recording studios. Great work by talented developers.

Here’s a quick example of how to run a daemon process at boot on four platforms:

• Ubuntu, using upstart
• CentOS/RHEL, using sysvinit
• Debian, using sysvinit
• Mac OS X, using launchd

Typically your system services- web servers, etc, start up through these mechanisms. The idea across all these platforms is that you put a script or text file describing your service in the right place, and the OS init apparatus starts everything.

Caveat: I’m not a initscript expert by any means. There are many options that vary across these platforms. What I’ve written here is a quick glance.

For the examples here – imagine we have a daemon called exd that we want to run at boot. The settings used in the initscripts here expect exd to stay in init’s foreground (that is, it doesn’t daemonize by forking again).

Ubuntu

Ubuntu uses “upstart” to manage jobs (at least for the modern releases).

1. Place this script in /etc/init/exd.conf.
2. It’s all ready to go. Start it up: start exd

CentOS/RHEL

This platform uses sysvinit-style scripts.

1. Place this script in /etc/rc.d/init.d/exd.
2. Symlink it to the runlevel-specific directories: chkconfig --add exd
3. Start up the job (on reboot it’ll be automatic): /etc/init.d/exd start

Debian

1. Place this script in /etc/init.d/exd
2. Symlink it into place: /usr/sbin/update-rc.d exd defaults
3. Start it: /usr/sbin/service exd start

Mac OS X

Mac uses “launchd” to start daemons.

1. Place this file in /Library/LaunchDaemons/exd.plist
2. Load and start it: launchctl load /Library/LaunchDaemons/exd.plist

That’s it

That’s a minimalistic look at initscripts on four platforms.

Edited to add: Debian example and replace the inline files with links.

Six years of uthash

I noticed on SourceForge that it’s been six years and a month since uthash was first released on March 28, 2006. Their counter shows 23,421 downloads. Thanks to everyone who has contributed features and suggestions. Over the years uthash grew offshoots, the “bonus” headers for dynamic strings, arrays and lists. These have become handy tools in their own right, judging by the emails I’ve received.

There’s more options for C hash libraries now than there were in 2006. It would be tempting to keep sharpening the axe and make a shiny new uthash, but API compatibility and time have kept things to incremental changes. So it is with the release of uthash v1.9.6.

This is a minor release of uthash with a few bug fixes and edits to the documentation in response to questions I’d received.

Version 1.9.6 (2012-04-28)

• add utarray_prev (thanks, Ben Hiett!)
• add parens/casts for greater compatibility (thanks, Atis, Debasis Ganguly, and Steve McClellan!)
• added ifndef to uthash_malloc and related hooks (thanks, Holger Machens!)
• edit examples so they do not leak memory (thanks, 任晶磊!)

Apparently, I forgot to post about the previous minor version, so for the record:

Version 1.9.5 (2011-11-16)

• added utarray_renew
• fixed memory leak in uthash_clear when using Bloom filter (thanks, Jan Hättig!)
• utarray now copies the UT_icd on array creation rather than storing a pointer
• add parentheses to HASH_ADD to fix preprocessing of certain arguments (thanks, Aaron Rosen!)
• more parenthesizations for greater macro argument flexibility

With that, I’m off to Hawaii. Aloha!

I’ve placed my kvspool library on GitHub.

kv-spool (“key-value” spool)

a Linux-based C library, with Perl, Python and Java bindings, to stream data between programs as key-value dictionaries. The data stream is backed to disk, supports rewind, snapshot, network replication, and bounded disk consumption.

It’s a tiny API. It’s been useful to me in streaming data applications, where you have a never-ending event stream. Besides making it trivial to generate or consume the event stream between different programs in different languages- I wanted kvspool to enable me to “copy off” a sample from a running stream, take it back to a laptop as canned data, and “rewind” and “replay” while developing newer versions of software that consumes the stream.

I also wanted to send the event stream over a network, effortlessly- in fact, invisibly- to the programs that generate or consume the events.

So there it is.

Thanks to the awesome Trevor Adams for his help on this.

Starting processes at boot time, and keeping them running, is not glamorous but it’s important (if your computer gets power cycled, it’s nice to have your applications bounce right back… but I’m sure you already do this).

I put together a utility called pmtr (process monitor) to run my application daemons at boot time, and restart them as needed. I wanted to keep all of these jobs defined in one place– one configuration file. No rooting around in sysvinit files. Just my application’s jobs, all in one place. Like this,

/etc/pmtr.conf

  job {
    name weather-server
    cmd /usr/local/bin/weather -v
  }

  # this one keeps stderr and stdout
  job {
    name doppler-update
    cmd /usr/bin/dop-up
    dir /var/log
    err dop.err
    out dop.out
  }

That’s a simple pmtr.conf with just two jobs. No rocket science – just two jobs, in one place. There’s a handful of job configuration options it supports.

A friend asked me to support “deltas” if you change the configuration file- and this turned into one of the most useful features. It’s really two handy features that go together: pmtr monitors the file (using inotify) for changes so you don’t have to signal it when you make a change; then, when it rescans the configuration, it only fiddles with the jobs you’ve actually changed. You can disable or reconfigure jobs, or add new ones, and have that take effect without disrupting the other ones.

It’s not sysvinit, it’s not upstart, it’s not sophisticated. It’s just … handy.

You can read more about pmtr on it’s GitHub page.

Example of using a control port

% ./zcon -e tcp://127.0.0.1:3333
Connecting to tcp://127.0.0.1:3333.
zcon> help
help                 this text
version              version info
shutdown             shutdown server
quit                 close session

zcon> version
Current 0MQ version is 3.0.3

zcon> quit

You can add commands, for example — to control or monitor your program — using the zcontrol API. Use the zcon client shown above to run these commands interactively. Or, you can invoke them from any ØMQ req socket. I’ve used zcontrol both ways, e.g. manually and from a PHP script.

For further information please see,

• the zcontrol documentation: http://troydhanson.github.com/zcontrol/
• or git clone git://github.com/troydhanson/zcontrol.git

Feedback or suggestions are welcome- feel free to leave comments.

Pending some final approvals, I’m looking forward to releasing a new library this week for streaming data processing. It’s a small API for spooling off events (key-value sets, aka a dictionary, hash, etc). From there, other programs can consume them (possibly at a slower rate, hence you can couple fast and slow processes, while limiting the total resource consumption of the buffer). Or you can publish the stream to remote subscribers, and it will continuously receive the events written to the local stream. It’s a tiny API in C, Python, Perl and Java.

Zero MQ

Just a few words of appreciation for Zero MQ (sorry, “ØMQ”). These guys have taken a new conceptual approach to sockets (layering an API over standard TCP/IP sockets) that takes care of most of the headaches in network programs. I won’t go into detail but I immediately used pub-sub, went from there to using req-rep, to push-pull, and I love that the API is tiny, the docs are great (I love the ASCII diagrams rendered nicely as PNG, I discovered they are using ditaa for that, by the way). I ported my controlport library for use with ZeroMq-message-loop based programs. Hope to post it here soon.

Entropy

For those who appreciate a dose of theory with their morning coffee, my notes about measuring entropy of a byte stream are on on Scripts & Snippets. Since I’m more interested in practical applications, here’s a program that calculates the ultimate percentage to which an input file could be compressed.

My notes about UNIX domain sockets are now on Scripts & Snippets. Here’s the condensed version.

Communication within a host

UNIX domain sockets are a method by which processes on the same host can communicate. Communication is bidirectional with stream sockets and unidirectional with datagram sockets.

Identity

Instead of identifying a server by an IP address and port, a UNIX domain socket is known by a pathname. Obviously the client and server have to agree on the pathname for them to find each other. The server binds the pathname to the socket.

File permissions control who can connect

For UNIX domain sockets, file and directory permissions restrict which processes on the host can open the file, and thus communicate with the server. Therefore, UNIX domain sockets provide an advantage over Internet sockets (to which anyone can connect, unless extra authentication logic is implemented).

Comparison with named pipes for IPC

IPC within a UNIX host by may be accomplished by several other means including named pipes. What circumstances favor UNIX domain sockets versus pipes? The choice is influenced by these factors:

Duplex

Stream sockets provide bi-directional communication while named pipes are uni-directional.

Distinct clients

Clients using sockets each have an independent connection to the server. With named pipes, many clients may write to the pipe, but the server cannot distinguish the clients from each other– the server has only one descriptor to read from the named pipe. Because the named pipe has only read descriptor and possibly-multiple writers, random interleaving can also occur if a client writes more than PIPE_BUF bytes in one operation. Since pipes have these limitations, UNIX domain sockets should be used if there are multiple clients that need to be distinguishable or which write long messages to the server.

Method of creating and opening

Sockets are created using socket and assigned their identity via bind. Named pipes are created using mkfifo. To connect to a UNIX domain socket the normal socket/connect calls are used, but a named pipe is written using regular file open and write. That makes them easier to use from a shell script for example.

Linux Abstract Socket Namespace

Linux has a special feature: if the pathname for a UNIX domain socket begins with a null byte , its name is not mapped into the filesystem. Thus it won’t collide with other names in the filesystem. Also, when a server closes its UNIX domain listening socket in the abstract namespace, its file is deleted; with regular UNIX domain sockets, the file persists after the server closes it.

Resources

Here are some C programs that implement a UNIX domain socket client and server. These are placed in the public domain.

• A server example
• A client example