Filed under: Software updates
kvspool, coming soon
Pending some final approvals, I’m looking forward to releasing a new library this week for streaming data processing. It’s a small API for spooling off events (key-value sets, aka a dictionary, hash, etc). From there, other programs can consume them (possibly at a slower rate, hence you can couple fast and slow processes, while limiting the total resource consumption of the buffer). Or you can publish the stream to remote subscribers, and it will continuously receive the events written to the local stream. It’s a tiny API in C, Python, Perl and Java.
Zero MQ
Just a few words of appreciation for Zero MQ (sorry, “ØMQ”). These guys have taken a new conceptual approach to sockets (layering an API over standard TCP/IP sockets) that takes care of most of the headaches in network programs. I won’t go into detail but I immediately used pub-sub, went from there to using req-rep, to push-pull, and I love that the API is tiny, the docs are great (I love the ASCII diagrams rendered nicely as PNG, I discovered they are using ditaa for that, by the way). I ported my controlport library for use with ZeroMq-message-loop based programs. Hope to post it here soon.
Entropy
For those who appreciate a dose of theory with their morning coffee, my notes about measuring entropy of a byte stream are on on Scripts & Snippets. Since I’m more interested in practical applications, here’s a program that calculates the ultimate percentage to which an input file could be compressed.
2 Comments so far
Leave a comment
Ooh, I like the idea of controlport for zeromq apps.
Comment by SQ October 10, 2011 @ 11:51 amI agree. There’s something about having interactive access to an application’s main loop that makes it come alive. It’s easy to integrate into ZeroMQ apps that already use zmq_poll to implement their main loop. I have a client called zcon that uses req/rep to talk to that main loop. Stay tuned for another post..
Comment by troydhanson October 10, 2011 @ 8:30 pm