Fifteen years ago I wrote the code I’m still proudest of, for a product category that no longer exists. This is its story, told now that I’ve finally given the repo a dignified closing README instead of letting it rot quietly.
2010, when video was hard
If you’re younger than the problem, here’s the world: web video meant Flash players doing progressive download over plain HTTP — and I do mean your own Flash player, because every video site had to write one; there was no HTML5 <video> coming to your help. Seeking worked by asking the server for the file from an offset (?start=). There was no HLS, no DASH, no adaptive bitrate; CDNs existed but were priced for companies with sales departments.
If you ran a video site, bandwidth was your business model — every wasted byte was money, and every rebuffer was a lost viewer.
I ran into this the honest way. My company operated peteava.ro, a Romanian video-sharing site (now defunct, like most of that era), and I had already written one streamer for it — psstream, a PHP extension from 2008. It did the job until it didn’t: none of our servers could hold more than about 150 simultaneous streams without the machine practically freezing.
That wall is where Loomiere begins.
First pass: the micro-streamers
The first Loomiere (0.x) was an exercise in minimalism: a set of tiny micro-streamers — one per format — written in C against dietlibc, statically linked, the MP4 streamer weighing about 21 KB on disk and 120 KB in memory, spawned one-process-per-request under ipsvd/runit.
It entered production on all of peteava’s streaming servers on Friday, January 29, 2010, at 14:31 (some moments you write down). The same day, a random check on the most strained server showed 653 simultaneous clients where the old streamer had managed 150 — and total bandwidth had dropped by roughly a third, because of the throttle I’ll get to in a moment.
Two months thinking, two weeks writing
The 0.x design had a ceiling — one process per connection is honest but profligate — so I rebuilt it. I spent about two months just thinking about Loomiere 2.0’s internals before writing it, and then wrote the whole thing in about two weeks.
Think long, write short — not always possible, but I’ve been trying to work like that ever since; the thinking is the work, the typing is transcription.
What I remember from those two months isn’t code at all: it’s the old whiteboard, so full of sketches that at some point I was defending it with my own body from the cleaning lady’s sponge. Improbably, a photo of it survives — the exact spot where the Lua and C universes cross inside the code, coroutines yielding on one side of the line, the event loop deciding on the other:
What came out: a C core on libev with SMP worker threads (“the hardware should be the only limit”), an in-memory cache, and — the part I’d still defend in any design review — Lua as the configuration and scripting surface. Virtual hosts, URL routing, rewrite logic: all Lua, so wiring Loomiere into an existing site was an afternoon, not a migration.
And that whiteboard sketch is exactly the machinery being born: each stream lived as a Lua coroutine — a blocking operation yielded it across the line to the C event loop and resumed it on readiness — with spinlocks at the few seams where worker threads had to touch. Thousands of streams juggled in parallel on one machine, in 2010, when Node.js was a year-old curiosity and the async-everything movement it would ignite hadn’t happened yet. It was, and I say this with the full bias of authorship, truly beautiful.
The throttle (the part worth stealing)
The economics lived in one mechanism: the VBR-aware bandwidth-conservation throttle. Serve at full speed only until the client holds N seconds of playback ahead of the play-head (default: 20), then pace delivery to the stream’s own bitrate. The subtlety is “seconds”: with variable-bitrate video a byte target is wrong at exactly the moments that matter, so the buffer was tracked in playback time, not bytes. Smooth playback, instant seeks, and — since most viewers don’t finish most videos — dramatically less bandwidth bought for content nobody watched.
On day one it cut peteava’s streaming bandwidth from roughly 1300 to 900 megabits while serving more clients.
The numbers
The run I still grin about was a single SSD-based machine in January 2011 — a 3 GHz Core i7, 12 GB RAM, six 256 GB SSDs, three gigabit NICs:
6500+ active concurrent streams, 2841 Mbps of traffic, with the SSDs under 20% load.
At that point we hit a wall that wasn’t Loomiere: somewhere between the I/O path and the sockets — interrupts, kernel, mainboard — the CPU saturated while memory and disks idled. \
My note from that night reads, in its entirety: “Hmm… this is a deep one.” I never did get to the bottom of it; the wall was comfortably above what any single machine needed to do.
In steady production the fleet told the real story: Loomiere 2.0 served all of peteava.ro’s video — about 800 TB every month, from 14 servers — and did so for over five years in total.
And none of it ran in a cloud, because there was no cloud worth the name: every one of those servers was physical iron that someone had to rack, cable, and keep alive (shout-out to Bogdan, who has since turned that craft into his own hosting business).
Even the best things end
Loomiere stood on libev, whose documentation contains a section titled — beautifully — “ev_cleanup — even the best things end”. Somewhere in those docs (I’ve never managed to find the exact passage again, but never managed to forget it either) its author left a thought along these lines: you always have the API to kill whatever needs killing — but there is something about a program written so well that, at end-time, all its threads close gracefully on their own… something close to perfect.
Loomiere delivered on that spectacularly: at one point it ran for three years without a restart — no leaks, and no creeping memory fragmentation either, which at that kind of uptime stops being an implementation detail and becomes a discipline.
And when asked to stop, it stopped the way the docs dreamed: every watcher, every worker, every connection closing in order, all the way down.
And the child in me will forever cherish that.
What happened
The category was absorbed, the way categories are. nginx shipped its mp4/flv modules and pseudo-streaming became a config flag on a server everyone already ran. Then adaptive streaming (HLS, DASH) and CDNs finished the job. Loomiere’s last release was December 2011 — right about when the ground finished moving.
Could I revive it? No. The industry has advanced enormously and I’d be fighting monsters — nginx, CDNs, twenty years of protocol evolution — over territory nobody disputes anymore. Reviving it would honor the code less than closing it well does.
What it left behind
Three things survived the product, all of them still in use:
The method: think long, write short. Not always possible — but I’ve been trying ever since, and Krbn, the pencil-rendering engine I released recently, was built exactly that way, fifteen years later.
The taste: performance as a design property, not an optimization pass; the conviction that software’s job is to get out of the hardware’s way.
And the lesson I’d offer anyone building infrastructure products: a product built on a protocol era ends with the era. Platforms absorb categories. If your product is a feature of the platform’s future self, your moat is time — spend it accordingly.
The repo is preserved at github.com/vpalos/loomiere, readable and honest about what it is. Start with options.lua — the whole design philosophy fits in its comments.
Rest well, old friend. You were fast.
Don't keep it to yourself!…