Loomiere/Stream – A high performance streaming server

UPDATE: http://vpalos.com/1165/loomiere-2-0-1-beta-finally-out/

The Loomiere (0.2.1) code is now freely available under GPLv3.
Please see this post for an update.

Are you killing psstream?

Well, yes! I am sure that many of you already know about psstream (the PHP streaming extension I made a while back). Well, many things happened since then and I came to realize I could do better; a whole lot better actually. As of now the ‘psstream’ project is officially no longer developed (see below). It will remain on the website for some time to come for archiving purposes but that is it.

But wait, why did you do it?

For some time now I have be looking into improving the streaming mechanism for a large video-sharing project run by my company. PSStream was a first effort, and it did the job but soon ran into problems. None of our servers was able to properly stream more than 150 clients simultaneously and the resources were grossly wasted, hence, Loomiere/Stream.

Care to share some caracteristics?

Sure. Unlike psstream (in which the moov parsing code from code-shop.com was used) Loomiere/Stream is written from scratch in C (and a bit of Bash) and uses a very minimalistic and lightweight approach. A lot of time was invested into optimizing this streamer; here are some delicious features I consider worth mentioning:

• Secure and configurable
  • Loomiere/Streamer is, in fact, a set of micro-streamers (mp4, flv etc.) that come together into a fast and secure HTTP server using the ipsvd and runit server tools which are very well optimized and stable. This means that I (and you) don’t have to go through the huge pain of making this server secure over time. It also means that the server’s security rules will be very flexible and easy to manage.
• Optimized to the extreme
  • Each micro-streamer is an extremely small program written in C/dietlibc (the mp4 streamer has at most 21k depending on system) and completely statically linked (no external libraries are linked when executed); this means that these processes can be started very quickly and the CPU is not strained by linking.
  • The micro-streamers have an extremely low memory footprint (mp4 has uses about 120k, never more);
  • Loomiere/Stream uses a “One process per request” approach, which is not usually very recommended on a web-server (due to the fact that it a process is usually heavier than a thread), but in the case of video streaming processes are better (IMO) than thread-pools. This is because each connection transfers a large amount of data and takes a very long time to complete, and would therefore fill-up the thread-pool rather quickly and keep it occupied for the duration of the streaming regardless of server load; here the process start-up time becomes irrelevant, and anyway the Loomiere/Strem micro-streamers startup extremely fast.
  • As a side-note, I think it would be beneficial to test a thread-based approach in the case of smooth-streaming, since in that case the requests would be short enough to make it feasible. Or maybe I could use unlimited threads to stream (i.e. use threads but not a thread pool), that would be worth trying out, I think since it is possible that less context switching will occur. I’ll have to do some tests. 🙂
• Saves bandwidth (throttling)
  • The streamer tries to send only enough data to the player so that it will always be up-to-date. This is, of course, configurable via a set of parameters.
• Versatile
  • Loomiere/Stream is able to serve MP4 and FLV files (flv, f4v, mp4, m4v, mov…) as well as image files (jpg, png, gif, bmp). Images are supported because you may have images to be served along with your movies on the same machine so you won’t have to install a second webserver to access them (however, see the recommendations further down).
  • The starting seek point can be given in bytes or seconds (only MP4 supports seconds yet, but FLV players always seek by byte position anyway).
  • The whole streaming system is very configurable via ‘/etc/loomiere/stream.conf’, and in real-time (you don’t have to restart the server for new settings to become active).
• Open source
  • Loomiere/Stream is released under the GPLv3 license with an important amendment: one can never use this software for streaming pornographic or sexual content of any kind! Aside from that, you’re clear :).

Did you test this anywhere?

The Loomiere/Stream server entered full production on Friday 29th of January, 2010 at 14:31 on all the content streaming servers of the peteava.ro website. Until now it has been running flawlessly.

A random check (while I’m writing this post) on one of the most strained servers yielded 653 simutaneous clients and a workload of: 8.81, 9.03, 22.47. The old streamer served at most 150 clients with extreme difficulty (the machine was almost frozen). The bandwidth usage dropped from the initial 1300MB/s to about 900 MB/s (approximate values), even though there were many more clients served in parallel.

Everything sounds good but do you think it could be better?

I know it can! Right now the server is in beta state! I can not guarantee anything at this moment (even unofficially) since I am still testing it myself under very heavy load. I will get back with more detailed statistics (and graphs), I promise!

Fantastic, where do I get it from?

ATTENTION: Due to the fact that me and my company are not yet satisfied with some issues regarding the legal terms of Loomiere/Stream, I think it is wise to take it (temporarily) offline until we resolve our concerns!

Giving any support?

I plan to provide commercial support for all Loomiere components (just the streamer for now), so yes! Soon I promise! How soon depends on how many people will use this server and request this specifically.

Until then, the support is basically open and free to anyone who wants to get involved. Practically, if you will have a problem with it and need my help, I’ll be here, I promise!

How do we use it?

The server: I don’t have a proper documentation written yet (that will be coming) but there is a “DETAILS” file in the source archive (above) that explains (in short) what you need to do to get it working. However you should have some knowledge about how ipsvd and runit work before tackling the streamer.

The clients: That’s easy, from any flash video player, you simply use the following url syntax:

http://yourserver.com/path/to/file.mp4 [ ?start=x.xxx [ &end=y.yyy ]]

Is there anything else?

I did not do any benchmarks against the Nginx and Lighttpd modules provided by the people at code-shop.com (they have renewed their modules as well). However, I hope my streamer will perform better considering that it was written with performance in mind, as a standalone server, thus not  being required to work with existing web-servers. Maybe someone could do some tests (maybe I’ll get around to it some day).

Well, for now, give me any feedback, bug reports or feature requests as a comment here.

Remember, streaming high-volume video is a very tricky business, there are many things involved! So if you have a single server that does other things beside the video streaming, I advise you to use your main web server as a front, redirecting streaming requests to the Loomiere/Stream server or simply stream your videos directly on a different port (like 81, or 8080) so that both servers can live on the same machine.

Although Loomiere/Stream is able to stream images, I only recommend you use it in case of large images and not for very small images that are heavily accessed (which is usually the case on the web). So if you can, use another web server for small images (with caching maybe?). I recommend: thttpd, nginx or fnord but any other (even Apache) would do just fine (depending on your requirements).

That’s it for now, I gotta sleep. Enjoy! 🙂