Loomiere/Stream: Revived!

29 comments

1. Hello to All,
   I tried to use loomiere streamer on a debian system.
   But I run into some trouble.

   If player or the browser itself try to get video from url via streamer, download startet, but nothing happends – No plugin plays the video or browser start playing it directly. It is possibile to save it to disk and open it via VLC.
    
   If I doit without loomiere streamer (direct acces to file from webspace) every thing works fine, but without pseudostreaming.

   Do I have an error in my configuration?

   Greetings from Germany

   1. Well, it’s hard to say what’s wrong. Does the logs show anything relevant? 

      tail -f /var/log/loomiere/current

      1. Hi Valeriu,

         loomiere log say:
         2011-08-17_14:33:24.12235 CLIENT: xxx.xxx.xxx.xxx:22500,REQUEST: /test.mp4

2. Hey there,

   thank you for the great work. I’m trying your server in a Webhosting Environment in Apache with Debian.

   Here are some things for the documentation that would help me:

   – on debian there’s no need to compile again
   – make sure you installed all the prerequirements before you start the step-by-step guide in DETAILS:
   apt-get install ipsvd runit realpath
   (recheck if i forgot something here)
   – in the stream.conf the port parameter is essential for the url to your file. For example: when your video files are in the folder /var/local/www/project/base/files/videos
   change in stream.conf SECURITY_SANDBOX according to this path. Your URL should be something like that (given that you set BIND_POT to 81)
   http://adnsnametoyourserver:81/somevideo.mp4
   So that ipsvd is routing the request and not your apache (which may be running on port 80)

   for the others that might be some trivia :), hope that helps. If you need more documentation look in the bash scripts in the /usr/lib/loomiere/stream/service folder. You can add more logs there and have a look in:
   /var/log/loomiere/stream/current
   for the output of the log file.

   Best regards
   Philipp

   1. Hi Philipp, thank you for the input. Unfortunately I am swamped in work an can barely take the time to write this message. However, the good news is that I am working on releasing Loomiere 2.0 to the public (which is used in a production environment for about 6 months). Hope I will not be too late with the release.
      Cheers!

3. Hi,

   this is a great piece of software, thank you!

   I had issues with seeking in mp4 files until I found out that it considered the unit to be bytes instead of seconds. I found no place where seconds would be defined for mp4, so I patched it myself.

   I added the variable STREAMUNITS to the config file. It goes along with the STREAMERS variable and defines the units used for the different streamers.

   Here you go:

   — sample-stream.conf-orig 2011-02-23 12:20:59.000000000 +0100
   +++ sample-stream.conf 2011-02-25 14:42:46.000000000 +0100
   @@ -68,3 +68,7 @@
   STREAMERS[0]=”mp4:mp4,m4v,m4a,mov,3gp,3g2″
   STREAMERS[1]=”flv:flv,f4v,f4a”
   STREAMERS[2]=”image:jpg,jpeg,png,gif,bmp”
   +
   +STREAMUNITS[0]=”seconds”
   +STREAMUNITS[1]=”bytes”
   +STREAMUNITS[2]=”bytes”

   — server-orig 2010-01-30 08:10:01.000000000 +0100
   +++ server 2011-02-23 13:14:18.000000000 +0100
   @@ -64,6 +64,9 @@
   line=”${STREAMERS[$i]}”
   [[ $line =~ (:|,)$type(,|$) ]] || continue
   streamer=”${line%%:*}”
   + if [[ ${STREAMUNITS[$i]} != ” ]]; then
   + arg_units=${STREAMUNITS[$i]}
   + fi
   break
   done
   fi

   Regards

4. How do you prevent hotlinking? how do the rewriter plugins work

   1. Hey Dave, I’m sorry for the scarce documentation. I am now working on releasing Loomiere 2.0 and it will have much better documentation, I promise.

5. Thanks for this it’s great. Got it setup on a centos server without too much hassle 🙂

   I’ve tried streaming to an iPhone through loomiere but get: Cannot Play Movie The server is not configured correctly.

   Is there a way to play to the iPhone through loomiere or should I just serve them up the old way (over apache)?

   Merry Christmas!

6. @valeriup: thx! that sounds great… I’ll give it a try 😉

7. Does anyone used it on Debian ?

   1. Yes of course, we use it in production on at least 7 streaming servers. Any Debian-based distribution is recommended for Loomiere, since Debian comes with pre-built binary packages for all the needed tools to make it run: runit, ipsvd etc.

8. Hei Valeriu,

   Anything about compiling on BSD ?

9. Sounds like one would need a dedicated server for Lua to be able to configure it.

   Can you email me?

10. How is this project coming?

    Will it allow offsetting the starting point of an h.264 video based on when the client logs on?
    …in effect emulating live streaming…

    thanks.

    1. The project is functional and in full testing (testing it started about 2 weeks ago). The results so far are awesome, the server essentially guaranteeing that the bottleneck is always in hardware (i.e. it ensures at all times that it will stream the maximum amount of data that can be physically moved by the hardware: HDDs, CPUs, RAIDs). For example, on a server with 2xCPU, 4x1TB HDD’s, 4GB RAM and about 400 simultaneous clients Loomiere 2.0 used about 2.6% of the available CPU (all threads combined). While, this is still very vague, it does show the level of performance that I aimed for. Tonight we will launch tests on a 48TB server (24x2TB HDDs, 2xCPU 3.0 GHz, 16 GB RAM). We’ll see how that goes.

       About the live streaming emulation, yes, it can be done easily from the config file (since the server is configured via Lua scripts), although this is not available out-of-the box. Could you tell me in a bit more detail how would you like to use such a feature?

       1. Hi Valeiup, I begine following your story since you started developing PS Streaming PHP, so Congratullations for all you have run in this 2 years…

          I have some unexpert questions, if you are so kind to help me with them.

          1.- Can I use Loomiere Streamer in a normal Web Page…??
          The point is that I don´t have access to a server, but just to a regular web page where you can´t install programs or compile things on the server.

          On the other hand, I am not an Linux server person… I am just a regular PC user that has a web page oriented to present text and videos, in spanish language, that have to do with the expansion of human conciousness and potential (www.vidaplena.net)

          By the way, the reason I am trying to move from FLV files to MP4, is that I publish the contence with enoght video quality so they can be displayed with video projectors, and be used by any body that wishes to do movie phorums or small discussion goups…

          In order to get a decent display quality, I encode videos with a 480xHHH size, and a bit rate of 450Kbps for video and 64Kbps for audio. That gives you 514Kbps total bitrate that is not achievable in my country (Venezuela) unless you play the movies between 1 Am and 7 Am… (not a good time for a meetings… jajajaja). So my only and great option was trying to move to MP4, where I can encode video at 250Kbps with a video quality much better than the one in FLV at 450Kbps.

          But the down side of that, is that because the videos are 40 to 90 min long, they “hang” in the middle of the playing, and seeking seems to be the only way to get them out of the “hang” condition.

          I find you, looking for a solution like Xmoov.php, witch I used in my web to stream FLV files… The neet thing about PHP scripts is that you can just upload them to the web page and they work fine (though I understand all the unefficient resource usage you found when you decided to drop PS Streaming PHP).

          So, againg my question is: can I use Loomiere Streamer in my web page, just uploading the server binary code, or something similar that can be done from my Web user side…??

          2.- If the answer to point 1 is yes, how can I get a compiled version of Loomiere Streamer, for Apache servers (the ones used by my hosting provider)…??

          3.- If the answer to point 2 is yes, and I can get that compiled, uploadable and web runable Apache version of Loomiere, then: what html code do I have to include in my html files to use Loomiere…?? The good thing is that I am using JWPlayer witch as far as I understand is one of the most widely use web players. I think the I am using JW Vers. 5.2

          4.- If the answere to Point 1 is NO, then, I ask you: is there another simple streamer that can be use by a plane web owner (like me), in a similar fashions as xmoov.php…??

          Thanks a lot in advance, for reading my replay and for any kind of help you can give me on this issues…

          1. Hi jgespinosa, I’m afraid I don’t have good news for you.

             In short the answer is that unless you have root SSH access to your hosting server, there is nothing you can do to stream directly from your website’s server. The issue here is that you need to *seek* into MP4 files, which is in fact the tricky bit (it is one of the reasons that you would need Loomiere in the first place). This seeking implies a complex algorithm that is (yet) unavailable in pure PHP.

             One solution would be to host your movies on a video sharing website (e.g. http://youtube.com, http://peteava.ro etc.).

             Another solution would be to simply stream your MP4 files without seeking but as I understand from what you are saying is that this way, your movies hang after sometime. I suspect this is a configuration issue of the hosting web-server and you should be able to fix it by talking to your hosting provider.

             The ideal solution for you would be to have a streaming server written in pure PHP, but I know of no such thing.

             Hope this helps…

             1. Hi Valeiu and Thanks a lot for your reply…
                For all I have read, I understand the seeking in MP4 requires a lot more procesing power than the one required to do so in FLV files… so I understand that writing that in PHP will never be an eficient solution, just as you discover with your first PSStreaming.

                It seems that my best solution will be to move from a web hosting to something like a Virtual Private Server (VPS)… With that, I will have access to a root directory and will be able to install the progarms that I need as well as the web files needed.

                I have seen some options like this one
                http://www.serverpoint.com/en/plans.phtml
                and it seems very aforable…
                The basic plan (about 9$ a month, signing for 1 year) is just 2$ more expensive than my actual host… So the trade off is very good.

                And they give me a lot more bandwidth and storage space than the one I need today… According to my hosting provider, today my average transfer usage is about 35GBs/month and visitors about 1000/month, and web space 25GBs…
                And Server point´s basic plan gives me 3,500 GBs of transfer, and 350 GBs of storage… So I will be have 10 times more the storage, and 100 times more data transfer, than the one I need today.

                However, I have never done something like Server Admin, and I know nothing about Linux. But as far as I understand, my plan is not to do fancy things with the server… Just a regular web hostin for http://www.vidaplena.net, and the streaming of the MP4 video files…

                I understand, that the provider Tech Support, can do things for me, or help me to do them in case I am not familiar with some specific tasks.

                Do you think it will be a good idea to take this jump…??

                Are the Server Admin task very complicated, considering that I will only use it for a regular web hosting like the one you see in http://www.vidaplena.net today, and that the only “extrange application” will be Loomiere Streaming Server…??

                If you can give me your opinion about the VPS prodider I mentioned above, or you can suggest me some other you consider more suitable for me, I will greatly appreciate your suggestions…

                Thanks a lot for your help and expert advice…
                Javier Espinosa

             2. Hi Javier.

                Yes! Seeking into MP4 is CPU-intensive and is not a good idea to implement using PHP.

                If you are not familiar with Linux, I think it would *not* be a good idea to move your website to a virtual server, it’s simply not feasible (even with tech support).

                However, it might be a good idea to move *only* the video streaming (i.e. Loomiere) to a VPS. That way you keep your regular web hosting as it always was, and just deal with the streaming problem. So, on the new machine, you only need to copy your movies and run Loomiere. Done! It does mean that you will pay for both services: regular web hosting *and* VPS. The choice is up to you although I think this is the best option (for now at least, until you learn some Linux 🙂 ).

                There might be another option you should consider. It is a bit unorthodox and usually not recommended but in your case it might make sense. I’m not sure if you know this but let me explain:

                FLV and MP4 are both “video container formats”. They are simply file formats that contain the encoded video data. The video data inside an FLV file (which is an old and depreciated container format) is compressed using the codec H263 (which is also old and not very performant), while the video data inside an MP4 is compressed using the new codec, H264, which makes movies look better and get smaller. So what you actually benefit from switching to MP4 files is the H264 codec that is (usually) associated with the MP4 container format.

                Further more, MP4 is a far more complicated container format than FLV, and it is because of *that* (and not because the H264 codec) that the seeking problem got complicated.

                However, there is *nothing* that stops you from encoding a movie using the H264 codec and putting it into an FLV container, thus benefiting from the high quality and smaller size of the movie but still being able to seek the old way.

                As I said, this is not recommended but it *will* work! It all depends on how your website will grow. If you have a small set of movies that doesn’t change much over time then you should consider this option.

                Encoding movies with H264 and FLV can be done using mencoder/ffmpeg, or maybe even VLC.

                Valeriu

             3. Wuuuaaaaooooo Valeriu…!!!!
                You don´t know how happy you have made me feel…!!!

                Thanks a lot for all that wonderfull explanations…
                I had some of the pieces, but now I have the puzzle¨s box, and I can
                understand better what I am assembling here… Jajajajajajaa

                As you said, my video contence does not change much with time… I am just
                simpli adding movies to my site, and they remain there, because people keeps
                finding the site and keep using it as a reference point to “study” all that
                wonderfull informations and sharing it with friends and “soul family”, as I
                usually refer to them…

                I figure that H264 was the real deal of the MP4 encoding, as well as I have
                found that Sorenson Spark is the real limitations for the standard FLV
                files, and that VP6 gives a little better video performance (Quality/Bit
                Rate) than Sorenson… I even found some encoding tools that I use to test
                FLV/H264 video encoding, but the results were quite bad…

                During those FLV/H264 encoding test, I could not manage to get better
                quality picture with file sizes smaller that the ones I create with
                Sorenson… For example, a Test video 5 min long, encoded at 480×270 and
                450Kbps, was 35MBs using FLV/H264 wich was the same I got from
                FLV/Sorenson… And the quality was not very diferent either… so, after
                about 1 day doing test encoding with diferent tools, I concluded (aparently
                wrong) that FLV/H264 was not an efficient combination… Unfortunatelly I
                never try with ffmpeg tool.

                But now, with all your help, I know that road is the best one for me… So
                back to the testing lab I go… Jajajajaja
                I am loading ans starting to get familiar with ffmpeg and WinFF Gui, and
                something inside me tell me that´s the right way to go…

                I again thanks you with all my hart for your help in clarifying me…

                I will get back to you with the results I get, but in the mean time or
                later, if there is something I can do for you, just let me know…

                Your thankfull virtual friend in Venezuela…
                Javier

11. I will try to install a BSD machine on which to attempt compilation, fix the bugs and make it work. There should be no reason why this should not work under BSD (except some small bugs).

12. I’m so looking forward to it.
    P.S.: I don’t see the bash script returning Content-type http attribute in response header.. (it can be changed to “play” a little with http headers , for caching purposes and so on.. but I guess this is up to us 🙂 ).

    Unfortunately *BSD is not a solution. diet won’t compile in any way. Not even with gmake.

13. @Andrei: the actual ‘server’ script and the binaries (i.e. mp4 and flv) will run on their own just fine, but runit and ipsvd were used to turn everything into an actively running service; of course, if you found another way, that’s also great!

    About wether it compiles on BSD systems I have to say I don’t really know; it should, but some tinkering may be needed. Also, the server relies heavily on the sendfile() system call which also exists on BSD systems but is implemented differently than on Linux, so by default it will fall back to a read/write emulation of sendfile() (which is slower). A proper wrapper for the BSD sendfile() should be implemented in stream.h in order for this to work on BSD.

    As a side-note, as we speak we are testing the new version of the Loomiere streamer, which is completely rewritten and greatly superior in every conceivable way. I will write a post about this next week.

14. I made it run without installing runit and ipsvd. with dietlibc installed, the source should be able to compile on *BSD also, right ?

15. Hi,
    I am a linux newbie. I have a VPS running centos 5.3 with apache. I would like to install loomiere but I am having trouble finding prerequisites:

    dietlibc bash 3+, realpath, runit, ipsvd

    Also I’m not sure about installation paths.
    Is there a step by step guide for installation on CENTOS 5.3?
    The truth is I’m in over my head and I need some help.
    Sorry to be a pain
    Frank.

    1. I’m afraid this can be a bit tricky if you’re not familiar with Linux, but I’ll try to help even though I don’t have the time to write a full-blown how-to. 🙂

       Bash should already be installed and should work just fine. Dietlibc is the library used to compile the streamer itself and it is statically compiled (i.e. it is included in the compiled binaries: mp4 and flv) so you don’t really need it at all.

       RUNIT and IPSVD are very important and you can find CentOS packages for both i386 and x86_64 architectures in the annvix repo.

       The tricky bit is the REALPATH utility for which there is no CentOS binary package (as far as I could tell). However you can simply replace realpath with the built-in command ‘readlink’ like so: in the ‘stream/service’ file change line 74 from
       path=`realpath -s "$sandbox/$path"`
       to
       path=`readlink -f "$sandbox/$path"`
       and it should work just fine.

       As for installation paths, everything should work as instructed in the DETAILS file (from the loomiere tarball) with two notable exceptions specific to CentOS: the ‘/etc/service/…’ path is changed to ‘/service’ and the ‘/usr/bin/sv’ executable is relocated at ‘/sbin/sv’.

       I admit that I have not tested this setup, but I see no reason why it should not work. If you can’t switch to Debian, this configuration should serve you well. Let me know how it goes and if you need more help. 🙂

       P.S.: Laurie managed to do it using Xinetd on Fedora, for which there are no existing RPM packages for RUNIT or IPSVD (so it’s even more complicated).

16. You’re very welcome! 🙂

17. Awesome!

    Time to re-do my psstream implementation 😛

    Thanks for the heads up on my blog too 🙂

Comments are closed.