ffmpeg processes limit

I the Flash setting for the video module, there is a setting for how many processes are run.

Yet in SSH I see this,

results,

Personally, I always thought 'Maximum number of processing files' meant the number of video file you wish to allow your web server to process at one time before moving on to other video files needing to be processed????

IDK

I thought that setting would determined how many files would move from pending to processing but when checking in the back end, only one file would move from pending to processing.  The question is if ffmpeg can process multiple files simultaneously and if so, does it save on processing time if you have the resources?  If it takes the same amount of time to process one file, load the next, process, load the next, process, as it does to load three files and process then it really does not matter if we process one at a time or three at a time.

Well, no matter the flash setting, the system will try to process all that are sent to it.

I tested by uploading 20 videos, I had as many as 10 processes running. The server load got so high, most failed during processing.

Mind you this was my test server, with nothing else going..

Actually "Maximum number of processing files" setting means that it will process up to 2 files per one run.

Dolphin runs video processing every 5 minutes (via cron jobs).

The problem maybe with big video files, if they are processing longer than 5 minutes:

for example you uploaded 5 big video files and processing of one file takes approximately 15 minutes, and cron runs every 5 minutes:

- after 1st cron run you will 2 files in processing and 3 files are pending processing

- after 2nd cron run (after 5 minutes) you will have 4 files processing and 1 file pending processing

- after 3rd cron run (after 10 minutes) you will have 5 files processing and 0 files pending

- then after 15 minutes you will have 3 files processing

- after 20 minutes 1 file processing

- after 25 minutes all video files should be ready

To limit the server load you can set to run video processing every 30 minutes (in table sys_cron_jobs)

 That's what I will do then; thanks.

Would it be "smarter" for dolphin to not process another file until after there is a free processing slot open?
For example: (5 videos to process)

after 1st cron run you will have 2 files in processing and 3 files are pending processing
after 2nd cron run (after 5 minutes) you will have 4 files processing and 1 file pending processing 

• If 2 files are still processing, then it does nothing.
• If 1 files are still processing, then it begins processing another file
• If 0 files are processing, then it loads 2 more files to process 

Then reduce the cron jobs to check/run every minute.  This is so that users do not have to wait until half an hour later (or even longer) to see their videos, and can see them instantly after they are done processing, which should be fairly quick (on a properly configured server)
This should also reduce the load on a server because more than 2 videos would never be processing, and how ever many cores you have can tackle processing just only 2 files, instead of trying to process 5 (or potentially more) videos.  

 so what if the first two videos are long videos. With your suggestion, users will still have to wait a significant amount of time. At least how it is now, if the first two videos are long then other videos will still get processed and more persons are happy instead of having to wait. And God forbid your video is the last one uploaded. You may be waiting for hours depending on the size of the videos before yours. It is like going to the store and there are only 2 registers open with 20 people waiting. The way it is now is like opening 2 new registers every 5 minutes thus allowing everyone to get through faster

 I agree, it's not going to work this way.

I'm going to have to do something, I mean really, my site is busy with videos.

Big wake up for me, Dolphin is not meant for a "big, active site," apparently.

 That's a good point, I didn't think about that the first 2 videos could be huge files.  But with saying that, my logic was that:

If a 4 core server is being used, and lets say there are 5 (big) videos being uploaded then:
Those 4 cores would process 2 videos much quicker than having 4 cores processing 5 videos.  Even if it takes 10 minutes to process a (big) video, then the first 2 videos would be processed in approx 20 minutes, versus 5 videos that will take 50 minutes (or longer because more videos being processed will add to the wait time).
 

But if my theory is wrong (which Im putting on my "to-test" list) then what would probably be a better way of going about processing them? (outside of having a beefy server) 

 Oh no Newt, please dont begin crushing my dreams, lol

The number of threads set is suppose to help with speeding up processing; Dolphin by default uses one thread.  I tried setting more than one thread in a test but did not seem to improve the speed; could be that I did not set it properly.  I checked and Boonex appears to have compiled ffmpeg to use more than one thread; check host tools on this.

I found the following on a site discussion:

FWIW I ran some x.264/ffmpeg encoding tests on a Dual Xeon 5620 (16 HT). My input file was an 1080i m2ts video file (I can provide it to you if you would like).

threads 1, simulate single thread/cpu system, CPU peaked at 100% of 1,600%, encoded approx 6fps:
time ffmpeg -y -i test.mts -deinterlace -r 24 -acodec libfaac -ab 96k -ac 2 -ar 44100 -crf 24 -s 1920x1080 -vcodec libx264 -vpre normal -threads 1 -f mp4 test.mp4
3m20.501s

threads 4, modern quad core, CPU peaked at 400% of 1,600%, encoded approx 20fps:
time ffmpeg -y -i test.mts -deinterlace -r 24 -acodec libfaac -ab 96k -ac 2 -ar 44100 -crf 24 -s 1920x1080 -vcodec libx264 -vpre normal -threads 4 -f mp4 test.mp4
0m54.704s

threads 0, CPU peaked at 800% of 1,600%, encoded approx 30fps:
time ffmpeg -y -i test.mts -deinterlace -r 24 -acodec libfaac -ab 96k -ac 2 -ar 44100 -crf 24 -s 1920x1080 -vcodec libx264 -vpre normal -threads 0 -f mp4 test.mp4
0m35.129s

The Dual Xeon really blasts through the encodes. But even with threads 0 ffmpeg/x.264 is only making use of approx half the available CPU power on the machine, so in this situation with these settings I would suggest running 2-3 concurrent conversions on this machine at once for optimal CPU usage (keeping it maxed out as much as possible while converting as quickly as possible).

If you don't upgrade your server but find ways to speed things up do let us know :-). When the use of GFX cards / CUDA comes to ffmpeg/x.264/?web-M? then you will be able to simply drop a NVIDIA GFX card in your server for 10x faster encodes. There are some cool desktop apps that do this already.

The key for a site with a lot of video encoding might be to have a separate server to process the video files.  With a single server, we also have all the other things running such as the webserver and MySQL server so we can not max out the CPU load on our server from just processing video files.

I had 162 videos to upload - sizes ranged from 22 mbs to 1.4 gbs. As a whole, it took about  6 hours to get all videos uploaded and processed. Looking at it originally, I did not think this was too bad of a time frame. But looking at it from a user standpoint of someone uploading one or two videos, the conversion time does suck - but it only sucks because everyone is used to the quickness of youtube. Without comparing to youtube, I do not think the time is really that drastic - even for large files. Most with common sense will understand that large files take longer to upload and process, just like they take longer to download. The only thing I would like to suggest would be a circling wait icon or something so that users know that their files are uploading (referring to uploading videos within a module such as Groups).

I was hoping someone would come in and talk about the threads.  There is a old post on Boonex from someone that did increase the threads and saw an improvement.  When I tested it it did not seem to make a difference so perhaps I did not do it correctly.  Dolphin has changed since that old post.

We currently have an eight core processor; If increasing the threads means that our members can see an improvement on processing their videos, then that would be a good thing.

 This is my testing server above... not my "real" site.

I have a much better commercial server for the site I'm concerned about.

I can't imagine the required server to process lot's of members videos, even if they are not all doing it at once.

IF they were to all do it on a busy day, say the afternoon when my site is going, I'm screwed. 

 Im having same issue when too many people upload too many videos and they go in processing at same time it spikes processor and they fail. I uploaded server and got better but still an issue. I will be upgrading again soon but it seems there needs to be a way to limit how many videos are being processed at the same time or it will fail videos. And even then when site gets big enough seems this script is not going to work this way. What have you found out?

Hello freespeechsocial! 

One possible way is to limit the threads of Ffmpeg. How to do it:

1) open the file flash\modules\video\inc\functions.inc.php

2) find the following code like

function prepareCommand($sTemplate, $aOptions)
{
    foreach($aOptions as $sKey => $sValue)
        $sTemplate = str_replace("#" . $sKey . "#", $sValue, $sTemplate);
    return $sTemplate;
}

3) change the last line like:

return $sTemplate . ' -threads 1';

4) save changes

 I don't check these posts as of late.

Here is a novel idea; don't process videos through ffmpeg if not needed to be processed.

There was a module which was able to integrate to YouTube. You filled out the firms and began upload, the video was processed by YouTube, placed in your account, then the link and description and details populated the Boonex site as an embedded file. 

Users could setup albums,  it would show on the site profile and there was a ton more features

Video zone extreme

https://www.boonex.com/m/Youtubes_Video_Zone_Xtreme_D7