XML.php spiking CPU Resources

over the past week or so, i have been analyzing the dolphin script Dolphin 7 to be exact. There seems to be a problem with XML.php such as it uses an enormous amount of CPU resources. This is not on every account, but on select accounts and the odd thing is that in some instances, there is no activity on the site whatsoever.

I have multiple screen shots if you would like to see them over a five hour period of no activity on the site at all. Even with a fair amount of cron jobs firing at the same time, they are not using a fraction of the resources that are being used on a consistent basis by the XML.php.

also dont understand what the process is running on /usr/bin/php when that is not the path for crons to fire on. so why is this file running a command that we have no control over?

Regards,

DosDawg

Flash messenger is asking this file for updates every N seconds, so if user just leave the computer with some dolphin page opened it will be some load on the server - the load is not much - but because of numerous of request it can cause some load in the sum.

Regarding php process - please provide full command with some php file as parameter.

@AlexT

thank you for your response, and that part of your answer is extremely understandable. however, this is something more where its 'hammering' the CPU. Here are a few screenshots sir with the command that is being processed on the /usr/bin/php which is not even the path to my php binary to run the script.

The following image shows a CPU load of 52%

The following shows xml php hitting the CPU for 31% and 21%

if you require more information please let me know. these spikes are when there was no users on the site. now as to whether or not some user left a window open, then there should be some mechanism that will kill that window, because CPU spikes of this nature, from one user account is unacceptable.

Thank you for looking into this.

Regards,

DosDawg

Also ffmpeg is running from this file too. Video conversion takes a lot of resources - so you may have almost %100 server load when video is converting.

I have noticed that in some rare cases ffmpeg process is hanging for a long time with high CPU usage - then there is only way to kill it manually.

understandably that processing a video would use up resources. your reference to "a lot of resources" can you be just a little more specific? there are other video processing functionality from other sites, that do not use 'a lot of resources', so just wondering what the definition of 'a lot of resources' would be on your end?

have there been any stress tests done by boonex to find the capacity threshold based on any server configuration in regards to scalability? if one video is taking a CPU upwards of 100% then the application as a whole is not scalable, and could prove to be extremely problematic on a larger scale environment of 1 million users?

do you have any documentation as to how many users simultaneously can use Dolphin 7? because this is now a huge concern for me, that one video could use 100% of resources on a conversion, that is just not logical, and maybe we need to look at some other methodology of converting, in order to refrain from resource usage of such a great magnitude?

at any rate AlexT, thanks for looking into this, and please understand these screen shots were provided when there were no users online. as to whether or not some user or many users left their browser open, of course i wouldnt have that type of information. is there not something to trigger the session to end after x(secs) of inactivity?

so what i am gathering here is that if your site has visitors, and those visitors leave the browser open, this could spike your CPU, then if other users who are uploading video, lets use a nominal number 100, so 100 users on the site uploading 20mb videos, at what point will this server be rendered useless? because if you have that number of users uploading video that is going to spike out the CPU to 100% then this is a huge problem?

Regards,
DosDawg

Re-encoding an uploaded video file to three different formats is going to take a lot of CPU. If the XML.php spikes are from encoding videos, then its not really Dolphin7's fault. Ffmpeg is designed to use all available resources (limited to one core per process unless compiled for support for more).

Longer videos will make bigger an longer spikes. By default D7 will allow two different videos to be encoded at the same time.

Two things to try if your host is having CPU issues due to ffmpeg, tell D7 to only encode one video at a time and one way or another make D7 call ffmpeg using the unix 'nice' command. Using the nice command ffmpeg will still try to use all the CPU, but if other programs need CPU ffmpeg will be throttled to let them run.

interestingly enough on this one, and several of the posts on the Dolphin 7 forum, is that this site was monitored for 5 hours, there was no traffic, and like others posted in the forum, this has happened to them as well. I do not think this has anything to do with processing a video, i was bringing up the video portion because AlexT mentioned it. There was no activity on the site when the script was hammering the CPU.

no videos were being uploaded, there was nobody in chat. so that leaves the door open, what can this be?

Regards,

DosDawg

Re-encoding an uploaded video file to three different formats is going to take a lot of CPU.

,

again i inquire for a definition of "a lot of CPU"

Regards,

DosDawg

How much air is in a room? It depends, how big is the room, how much stuff is in the room, what altitude is the room at...

How much CPU ffmpeg needs depends on the file size and encoding of the video uploaded. Even the type of server and OS matters. The Intel iCore7 CPUs are more efficient at video encoding then a P4. Some Virtual Machine hosts ( VPS ) don't allow use of CPU extensions.

Might as well ask how much a rock weighs.

But your right, the issues you are describing is not likely to be ffmpeg. On my server I see ffmpeg itself show up in the process list when encoding videos.

your analogy is exactly what i am getting at. in using the term freely 'a lot' that does not present any methodology as to how to remedy the situation. might we well say it uses ' a rock ' to process video.

the statement is vague and unuseful.

Regards,

DosDawg

You are not reading all my posts, or read it partially, I have already told about this:

I have noticed that in some rare cases ffmpeg process is hanging for a long time with high CPU usage - then there is only way to kill it manually.

It is like in case of login problem to the trac - it was clearly described how to login to the trac, in blogs and forums, you just skip this - and still have the problem, because just of the ...

Regarding high CPU usage - if ffmpeg is compiled without multithreading support then it can take up to 100% of one of the processors - so other processors are free for other tasks.  If it supports multithreading it can use all processors - and can cause considerable load on the server.

AlexT,

you are not the only one posting in this thread, and for you to say that in some "rare" cases it uses 100% of the CPU is not factual. please go read the forum where there are many who are complaining that this file xml.php is using an enormous amount of resources. i did read that, you mention video conversion takes 'a lot of resources' i am asking that you define 'a lot of resources' .

as for your remark about the trac login you are a straight up ASS for referring to that post where i ignored your first statement. if you are so f*cking brilliant i will send you my login credentials and you go login to trac with them. the problem is not logging into trac, it accepts my username and password, then it says i dont have permission.

one more remark from you with regards to my intellegence, we are going to have some serious issues. fix the f*cking script so its not using 'a lot of resources' because that is not normal. get off your high horse and understand that you are not GOD, and your script is not the greatest. it has many issues, and i for one have been trying to help you get things sorted. if you are going to use vagueness in your responses, please save those for others who do not read or cannot read, I can read, and there is no definitive construct to the term 'a lot'.

again, i ask because you have eluded to this answer so you must be the one who is not reading.

/usr/bin/php why is that file xml.php firing on that path, when my php binary is not listed as such? where is this configured?

i understand that at some point ffmpeg.exe could hang up on a large video processing job, but that is not what was going on when i took those screen shots, guess you failed to read that one as well. there was 0 (zero) activity on that site, and that script xml.php kept firing and kept using in excess of 30% CPU. so even if i had killed it as you suggest, that was monitored for 5 hours, so it is your suggestion that i should have killed the process every time it fired.

yeah ok, fix the file, it has a problem

Regards,

DosDawg

AlexT,

ok i see you posted on here then just ignored the statement. so you noticed in some rare cases that ffmpeg spikes the CPU, i am not convinced this is ffmpeg, since i clearly stipulated that there was nobody on the site, thus what would ffmpeg be spiking anything for.

so you noticed on a few rare occasions, that means you have seen what number of times? because there are literally hundreds that have written and said this was happening.

the issue here is not does it happen, or on what frequency, and not everybody has the ability to as you state 'kill the process'. so what is the fix, and when can we expect this to be addressed in trac and be fixed?

Regards,

DosDawg

On some systems /usr/bin/php is a symbolic path to the actual php binary s, and appears in the system path. So maybe something is calling php from the system path...

I can confirm, this file has been spiking (albeit at a small amount) for myself recently. I can also confirm that this is without the usage of any Flash applications, and without any access to the web site (including the addition of a robots.txt file).

It would seem that this file likes to run during cron jobs, from what I've seen. Perhaps we should take a look there and see if it's firing it? I'm only suggesting, since it seems like AlexT has no idea himself how it's really running.

Edit: Hmm, I'm trying to see where it's called to run. I've only found a link to it in BxDolCmts.js.

I'm needing an answer also, to why my dolphin sites are spiking during cron. with the xml.php

dual quad core xeon's with 12 gig ram

Time:         Sun Mar 14 14:00:30 2010 -0600
Account:      *****
Resource:     Process Time
Exceeded:     74727 > 3600 (seconds)
Executable:   /usr/local/bin/php
Command Line: /usr/local/bin/php -q /home/******/public_html/periodic/cron.php

AlexT,

this still appears to be a 'rare' problem. this is not going away unless it is looked into and fixed. ignoring it will not fix it.

Regards,
DosDawg

Any resolution to this? The cron jobs are playing havoc with my server also. I have one D7 website which gets about 1 hit a week (don't laugh) and the cron jobs are running and spiking at roughly 29% of the CPU each day. I also have one D6 website which get anywhere between 100-200 hits daily which consumes around 0.2% of the CPU.

Boonex needs to respond to this and do something by it.  Anyone with full server logs will see that program is causing major server resource usage for no reason.  It is not an issue of converting files, it is running when we have our site in maintenance mode.  I have a feeling they won't resolve it because they simply can't resolve it and don't know why it is causing it, thus labeling it as "rare".  It is not rare and users, especially shared account users, are paying for it.

It is time to start a campaign to simply boonex coding and structure...

You are right, it is not rare. It happens "every" single day. My WHM logs show this every day for weeks. The D7 website(s), which have no users, are the top CPU usage processes of the each day. Sometimes the load is such that the server restarts. The website I speak of has had 27 visits (a 48.15% bounce rate) in the last month and 47 pages browsed from those visits  (according to google analytics). When I switch the crons off, calm then returns the the server.

Thanks Alex. I have applied the changes and I will check out the server performance over the next few days.

I can say i have seen this on 6 different hosts most of witch are on the hosting list here.  Most of the hosts say the same thing when i speak to them that the xml.php file is killing the server. 

At home i just built a quad core amd 965 running at 4.1 ghz and 8 gigs of ram and that file damn near brings the system down.  I am the only one on the machine and it is not on the internet in any way shape or form.  No videos being played no videos being converted. When i do convert a video it only takes up 2% of the total cpu power.

So something is way of with the way dolphin is calling that file. 

I have not applied the patch you mentioned above yet but, I sure hope you guys got this one under control becuase i must have a good 20 emails in my inbox from people asking me to fix this before they get kicked off from recommended hosts here on boonex.

EDIT: Also no third party apps are installed  i can show you atleast 10 dolphin installs on different server that are clean installs out of the box that get killed by that damn call to xml.php if need be.

So the fix is as followed:

Open   /inc/security.inc.php

And replace line 5 from   if (-1 != $iImpactLog || -1 != $iImpactBlock) {

To    if ((-1 != $iImpactLog || -1 != $iImpactBlock) && !defined('BX_DOL_CRON_EXECUTE')) {

Save the file

Open   /periodic/cron.php 

Remove  define('BX_DOL_CRON_EXECUTE', '1');  

from line 28 and move it to line 21 just under 

$GLOBALS['bx_profiler_disable'] = true;

Save the file.  

Now keep in mind this has nothing to do with xml.php and the way its called.

This has to do with  "PHPIDS can cause crontab to consuming more CPU resources that it needs, so disabling PHPIDS for cron execution can save some CPU resources"

According to the ticket from Alex.

Also keep in mind that phpids will be disabled be defualt in upcoming versions of dolphin from my understanding.

Please can you provide access to the server with the problem ?

Also I need to know situation when this problem occurs  - to be able to reproduce it.

I cannot believe how long this problem has been left unchecked. There still seems to be no real solution in place.

Has anyone found a way of fixing the spiking problem as it is becoming quite obvious that Boonex don't know what is really happening within their own script.

I just received an email from my agent pushing all the blame on my host as it seems that all the complaints are coming from unity members that only use that host.

What a load of rubbish is that. It seems that there are unity members using a variety of hosts that are having the same issue.

Boonex really need to start pointing the finger at themself. Once they do that then maybe they can finally solve this CPU problem.

I apologise for pissed off tone but hey......I am just that!

Bump. I keep seeing this one pop up. Any progress on this one guys and girls? I know there is another thread on this or was it a blog? Last I heard it was being looked into by others and Boonex but all has gone quiet. One this is for sure. It is a convenient way for webhosts to recommend people get dedicated hosting and then they can just forget about it. The issue may still be there but it is not affecting their other clients on the same server. Call me cynical, but it is happening. $$$$$$$$$$

You know too much. Expect a visit from several Kyrgyz over the weekend.

The sound you hear is that of me rapidly packing a suitcase. Destination unknown. :P

Well my site has now been suspended and only just after having upgraded to a VPS I am left with no choice but to upgrade again, but to a dedicated server. Lets just hope that by the time my site goes back up that I still have enough members left that will forgive the ridiculous amount of downtime and continue to stay on as paying members.

I would hope so. Thanks Boonex for all your help on this. NOT!

Is it true that you guys took the CPU spiking problem on a trip to Egypt and once you arrived there, appropriately went into denial? (I know its an oldie but its still a goodie)

Maybe its time to come back now before someone else is forced down a path of expenditure, way before they were due to.

As always, trying to stay positive and am hoping that after this change my site will perform a hell of a lot faster.

My auto install of D7 already had these changes. Howeve, ARVIXE keep suspending my account saying that it uses too much CPU and there might be a problem with SQL and CRONS usage. For example, applications being opened and not closed. I really am hitting my head a against a brick wall here. I have less members and less media than I did when I was running on D6. This is shocking and I don't know what to do. ANY HELP or ADVICE would be appreciated.

Try to change the following settings:

1. Change the following line in inc\classes\BxDolMemberMenu.php from

var $iBubblesUpdateTime  = 6000; // in milliseconds;to
var $iBubblesUpdateTime  = 30000; // in milliseconds;

2. Change the following setting:

Admin -> Modules -> Simple messenger -> Simple messenger update time (in milliseconds) : 8000

3. And the following:

Admin -> modules -> flash apps -> messenger -> settings -> Update interval (sec) : 7

LyubovL suggested I try this but with:

var $iBubblesUpdateTime  = 20000; // in milliseconds;

I have tried but a week later and ARVIXE are still saying I'm using too much CPU / SQL despite having low traffic on my site. PLEASE HELP! ARGHHH!

Still having this problem,  server spiking up to 98% and then crashing server.

 Anyone is having this issue ?  Who else ?

 BONNEX, THERE IS A FIX YET?

Boonex DO NOT KNOW how to resolve this problem.

I pleaded for a solution but there was none so I upgraded to a VPS.

The problem continued I searched and searched but still found nothing that could resolve the issue. Boonex still DID NOT KNOW. So I then had to upgrade to a dedicated server.

Now a couple of months have passed and am starting to have overload issues once more. The lack of understanding by Boonex of the absolute urgency of this issue is way beyond ridiculous.

We need to force their hand by recreating the problem on Boonex.us by arranging a large group of people to use simple messenger and the chat room at the same time.

Why else do you think they have not made these services available on boonex.com? Then they would have to face the fact that there IS A PROBLEM!!!

This issue is very hard to debug, not too much people provided necessary access details to investigate this problem, also it is very difficult to check the site during the load, because it is hard to predict when it is most loaded.

If someone experienced this issue, please provide your site access(root access is necessary) via PM - to investigate it.

Also another speed related issue was fixed: http://www.boonex.com/trac/dolphin/ticket/2077

Ok, we have the ticket number, so where's the fix?

http://www.boonex.com/trac/dolphin/changeset/14149

Thanks Magnus. Why couldn't AlexT put that address there to begin with???

You have a point. But it doesn't change anything, since you can find all your needed information by including the ticket number in a search of Trac.

"You have a point. But it doesn't change anything, since you can find all your needed information by including the ticket number in a search of Trac."

Good to know, but still seems a bit silly not to put the actual destination rather than the ticket number.

To be honest, it's also a bit silly to be arguing this. You have a point, though, and I agree. Before I realized that you could just use search, I always complained about them not adding the revision number to the ticket after they're done.

Are there any recent updates to this? I just got the "shut down" notice from host today due to "excessive use of server resources" by flash/XML.php

Maybe your server setup is not efficient, please refer to this doc to optimize your server:

http://www.boonex.com/trac/dolphin/wiki/HostingServerSetupRecommendations