mySQL Server Shut, Possible Causes Things to Check

One evening I was doing a test on the site when it became very unresponsive with the page load spinning thingy just going round and round.  Then I got the red database connect error.  I quickly logged into the server and the mySQL server was shut.  I clicked the button to start it and it went through the motions but then reported back that it was not running.  After a few tries, I went to the command line and typed the command in to start mySQL server and it started that time.  It has been up since but I am uncertain as to what caused it to shut in the first place.  The sql log file shows nothing.  The server log seems to indicated that php-fpm may have crashed but both Niginx and PHP-FPM were running when I logged into the server.

So what could be the possible causes and what are some of the things I should check?  I am worried that I will get a repeat of this.  The server ram is 16 gigs.

No sql server guys here?

I don't think it is an issue with mysql itself, but since I am not familiar with NGINX, I can't even tell you where to begin. On my linux machine, my database usually has issues when my server does an automatic update. This has happened and caused my mysql to do something funky. But, I am using database encryption so that plays a big part in my database issues. Hopefully someone else who uses NGINX will have an answer for you. I may setup a computer specifically for NGINX and see what happens. I have an extra one around here I was gonna use as a local web server for my son who wants to create his own Dolphin site (yes, I am passing it down. Hopefully he can grasp the coding easier than I did).

I don't think it was either Nginx or PHP-FPM that caused the shutdown.  The logs don't indicate anything either.

Check if mysqld is set to start on system boot. It may be that your server was restarted for some reason.

Yes, that was one of the first things I thought; I did have it set to autostart on boot; along with nginx and php-fpm.

However, the system log showed the system had been up for over 256 days.  We had a system reboot to occur back then and the host had no answer for why our server was rebooted; so I thought maybe that had occurred again and that mysql had failed to start but it was not the case.

It's always nice to know what caused something.

But unless there is a reason sticking out like a sore thumb in the logs you will not find one.

Quite often looking for the cause of something is like looking for a needle in the proverbial haystack.

To find out why a airplane crashes or why a bridge collapses usually requires a huge team of specialists to figure it out. And who the hell knows if they are actually right.

You may have to blame gremlins.

Yes, I know you are correct in this.  The problem is that I worry it will happen again although the site had been up since that 256 days ago reboot until the sql server when down.  We are a small volunteer team and can not always monitor the site.  Perhaps there is a way we can get email alerts if one of the three main bits, nginx, mysql, or php-fpm shuts down.  

On my server, there is an app or service or something that constantly checks to make sure that everything is running. If something crashes, or closes for any reason, it will automatically restart it. I tested this by manually stopping my mysql server and it restarted it within 5 seconds after I stopped it. I will have to find the name of it so you can see if you can install it on your server, or ask your server company if there is a program that can do that for you.

Yes please, I am interested to learn more.  This server is unmanaged but I do have some basic tech support.

As Prof pointed out. There are programs to do it.

Way back when i used to own and run my own ISP and hosted servers as well i had scripts that would run on the servers that would page me when a service was not responding. I also had two servers that had only one job and that was to ping the other servers on a regular basis and page me when one of the servers went down. That was back in the day of numeric pagers.

It could still be done now. Most cell phone providers provide a email address you can send a message to that will send a text message to your phone. So yea. You could set anything up you need.

@GG - just called my server company and ran my ssh history - I Have two things going on that work hand in hand: chkservd and tailwatchd. So - from my understanding, there is a server cron that runs every minute, and runs the chkserv and tailwatchd scripts. These scripts check and make sure that all listed services are running, and if any are not, then start that service. I don't know if that will be reflected in the logs.

So, check with your server company and see if they have anything similar.

I guess if I dug deeper I could find that actual scripts themselves as well as the cron job. Let us know what your server company says. If they are confused, then I will look them up.

I use www.pingdom.com to notify me by text if my site goes down. That would not work in your case. My plesk notifies me If mysql goes down. 

just wondering, and I never really paid attention to it till now. When installing dolphin there is a setting for where to send bug emails. I have created an email address called bugs@mydomain.net specifically for this. Then set up email clients on my pc and all of my mobile devices, with the refresh rate of every 5 minutes. So, if there is an issue, I get a bug email. This should work for your issue GG because if the mysql goes down, there will be a database connect error, thus sending an email notification.

 what if php-fpm goes down? 

I have no clue about that. Will it cause any bugs within dolphin I guess would be the question. If it does, then a bug email should be sent. The only thing I can say is turn that off, and try to access your website, then see if you get a bug email.

Looking at my bug emails, I get them for a lot of different reasons, so it may send one. (crosses fingers)

 wait...what?

if php is turned off, it will not cause a bug cuz it will not run "At all"

see, I have no clue what pfp-fpm is, or what it does, so wouldn't know what happens.

php-fpm is a fast cgi for php.  You still have php installed, and still configure it through php.ini.  With Nginx, you will hand off the processing of php scripts to php-fpm; it is not a webserver module like how you run php with Apache.  If the php-fpm service goes down; then Dolphin won't run and the webserver will return an error code that the downstream service could not be reached or some such thing.  I don't want to turn off php-fpm to find out.