If you are on shared it will be almost impossible to setup/install your own rms. It will require server access that you won't be able to access. Even if you could it would consume more power and resources than most shared hosts will allow.

Get a vps or dedicated box? Install your own? This can be very confusing and challenging for someone not used to working with the full root access environment of a server. There are many things that become available to you that you are not used to dealing with see more in a shared hosting setting. It can also potentially cause security issues if a vps or dedicated is not properly setup/configured, and maintained. These are not for everybody, and I wouldn't jump into one unless you have some pretty good shared hosting skills, and more important linux knowledge, and security familiarity at the server side. Expect to spend some time learning how it works. It's not something you learn over night.

3rd party rms. You can go with a 3rd party rms for $3-$10 per month range. This is very reasonable and would still be cheaper than your own vps or dedicated. Also you would not have to worry about maintaing a server, fixing a server, upgrading or any of the confusing linux stuff. You just enter the rms ip and ports into your ray base settings the 3rd party rms service gives you and your good to go.

Good Luck!!