How To Use Amazon EC2 for offloading CPU intensive tasks
I’ve got some processes that I use to test historical data. I like to optimize them till they are running very fast (as can be expected)…. Normally this will suffice, but on this test I’ve got several variables to test — essentially I am looking at about 200 days of processing to get my results; but I don’t want to wait 200 days.
So.. the solution [a solution] in cases like this is to use something like EC2. There are other solutions, use a bunch of old laptops, or a ec2 competitor, but since all my code is in C# w/ MSSQL server, ec2 is my first choice.
Step 1 (since I am doing this on a small windows instance) the code must be converted to a service. This is so you can launch an instance and have your code start right up. A few things to watch out for: You can’t just throw a timer control on the service and expect it to work, you actually have to write one.
private Timer oTimer;
private int _interval = 1000;
private string currentTask = ""; // Lazy way of making sure I don't step on my own toes...
protected override void OnStart(string[] args)
{
TimerCallback timerDelegate = new TimerCallback(OnTimedEvent);
// create timer and attach our method delegate to it
oTimer = new Timer(timerDelegate, null, 1000, _interval);
}
private void OnTimedEvent(Object state)
{
if (currentTask == "")
{
currentTask = "Working";
//Do your processes
currentTask = "";
}
Don’t forget dependencies (in my case I had to add a dependency for MSSQL$SQLEXPRESS (so it won’t start will the database server is up and running) — I also added a test in the code to make sure I could connect just in case, since in one of my realworld tests I had some racing issues.
Ok… service is all fine and dandy. You have tested your EC2 instance (windows small w/ sql express is cheapest for this situation at $0.12 per hour). Make sure that you have it set to automatic, and that there are no weird issues on reboot to keep your service from starting how you want.
In this case I have them all communicating with a central Non EC2 database server to track which server is doing what tests and to store the results. This allows me to start 5, 20, 100, etc instances at once and to just let them run. [Actually amazon limits you to 20 instances, unless you get special approval -- which doesn't seem hard to get, I just haven't done it yet, since I don't know if I will actually need it]
Now that you have your serer all updated (windows update, and starting it starts up your service and the logging and tracking works, etc.. it is time to make this into an AMI (instance that you can use to start up as many servers as you want)
I left mine at the default size of 30 gigs, but if I were to do it over again, I would research how to lower the size to say 8 or 10 gigs (of the original hard disk) since I don’t need that much space, and I bet amazon is charging me for all that space even though I am not using it. Amazon ec2 has changed quite a bit over the past 12 months. I personally use a combination of elasticfox and the AWS web console to manage my instances. elastic fox doesn’t have a “start” option for a shut down instance, but AWS does, and there are a few other quirks…
Anyways, I would recommend shutting down the server first, then using AWS to make an EBS AMI Instance (right click on your instance). This is less hassle then doing it via S3 if you really havent bothered to setup S3 yet (like me).
This will take a while (30-60 minutes). I didn’t keep track, but I know it took more then 20 minutes. Once that is done, it is just a matter of starting 20 instances of your private AMI.
Then sit back and wait while the results come in.
