With the Digg versus Thoof commercials posted to the digg front page, it seemed like the Thoof servers have difficulty with the load. But in one hour, the servers seem like they are up, running and humming.
I’m wondering if this was a load test they were expecting. It would be interesting to read a blog entry on the effects of being dugg.
We had some trouble, initially because we have a queue of events being written to the database, and it was filling up. We’ve temporarily solved that one by dropping events when the queue is full (the consequences of this are not that serious), but this is only an interim solution.
We are currently experiencing memory issues, possibly a leak. We are still debugging that one using a heap profiler on a live webnode (but only directing a tenth of the normal traffic to it).
Another update, for anyone that is interested!
Ok, so the memory issue was basically that we were allocating too much memory to the JVM heap, and not leaving enough memory for other Java-related requirements. We’ve now allocating 2 of 8GB to the JVM heap on our webnodes, and this seems to be working well.
Another issue is an expensive database call we were doing on new-user creation. We took some quick measures to remove this (thus temporarily reducing the effectiveness of our recommendation algorithm), and are now working to move this workload into the webnodes through caching of the necessary data.
A JVM heap of 2 GB is the biggest that is effective, at least on sun JVM’s. Bigger than that and your GC pauses are taking too long.
It would be nice to read about how you tune the JVM to handle a load, for instance what GC you chose, and more importantly why, what the sizes are for the different generations.
A traffic analysis of the digg onslaught would be nice too. I guess you were fortunate that there wasn’t a direct link to thoof.com in the main article, so that people had to type the link in manually.
Other interesting tidbits are:
– number of parallel sessions per vm and its memory usage
– requests per second
But I understand if you are too busy with tweaking a system. I suspect you are monitoring the jconsole and look at the heartbeat of thoof. I know I did when I was monitoring our own application: looking at response times, number of concurrent connections to database, number of garbage collections, all interesting stuff if you want to know the status of your application.