It would be nice to read about how you tune the JVM to handle a load, for instance what GC you chose, and more importantly why, what the sizes are for the different generations.

A traffic analysis of the digg onslaught would be nice too. I guess you were fortunate that there wasn’t a direct link to thoof.com in the main article, so that people had to type the link in manually.

Other interesting tidbits are:
– number of parallel sessions per vm and its memory usage
– requests per second

But I understand if you are too busy with tweaking a system. I suspect you are monitoring the jconsole and look at the heartbeat of thoof. I know I did when I was monitoring our own application: looking at response times, number of concurrent connections to database, number of garbage collections, all interesting stuff if you want to know the status of your application.

Ok, so the memory issue was basically that we were allocating too much memory to the JVM heap, and not leaving enough memory for other Java-related requirements. We’ve now allocating 2 of 8GB to the JVM heap on our webnodes, and this seems to be working well.

Another issue is an expensive database call we were doing on new-user creation. We took some quick measures to remove this (thus temporarily reducing the effectiveness of our recommendation algorithm), and are now working to move this workload into the webnodes through caching of the necessary data.

We are currently experiencing memory issues, possibly a leak. We are still debugging that one using a heap profiler on a live webnode (but only directing a tenth of the normal traffic to it).