RIght2Water.eu is hosted on a dedicated server with linux+mysql+glassfish. Since about a week ago, we started to get an order of magnitude more traffic and signatures than we (and the other ECIs) had experienced perviously.
We are currently getting more than 30'000 signatures a day, with peaks of 3000 signatures by hour.
And unfortunately, the server doesn't handle the load very well. It crashed or became extermely slow a lot of time. We had to restart the appiication server at least 50 times. We know for a fact these downtime cost 1000 of signatures.
The team at the European Commission has been extremely supportive and we are working together to try identifying the botlenecks and remove them. We have probably improved a bit the situation, but as of today, we are still not ok.
If you plan to host the OCS software yourself, plan a lot of time to read all the best practice and tuning suggestions before the start. Obviously, you won't be able to know if it was able to make it more scalable. So plan that you might need a lot of system administrator time when your ECI becomes successful.
If you are an expert in glassfish tuning and monitoring, could you please contact me? I would love that me and my team go back to normal work hours ;)
It's too early to see if it the bottleneck is in the OCS software (at least 2 requests could be avoided per visits for instance) or the configuration of glassfish, probably both. I'll update this post when we have solved it.
Edit: so we have passed the million signatures and it does crash only every 10 days or so. We had a peak day of nearly 100'000 signatures in a single day.
The OCS team at the european commission did find the cause of the problem: the application was having a timeout for the session of 10 hours. We reverted back to the default (30 min) and now the system works under load.
I'm not sure a patch has been created yet, but if you host OCS, I'd suggest you do it manually (you need to change the config and re-deploy)