Tens of thousands of captains showed up, and my database buckled

I need to say the grateful part first, because it is genuinely the most important part. I did not build this expecting a crowd. One person, evenings and weekends, a database on a plan that costs nothing. When I shipped it I told myself that if a few hundred people tried it I would be thrilled. Tens of thousands of unique people used it in the first month, and I still do not entirely believe that sentence when I read it back.

So the fact that the servers were struggling was, in a strange way, wonderful news. Nothing was on fire because I had done something clever. Everything was straining because real people, in real numbers, were actually using the thing I made. That is the dream. It just arrived faster than my infrastructure was ready for, and it caught me completely off guard.

The trouble is that being flattered does not keep the lights on. Some of those people were getting locked out, and they had done nothing wrong. They just showed up at a busy moment. I owed them a fix, not a thank-you note, so I went looking for what was actually wrong.

The symptom was maddening precisely because it was inconsistent. On average the site was not slow at all. If you had asked me for the typical response time I would have shown you a perfectly healthy number and shrugged. The pain only appeared at the peaks. During a spike, the database CPU would climb to nearly 100 percent, and in that window some users would hit a timeout and get bounced out while others sailed through without noticing anything.

That intermittency is exactly what makes this kind of problem hard to diagnose as one person. There was no single broken page I could open and watch fail. The failure lived in the load, not in the code path, so I could not reliably reproduce it by clicking around myself. It only showed up when enough people arrived at once, which of course is the moment you least want to be experimenting.

So I stopped trying to feel my way to the answer and started measuring where the database was actually spending its time. When I did that, the culprit was not subtle at all. One thing was eating the machine alive.

Here is the part I have to own. The single most expensive thing in the entire system was one query that recomputed a captain's whole match history from scratch, on every single lobby load, and then again after every match finished. Every time you opened your lobby, the database went and re-added up your entire past, from the beginning, as if it had never done it before. That one query was roughly two-thirds of all the work my database was doing.

Because it was different for every user, I could not lean on a cache to save me. Each person needed their own fresh recomputation, so the cost grew directly with the number of people playing. And the ranked PvP stats had the exact same flaw hiding inside them: scanning up to a thousand match rows and crunching the numbers in code, per user, on every lobby load. Lower volume had kept it quiet, right up until volume was no longer low.

Writing that out is a little painful, because it is such a basic mistake. But it is the honest answer, and I would rather tell you the plain truth than dress it up. The database was doing an enormous amount of pointless work, over and over, and I simply had not noticed while the crowd was small.

The fix is almost boring once you see the problem, which is usually how these things go. Instead of recomputing your totals every time you read them, I now maintain those totals as matches come in. A database trigger updates a small totals table every single time a match is inserted, doing a tiny bit of work once, at the moment the new data arrives. When your lobby loads, it no longer computes anything. It just reads an already-finished number and hands it to you instantly. I applied the same treatment to the ranked stats.

The lesson underneath it is one I will not forget, and it is simple enough that you do not need to be an engineer to feel why it is true. Do the work when the data changes, not every time someone looks at it. A match finishes a few times per player. A lobby gets opened constantly. Paying the cost on the rare event (the write) instead of the common one (the read) is the whole game. Never compute on read what you can maintain on write.

There were smaller supporting fixes too, each closing a leak I had not seen. I added a missing database index for the global draft-intel query, which had been scanning the latest matches with nothing to sort on. I extended a shared cache on an ambient trend panel from 60 seconds to 15 minutes, since that data barely changes. And I wrapped the auth check in a 1.5 second timeout, so that when the database is slow it now degrades gracefully instead of hard-locking people out.

I want to be plain about my limits here. I am a software engineer by trade, but keeping a live service healthy at this scale is its own discipline, and I am learning the operations side of it in real time, on a free tier, with no team behind me. I got real things wrong, and those wrong things inconvenienced real people at the exact moment they were excited to try what I had made. That stings more than any graph.

So to everyone who hit a timeout during those spikes, who refreshed and got nothing, who wondered if the site was broken: I am sorry. You were not doing anything wrong. My database was, and it was my job to see it sooner. Thank you for your patience while I dug it out, and thank you even more for showing up in numbers I never dared to expect. I will keep learning this in the open, and I will keep telling you honestly when I get it wrong.