Enterprise IT is awash in buzzwords: scalability, performance, responsiveness. If you worry about databases, you’ll have words like these etched on your eyeballs. Sometimes, though, they stop being words and become critical realities — not to be rolled out over months of planning and testing, but things that must work, if not this hour, then the next. And not just for your company, but for everyone who relies on you.
This is where Bingel, the online learning platform for primary education, found itself on Friday the 13th, March 2020, the day Belgium went into lockdown. Bingel, part of the leading pan-European Sanoma Learning group of educational technology companies, had been having a good few years until then. It had been the first to deploy — and was easily the leader in — interactive online learning for Belgian primary school children. It was expanding into neighboring territories. Its colorful, adaptive, thoroughly gamified experience of exercises proved very popular with children, parents and teachers alike. Growth was good — very good, with 1.5 million exercises completed daily.
Then came Friday the 13th. The Belgian government had looked at the data for the COVID-19 pandemic, and reached the only possible conclusion: shut down, go home, stay home. Suddenly, there were no classrooms to go to. For Belgium’s teachers, students and parents, they all went to the other familiar place they had come to trust and love over the past years, Bingel.
Becoming ‘critical infrastructure’
The company knew it had a role to play. Koen Rousseau, Digital Lead of Primary Education at Sanoma Learning, was leading the response. “Teachers had a million questions. We had to reach out and help. Over the weekend of the 14th and 15th, we opened up all our digital platforms, making everything freely available, reaching out to schools and saying, ‘We’re here, let’s do it together.’” By Sunday evening, there was nothing left to do but wait.
“Monday morning, 9 am, all hell broke loose,” says Rousseau. “We’d seen traffic skyrocketing since 8 am, but by 10 the first cracks started to appear. Parts of the system started to go down.”
One early sign came when his youngest daughter, who also had been using Bingel that morning, suddenly had trouble with the service and burst into the room yelling "Daddy, Bingel is broken, fix it! Rousseau knew he had to act quickly.
At first, the standard fallback procedure worked: “We switched it off and on again, and for a while it was running fine. Then at 1 pm it went down again.” Disk utilization reached saturation, with 12,000 queries per second and up to 96,000 connections. As the day progressed, the platform would crash three times. The magnitude of the problem — and the consequences — became clear. “I started to get calls from journalists. I once worked in the power industry, where I learned phrases like ‘critical infrastructure’ for things that, if you lose them, the nation’s in trouble. That afternoon, I realized that Bingel had become part of the nation’s critical infrastructure.” Rousseau needed a plan — and fast.
Fortunately, there was a plan. Not for this exactly, but the company had looked ahead to serving a lot more students. It just didn’t expect to have to do it overnight.
“We’d had a major crunch before, when we were much newer,” says Rousseau. “We’d started out with a very basic three-layer architecture, a front-end web server, some business logic and a plain relational database holding exercise resources.”
When the service was launched in 2011, there were low expectations. “We thought we were a bit too far into the future for our audience. It had so many new features, a digital currency to buy games, colorful islands you could leap between — it wasn’t like anything else.” But it turned out lots of children wanted the future to happen straight away. “Suddenly we had thousands upon thousands of children doing exercises at the same time, and the system couldn’t cope.”
#mongodb
