Another Black Friday, Another Record Broken – Thanks, ScaleArc!
Black Friday is the Super Bowl for eCommerce. Every Thanksgiving weekend, while our families are eating leftovers and watching football, our whole Ops team is huddled in a back room fighting fires to keep our websites up under the load of 10x traffic.
Craig Thayer is vice president of technology for Sazze where he is responsible for engineering and operations. Sazze is the parent company to a number of fast-growing eCommerce sites, including DealsPlus, BlackFriday.fm, Whym, Pocketly, and Progrids.
This year was different.With our staff on call and our eyes on every monitoring system, nothing happened.Not one fire or outage this Black Friday.Zero. We actually spent Thanksgiving with our families. That’s amazing. And that was with this year’s Black Friday traffic being 2.5x over last year. That’s a spike of millions. We serve 400k users on a typical day – we supported 4 million users on Black Friday.
So what’s responsible for our shift from huddling with laptops to huddling with family? Classically, our source of meltdown has always been the database – it’s a bottleneck. We can apply webscale functionality to every other part of our infrastructure but the database. We did some cool stuff with clusters and Galera and super fast hardware, but the key to easy scaling and new levels of uptime has been ScaleArc. ScaleArc’s software sits between our applications and our MySQL clusters. t helps us scale out, distribute the database load, and endures failovers without our website ever feeling it.
This Black Friday, our 4 million users ran 3 billion queries – that’s10x our norm as well – and ScaleArc cached 70% of them. That’s over 2 billion queries that did NOT hit our database. That gave us a huge benefit for performance and customer experience.
Crucial to allowing us to utilize the max capacity of the database cluster without recoding the app was ScaleArc’s read/write split and caching for prepared statements – and we took full advantage. ScaleArc automatically identifies reads vs. writes and it distributed our reads across our secondary databases, offloading the primary. This combined with ScaleArc’s alacrity to update its software in time to support our deployment kept our website performance blazingly fast. Even though we use prepared statements a bit differently, ScaleArc delivered exactly what we needed on that front.
To give ourselves an extra challenge in addition to 10x traffic this year, we decided to completely rewrite BlackFriday.fm, our award-winning Black Friday shopping website and mobile app. It’s our biggest traffic driver and two days before Black Friday we started to see performance issues in the run up to all the good deals. We used ScaleArc’s analytics to look at the queries and it instantly showed us that, under load, the new app had a constant write queue even though the website requests were nearly all reads. The ScaleArc analytics pinpointed the offending write query. It turned out that the developers wrote a query where every time the site displayed an ad, it wrote to the database to track the impression. This tracking feature wasn’t a problem under typical traffic loads, but it became serious under heavy holiday traffic. Armed with the information from ScaleArc’s analytics, the developers were able to offload the tracking process and batch up the writes, netting a 10x reduction in these database write queries. Without ScaleArc showing us that a write queue was developing, we would’ve had a very difficult time finding and fixing the problem before Black Friday – and if left unfixed, those 10x traffic loads would have melted that app.
This wasn’t our first Black Friday rodeo here at Sazze. We strive to see what’s coming, see where things are starting to break, and address them before they do. Our team worked for two weeks straight on about four hours of sleep a night – including weekends – to prepare for Black Friday, the weekend, and Cyber Monday. Just like the teams in all those Thanksgiving football games we used to miss, our staff understands you just can’t simulate the game. You practice, prepare, and play – hustle and sweat. You win or you don’t. This year we won big. And we thank ScaleArc for helping our team with that win. ScaleArc enabled us to overcome the weakest link in scalable infrastructure and bring webscale functionality to the database.
For more, watch our video on Sazze's tips on prepping for Black Friday.
comments powered by Disqus