Best practices for peak traffic

Here in Australia, the end of financial year rush during the last two weeks of June and the holiday season in the month of December are the two main peak traffic periods which many charitable and NFP organisations experience. In addition, there are other specific peak traffic periods which can be attributed to various holidays, events, campaigns and public sentiment. 

The largest known enemy of all – simultaneous prime-time television, often attracts an onslaught of heavy traffic.

There are many misconceptions that simply selecting a quality hosting provider such as Azure or AWS will somehow guarantee automatic scaling and load readiness. Without careful consideration and preparation, this is rarely the case - as is often witnessed with even the largest of Government or e-Commerce sites crashing under unexpected load.

At Luminary, we develop tailored strategies for each of our clients to ensure they can support peak traffic inline with their specific needs and anticipated load. Below we have outlined 12 of our most common practices that we could broadly recommend for any client seeking to seize the opportunity attributed to those big moments.

#1 Identify peak traffic periods and set out a plan

Looking into traffic analytics from previous years will help you set expectations for this year. With COVID-19, the usual peak traffic period for Christmas 2020 maybe a little different to other years. Due to delivery delays, you may see traffic peaks in November rather than December this year. 

Once those peak traffic periods have been identified, plan for content freezes and code freezes on your heaviest traffic days. Make sure to put this in a calendar and share with all your stakeholders and partners.

#2 Make your site load faster on mobile devices

In 2020, websites we build for clients use mobile-first UI frameworks and work well across the different screen sizes. But you need to probe even deeper around latency, performance and speed on mobile devices. Personalised pages may help with conversions, but they could be slower than pages with static content and images. With Google's mobile-first indexing policy, make sure that your mobile site is fast and SEO-friendly. A case in point is that on Black Friday 2019, as much as 64% of all payment transactions were initiated through a mobile device [source: statista].

Optimising your mobile website and your mobile checkout flow is a task which can be done in conjunction with your agency long before you hit peak traffic. Google's PageSpeed Insights is a good starting point for this.

#3 Thwart DDoS attacks, bots and account takeovers

Is your site ready to thwart a Distributed Denial of Service (DDoS) attack? If not, make sure your website has Cloud-based DDoS Mitigation in place. Having protection against form spamming is also crucial during peak traffic as that could slow you down. Implementing a captcha solution to distinguish between bots and humans is one possible solution. Malicious actors will also use bots to take over accounts by brute force. Peak traffic periods are the perfect opportunity for cybercriminals to take over legitimate customer accounts and make illegal purchases. In the case of NFP organisations, cybercriminals use bot attacks for card testing and alternative refund scams.

Luminary recommends using a service such as Cloudflare for boosting site security, preventing DDoS and increasing performance. 

#4 Secure PII data

A person’s name, date of birth, driver's licence number, address, email, phone number, and credit card details will generally constitute Personally Identifiable Information (PII). Increased traffic during peak times also allows malicious actors to go undetected as they try to steal PII for nefarious purposes. 

Collecting the minimal amount of PII from your customers, encrypting data at rest and during transit, and reviewing OWASP recommendations are the easiest forms of security checks to apply. In addition, you should run regular penetration tests against your production environment and have automated security checks in your CI/CD (Continuous Integration/Continuous Delivery) pipelines.

#5 Set up monitoring and alerts

Monitoring your website's uptime and page speed are the minimal Synthetic Monitoring checks you need to have in place. Using third party services such as Pingdom or UptimeRobot allows you to set up monitoring and alerts within a few minutes. Going further, Real User Monitoring (RUM) allows you to monitor and rectify issues on your production environment in near real-time. In addition, monitoring the health of your infrastructure is critical as it allows you to take proactive measures before your customers notice anything is amiss.

E-commerce and not-for-profit (NFP) websites can also implement monitoring based on number of transactions and/or transaction value per hour (or a different duration) to ensure that the online store, checkout flow and payment gateway are working as expected. Monitoring of orders within your e-commerce platform or transactions on your payment provider or even in your Donor Management Portal or CRM are possible options which might require some custom development.

All monitors and alerts should be reviewed periodically and especially before a known peak traffic period. You should make sure that alert email addresses, mobile phone numbers, and Slack channels are up to date and that all notifications get through without ending up as spam. Where possible, test these monitors and alerts against a non-production environment.

#6 Update Run books and contact information

What should you do in case you get an alert? Who should you contact? What is your escalation path? What should you do in case of a known issue? All of these questions and more should be answered and documented in a run book which can be accessed securely by your support engineers. A run book also enables the onboarding of new support staff easily and makes sure that no knowledge is lost.

Having up-to-date contact information allows for escalation of support requests to reach the right people quickly. As with monitoring, run books should be reviewed periodically and especially before a known peak traffic period. 

#7 Load test

We do not recommend load testing on a production site when you have peak traffic. Rather, you should load test your pre-production site in advance to have metrics to compare with and to know how much your current infrastructure can handle. Load test results should be recorded and used to guide Synthetic and Real User Monitoring. The load testing results can also be used to tweak back-end code, caching, front-end performance and even database indexes.

Load testing should be carried out regularly and especially whenever new features are introduced to your website.

#8 Scale-up infrastructure on-demand

Maybe a decade ago, you needed to budget for and provision your infrastructure almost a year ahead. But with cloud-based infrastructure, provisioning your infrastructure happens at the touch of a button. Luminary works predominantly with Microsoft Azure and we advise that you don't over-provision or under-provision your infrastructure. The load testing results we spoke about earlier should guide you in hitting the sweet spot when provisioning your infrastructure for your daily traffic.

During peak traffic, rules can be put in place to automatically scale-up or scale-out your infrastructure depending on usage metrics. As easy as it sounds, there are gotchas such as file locking, database concurrency, session handling and stale data which can impact user experiences. So it's always good to test this out beforehand to make sure you are ready for peak traffic. Once your peak traffic period is over, those same rules can scale-down your infrastructure to handle normal daily traffic.

#9 Tune your CDN

A Content Delivery Network (CDN) allows you to offload and cache your web assets on edge nodes which are closer to your customers. This frees up your core infrastructure and speeds up the website load time for your customer. During a peak traffic period, getting a CDN to work harder than your own core infrastructure is the way to go. 

Tuning your CDN's cache duration, checking configuration rules and enabling tiered distribution should be done prior to your next peak traffic period.  

#10 Review third-party services and apps

We have seen a rogue Google Tag Manager (GTM) script change bring an entire site down. Seems impossible and initially, that's what we thought too. Third-party inclusions such as GTM, Facebook Pixel tags and others could make or break the experience for your customers. Your uptime and infrastructure monitors are green. But your customers are unable to make a purchase or a donation. This could be simple as a third party not serving a web-font or a required Javascript file.

At Luminary, we are advocates for micro-service architecture and using third-party services and apps are a part of that strategy. Having a register of third-party services used on your website is a start. From there, you need to plan how to mitigate issues if any of these services are unavailable or slow. It could be as easy as deferring a script file loading in the front-end.

Oh! Another piece of advice – treat any GTM changes as a production deployment and give your agency a heads-up if they were not involved in that change. 😜

#11 Plan for contingency and disaster recovery

You have followed all of the steps above, planned well, done all the proactive actions you can think of. In an ideal world, you should be set for worry-free peak traffic. But the unexpected could happen. Your cloud service provider could go down or one service in a particular region could be unresponsive. An urgent code fix could have unexpected knock-on consequences.

Contingency planning is key to this. Start from a backup policy in place for your databases and assets. In case of a disaster, having infrastructure as code with tooling such as Terraform will support in spinning up brand new infrastructure in a matter of a few minutes. Fully automated CI/CD pipelines will allow deploying code to any new infrastructure as well. Cloud providers such as Microsoft Azure and Amazon AWS allow for geo-replication which allows for your website to function even if a data centre ceases to exist. Decide how much down-time you can tolerate and work your way from there when planning for contingency and disaster recovery. At Luminary, we are happy to assist you in planning this to suit your requirements and budget.

#12 Retrospect – what can we do better next time?

Planning for peak traffic and the actual execution of that plan can differ in many ways. With our Agile ways of working at Luminary, it makes sense to run a retrospective for everyone involved to reflect on the past to improve the future. This should not be a purely technical discussion and should involve the key stakeholders for your website from across all departments. We recommend setting up a retrospective at least twice a year in January and July to reflect on how peak traffic was handled and what could be done better next time.  

We hope our twelve most common practices above set you up to win the next time you have a significant surge in website traffic.