
A usually overlooked, yet fundamental part of designing high performing systems is capacity planning. I particularly didn't pay much attention to this topic in the past. I had a tendency to only think of it only at a very early stage of designing systems and thought it was only valuable for massive scale systems.

In fact, capacity planning is an incredibly informative exercise. It can help you understand the feasibility of a particular design and can also hint you better options. Approaching it as part of the design process in any considerable change – "considerable" defined as a function of how the change impacts relative scale – can be tremendously helpful.

That said, I thought I'd share some of the basics of capacity planning for system design.

### Clarify the design scope

If you're building a new system from scratch or introducing change, the usual process is understanding what the software is supposed to do. **Understanding [functional requirements](https://en.wikipedia.org/wiki/Functional_requirement), as what features are needed, and [non-functional requirements](https://en.wikipedia.org/wiki/Non-functional_requirement), as constraints the system should abide by** are a good place to start. An example of a functional requirement is adding video upload to a social media post/feed that only supports pictures. A non-functional requirement could be a need that these videos should work in 2g connection.

**Once you understand what the change entails, explore its characteristics.** For instance, observe read vs write ratios. Different systems will need different optimisations because of these characteristics. For example, a news feed will likely have more reads than writes and a wellness app that tracks your activity while exercising will probably be the other way around.

Another set of characteristics that will impact the overall capacity planning exercise are **understanding how non-functional requirements impact storage or bandwidth.** For instance, for a messaging app, choosing client over server persistence for messages will drastically impact storage investment. Nonetheless, it will also have its own set of challenges when syncing up data across devices.

At this stage, clarifying what matters will support defining the best trade-offs and potentially will simplify implementation.

### Estimate a few options

Once you know the scope, you are ready to move to some estimates. At this stage, you likely have a rough idea of which features to pick for estimation. **Your initial measure for quantifying capacity is usually an assumed throughput**, inferred from a similar feature or another number such as daily active users.

Firstly **come up with a ballpark estimate of how much a request costs in bytes.** Rounding up is a good idea to keep figures conservative. Estimates for text should be straightforward as you can assume a set of fields and their sizes in bytes. For object storage, such as images or videos, think of versioning, for instance, the average size of a thumbnail vs full-size, or video in 480p vs 1080p. Note how this interacts with non-functional requirements and which options you have available to fulfil the specification.

**Once you know how much each record costs, think of bandwidth per operation.** Think of how you can potentially reduce the amount of data sent both ways: client to server and server to client. Especially for reads, you usually have opportunities for savings. For instance, instead of fetching the entire record and all of its fields every time, you can likely send a fraction of the content from the server sometimes. Examples of that are truncating content or rendering versions. In short, not every operation will need the whole record so you can factor it and discount accordingly.

From there on it is just byte maths and [back of the envelope calculations](http://highscalability.com/blog/2011/1/26/google-pro-tip-use-back-of-the-envelope-calculations-to-choo.html). **With the average record size per operation you can get estimate measures** and understand trade-offs for different design options.

### Seek opportunities for optimisation

Next up, optimisation. **A very typical opportunity for storage optimisation is defining data retention.** Not all systems need – or even can, because of compliance – keep data forever. A typical opportunity is looking into data granularity, for instance, aggregating data by a dimension such as time or user. Especially at large scale, an appropriate data retention policy can drastically improve systems performance and save costs.

Another opportunity to reduce overhead is caching. **For read-heavy systems, usually, a small portion of the content gets a disproportionate amount of reads**, and caching can reduce bandwidth dramatically for such cases. There's a myriad of ways you can explore optimisation for reads such as lazily loading content based on what's on the viewport, making use of content delivery networks, incrementally improving video/audio quality over user engagement time, and so forth. These are very much dependent on non-functional requirements as well.

Similarly, you can **leverage caching for write-heavy systems as a way to reduce storage capacity needs**. For instance, in a time-series event-based architecture, omitting writes where there is no state change. Temperature-reading sensors which push updates on a fixed time interval to a server are an excellent example of that. Temperatures aren't prone to change drastically within a short timespan, so you can cache a sensor/temperature pair and only write to longer-term storage when a change happens.

In short, there are many ways to go about capacity optimisation. I believe two key aspects to keep in mind are how optimisation impacts relative capacity, and what you are optimising for.

### Be prepared to operate

Much of the truly challenging work for capacity planning, in my opinion, is in operating capacity. **Sometimes, assumptions we base off capacity planning at the design stage are untrue in practice**. That's why being prepared to operate and thus, continuously plan, is extremely important.

**A good practice for detecting misconceptions is writing some of the assumptions made at the design stage in the form of alerts**. Let's say at the design stage you assume the system will have 20% cache hit, and in practice, you have 5%. Even though you likely will have monitoring in place, the assumption made can slip off. Monitoring the assumptions made previously gives you the ability to act proactively instead of only knowing it on your next infra bill arrives or when an incident happens.

Even though you have ways to rely on data to forecast and know how load evolves, you can't predict every single scenario. To reduce this risk, what you can do on an ongoing basis is **using some concepts of [threat modeling](https://en.wikipedia.org/wiki/Threat_model) to understand which safeguards are missing**, what happens if a certain characteristic changes, or if a non-functional requirement changes. These help to indicate where you might have blind spots.

___

With this said, capacity planning is a deep and fascinating subject. It can inform design decisions, support systems operations, and even help you challenge your initial requirements. Having this as part of the design process can help increasing awareness of the trade-offs you make, and puts you in a stronger position to approach change.


Capacity planning is an underemphasized subject in system design. Approaching it as part of the design process can be tremendously helpful. Here are some of the basics.

System Design: Capacity Planning basics


If I had to pick only one interview method to assess engineers, it would likely be system design. That's not because coding, behavioural, role-play or case interviews are less valuable – they are revealing and help to build a bigger picture of the candidate. If you have resources to create a process that will support diversity, a fair framework to avoid bias, and the time, people and training to run a multi-stage process it will likely reduce the propensity of false positives. Nonetheless, if I were constrained to pick just one, I'd pick system design.

I find the mix between subjective and objective criteria that the system design interview imposes very useful for assessing candidates from multiple angles. There is a reasonably well-set frame for technical assessment, but just coming up with a solid design is not necessarily a pass. Similarly, candidate behaviour has a vital role to play, but the lack of causal nexus and technical-ability can reveal lack of fit to the position quite clearly.

There's a lot of resources out there which are an excellent foundation for the more objective parts. I curated a few of my favourite ones in the [appendix](#appendix) of this article. These deserve great focus when studying for an interview, as they are the most fundamental part of the assessment being made. However, having undergone some interviews recently and practicing it, I thought I'd write about a few points I have learned and that I believe can help on the subjective aspects of it, especially in regards to posture.


### Don't just communicate: collaborate with the interviewers

One of the things I find very helpful in technical interviews is turning it into collaboration rather than just an assessment. The role interviewers are trusted to play fundamentally is assessing whether a candidate is a good fit to be a colleague of theirs. I believe that when you achieve a collaborative atmosphere at the interview you are giving people a much better perspective to assess your fit fairly.

With that in mind, once the interviewer shares what the problem to be solved is, a technique which can help is **rephrasing the objective in your own words.** I find this helps to avoid situations where you presume understanding what is being asked. It is much better to clarify with the interviewer than design the right solution to the wrong problem. Ask questions as obvious as they can sound, and try to engage in gathering all the information you can before proceeding.

As the objective is better defined, it is essential to **establish what is in scope and what is out.** Narrowing down what are the scope boundaries for the interview problem will help you to focus on what matters the most for the design in question. Making compromises on scope early on will prevent you from spending the very little time you have on going deep in off-scope topics of the problem at hand.

When you infer information from what you are told, or when you reach to a conclusion **voice your assumptions and conclusions.** You need to bring your interviewers to see the problem from the angle that you are looking at to make a fair judgement if you are going off track and nudge you on the right direction if it needs to be.

Then, when working on the proposed solution, **avoid turning the experience into a monologue by leaving spaces for feedback**. Interviewers are usually fine to signal when you might be going off-track. Have them contribute to the discussion even if in a non-verbal way, regardless if it is about implications of the design decisions you are making and why, or if it is more specific such as whether you should split a component into two.

In summary, communication goes a very long way for technical interviews. Even further, actively seeking collaboration improves your chances of performing well in the interview, and also helps interviewers gauge whether they would like to work with you – and vice-versa.

### Go wide first and then deep afterwards

It is easy to go deep prematurely as design interview questions are unlikely to fit within the timespan available, conversely to a coding interview, for example, where the problem likely fits within the time available. Nobody will expect you to come up with the perfect solution for a system which is run by dozens, hundreds or even thousands of people – although ideally, you should have a rough sketch to solve the problem end-to-end.

As in real-life systems, the design will evolve. Done is usually better than perfect. **Start simple before going complex by favouring extensibility rather than perfection**. Put forward the most straightforward design which won't pose obvious scalability issues first, while keeping the interviewer informed of your strategy, and then as you cover the requirements, get into more in-depth optimisation.

A way I have found to frame the problem at the right level at the starting point is to **put yourself in the shoes of the user of the system you are about to design.** Basically, talk with the interviewer about the steps the user would take to use the product/system you are designing for the features in scope. Being mindful of the user's needs will help to clarify the characteristics of this system, for example, what are the capabilities which will need public-facing APIs, which components will have higher reads vs writes, or when to ebb on consistency over availability. Looking into the system from a user experience perspective usually hints what is fundamental and will help define a few whys further down the path.

Once you feel like you understood the whole picture, came up with a high-level design and have shown the interviewer you have a hold of the overall problem, pick the part which you are most comfortable with to go deep and explore it. **Focus on your strengths rather than your areas for development.** Most likely, the problem at hand has some exciting scale challenges for any discipline. Pick your favourite aspect of the problem that you can speak deeply of, regardless if it is client-side bandwidth optimisation, data model and partitioning, machine learning infra automation or whatever you want.

That said, resist the urge to get into the detail before you fully understand the problem. Attaining the conversation to motivations and expected outcomes first will help you make more assertive design decisions and make a few whys more evident to both you and the interviewer later.

### Talk about caveats and edge cases

What will happen if your system stops sending data to a downstream system? How do you notify upstream systems to rollback something which was mistakenly changed? How does the system work for the 99th percentile of your power users? These are all questions which can influence your design choices, so thinking of what could go wrong gives you either the opportunity to change it or to confirm that you made the right design decisions.

Especially in large scale distributed systems, you usually will have to optimise for specific characteristics. That inexorably introduces unwanted behaviours – which you need to be aware of to make the right decision. **Talk about trade-offs you've made and why** you chose an approach instead of all the other options you had.

Further, **exploring failure scenarios and recovery options** will show you understand the design that you have committed. This can get especially interesting when you can cover not only system failures but how users could abuse the system, which potential risks are introduced that you will have to accept, or even human error. When possible, talk about how you recover or prevent these from happening.

**Talk about how to make things right when things go wrong**. One example of this is a personal experience where I overestimated capacity in one interview – instead of going with 1GB storage for a free-tier product quota, I went with 1TB. I realised it later in the process, as storage and bandwidth started to feel absurd for the problem at hand with the volume the interviewer and I had committed to. After acknowledging something was wrong, I spoke about the steps I'd take to revert the mistake if this went live such as keeping users who signed up to the 1TB proposition with the benefits they paid for indefinitely, talking to teams responsible for advertising and intake about the mistake, and other damage control activities that could be done. I think this turned out to be an even more interesting conversation than it would be if I had calculated the right capacity at first.

In short, things will go wrong at some point – displaying readiness on walking the not-so-happy path will increase your confidence in your design choices, and if you made a wrong decision, you still have time to adapt and make it better or at least acknowledge it.

### Touch on adjacencies of the problem

Again, given the system design interview is extremely time-constrained, you need to be mindful of what you want to emphasise. Whilst it is great to have a narrow scope to work through, touching lightly on adjacent topics to the design itself can show even further breadth of knowledge. Here are some examples of subjects to cover, which might be off-scope but can increase the number of signals, thereby increasing your chances of performing well.

I believe most engineers appreciate that operating large scale systems is very hard. I personally like to **cover operations, not only design**. Especially at large companies, where you usually have dedicated infrastructure engineering functions, it is easy to take a lot of ops for granted. Briefly talking about subjects such as alerting, monitoring, tracing, observability can reveal problems on scaling a specific design.

Especially for highly regulated environments such as payments and abuse prevention, touching on **security, compliance & regulatory constraints** can be a plus. Some ideas of subjects to touch upon are data retention or data anonymisation for systems with personally identifiable information, or risk management by splitting components in different networks which are in scope for audits.

Maybe a subject more on the technical leadership or management side, design decisions will inherently dictate how teams will be mobilised for specific domains. A pretty large upside of microservices is making scaling independent – people and organisational design included. Talking about how the design will likely impact **how you mobilise folks in the set of teams owning the design** and showing awareness of scaling not only infra and systems but also organisations can spark engaging conversations.

My examples here are highly biased to my own experience, but making a short aside whilst you cover something within the scope of the interview can serve as an additional signal and increase the interviewer's confidence in you.

___

In closing, it is worth noting that all of these tips are based on my own experiences and preferences. Different organisations will focus on different things, and your best starting point is understanding the expectations of the company you are applying to. I believe incorporating the topics above to my posture when interviewing has helped me, and I hope it will help you too.

___


### Appendix

Below I've added a small appendix for some resources on the technical side of things. I hope you like it!

- [Grokking the System Design Interview](https://www.educative.io/courses/grokking-the-system-design-interview) course by [Design Gurus](https://designgurus.org) at educative.io;
- [Designing Data-Intensive Applications](https://dataintensive.net/) book by Martin Kleppmann;
- [System Design Primer](https://github.com/donnemartin/system-design-primer) by Donne Martin;
- [High Scalability](http://highscalability.com/all-time-favorites/) blog;
- [How the Web Works](https://github.com/vasanthk/how-web-works) by @vasanthk;
- [SuccessInTech's](https://www.youtube.com/c/SuccessinTech/videos) YouTube channel;
- [TechDummies'](https://www.youtube.com/c/TechDummiesNarendraL/videos) YouTube channel;
- [How we've scaled Dropbox](https://www.youtube.com/watch?v=PE4gwstWhmc) video;
- Amazon's [DynamoDB](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf) paper;
- Google's [MapReduce](https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf) paper;
- Google's [Hypertextual Web Search Engines](http://infolab.stanford.edu/~backrub/google.html) paper;
- Google's [GFS/Google File System](https://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf) paper;
- Facebook's [Cassandra](https://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf) paper;
- Facebook's [TAO, a social graph data store](https://www.usenix.org/system/files/conference/atc13/atc13-bronson.pdf) paper;


System Design interviews are my favourite interview type. Taking the technical part aside, such interviews have a large behavioural component. I wrote down some ideas on how to perform better in the more subjective bits.

Software Engineering

System Design: Capacity Planning basics

Posture in system design interviews