API design is Hard. I’ve made many good and bad decisions when designing APIs and I hope this blog post will guide people into designing better APIs. It’s a pretty meaty subject, so I’ve broken it down into two parts. This first part focuses on the actual design process for APIs. The second part focuses on the tools and techniques you can use when designing APIs.
What is an API?
I don’t mean a textbook definition, which Google defines as:
a set of functions and procedures that allow the creation of applications which access the features or data of an operating system, application, or other service.
I don’t find this particularly helpful, as it’s hard to map this statement to what I do every day. My hand-wavey definition would be that “An API is the way a developer interacts with your code.”
This is short and sweet but has a few key points:
- It is “The One True Way“. The API is the de jure way of accessing your code and logic. This is the way you intend for developers to use your code. This doesn’t always match reality. If the API is lacking or unclear, all too often practices like accessing private fields or functions or relying on undocumented side-effects creep in. People start to rely on this, and it ends up becoming the de facto way developers interact with your API. At this point, it is no longer your API, but a modified version of it which is often more practical to the problem domain.
- It is for developers. Ultimately, other people in the profession are your target audience. They may be web developers half way around the world in the case of REST APIs. They may be a work colleague in the case of internal company APIs. It may even be yourself several years later when you’ve forgotten what you’ve done. More generally, the person using your API has some knowledge of the underlying domain in question (e.g. it’s a REST API, so you’re using HTTP) but may not have knowledge of the specific domain that your API is in (e.g. knowledge of all the Facebook Graph entities/relationships).
- It is to interact with some code. This code may not be defined yet (in the case of specification writing), or it may be concrete. The important point to note is that APIs implies a concrete implementation somewhere. This has implications on the first two points. If you are not the one writing the concrete implementation, you need to be extremely thorough when specifying the interfaces in order to convey your intent correctly. Common pitfalls here are not specifying edge cases absolutely unambiguously (e.g. can this be null? Is 0 a valid value?)
What does the API do?
This is the single most important question when designing an API. What is your goal/objective/aim here? This should ideally be represented as use cases which are agnostic to the API itself. You want use cases to be clear and concise to a non-technical audience. This is not just for communication purposes. Specifying use cases in pseudo-code like terms immediately restricts how you think about the problem on a sub-conscious level and affects your ability to come up with potentially better designs. For example, a bad use case is something like “Get the account, add some money to it, store the account”. This immediately evokes 3 API calls (getAccount, addMoney, setAccount), whereas 1 may be the better solution (depositMoney). A better description would be to “deposit money into an existing account”. Use domain-specific words whenever you can, as it will likely affect the names of your objects/entities.
A particularly gnarly problem comes when trying to extract use cases from other people. People will often announce solutions or steps to perform to solve the problem rather than say what the problem really is. This is problematic because:
- It prevents you as a developer from seeing the whole problem. This can lead to bad designs because you simply weren’t aware of some additional constraint or couldn’t leverage a useful bit of information.
- The solutions presented may not be appropriate or well-thought out. Rarely is the first design the best design.
- You may miss important “behind the scenes” information such as assumptions made and trade-offs considered.
Always, always dig deeper. It is sometimes hard work to do this, but you can save yourself a lot of time by getting clear and concise use cases first, so you know what the aim of your API is. Now that you have your use cases, you need to start designing the API…
What you should think about when designing an API
A lot of the guidelines when designing APIs go something like: “Be Consistent”, “Do one thing and do it well” and “Don’t have surprising behaviour”. I couldn’t find a comprehensive list of tips which I should be thinking about when designing APIs. Then, I stumbled upon the 1996 paper “Usability Analysis of Visual Programming Environments: A ‘Cognitive Dimensions’ Framework” by TRG Green and M Petre. They focused on visual programming, but their cognitive dimensions apply uncannily well to API design (Source: wikipedia):
- Abstraction gradient : What are the minimum and maximum levels of abstraction exposed by the notation? Can details be encapsulated?
- Closeness of mapping : How closely does the notation correspond to the problem world?
- Consistency : After part of the notation has been learned, how much of the rest can be successfully guessed?
- Diffuseness / terseness : How many symbols or how much space does the notation require to produce a certain result or express a meaning?
- Error-proneness : To what extent does the notation influence the likelihood of the user making a mistake?
- Hard mental operations : How much hard mental processing lies at the notational level, rather than at the semantic level? Are there places where the user needs to resort to fingers or penciled annotation to keep track of what’s happening?
- Hidden dependencies : Are dependencies between entities in the notation visible or hidden? Is every dependency indicated in both directions? Does a change in one area of the notation lead to unexpected consequences?
- Juxtaposability : Can different parts of the notation be compared side-by-side at the same time?
- Premature commitment : Are there strong constraints on the order with which tasks must be accomplished? Are there decisions that must be made before all the necessary information is available? Can those decisions be reversed or corrected later?
- Progressive evaluation : How easy is it to evaluate and obtain feedback on an incomplete solution?
- Role-expressiveness : How obvious is the role of each component of the notation in the solution as a whole?
- Secondary notation and escape from formalism : Can the notation carry extra information by means not related to syntax, such as layout, color, or other cues?
- Viscosity : Are there in the notation any inherent barriers to change? How much effort is required to make a change to a program expressed in the notation? There are 3 types: “Knock-on”: a change in the code violates internal constraints in the program, whose resolution may violate further internal constraints. “Repetition”: a single action within the user’s conceptual model requires many, repetitive device actions. “Scope”: a change in the size of the input data set requires changes to the program structure itself.
- Visibility : How readily can required parts of the notation be identified, accessed and made visible?
To highlight some examples of what these dimensions mean:
- If you’re writing a pet store API, you would expect to see a Pet object and a Customer object (#2 – Closeness of mapping).
- An API with an addFoo function should have a removeFoo function (#3 – Consistency)
- To send instant messages to another user, a bad API may accept 2x User, a Route, a TargetDevice, MessageContents and MessageMetadata. This requires far too many moving parts in order to produce a result (#4 – Diffuseness / terseness). It’s worth noting that having too few objects is also bad, if it isn’t representative of the problem domain.
- APIs which rely on being called in a certain order (#5 – Error proneness, #9 – Premature commitment)
- Errors should be informative and clearly identify the problem (#10 – Progressive evaluation). Note that this clarity applies to the API and not to the implementation, e.g. returning HTTP 500 when there is an implementation error is appropriate.
- Tightly coupled components (#13 Viscosity (Knock-on)).
For a good talk on API design (with an emphasis on Java), I would recommend Joshua Bloch’s Google Tech Talk. It’s an hour long but has a lot of practical advice which I’m not going to (badly) rehash here in this blog post. Once you’ve distilled this information, you need to actually come up with a proposal. Don’t try to be perfect, and instead aim to keep it as simple as possible.
When you have a proposed design, you need to see if it is fit for purpose. You should quickly write up a prototype which demonstrates the proposed design. This is hard to do because there aren’t a lot of tools to do this currently (I’ll cover this in part two). The prototype need not be code. Worked examples on paper using the proposed design can work just as well.
The whole point of trying out the design is to be able to prove that the 14 dimensions outlined earlier are met satisfactorily. Often, there will not be a simple pass/fail to each dimension. A lot of these dimensions are subjective e.g. #5 – estimating the likelihood of the user making a mistake. You’ll probably find some flaws in the original design. There will be disagreements among people about which dimensions are important. That’s the point. Iterate on the design a few times, and when you (and your team) are happy with it, write up a prototype in code if you haven’t already.
At this point, it’s extremely tempting to say that the API is complete and act as if the API is set in stone. This is a mistake. The point of writing a prototype in code is not to “finalise” the API. It is to exercise parts of the API which people may not have thought about. You may find yourself calling the same functions in the same order over and over, which you never thought about before. You may find edge cases where you lack all the information you need to make a decision due to some race conditions. The prototype is serving as the ultimate test of your API. Modify it liberally. Don’t be tempted to set your API in stone yet. How long you remain in this process depends on the time constraints. When the API hasn’t changed for a while, it’s a good indication that the API has reached a natural equilibrium. This doesn’t necessarily mean it is the best solution, it just means it has reached a local minima. This is the ideal point to make the API public.
Future-proof your API
You should assume that the first public API will be replaced completely. This means designing in version numbers from day 1. You should consider how flexible the API is to modification. For example, it is a lot easier to add new JSON keys to a JSON object than it is to rename keys. That being said, try not to “organically grow” your API or else you can end up with some odd, confusing or surprising behaviour.
An example of an inflexible API is Android’s permissions model, in particular push notifications. The permission depends on a string constant which was originally set to
com.google.android.c2dm.permission.RECEIVE. This was named after Cloud-to-Device Messaging (C2DM) which was the API they presented at the time to send push notifications. Fast forward several years and this is the only reference to C2DM remaining. The API was superceded by Google Cloud Messaging (GCM) but because they baked-in the name of the API into the Android permission constant, they couldn’t ever update it. Now in practice, this doesn’t really matter because most people just copy and paste the string and couldn’t care less about what it means. However, anyone looking at the API can see this relic which developers are stuck with forevermore. Google had a choice to either leave the wart alone or break backwards-compatibility with apps which used the old string constant. They decided to leave it as it was, which I think was the right choice.
Always be thinking about extensibility when modifying the API. You may have heard the saying “It’s easier to add than it is to take away” used in reference to API design. This is absolutely true. You can always add more methods, more REST endpoints, more helper functions, but the moment you want to remove a REST endpoint or an obsolete interface function, things break and you start getting hate mail. Add things sparingly. This should be thought of as a stronger version of YAGNI, because the scope for things to go wrong is that much bigger. If you’re thinking you “probably” will need this in the future, don’t add it. You need to be certain; if it isn’t blocking progress then you don’t need it. It’s easy to add it in later and you don’t break other people’s code when you push a new version out. Rather than getting hate mail, you’ll be getting praise because you’ve added support for super-awesome-thing-#4.
Tools and techniques
Part two will outline some useful tools and techniques you can use when designing APIs. This will focus more on the actual “doing” part of API design rather than the ideals you should be shooting for.