Creating Solid APIs

Speaker 1: APIs are not just used by humans, did you know that? You need to make sure that there's authentication, versioning, documentation, and a proper response structure for all the robots. So Riva 's going to tell us all about that.

Speaker 2: Thank you and good afternoon. I am Riva and this is my talk on how to create solid APIs. Our applications are increasingly being used, not by humans, but by other applications, via their APIs. APIs are eating the world, they say, but ironically, APIs themselves are first always used by humans, by the other developers who are integrating with your application. And this means that your APIs must target not just machines, but even more importantly, humans as well. I come from Thorgate, a product development agency based in Estonia. We're working on various applications, mostly using Python, Django, and React, and

Speaker 2: various other technologies. This talk was inspired by a project where API had top priority from day one. The focus was on creating a platform for managing forestry-related data. And other developers had to be able to interface with it, fetch the data, do various operations on it And the UI itself also had to be built on top of the publicly available API provided by us Now, as most developers, I had used and even created APIs before, but this project had higher demands, and so it got me thinking What uh does it take to uh create APIs that other developers would love to use?

Speaker 2: And this talk aims to share what I found. So let's begin with the definition of API. Usually it's defined as application programming interface. But a better definition might be application programmer interface. Because when you think about it then API is basically a user interface for other developers. And that brings us to the question, how do you make that user interface really good one? Quickly covering what I'm going to talk more in depth about, I think that good API should have documentation that is not just out there

Speaker 2: but also helpful. I think that it's very important to use standards to bring familiarity to your users and get them started faster. And you should make sure that the user has to deal with as little issues as possible, meaning lack of friction. This talk uh focuses on the web APIs, but much of uh the topic is also applicable to uh packages and libraries and code bases in general as well. So let's begin with the documentation. It's too often overlooked. We don't really want to do this because it's not the fun part, at least when we're developing. But when you're trying to make sense of something created by others

Speaker 2: or even by yourself a while ago, then documentation becomes much more important. Documentation is also often the first point of contact that people have with your API. So That means that they could decide whether to uh stick with your API or go looking go on looking for alternatives based on just that. I do the same myself all the time when I'm looking for a solution to some common problem, and there are usually a variety of packages available And then the first impression of the README as well as the documentation is quite important deciding factor And documentation does take effort, and I admit that I'm not that good at doing them myself either, but that does not mean that we shouldn't try.

Speaker 2: And if you only took just single thing from my talk, then I guess then I think that it should be that documentation is really worth putting some effort into. So now that we know that documentation is sort of like sales pages for your API or packages , let's think about how to make those sales pages really awesome. When I first start looking at some API or API documentation What do I want there to be? What should go in this documentation? One of my first questions is how do I even access it? Uh

Speaker 2: do I need to sign up for some developer account before I can start uh poking at it? Uh what is the root URL of the API? If your API has browsable interface and does not require any authentication, then I can just take this root URL, put it into the browser and start looking around immediately. I also want to know what the authentication options or requirements are. Do I need to use something like GoAuth to have some custom token authentication? And then there are some generic sort of mundane stuff that often goes overlooked, like character formats or encodings.

Speaker 2: In Python world we're quite used to UTF-8 everywhere, but uh when you have external developers interfacing with your code, then they might have quite different uh options. or experiences about uh the character encodings and so you should always make it explicit in your documentation The same goes for stuff like timestamp formats. ISO 8601 is quite commonly used, but again, it's better made explicit in the documentation. I also want to know how stuff like pagination or versioning works. And I want to know about the common errors that I might encounter. This means that if I do get an error, I can immediately come back to your documentation and for example

Speaker 2: find out that I'm sending the authentication token in an in an invalid way. And instead of spending my time trying to work out the solution myself, I immediately have your documentation as the main anchor point of sorts. And perhaps most importantly, include some code that I can immediately copy and paste and get started as soon as possible. Because even if it's just copy-pasted code, if it's uh running and works, then it gives this sort of warm and fuzzy feeling to your users and they are much more likely to keep using your API. You probably also have different endpoints for different resources in your API.

Speaker 2: So what do I want to know about each of those? First of all, the URL of the endpoint. I also want to know what operations can be done with that resource. For example, you might have listing and perhaps a detail view. And perhaps I can also update the data of the objects. For each of those, I also want to know what the request and response data looks like. So this is this is especially important for uh more complicated operations like update, uh where I have to potentially specify lots of data. Which can be in some quite complicated format. And I want to immediately

Speaker 2: see the structure of that format There might also be some optional parameters available. For example the list view might be sortable and I want to know how I can specify what to sort it by. Or if we're talking about filtering, then you might have uh different filtering options available, but with some constraints. And it always pays to list those out There might also be some permissions involved. For example, as an administrator, I might be able to change all the objects, but less privileged users can only view them. W or perhaps update

Speaker 2: uh only a partial set of the attributes. The only thing that's worse than lack of documentation though is when that documentation is outdated. So how do you keep it fresh? The answer for me is in auto generation. I think that your documentation should always be automatically generated based on the code itself, which sort of uh implicitly means that the two are always in sync. And usually the approach that I'm using at least is that you start with the code itself Then based on the code you generate schema, which is basically machine readable documentation for your API

Speaker 2: Listing all the same endpoints as well as their potential parameters and so on and so forth. And then based on that schema, you can further generate the user user-oriented documentation. There are different standards available for those. The most common perhaps are OpenAPI and Swagger, the front end, that gives you nice interactive documentation. But I would say that most importantly you should just figure out what your tools support. For example, if you're using Django REST framework, then there are a variety of packages available which either provide swagger integration or a different form of auto-generated and or interactive documentation

Speaker 2: And it's nice if your documentation also provides those same uh code examples next to the text. so that if I'm dealing with some endpoint that perhaps I haven't used before, then I can immediately copy-paste the Python code and again get rolling much faster. Finally, if you already have that schema available, then you can also use it for some more esoteric things, like you can auto-generate client libraries. This means that your users don't have to make the HTTP requests themselves anymore, but instead they can use a friendlier Python package that again is automatically in sync with your code.

Speaker 2: Next, let's talk about standard standardization and why that matters. Following standards is good because it gives your users a sense of familiarity. If you create something that's completely unique and handcrafted, then Your users will have to learn everything about it. But if instead you follow some widespread standards, then they probably already have have experience with something that is either using the same standard or something similar and then they can transfer this knowledge onto using your API. And just as importantly, standards usually have some thought already put into them and they will help you avoid some common pitfalls.

Speaker 2: This is quite similar to how frameworks make decisions for you, such as how to store passwords, for example, and they thus keep you safe from storing them insecurely. And when it comes to APIs, my current standard of choice is JSON API. Despite the perhaps unfortunate name, JSON API is not just an API that uses JSON in its responses, but instead it's an actual specification for building APIs. It was created by authors of Ember and it offers quite comprehensive solution to building efficient APIs. And I should stress that this is just one option of

Speaker 2: several which are available. For example, GraphQL largely accomplishes the same goals. And there is a CraftQL talk coming up on Friday, so you might be interested in that one as well One of the most important aspects of JSON API is that it defines a generic yet flexible structure for the API responses. So let's look at how those structures, how those responses are structured. I'm going to use a project management tool of sorts as an example project. It basically lets users define projects and then epics within those projects and stories within the epics, kind of similar to how Paccamp

Speaker 2: works. So here we can see that uh the client makes request to the project's listing And the response document has three top-level members, the links, the data, and the included. And let's look at those one by one. First, the links. They are important because they enable discovery of related endpoints. In this case, because we asked for a list of projects, then the response is paginated, and uh the next link allows us to very easily get the next page of results. The client here doesn't really need to know about how the server side handles this.

Speaker 2: It just has to know that it needs to follow the link In the same fashion, uh your root URL of the API uh should respond with links to each of the individual resource pages. So that means that as a user of the API I don't need to know which endpoints or resources are available. I can just ask the root URL and find out that way. Next up, we have data, which contains the so-called primary data, uh the resource or resources that you asked for. And in this case we asked for a list of projects, so the data itself is a list as well. If we asked for one specific project, it would be just

Speaker 2: one JSON object. And as you can see each resource has type and ID which uniquely identify it. And they can also include links. In this case we get a link to the detail page of this one projekt and we can use this same detail link to update it for example. or delete it or do anything else that requires this one specific project. Next up are the attributes which are quite self-explanatory. In this case, we get the project's creation timestamp, uh which uses the ISO standard, as well as the project's name and description. And the resources can also have

Speaker 2: related objects under the relationships object. So let's take a look at those next. You can see that the project here has first of all created by a sort of foreign key or relation. which is a user with ID 199. And then it has a list of epics. This list is only a single has only a single element at the moment, the epic with ID 3101. And the idea between those relationships is that you can use the type and ID, which again uniquely identify this resource. To look up

Speaker 2: the related objects in the included resources. And included resources is the third top-level key that we looked at In this case, we have two uh related objects included in the data here. The first is the same epic and the second is the user. And both of those look exactly like the primary data looked. They have type and ID and they can have attributes and links and also relationships of their own. Now why is that important? Because it means that you only have to make a single network request most of the time. Meaning that if I want

Speaker 2: to show details page of a project, for example, and I want to list the epics of each project on that details page. Then using this approach, I only need to make a single network request and I get all the data that I'm interested in back immediately without having to make follow-up requests. And this is very important if your application is, for example, a mobile network mobile application which is working on a potentially slow network. So, how did that make you feel? If you haven't used JSON API before, then perhaps it looked a bit weird, or bloated even. If if I wanted to receive

Speaker 2: the name of the user that created the project, then there are quite a few layers of indirection to jump through And yet, if I now gave you response for another object from that same API and told you that it had updated Pyfield, which is also a user, and I'm interested in the email of that user. Then you would know exactly how to do that because the data would be structured in exactly the same way. Furthermore, if I gave you a different API completely unrelated to this one And tilt that that one uses the JSON API as well, then you would know how to use that other API as well. And that is the power of standardization.

Speaker 2: It brings familiarity and makes concepts that you already know applicable to something new as well So let's look at some more features of JSON API. The included objects or related objects, they are actually configurable. So the first get request here gives you the results that I already showed you But if you're interested in comments of each project, then you can say that you want the comments to be included as well. And again, you don't have to make any extra network requests to get them And you can customize not only the included data but fields for each uh

Speaker 2: resource type as well So the third example here says that for projects you're only interested in names and the comments. And again, this is important if you're building clients for slow networks where basically each byte matters and you don't want to send data over the network that you won't be using. You already briefly saw the pagination style. The list responses basically have next and prev links, which the client only has to follow. And this means that the client doesn't really have to know or care about uh how the server implements pagination or what pagination style is used.

Speaker 2: In my example I'm using cursor-based pagination because I sort of like it. It solves many issues For example, when you get new objects added into the database. But depending on the application type, it might be that you want to use page number-based page nation. And again, for the clients it would only it would only mean that the next link would look slightly different, but they wouldn't need to know how to implement this particular paging style themselves. JSON API also uh specifies to some extent how to do stuff like filtering and ordering on list views. But also

Speaker 2: they are sort of implementation dependent uh to some extent anyway Let's talk about errors. Errors happen and you can't really protect against that, but what you can do is making it easy for the user to figure out why something happened and how to fix it. And again the goal here is to basically reduce friction so that the user wouldn't throw up their hands and walk away, but instead would get help as uh as easily and as quickly as possible. And the way JSON API uh returns errors is that if something goes wrong, the results or the response will contain top-level error

Speaker 2: scheme which is list of everything that went wrong. And in this case we can see that there was some invalid attributes The detail here is something that you might show to the user saying name must contain at least three letters. And then the source is something that's machine readable and which you can use to find out, for example, the exact form field where the problem originated from. There are also some special cases. For example, when you want to transmit a large amount of data And in those cases, maybe JSON API does not make sense and you need a different, perhaps more specialized format.

Speaker 2: I should note though that the JSON API's responses, or rather, all the JSON responses should be compressed. And that means that the bloatiness that you sort of see when looking at the requests, it might not actually be a problem, or at least not as much of a problem as you might think. But an even better solution might be to still use JSON API and just include link to the actual raw data in your main API response. So sort of moving the data out of band. And JSON API already has the links object that you could use to implement it And here's an example of an application that does something with datasets.

Speaker 2: And we ask for a specific dataset here. What we get in response is sort of metadata And then there is the link to data DGZ, which is the actual raw data that the client can then follow and download and use. So to wrap up the standardization part, I want to once again uh stress that that specific standard is not that important. I like JSON API. If you like GraphQL, for example, that's cool too. But the important part is that you give users something that they might already be familiar with.

Speaker 2: Now, most APIs don't deal with just public data, or even if they do, you still might want to be able to identify the clients for various purposes like request limits or something similar. And that means that you need to think about authentication. How do you identify who is making the request? as well as authorization, which is what is this particular user allowed to access. The best practice here depends largely on the use case. I will be covering two major options. The first one is token authentication. This is the simple one where clients basically send an HTTP header containing a simple token with each request.

Speaker 2: Token authentication is useful for client-server situations where the client is, for example, a native mobile application. And when the user logs into that mobile application, then the application will get the user's authorization token. Which is then sent with each subsequent request and thus the server knows that the request comes from this specific user And if you think about it then session cookies are also basically one kind of tokens. So It might be that your API is only ever accessed from the browsers and in that case session cookies might actually be everything you need. If you're using Django REST framework, then once again,

Speaker 2: know your tooling. REST Framework already has built-in support for token authentication. For more complicated situations though, there is OAuth too. OAuth is meant for creating platforms. Think Facebook, where a third-party application can request access to the user's data, and then the platform verifies this request. And asks for the user's permission and then grants the application a token which is both application specific as well as user specific. OAuth2 is a quite complex protocol. It covers many different use cases and flows

Speaker 2: like for mobile applications, web applications. Applications which might be in your living room and have very limited UI. And it's good because once again you will be you you will be using proven standards that have evolved over the years and uh have a lot of thought put into and many corner cases solved. But the downside is that it also requires quite a lot of attention when you're implementing it. Luckily, there are various libraries available that take care most of that low-level plumbing work. If you're using Django and Django Rest framework then there is Django OAuth toolkit which takes care of uh basically making OAuth

Speaker 2: quite easy. If you're not using Django, then there is O outleap which uh the Django of Toolkit itself builds upon And uh OAuth leap is a bit more difficult to work with, but on the other hand, if you need to change something that's relatively low level, then you have to go there For our own API project, we had to add some functionality on top of the Django OAuth toolkit. And some of this was due to the missing features. For example, we needed better redirect URI validation. But most of the problems or missing features were due to the requirements of the project itself. For example, we needed to

Speaker 2: pass around more info about the token and the user that it's connected with. And we also had to re-implement the pages where developers could register their applications because we wanted them to be more user-friendly. Once you have the structure and authentic authentication uh figured out, you should also think about versioning. Versioning is really something that you should think about from day one because it's very important to bolt it on later. Because once people will be using your API, they will be assuming that it never changes

Speaker 2: because you didn't tell them otherwise But if if you have versioning from the beginning, then it will be easier to manage these expectations. And you should also make it clear how long the old versions will be supported and how developers can uh find out what changed and uh and those what those support schedules are. So let's look at how the clients can specify versions in their API requests. There are once again different options here, and I'll cover two of the most popular ones. The first one is header-based versioning, where the clients specify the version they're interested in as part of the accept

Speaker 2: HTTP header. In this example, the client once again asks for the list of projects and says that it wants the response to be in JSON format and the version to be 1. 0 Uh headers are more idealistic approach because the version that you use is sort of meta information. And with header-based uh versioning you keep it out of the URL pods. But headers are also a bit harder to use and test For example, you can't specify the headers when you're just browsing the API. So in the real world, path-based versioning might make more sense and be a more pragmatic choice.

Speaker 2: This is when you basically uh prefix the URL with the version itself. So in this case we have the slash V1 prepended to the URL that we're using. And this can make debugging easier as well because if your server logs contain URLs, then you automatically sort of have the versioning information attached to those URLs. And as you can see once again Django Rest framework provides out-of-the-box uh functionality for both of those cases. Once you've chosen uh which uh variant to go with Then you need need to think about what the version should be.

Speaker 2: Some people prefer to use integers like v1 and v2, others prefer dates And I'm also a fan of the dates lately because they're sort of less emotional, which means that you don't have to think about whether the next version will be B1. 1 Or is it big enough chains to uh justify version two? And I think that's uh a good thing because it lets you focus on the API itself. You should also once again assure that the upgrades are easy to make by the developers and they have access to change logs and upgrading information.

Speaker 2: So this covered the client side of things, but how do you handle versioning on the server side? For incremental changes, a nice approach is to use version transformers. This is actually quite similar to how Django middlewares work. So basically you would write your core API code only for the latest version, and then if a request comes in using an older version Then you would have the version transformer sort of transform that request into a newer version, which could then be processed by the core code itself And once the core code gives you a response, again for the latest version, then you can use the transformer to transform that latest version back into something that the client understands.

Speaker 2: And this approach is also stackable in the sense that if you're if you have multiple upgrades or say three versions Then if the uh client is asking for version one and your latest is version three, then you can have two transformers, one that knows how to go from V1 to V2 and the other way and then the second transformer that knows what the changes between v2 and v three are And this approach makes it quite easy to do smaller smaller changes like uh changing some field names or adding new fields. And notably

Speaker 2: Stripe is also using their uh this approach in their API, and they have a blog post about it if you're more interested. But this won't really work for mass event breaking changes. In that case, you might just need to create a completely new API implementation and duplicate some code in the progress. Getting to the more practical side of things, uh we have put a lot of what I just talked about into a package called DG API Core. It's basically an add-on built on top of uh Django Rest API JSON API package uh which

Speaker 2: in turn is built on top of the Django REST framework package, and where the JSON API package basically makes REST framework uh compatible with the JSON API spec. The DGAPI core package adds some additional stuff on top of that. Some of those features include documentation generation It also comes with pre-configured settings so that you don't have to configure all the uh response and uh request and response processors for the for the REST framework. And it also contains some utilities for viewsets and serializers

Speaker 2: so that you can, for example, uh return different fields. for listing and detail requests, so that the listing request uh returns some summary data and then the detail uh requests uh return the full details of each object. And there are also similar packages out there. Which one is the best for you again depends on the use case. You should really just once again know what your tooling has and uh what uh other alternatives or add-ons are out there. Finally, let's also look at the same thing from the client's perspective.

Speaker 2: Uh if we're trying to use an API And see how all of that is actually in use. So as an example scenario Let's say that I have some audio and I want to do speech recognition on it. I will be using Amazon Web Services and Google Cloud Platform as examples. They're both quite good because they have a comprehensive list of services and they cover all of those services with a single library that provides access to them in a unified and familiar way. So first of all, of course I took a look at that documentation

Speaker 2: In both cases the documentation was uh out there of course, quite easy to find, but perhaps slightly overwhelming just because of the immense amount of services provided by both Google as well as Amazon. But importantly in both cases they also provide code examples. So once again, yeah, if I want to I can just copy and paste some uh starter code and get moving really really fast They uh provide quite comprehensive client packages. For the Google, there is the Google Cloud package for Amazon's For Amazon there's Podo 3, uh both are installable from pip.

Speaker 2: Once uh that part is done, uh you need to run through authentication. They both provide some command line tools to get that done. And then you have quite thorough documentation that lists all the different operations that the APIs support. as well as all the different parameters for each of the endpoints. So this is an example of how it would work in case of Amazon First, you import the package itself, the Poto3. Then you ask it for a transcribe client And once you have that client, you can just call methods on it like start transcription job

Speaker 2: and uh pass it your audio data as well as perhaps some other optional parameters And you get the response with the results back. The Google 's case is quite similar You once again import the speech client and then instantiate it and then you can call methods on it like recognize here and again posit your audio data get the results back And as I mentioned, both of those SDKs provide common interface for all of the included services. For example, in Amazon's case, S3, EC2 uh the transcribe that uh you see here. And

Speaker 2: again it uh plays on the familiarity uh side of things. So once you know how to do transcriptions for example It would be quite easy for you to use some other service as well. And as you can see, both of those SDKs are quite similar to each other as well They use common and familiar patterns, they don't try to invent something really, really unique And again, this means that the potential pool of developers who are able to quickly and easily start using it is so much bigger. Their documentation is thorough, they provide getting started pages and code examples.

Speaker 2: And what you can't see here is that at least partially they are both automatically generated from the same schema. So basically they take the API schema and then from that they generate both the documentation as well as the client libraries for Python and some other languages as well. And again, this means that everything is nicely in sync. So let's wrap this up. Documentation matters because it's usually the first impression that users will get about your API. So make sure you invest into it. Embrace standards because they bring familiarity

Speaker 2: and make it easier to use your API. And use automation to ensure that things don't go out of date. This applies to both documentation as well as perhaps client libraries. And in general, reduce friction as much as possible. By friction I mean all the small potential issues that drive people away from your API or your product. and think of the humans basically. Here you can see my contacts. I will be tweeting my slides later and perhaps do a blog post on the same topic. Thanks for listening

Speaker 1: We have a couple of minutes for questions. If people want to go up to one of the mini microphones that I've just discovered, they go all the way up and down. And you can get your present after you've answered some questions.

Speaker 3: Hi.

Speaker 2: Hello.

Speaker 3: I was wondering on whether you had any thoughts on um clients returning Python objects versus dictionaries.

Speaker 2: I think if you have a good client library, then uh returning Python objects is uh preferable because if you have a li if you have a dictionary then you had you sort of have to go into that dictionary manually. and uh using an actual Python object can provide shortcuts. So I think that was the preferable way Uh

Speaker 4: since you're using Django Rest framework, you surely have seen a core API. Which is uh integrated. So what's the main difference between JSON API and Core API?

Speaker 2: Uh I'm not entirely sure, but I think Core API is more about the schema itself. Uh feel free to correct me if I'm wrong. And uh JSON API really defines the responses that your API uh outputs. So I think core API solves the schema part, but uh JSON API is for the request and response formats and related things. But I might be wrong.

Speaker 4: So um I just want to use the

Speaker 2: uh no I did not.

Speaker 4: Okay.

Speaker 5: Thank you. Thank you for the talk. Um I wonder if you would design your APIs for mobile clients differently because of the si the size of the data matters there a lot.

Speaker 2: I think that uh JSON API uh already gives you rather good tools because you can customize uh which related objects are included in the response as well as which fields are included. So depending on your specific use case you might want to go farther from that and perhaps uh go as far as to using some binary protocol instead of JSON. But uh yeah it really depends on uh what your use cases are. The advantage of JSON API is that it works for a variety of use cases and it works quite well for those So if you have the mobile clients as well as some other uh API integrations with uh other services, for example, then you don't need to create too many specific

Speaker 2: uh APIs.

Speaker 5: Thank you.

Speaker 2: Thanks.

Speaker 1: We still have a few more minutes if people have questions. Going once, going twice. Well let's thank our speaker again.

Speaker 2: Thanks again.