REST: It's not just for servers

Hi. Um Yeah, as I said, uh my name's Mark. Uh I work at Cactus Consulting Group. I didn't just get a free t-shirt. Um There's my Twitter handle if you want to tweet at me any any feedback that you have. This is the part of the talk where I'm supposed to tell you I'm really smart and you should listen to me. So I'm a technical technical director at Cactus Group, I'm a co-author of Lightweight Django, and we we build REST APIs at Cactus. We interact with a lot of REST APIs from Cactus. And this is really um you know about both sides. I mean the uh

and my examples here are are in Python as as you might example, uh use as you might imagine But it's nothing really Django specific about this. It's more about REST as a concept. That's not really like a full introduction. Like this is this is the real me. This is me with my family. uh completely broken with a grimaced smile in an unbelievable amount of pain like this is when I'm real and it only happens for like a few fleeting moments every year. Um but this talks about rest. What

what is rest Sometimes it feels like just a marketing term. It feels like something like responsive. Like people use it and like I don't know what it means. Sounds good. Let's call our site responsive. Let's call our API RESTful. But it means something. It stands for representational state transfer. This concept is defined by Roy Fielding's PhD thesis at UC Urbine from 2000. Again, it's a term that's been abused by web services and marketing teams basically every day since then. And REST

is not a format or a protocol. It's an architectural style It's a way of building applications. In his thesis, it stated it's an architectural style for building distributed hypermedia systems. It emphasizes scalability , generality, independent deployment, reusable components, even more buzzwords, things people want. But there are some constraints. If you want your API to be RESTful, you need to satisfy some rules.

First of these rules is that it needs to be a client-server model. And server needs to be stateless. So no client context should be stored on the server between requests. The system should be cachable, layered. A client shouldn't know whether it's talking directly to the server or an intermediary proxy. There should be a uniform interface on how the client and server talk to one another. There should be a unique identification of resources, self-descriptive messages passed between the client and the server. And there's an optional code on demand.

It's kind of weird. Doesn't really make a lot of sense. Not going to touch on it in this talk. So again, when you when you buy into this, when you commit and say, I'm going to build a restful service. I'm going to build a client server model and I'm going to build this stateless server , these amazingly self-descriptive messages. What do you get for that? Well, you're supposed to get performance in terms of scalability, simplicity, modifiability. This comes from again this cache ability , this separation of concerns, that pieces can be you know

scaled independently So it's not surprising that this is an extraordinarily popular architectural style of late If you name a web service, it probably has what it calls a RESTful API. Social media sites, Twitter, Facebook, Instagram, Pinterest. cloud services like AWS or Rackspace or OpenShift Cloud Foundry. The list just goes on and on. And those are just the ones that have public APIs. Many people use this just internally. And they build mobile applications on top of a RESTful

API. And Public APIs just love open source clients. And that's what this talk is about. It's about the client This is one of the coolest things I think that that is happening right now on the web. Opening up public APIs. Through OAuth or whatever mechanisms you have is why there were Twitter clients before there was the one true Twitter client. And why you have services like Travis that run on top of GitHub, and how Travis

can call coveralls and Travis can deploy to Heroku. These decoupling of services that again scale independently, these are built independently. It's uh that's that's what the dream of rest is. Uh and it happens again with two components. One, you need a server. If you want to learn how to build serve uh RESTful servers in Django, um There have been like half a dozen talks already about it. That's not this talk. This talk is about the clients

and particularly the challenges for clients Some of these come from being the client and some of these come from just the interaction between the client and the server. Sometimes you think, well, I mean this is These are HTTP. HTTP I understand as a web developer. I can just import URL lib or import requests and I can just start going You know, writing a client that sort of works, like you know, Brandon said, not really hard. Getting one that works well is hard. And maintaining one as APIs change. is hard. There's

technical challenges that we're gonna cover and look through. And there's non-technical challenges, like really terrible terms of service, which I'm not gonna cover. So what are some of the technical challenges? Uh and these are things like changing APIs. When you're talking to a public API, you don't have the choice of how the API will change. It's a client-server model and you don't get to have any say on the other half. That's frustrating. That's hard. It means potentially supporting multiple versions of the API in a client wrapper

, making it clear which versions you are supporting. understanding how the server versions content, which is a hotly debated topic in REST. So for example, you know, Twitter deprecated their 1. 0 API, and now all the URLs have a 1. 1. And these two URLs in particular are the ones that bit me in a project. They changed. They have identical responses, which is even more frustrating. Uh there's a two-character change that that broke because I hit a 404. Um And

but these aren't this is a relatively simple and easy change to make. The larger changes uh To APIs can be harder to deal with. A few years ago, Twitter got rid of their basic auth and they switched everything to OAuth. And a funny story is that we actually had a client a couple of years ago and wanted us to upgrade a site. The site was running Django 1. 0. We upgraded it to 1. 5. And there was a piece of the site where they could tweet things that happened. Like a new blog post would come up and they could hit a button and tweet about it. And it used Basic Auth. And I, as

diplomatically as I could, said, uh, is anyone using this piece? And it said they would check. whether anyone was using this piece, but I was fairly certain they were not using the piece because it hadn't worked in two years. So they eventually agreed that we could remove it rather than updating the client Uh because it hadn't been used. A big challenge for clients is servers. uh in particular servers that really don't meet what I would say all of the uh All the constraints that are necessary.

In particular, the uniform interface constraint, as it's defined, defines the concept of hypermedia as the engine of application state. It's usually shortened like this. I'm not entirely sure how it's pronounced. But it's not usually implemented in a way that's helpful for the client. And there's Again, debate as to whether it's necessary, whether it's helpful. I will tell you it is helpful, and I will show you how it's helpful. Uh idea is that

you should build discoverable APIs. That the server should tell the client how it can navigate through the API, how it can find resources that exist. and how it can navigate through the API. This shouldn't really be a uh such a controversial topic. This is exactly how we build websites We tell we have links on pages and you navigate through them to find related pages. You have forms and the forms have actions and they tell the browser where to submit and how to submit. These are the same types of concepts here.

And when you don't have a discoverable API, how is the API discovered? It's discovered by humans. And humans are terrible. You have to read a giant pile of docs, and again, when APIs change Clients don't know. Humans have to know. And humans have to read more docs. So instead of relying on documentation to build discoverable APIs, how do you build a discoverable API? I have uh an example of a change that Bitbucket made You're not you don't have to see the whole example, but this is my Bitbucket profile on version one of the Bitbucket

API There's a tiny little section at the top, which is my user profile information, which is what I asked for. And then there's a whole pile of information that I didn't ask for. Which is every repository that I have, all of the information about every repository that I have, and all the sub -information about all the forks that I've created. Down the line. You can see the scroll bar of how ridiculously long this response is. And I don't even use Bitbucket that much. I have like five repositories. What they did in version two was they normalized this, if you would, like in the in a database sense.

When I ask for my profile in version two of the API, I actually get my profile information at the top level. It's not buried into a user. key. And then beyond that there's a set of links. They say, if you want other information about this user, here's where you can find it. You can find your repositories here. You can find Mark's followers here. You can find my avatar over here And again, going back to the keynote, you can think about how these can be cached differently. In this first response,

When anything about any one of my repositories changes, this cached response has to be invalidated. Now here I have a small concise payload that can be cached when I ask for my profile. When my repositories change, this response doesn't change So those are those are a couple challenges on the server side. Building RESTful clients is also a challenge because some environments have weak HTTP support. And this is typically in a browser-based environment.

As web developers, you probably don't get to always use Python to do your API interactions. And when I say that some browsers have weak HTTP support, I mean IE has terrible HTTP support when you do cores or cross-domain request. It's basically completely broken. There's no delete or put when you do cross-domain requests. The content type is broken. All the things that you would want when really building a robust uh API is not there in IE. And

so servers need to work around this. And uh Django Rest framework, as many have talked on here today, have ways of doing this. There's an a commonly used uh HTTP header called method override to say this is a post, but it should have been a delete, so treat it like a delete. If you control the server, that's a facility that you have. Uh if you don't control the server, uh sorry, you're out of luck. One of the biggest problems for clients, though , is managing state. I said that it's state, you know, the client-server model and it should be stateless.

Well, not entirely stateless. The protocol is stateless. HTT is stateless The server is stateless. Guess who has to manage state? The client. It's like herding cats. It's a pain. And particularly when you're trying to build a general client library, managing state is difficult. You don't know what state the person who's going to use your client is interested in. But you need to help them with some basic pieces. And that's what I'm going to talk about. So what should you do to build a good client? What are some best practices in building a REST API client?

Well, first thing is to build useful objects And this almost sounds like a tautology or uh you know something that's stupidly obvious. If you want to build something useful, it should return useful objects. But you would be surprised if you went through GitHub And saw how many API clients basically import requests, do the request, spit back a dictionary blob. You should provide useful objects that translate these dictionaries, these JSON blobs. Into meaningful business objects for the API. And they should help you link to related resources. They should help you perform actions

Because you're not just asking for a dictionary. I'm asking for a thing. I'm asking for my user profile. I'm asking for a repository. There are things I want to do with my My profile and there are things I want to do with my repository. Maybe I want to delete my repository. Maybe I want to see what the last commit on my repository is. Maybe I want to update my user profile. Those types of things is what you should be building in your Python clients. So here's an example from uh Twilio Python. They're right out in the hole if you want to bug them. So Twilio, if you haven't gone to their table, I don't have any affiliation with Twilio, just to be clear.

They didn't pay me to put this on there. I didn't know they would be out there. Twilio is an S whatio is an SMS gateway and you can purchase numbers. Well they do more than just they're a telephony gateway. They translate SMS and voice into HTTP. So you can purchase new numbers, you can send SMSs, these are all the things that you might want to do. And when you use the Python Twilio wrapper and you search for available phone numbers, you get back one of these. You get an instance of available phone number. And available phone numbers do

one thing that's really helpful, which is they know how to purchase themselves, which is probably why you are searching for available phone numbers. And how do you use it? How you you construct an instance of the client with your credentials? I'd say search for phone numbers and 919 area code. And then if there's a number, just buy the first one. I don't care what the other digits are, I just want it in this area code. I don't know anything about the URLs. As the person using this client, I don't want to know anything about the URL. That's why I'm using the client. I don't want to know that to do a purchase I need to take the response from the search and I need to do a post

to another place. I just want to purchase the thing. That's what this wrapper does. This is a great example. of writing a REST client. Another piece of useful information that you want to track, Brandon touched on this a lot, cache headers This is a piece of state that the client needs to track. If the server is giving you e-tags and last modified headers, and you don't send them back. You're not holding up your end of this cashable bargain. It's not uh the system isn't cashable if you're not respecting the cache header.

So these useful objects that you create for your API clients should help you track the cache headers. This is something that's easy to take for granted in Python because it just kind of happens in the browser. And your browser is really smart about tracking e-text. tags, tracking last modified, knowing where the resource was held locally. But in Python you need to take a little more care. So an example of a of a client library that does this is uh the GitHub 3 Python wrapper. They have a series of objects that build upon one another and the core GitHub object has a refresh.

You know, I fetched I fetched Mark's profile and I did some things and I'm gonna update it. But I want to make sure I've got the most recent copy, you know, before I make my update, because I only want to update one field. But this is rest, so I have to send the whole thing So uh the body has the last modified header uh and and the le the e-tag header that's that's actually handled by the parent class when the self last modified and self. of e tagger said. And this is used like this. I would log in and I can get the user Who is currently logged in?

That's not my GitHub password. Don't try to log in with me. Maybe it should be. No one would ever suspect that that's my password. But I get the user who's currently logged in. And then I can see my e-tag. That's so cool. I can, I don't know why, but I like that. I want to know what what e-tag did GitHub give me? Like what it what do they think of me? What MD5 hash like really represents me as a as GitHub user? And I can do a conditional refresh. I can say get me my profile if it hasn't changed.

This is the version I have. What? Is there a newer version of Mark's profile? It doesn't update that much. And the cool thing about this, for for GitHub in particular, if you get a 304, if you get a uh a not modified response, it does not count towards your API rate limit. And that's probably because it doesn't hit their application servers. It probably just hits their varnish server. And they they repay that, they pay it forward and say, this doesn't count towards your API limit because You didn't hit. You didn't get a response. You it was not modified. So you want to avoid hard coding paths. This is another thing to do

when building API clients. You should use the URLs and links that are returned from the server when they're given. And we saw they're given by Bitbucket. Sadly, most of the Bitbucket clients haven't been updated for version two. They're used by GitHub. GitHub 3Py uses the the responses sent back by by GitHub, but I already use them as an example. So a more common place where these are given is in pagination. So if you get a list of things

Many APIs, while they don't give you a nice block of related links on detail pages, Do usually provide a next or previous URL when you paginate large objects. So this is Pyracks, this is the Rackspace API. The entire method here isn't shown. It's kind of long. But this is their Cloud DNS manager. So you can Search through all the DNS records that you have managed by Rackspace. I don't know how many DNS records you have that it fits on more than one page, but I don't know what you do.

Here you can see it parses out the links from the bodies and says, like, is there a next URL, is there a previous URL, and hold on to it. And then when it lists through them, it just iterates through them. You have this option to say list all. And it'll just yield. It'll just keep making API calls. It'll go through all the things in the response and it says, oh, I need to fetch the next one. And the next one and the next one. This is pretty fantastic. I can tell you that this hasn't worked for me uh before uh on Facebook where they actually gave me the wrong next URL and I just looped. for eternity until I hit the rate limit.

But when your server works, this works. So here's how you use it. Uh setting up the credentials is kind of weird on on PyRax, but that's what this is about. This is about you know, iterating through all the reser all the all the things. So in their cloud DNS, you can search for records. And rightfully so, the search by default lists all the results. It doesn't just search the first page of results. It'll yield all the results. So I can find all the C names that I've set up for example. com, which is probably not very many. But if you're building a software as a service platform where you may have a lot of

CNAME records, where you're mapping your domain to a client domain, you may have a lot. And this is again kind of good and bad. Makes it really easy to use. It abstracts away the fact that there is pagination here. It also hides how many API calls this is going to take. It's not immediately obvious how many API calls that'll that will make or how fast it will make them. We'll basically make them as fast as I can iterate through the list. But again, as as someone using the client, I don't I don't want to do that pagination myself, so I'm happy that they do this for me.

So those are some some things to do. There's one thing that I really want you to stop doing. This is my plea for the Python community and the Django community. is to stop making REST clients, which are basically glorified URL builders. Uh if your REST client only builds URLs, like the server has failed, the client has failed. You're not really doing anything more helpful than just string formatting, but you're adding like syntactic sugar on top of it. But there's not you can't really make a general REST API client at this point.

There's not enough standardization of message formats yet. You know, there's some work in that in that space. Anything claiming to be a general REST API client is kind of missing the point and they're lying to you. There's no shortcuts, right? The business objects that come back from a REST API, you have to understand the API. There's no generality to that. Um and each a API is a little bit different. So what what's what's an example? And I don't mean to pick on the developers of Slumber. They're probably very well-meaning people. But this is one that attempts to be a general REST API client.

And it's very clever Python. This is exceptionally clever Python. It translates method calls and attributes into URLs. It's fantastic. It looks so promising, but it gives you back dictionary blobs. Which you then need to translate and uh to make additional calls like this the put call here to do the update. To do the delete. You know, these objects don't know how to delete themselves. They don't know how to update themselves. And when you look at this, again, it looks so elegant. It looks like the Python I love to read.

But what is kind of hidden here is that when the API changes from note to notes plural, because it's kind of awkward to have these like singular resource names, I have to change all of these calls. This client hasn't saved me anything. In fact, it's added a layer of abstraction that I'm still building the URLs. I could have done this with string formatting. Uh I still have to translate all the objects. It doesn't help me with caching.

Uh it's just like I said added syntactic sugar over building URLs. So please stop doing this. Just to summarize what we talked about, we talked about rest. REST is a client-server model. And servers are completely useless without clients. In fact, I don't know that you can really call something a REST API if it doesn't have a client. Because it's a client-server model. So if you're gonna build a REST API, I would encourage you to try to write a client Understand the pain

of navigating your API with a client. And don't just show examples that have one request. It's really easy to make one request. It's hard to manage state over requests. Do you know tracking this profile that I want to later update So show large examples. Treat your API just like you treat your website Make it discoverable, browsable. Think about how clients are gonna navigate and find the data that they want, how are they gonna do the actions that they need to do I again I think uh REST API, the

Django REST framework does a great job with this with the browsable API because you get this experience in the browser to kind of click through, like how do I get to the next piece of data And again, this is like documentation. When you have to explain how something works, you realize how terrible it works, and you get to redesign it before it's too late. So some some handy uh resources. Uh there are some links here. I'll have links to the slides. The original doctoral thesis and this uh rant uh by uh Roy Fielding about REST APIs must be hypertext driven. There's an RFC about constructing URI templates if you have more uh complex templates.

This is something that GitHub does and the Python library uses. the links to the the example ones I didn't link to Slumber because you shouldn't use it. Here are the photo credits. Thank you for listening. And I'm Mark. I'm writing Lightweight Django. I actually have a few pre-release copies. I'm going to be signing them at 12. 30 at the cactus table if you wanna if you wanna come by. We'll have one on the table if you want to flip through it and look at it. Uh thanks for listening and uh build great APIs.