Navigating the maze of Django's URL routing: a deep dive

So yes, welcome. You've uh made it to the final funnel talk where I'll be talking about um URLs So um thank you for coming. Um and before we get going, a little bit about me. So my name is Tim Um I am a React and Django developer. I love both. I wish I had more time to contribute to both ecosystems Um I work for UNOJuno, we're a freelancer management solution and freelancer marketplace. So if you're a freelancer, check us out, or if you work with freelancers. Check us out. In my spare time, I like to do lots of different things. Most recently I've really got into bouldering. It's a great sport. I recommend it, really friendly community. So enough about me.

Why talk about URLs? I mean, you come to DjangoCon to um learn how to make the ORM blazingly fast. or to learn about the latest cool thing you can do with AI. Why why think about URLs? They're a a bit dull maybe. Well, um behind me on the screen I've got the first two sentences. from the uh URL's page in the Django docs. Um and this idea that URL should be clean and usable is is really important. because clean and usable URLs provide a better user experience, but I hope to show you that they provide a much better developer experience. um and they're less buggy and and there's loads of advantages and we tend to get really into kind of optimizing for performance and implementing cool features

And certainly for me at least, uh URLs can be a bit of a an afterthought. Um and I think we should be giving them the uh due care and attention that they deserve. So uh why a deep dive? Surely all I really need to know is the the tools that Django gives me to use the URLs. Why do I need to look under the hood and figure out what's going on underneath? Well, um I'm a firm believer in the better you understand something um uh the understand the way something works, the better you're able to use it. You can start unlooking all these cool tricks and things that you didn't realize you could do. uh with Django. Um I also hope to show that the Django source code isn't that hard. You know, it's not easy, um that would be a lie, but um

it's just Python. Um and we can all read Python and as you kind of take care and just read it slowly, um it makes sense. And so I'm hoping to show you that um it's not something to be scared of Um so what's the talk gonna look like? Well we're gonna build the URL routing system from the ground up, starting from nothing Um it will be a a much simplified uh version. Um obviously can fit all of that in a half hour talk. It's difficult to get the talk that I've got down to half an hour. So it'll be uh kind of MVP version. But all the key moving parts, all the key classes that you need to know to understand what's going on will expose. And so if you can if you can get that, then you'll be able to maybe dig into the source code yourself

and um be a little bit a little bit more confident doing so. Along the way, we'll also think about what some good URLs look like. What are some good practices so that we can have clean and usable URLs? Where do we start? Well, when you create a fresh Django project, you have a settings file, and it will have a setting like this, which will point to a module. uh for your url conf and your module might look something a little bit like this. It will have a URL patterns, which will be an iterable And that's basically what an URL conf is. It's a module with a URL patterns where we can search through

each of the patterns. And when a request comes along, we can match against each pattern until we get a view. And hopefully we'll then have a view along with some arguments and keyword arguments. And we can we can do something with that. Um that's not 100% true what I said about what a URL comp is, but we'll roll with that definition for now. Um So how does it work? Well in the base handler when a request comes along we want to um resolve that request and the key bits we're going to look at are this line here. So we get a resolver that's going to return a resolver class That's going to do all of the heavy lifting. And it's got a resolve method, as you might expect. We're not going to spend too much longer in this in this method, but I think it is worth pausing and um observing a few things.

Notice what gets passed into that resolve method. It's the request path info. So that's just the bit after your. com and before your query parameters. Um last DjangoCon in Edinburgh during the sprints I was chatting to someone and he was saying he really wished there was a uh a method-based kind of URL routing system so we could root to different views depending on if it was a post or a get. And we can see that's a limitation. We can't do that. I mean we could subclass lots of things and reconfigure things. It's not impossible, but it's certainly difficult. Um likewise you go on Stack Overflow, I see lots of questions saying can I root to different views depending upon what type of user you are Well no, we don't have the request. We've just got that path info.

Having said that, it is worth noticing here, you can set the um URL conf dynamically. So if I wanted to, um I could set up some middleware. It might look something like this, and I'll have a separate um URLs pile for maybe a maintenance mode where we only show a few views. Um and and that's one thing you can do And it's just an example of how just being able to read one method you can understand a few tricks that um you didn't realize you could do before. Which I think is one of the reasons why it's really worth um read reading the source code. So let's have a look at this getResolver method. It's nice and simple. It returns a URL resolver class, and that's the thing we're going to build out.

So let's get going with that. So we need the class , and we saw that we passed in a URL conf, so we better do that. And we also saw we need a resolve method. So that that's a good start. But what was that URL comp? That was a a string, right, that pointed to a module. And we need the module itself. So we better get that. Let's add a property. And we can uh look in the Django source code and we can see what that property looks like. Um and there's a few interesting things to notice here. Um if we have a string, we import the module. Great, that's what we want. But if we don't have a string We actually just returned the object itself. And so our URL conf, we could put anything there. It doesn't have to be a string.

It could be an object, so long as it's got the appropriate prop properties to behave correctly. So now we've got the module. The next thing we need is we need that URL patterns list where we have all the paths. So let's get that and we can add another property to do that. Um oh I need to I'm gonna need to zoom out a bit there. That's too small. There we go. Um And again, there's some surprises here when we look at what happens. We use the getAttribute method to Extract the URL patterns from the from the module, but if it doesn't exist, we fall back to whatever that URL module variable is.

Which means it We could have any um even an iterable there. The only thing that we care about is that it's an iterable. Um and this is uh just another example of how reading the source code we realise we can do a few tricks. So we don't even need to point to a module here. We could define some URLs in a list , set that to our URL conf um and it will and it will work. You're probably not going to want to do that. I think that's a bad idea. And disclaimer to lots of things here, it's an implementation detail, it's not in the docs. So if you're going to read the source code and do some cool tricks Make sure you write some tests so that when Django, uh when you update Django and something's changed in the implementation details, your project doesn't break.

So that's something that's important to remember But now we've got um our URL patterns, which is the key thing we want to do. So now we can start this um resolve method. And what does the resolve method need to do? Well it needs to get a view and some arguments and keyword arguments and return an object. We saw that in the resolve request method. So let let's do that. We can add a view. We can add a add a class that's going to encapsulate all that information And we'll add this getItem magic method, which just enables us to do this sort of assignment destructuring, which will make things a bit easier later on. But all this really is, is a class that's going to encapsulate the information we want to get when we've resolved

the URL. And now we can do some actual uh resolving. So we're going to iterate through our URL patterns. Um but what what are our URL patterns? Well, um they look something like this normally um and we've got these path functions. So what do those path functions return? Well if we look in the source code we'll see they return another class called URL pattern So let's build up that class. And this class is going to be responsible for matching a pattern and returning a resolve a match method. So it's it's a simple method. Um I've just put in a uh a dummy function there for the time being, and we either resolve and return a resolve match or we return none.

And then if we go back to the URL resolver class, well all we need to do is iterate through each of those URL patterns. We can see if it resolves. If it does, we return the match. Otherwise, we raise a 404. So we've got a really minimal sort of MVP URL resolver there. We can also put in some extra bits. We can make a list of all the patterns that we've tried which will be useful for the debug 404 page when you've got debug on. And so here we've got the the building blocks of our URL kind of system. We've got the resolver which stores a list of patterns Then we've got the patterns which are responsible for returning the match back to the resolver, and we've got the match object itself.

One thing that we left undone was this path matches. So let's fix that up. And actually Django delegates responsibility for the actual matching to another class. So we'll introduce this. regex pattern. We'll do um regular expression pattern matching just to keep things simple for now. And again, we can this this pattern this class is going to have a a match method. Which we can tull call to do the matching, and we can pass back um any arguments or keyword arguments. Um so what does this uh regex pattern class need to do? Well it just needs to do basic um Python where we use built-in regular expression functionality, we search for the path, if we get a match, we extract out the keyword arguments and arguments.

um and we we return them. Notice by the way that um if you have keyword arguments we don't have um any arguments at all. And that's Django kind of helping us to Have a healthy pattern. Keyword arguments I think are much better than arguments for views and URLs anyway, because um It's so easy to introduce bugs when kind of refactoring your view. Um and a mixture of arguments and keyword arguments is a total mess, so Django just doesn't let us do that. We return the arguments and the keyword arguments, and we'll forget about that first element of the cheaple for now. So where does that leave us? Well it leaves us with a a basic URL um

resolving system that's just a few basic clubs. Now, um in the code base they're a lot bigger than this and there's lots of extra functionality, but the basic roles haven't changed. The basic kind of key things that these classes do is the same. If you were to go and look in the code base now. Um but it's twenty twenty-four and hopefully we don't write um uh paths like that anymore. We use the the nice Django syntax where we just say the type of the uh parameter we're trying to catch and we give it a name a name for it And how does this work? Well this uses something called path converters. So a path converter is a really simple class.

It 's three things. Um it's got a regular expression which we're going to um use to match our paths against And then we register that class in a registry. And then when we when when Django comes along and sees this path, it knows how to turn that into a regular expression. It does a couple of other nice things too, which it has this toPython method, which converts it to a Python type when it gets passed to the view. So with your old regular expression um pattern matching, you're always going to get strings in being passed forward to your view and then in your view you're going to have to maybe pass those strings a little bit to get to the right types, which is a load of boilerplate we don't want

And so with these path converters, we can do that outside of the view to keep our views clean. And the URL is going to do the reverse for when you're calling a reverse function. So I think path converters are really cool. They're easy to write. More importantly, they're easy to read, which is really, really key. And when you've got a URLs. py with you know fifty different paths, being able to easily see what they're all doing is important. They keep your views clean and who doesn't want clean views You can write your own. They're really easy. Just an example here. So here's a date converter. We write a regular expression for what a date in a URL might look like. and we write a method to convert that um that string into a date.

And then we register it. We register the um uh the path converter and we give the the um kind of prefix we want before the colon uh to kind of match the path converter to to the path Um and then instead of writing this, we can write this, which is much nicer, much cleaner. Um so path converters are really cool. But I think path converters are really cool for um another reason, which is they allow human-centric URLs, and this I think is key So um three or four years ago at UnoTuno, we realized we had lots of IDs in our URLs and we decided we were going to get rid of all of those. Before I talk about that, I want to give credit where credit is due.

A lot of this was inspired by work that Stripe has done and lots of the thoughts in this talk are taken from this article. I'd go read it, it's excellent I don't have any affiliation with Stripe, but I am a fan. So what's wrong with IDs in URLs? Well, they're meaningless. Um let's take a URL here. Um maybe it's for a DjangoCon ticketing site. Now 38921, what does that mean? It's clearly an ID for something. Uh maybe it's an ID for uh a user type. But maybe it's not. Maybe it's a ID for a profile entity. I can't tell. There's no context there to tell me what it is. Maybe it's neither. Maybe it's something other that's related. Um They're sequential, which opens you up to all kinds of um enumeration attacks.

So um let's say I've got a site and I upload some content and I see my content has a a number in the URL, 784. And I say, well I wonder what happens if I change that number to seven eight three. And all of a sudden I can see somebody else's content that another user has uploaded that I shouldn't be seeing. And there's all kinds of attacks like that. Obviously, you know, you're gonna have your permission set up and hopefully um that's not gonna happen. But um you know accidents happen and we should design our URLs and um in a fashion that steers us away from those accidents Um they're easily scrapable. Um even if you do have all of your authorization and authentication and permissions all really locked down, you've got a secure site, they still leak information.

If I create a user account for a site and I see I'm ID7852 , and then next week I create another user with another email address And I see I'm 8582. Um, I know that in a week a thousand people have signed up. Um and that's really valuable information for me as a competitor, and it's information that you as the owner of the website wants to keep private. So um IDs um in URLs are bad. Um they're also quite buggy when compared to alternatives, as we'll see. So what's what's what are the alternatives? Well we could use um UDU IDs, some kind of unique universal ID. Um these come in various shapes and forms, a string of random characters that are hopefully not going to clash Um

that's better. Um it solves the enumeration um attacks, it solves the um leaky information, but they're still not particularly meaningful Looking at that URL as a human, I still don't really know what that string of numbers and letters actually means. Um so how do we make UUIDs human readable? Well the solution is really simple but really effective We just put a prefix in front of it. And then straight away, each part of that URL has meaning to me as a human. And this is something we've done and it's had loads of benefits, not just technical. So it's aided communication across the whole company. Because people are used to seeing these numbers in the in the URLs and when another member of uh

staff maybe comes to me for some support to do with to do with um maybe a bug that's happening. Um they'll say it's you know this entity and and they'll know straight away um the relevant information to to pass to me because they're used to seeing these prefix UUIDs Um they avoid some human errors. So supposing someone says, Oh, can you check out this entity for me? And I look at the prefix and I see it's not the right type. You know, they say, here's a user, and there's a A prefix that starts BCK. I know that's not a user at UUID, so I can reply straight away and say, oh, I think you've probably given me the wrong number there Um they help with debugging. So when I'm looking in Sentry or my logs and um I'm looking at uh a page that's gone wrong, straight away just by looking at the URLs, I know what are the key objects that I'm going to investigate here.

Um that that's a really useful thing. This one's a big one. They allow polymorphic lookups. So um if I'm using um an integer or a non-prefix UUID, I can't do something like this I can't have um Django Con tickets where I could put in a user um ID or say a profile ID. Django has no way of distinguishing between those two numbers. This isn't strictly URLs, but a really really cool feature that we have on our back office is we have a search bar. And you can put any prefixed UUID into that search bar and it will direct you to the relevant back office page. Um which is just such an amazing feature. It gives me joy every time I use it. Um I think that's something that every Django site could utilize.

Think about putting that in the Django admin. You just Plug in your UUID and it takes you to the detail page straight away. That would be such a big time saver. It would be a big accessibility boost as well. Far fewer clicks and so on. So how does this work? Well, um any class that we want to have a prefixed um UUID associated with, um we add a prefix attribute and we add a UUID field. And then all we need to do is loop through um all of the models and generate a path converter for each of them. So what does this look like? Well we start off with a a simple path converter Um and this code is actually

it's not too bad. We get all of the Django models using get models. We loop through them and we check for the prefix attribute If there is a prefix attribute, we generate some appropriate regex. We use the built-in type function to dynamically create um a path converter with the correct regex attributes and then we register that model. And now all of a sudden, um where before we had this kind of thing which we couldn't do Now we can. We just say the user prefix um and it's going to it should it just works. Um This is also going to catch us some bugs.

We have a sort of built-in almost type safety to our URLs. Before, if I wanted to um reverse a URL Maybe I do something like that and that works fine, that's great. But maybe I've um been up late and I am not thinking straight and I make a an error and I put ticket. id Well, that's going to work in the sense it's not going to crash and it's going to generate a URL, but it's going to generate a URL that is either pointing to the wrong place or will result in a 404. And this is probably won't get caught until the user clicks on it. With the prefix UUIDs, if I try and do this the um the regular expression isn't going to match and it's gonna raise straight away.

So I'm gonna catch this error much earlier on and avoid those kinds of bugs So um park converters are cool. So let's um see how we can implement them. Um I've been tired, I've got five minutes. So I'm gonna go this through this quickly Basically we create another class called root pattern, which is going to be like a sister class of regex pattern The way it works is it generates the regex based on your Django style roots. I'm going to skip through all of this. We have a function Um you can download my slides and and kind of look through this. It's not too difficult. But it takes your Django style roots and converts it to a regular expression.

Our URL pattern that we set up before was relying on regex pattern, so let's delete that and we'll pass in the appropriate um pattern matcher we want to use, whether it's a regex pattern or a root pattern. And the way that works is well we have both of them and we have these helper functions, path and dpath, that you're familiar with. And all we're doing there is we're saying, okay, use a root pattern or use a regex pattern, we instantiate it, we pass it into our URL pattern, and everything works as before Um so so what next? Well I think with just those basic building blocks we can do some cool things. So here is a ticket that's actually an an open ticket um on um

you go to the Django um you know work that needs being done. Um backtracking URL resolver. And the basic idea that's being asked for is when you get to a view, is it possible to say, oh actually I don't want this view and go back into the URL resolution process Now there are arguments for and against this. I think it's probably a bad idea. But we could do something similar if we really, really wanted to. So let's subclass our root pattern and that match method we could stick all the additional logic we want in there. And so if we have some use case where we really want to actually add some little bit of extra checking once we've matched a path, we can do so. And then all we need to do is pass that to

the path function that we had before. And we've got a system where we can do that if we want to. This isn't something I've done before, but I think all new Django projects, I'm going to start with this and I'm going to re-export Path rather rather than using Django's built-in one. Why? Well there's a a check method um in the um root uh class that um Django utilizes to do URL checks. There's a check method in the URL pattern as well, which is useful. And so I can write my own checks, which is super helpful. Maybe I want to make sure that it's a large company, I want to make sure people aren't accidentally adding in IDs. So I can search for anything that looks a little bit like um an ID.

So maybe there's an underscore ID in the parameter name, and I see they're using an integer. And I can put up a friendly warning to say, check out our docs, this is a better way to do things. And so I think starting projects like this is a is a really useful thing to do. I'm not going to have time to talk about includes. I apologize for that. But the basic idea is that as well as URL patterns We can put UL resolvers themselves in this list. That's what an include is essentially going to do. And we can update the resolve method um so that it becomes recursive and we have a tree of URL resolvers and URL patterns. And that that's how we can include submodules

as you all know how we do. So where does that leave us? There are a few um a few examples that uh I wanted to include, but I knew I wouldn't have time for Um and I used to be a maths teacher before I was a um developer. Um and old habits die hard. So I'm gonna leave you with some homework All of these things are things that I think are useful for most Django sites. They are things that I think you should be able to do with just a basic understanding of what the major building blocks are. and they enable you to kind of ship with more confidence.

So see if you can have a go. Dig into the um the source code, it's not it's not as bad as it it might look at first. Um thank you for listening.