Scaling Django to 500 apps

Hi everyone, my name is Dan and I'm here to talk about how we scale our Django codebase at Thread. In this talk, I'm going to cover what Django apps are, why they are a good way to scale a Django codebase, how to keep it manageable when we have many Django apps. I'll also talk briefly about alternative approaches and I'll finish up with my wish list for app improvements in future versions of Django. Before I get started, I want to clarify that this talk is about managing complexity and code so that you can remain productive as you have more engineers working on the same code base. particularly in a product-driven company where there might be a lot of changing requirements.

This talk is not about scaling to more users, more servers, or improving performance of a Django site, Who am I? Well, I'm Dan and I've been using Django since the launch of version 1. 1. I work at Thread, and in my role over the last seven years, I've worked across the whole stack. from front end to infrastructure, mobile and backend. Back-end development with Django has been about 80% of my work. And what is Thread? We're a clothes discovery service combining stylists and AI to recommend clothes. We're live in the UK and the US with menswear and women's wear, and we're on the web and on iOS. And with about 110,000 in-stock products, we're one of the biggest rangers in the world.

But what about the tech? And why do we have the experience to talk about scaling Django codebases? Well, we have a fair bit of code We have a great machine learning system for recommendations that I'm not going to cover in this talk because it's outside of our main Django codebase. But our Django codebase manages a lot. It manages the whole customer-facing site where users can view recommendations. Users can buy clothes, so it's a fully featured e-commerce platform with checkout, payments, promotions, and so on. We handle the order management, fulfillment, warehouse integration, logistics integrations. There are curation, review, and content authoring tools behind the scenes. We have a CRM, a customer relationship management tool We have marketing automation

and lots more. This all adds up to about 400,000 lines of Python code, around 450 models or about 500 tables Around 1000 URL patterns and that doesn't include the Django admin and we don't have a REST API, so that's not included in that. and about 500 Django apps. This is a big code base, but it's not out of the ordinary, except maybe for the apps. For context, I'm sure many of you will know Sentry. The Sentry code base is a similar sort of size to ours, but it only has 18 apps. So, how do we normally make this many features manageable across this much code? Microservices. We take each new feature, we create a new code

base, set up infrastructure, create a database for it, add an authentication system, build process, testing. Hang on, that's a lot of overhead. In fact, each new service introduces more overhead. There's overhead to create it, overhead to use its features calling between services, and overhead to deploy and maintain it. If we get the wrong boundary, maybe because we think something should be separate, but it needs more data from somewhere else, we need more links between services, and that introduces more problems. The cost of getting it wrong is high. Making this problem worse, many companies, many products, especially startups or companies working in agile ways, don't really know what they're building yet.

They're experimenting and figuring it out as they go. So they're rarely going to get those boundaries right. Our answer is to use Django apps. You don't get the force separation that separate services gives you, so you can't use different languages and different deployments. Scalability can be a little bit trickier. However, you do get some separation of concerns, and I believe you can still scale the complexity of the code and the amount of development work being done on the code base while keeping everything manageable. Above all, you keep the rapid development that Django enables. So let's get into it. First off, let's be clear about what we mean by an app here. I use the term in the same way as the Django documentation, that is not to mean a whole site or a web app, but rather a small component in the site.

These are listed in the installed apps setting in your site settings. Django gives you five or six by default, things like the Django admin. The Django tutorial walks you through creating a polls app. This is what Django says about apps. The first paragraph links to a tutorial on how to package the polls app to be reused, and the emphasis is on reuse throughout much of the Django documentation. Reuse of apps is important. It's how we get useful packages like Django extensions or Django Rest framework, but it's not the only use. At Thread, we use eight built-in Django apps like admin or sessions. We use 19 third-party packages like Django extensions, and we have 496 apps that we created ourselves.

These ones that we have created aren't reused for different purposes much. A few have some common parts like our Money app that provides database fields and utilities for handling money, but most are just the code for one small area of the site. There are three broad reasons that having many small apps is a good idea. Separation of concerns is the main one. You can often hold just one app in your head while working and ignore the rest. It's possible to hold a feature in your head when it's split across many places in the code, but it is harder. Consistency is very useful as well, by making every app look the same, putting things in the same place. An engineer can start work on an app that they've never used before and know roughly how it's going to work. Apps can give us a nice structure to form around for this.

And lastly, a bit more vague but I'll go into more detail later, if you have many apps, you can start to build automation around the app boundaries, things like auto-discovering or auto-testing per app functionality. Let's go through some of the ways in which we use apps to scale our codebase and how we make sure the large number of apps is still manageable. First off, when do we make a new app? The basic guideline that we use is that most new things should be a new app. The overhead of an app is very small and it's easier to combine multiple apps later, if we need to, than it is to split up a big app into smaller ones We rarely need to combine apps, but it's not uncommon for us to wish we had split a feature into a new app when we didn't initially.

Beyond this general rule, we find there are three common patterns in the ways that we're splitting up apps. The first is new features, for example, adding comments to a blog website. We'd typically create a new app for the comments. Most of our apps are like this. Secondly, internal versus external This may differ between codebases, but we have a lot of internal tools and it's very common for us to create an internal tool for an external feature. An example of this is if we added a comment moderation feature to our comments app. We'd likely build this as a new app. Why? The moderation tool is likely to have a different permissions system. Maybe it requires a staff account to use. It might use a different base template to match other internal tooling. Well there might be other pieces of functionality we want, like an audit log.

On the other hand, the user-facing comments app might need caching for performance or other things that we don't need internally. The differences you focus around here will vary, but the important bit is consistency within an app. This app separation would make it harder to accidentally expose the moderation tool to users by using the wrong permissions. The last pattern we see commonly is multi-part features. A good example of this is social sign -in systems. Each one may need some views, a model, maybe an API client. If we make each one its own app, we can more easily isolate their differences and expose each to the rest of the site with a consistent interface, separating concerns and making it easier to scale the number of features, in this case sign-in providers.

While we have 500 apps, they aren't all in one top-level directory. We nest our apps in other apps. We have around 50 top level apps, with the rest nested within these. The tree gets as deep as four levels, but we don't have a limit, we just haven't needed to go any deeper than that. Let's walk through an example. Orders is a top-level app. It's a very high-level area of the codebase. Another example would be accounts. We've then got a few logistics integrations. These are fairly isolated and don't need to be in the main orders app, so we split them out into separate apps below orders. Next up we have orders fraud. This is an internal system for flagging potentially fraudulent orders. It's called orders fraud rather than just fraud, because Django's app namespace is flat.

and a fraud app in orders would conflict with a fraud app in say accounts. Lastly, nested within that we have orders, fraud, postcode flags This is an internal tool that our operations team can use to flag postcodes for fraud checks. It has its own views and models and so on, so we made a separate app nested under the main orders fraud app. And there are a few others there too. So what's this about a flat namespace of apps? When we nest an app, we're nesting a Python module. We're putting a directory of files with an init. py inside another directory with its own init. py, and Python is happy with this. Unfortunately, when we register this with Django, Django only considers the last part of the path.

to be the app name. So for our Postcode Flags app, the name is orders fraud postcode flags. There are utility functions in Django to get a reference to the app or to a model in the app, and you may be familiar with defining foreign keys by the name of the app and model in the app. That's an example of the flat namespace in action We don't want two apps to have the same name, so we tend to prefix our nested apps with the name of the parent app. One of the most important things for us in splitting up our apps is making sure that each works in a consistent way. A key part of that is defining the files in which everything lives. If you've done the Django tutorial or seen some of the internals of Django, you'll know that it encourages using a core set of files or Python modules within an app,

models, URLs, views, forms, admin, and so on. While we tend to use guidelines much more than rules, putting things in the correct place in an app is much more of a rule for us. For example, in a views. py file, Every top-level class or function is a view, no utilities. In a models. py file, every class is a model, no exceptions. In a forms. py file, everything is a form. URLs. py only contains the URL patterns and nothing else. This takes some discipline, but means that an engineer opening an app for the first time knows where to find everything. To understand how the app works and what it does, they know they can probably just read the URLs file and the models

file and have a pretty good understanding. It reduces the chance of surprises, it means less to communicate between engineers, and it's one way that we stay productive Not everything fits into these categories, so we introduce new files where necessary, but we do it carefully, only when really needed. Tasks, schema for our GraphQL API, factories for tests. Reports for our internal reporting system. These are all extra files that we've decided many apps will have, and so should be common patterns, and we have many more. There are still bits that don't fit into those places, and so we do have a fallback utils. py, but at least engineers know that utils will never have any models, views, forms, tasks, and so on in it, so it ends up being pretty manageable.

Moving on to templates. With Django, you can either put your templates in one global directory, or you can have a directory per app that will be searched to find the template. It's worth noting that even when you're using the per app template directories, all apps are searched for a given template, so you do need to namespace your templates within the template directories. My recommendation is to match the app names exactly, but not to nest templates in the same way as nesting the apps. You're essentially matching Django's flat namespace of app names, but using it for templates as well So they'll be unique, but you don't have to type out the full path of the nested app to reference a template in it. An easy example of this is this blog app. It's not nested, the name is blog.

So we can just put the index in the blog directory within templates. In the second example though, we have an app called Comments Moderation. Even though it is nested within the Comments app, our template just uses the app name for namespacing, rather than the full path When I talk to engineers outside of Thread about our use of apps, one of the most common questions I get is, doesn't this all just become a mess of imports? Well this is hard to get right in any code base, but in the same way that microservices force separation with an API, we can apply a similar set of principles with apps, just a Python API instead of one over the network. Open source Python packages often define their public API by exposing things from their main module. For example, the requests package exposes a function

get. But it doesn't expose all the classes and functions involved in doing that request. Much of that is kept private. Normally, this is done by importing and re-exporting in an init. py file. But we can't do that with apps in Django because of Django's startup lifecycle. If you've ever seen an error message about models not being ready for use in Django. That's the same issue. Django must be able to import all the models first before anything that uses those models. And unfortunately this means we can't use any models in an init. py file or anything that they import which pretty much makes them useless. At Thread, we solve this by just putting the public API of an app into an API. py file

This means that to check an order for fraud, I might say from orders. ordersFraud. api import check order. Aside from this exposed functionality, we also define the models and URL patterns of an app to be public. Another potential problem in becoming a mess of imports is how apps depend on each other. We define the public API, which helps here, but we still don't want every app to depend on every other app's public API. Because of this, we have a few guidelines for how apps should depend on each other. Apps may depend on their parent app's public API. The fraud system nested in orders can use pieces from the orders app to understand the order that it may be looking at

Apps should not depend on their child apps API too much. The order system should mostly not know about the fraud system, apart from potentially the entry point to say check this order. Typically there will be just a few lines of code to hook into the child app. In the case of the multi-feature apps that I mentioned earlier, like social sign-in providers, The child apps may all conform to some common API so that the parent can treat them all the same. Apps can depend on other top-level apps. Our top-level apps are usually such core concepts. like orders or accounts, that we do need those interdependencies. And lastly, apps should not depend on other nested apps. This is an attempt to keep things loosely coupled.

While we have these guidelines, we don't enforce them. We look at dependencies in code review, we may encourage refactoring to improve things, but we take a pragmatic approach. Tools like import Linter can be used to enforce these dependency rules, and I would probably consider that on a new codebase. One of our other smaller codebases uses a similar tool, and it was beneficial in finding opportunities for refactoring. With so much code in so many different places, it's easy to run into issues with circular imports, when one module depends on another which depends on the first. We do have our fair share of these, but considering the size of the code base, it's not too bad, and they're often introduced when we've not built an app as well as we should have. All big Python code bases have to deal with this. But the app separation helps us.

Using separate files with well-defined purposes inside apps helps here, because it means that each app tends to have layers to it. URLs. py tends to only depend on views. Views tends to depend on models and forms. Those tend to depend on utils and so on. It rarely goes in the other direction. There should be no need for a model to be aware of a view or a form. When combining different responsibilities into the same files, particularly utilities, business logic, or views, circular imports tend to crop up more. As for dependencies between apps, the guidelines about how apps should depend on other apps help us manage import issues. Moving on to app configs.

If you have an app that needs to do something on startup, you can add an appconfig class and put it in the ready method. This can be good for things that need importing after the models have been imported, or doing other registration and setup processes. If you use signals, this can be a good place to put signal registration too, but we don't use signals at Thread. App configs can also be a way to attach metadata to your apps. This could let you scan your apps and apply tests or consistency checks to each one that matches certain rules. You could, for example, build a test that ensures that all apps marked as internal in their app config apply certain permission checks to their views. Through app configs, you can start to treat your apps less like static pieces of code and more like introspectable units of functionality.

You're probably already familiar with three examples of autodiscovery of functionality in apps. Django automatically finds all of your models if they are in a models. py file in the top level of an app. The Django admin also finds all configuration in admin. py files. And tests are also auto-discovered by PyTest or UnitTest, but this is based on file name, not based on being in an app. You can build on this same functionality yourself. If you find yourself writing similar pieces of code in many apps, but having to manually import these everywhere is a pain or is causing circular import issues. You may be able to use Django's auto discovery utility to simplify things. We use this to find all reports across our codebase.

Apps can define reports, simple classes that query the database to generate some stats and display on a graph. One central reports app uses the auto-discovery to import all of the reports. py files so that it can provide a list of all of the reports in our internal tools and so that it can run them on a schedule. This is great because it lets you define a report in the app that it relates to, right next to the models that it might be reporting on. We have several other things in our code base that do this, but all of them have this same benefit, moving functionality into apps so that engineers working on a feature can focus on that feature alone and not worry about how it may hook into other parts of the codebase. ThreadWhite labels our service to a large fashion

brand, and they use about 60% of our functionality and have a bit of their own custom code as well. This presented a big engineering challenge. How could we turn off a large number of features across the code base without the pain of adding flags to each one and without adding loads of overhead to new feature development? Because of our use of apps, we were perfectly set up for this already. We just used a different list of installed apps in the Django settings. Unfortunately, this doesn't just work. Firstly, apps can import from other apps because they're all just Python modules. When you do an import, Python doesn't know that you might be importing an app that isn't installed. Secondly, Django URL patterns used in an include are imported regardless of whether the app is installed or not

Those URL patterns will import views, which import more things, and suddenly most of the app is there and the URLs and views will work even though the app isn't installed. Lastly, test runners like Django's UnitTest-based runner or PyTest will find test files by name, ignoring whether they are in an app and whether that app may be installed This means you will end up running tests for apps that aren't installed, and those may use models that therefore aren't in the database. Since we had such granular apps at the level of all the functionality we wanted to disable, we decided to go down this route anyway, and we created a series of utilities to help us make it work. When using a lot of apps like this, the fact that they bundle up models and related functionality means that they make an ideal sort of plug-in

system. How do we do this? We created a custom test runner to exclude tests from apps that weren't installed. We created a custom URL include function that checks if the app is installed before including the URLs. And we created a utility function to check if an app is installed that we can use to enable or disable connections between apps. That's it for how we use Django apps, but I think it's important to note that this approach may not be right for everyone. I hope everyone can learn something from it, but there are alternatives that you may wish to consider. Another simple approach is to have a roughly similar structure of nested components, but just not make them all Django apps. This is what most non-Django Python codebases do.

This is suddenly possible, but it requires a bit more work. For a start, Django's model and admin discovery won't just work. You'll need to do some plumbing for it. It may also introduce more import issues as Django won't be managing as much of the import lifecycle via its startup process. You'll also lose the benefits of things like custom feature auto discovery, although you could build this with Python modules if you wanted to. There is one other disadvantage. Building a plugin system like I mentioned before may be a little bit more difficult or require more third-party libraries to make work. Another alternative is to slice functionality horizontally rather than vertically. Rails is a good example of this.

All models from the whole site go in one place. All views go in another, and so on. The advantage of this is that maintaining the separation between these layers is easier, and you don't get the same problems with dependencies between apps. Enforcing separation with tools is also easier. The disadvantage is that to hold a whole feature in your head at once requires looking across large parts of a code base, and at pieces that may be intermixed with other code. In practice, Threads App structure ends up doing the layering within apps and the separation outside of apps, whereas this approach is the opposite, layering at the top level and then separating within each section We prefer the trade-offs that our approach gives us, but do consider this approach and figure out what works for you

I'd like to finish up by covering a few issues that we have had along the way with apps and some things we'd like to see in the future. You might have guessed this one based on what I've mentioned before, but we'd really like apps to be namespaced in the same way as Python modules It's confusing enough that apps and the Python modules they live in are separate concepts. On top of that, it's annoying to have to maintain uniqueness in app names. There are a few ways this could be done. It could be a braking change, but we're not keen on this. It makes Django upgrades hard. It could be an opt-in feature. Or it could be something that's implemented via the app configs in some way. As app configs are a slightly more advanced feature, this is probably the best way to do it while maintaining backwards compatibility. As mentioned earlier, being able to emit an app from installed apps and have the code

base behave as expected was a big productivity win for us with our white-labeled service. Unfortunately, it wasn't all smooth sailing. We'd love to see Django provide more support for this case along the lines of these modifications that we've used. We know that this is not necessarily a common case. But white-labelled services aren't rare, and Django apps provide a great boundary for splitting up which features are included or excluded This could also have benefits for third-party apps on Pypei, which might be able to enable or disable integrations with other third-party apps. Django already does auto-discovery, and we can use the built-in utilities to do this, but they're not enough by themselves. A common pattern that we do is implement a base class with a meta class that maintains a registry of subclasses.

say a report base class. Use Django's Auto Discovery to import all files from apps with a particular name, say reports. py. and then access our registry to get all the classes. This is common enough that we have our own set of utilities to do this. This might be a bit specific for Django, but there's certainly room for an open source package to do something along these lines. On the other hand, a refactor of Django to use one unified discovery system that could be exposed to users would be really neat. So to bring this all to a conclusion, granular use of Django apps can enable a better separation of concerns. This can make it easier to understand a whole feature. Apps are a good layer at which to encourage consistency.

If every app looks the same, engineers can easily learn new areas and it's easier to review and test consistent patterns. And lastly, apps can enable pluggability of functionality. Auto-discovery is one mechanism for this that we found to be particularly powerful. Optionally installed apps is a bit more niche, but maybe useful for those of you white labeling your sites if you're up for a bit of a challenge. Between all of this, using lots of Django apps in your code base is a great way to manage complexity, to delay the need to move to microservices, and to keep development productive. I hope this talk has shown you some ways in which you can make Django apps work for you.

That's all I've got time to cover. Thank you for listening and I hope you've managed to get something out of this that you can use to manage complexity as you continue to scale your Django code bases. I'm Dan Palmer on Twitter and if you have any questions feel free to reach out. Thread is also hiring software engineers across a number of disciplines. We do a lot of Django, as you can hopefully see, so if you're looking for a job, we'd love to chat.