Managing multiple Django services in a single repo | Benjy Weinberger

Hi

Speaker 1: everyone, thanks for joining, and welcome to my workshop where we will talk about why it's often a good idea to manage multiple Django services in a single repo and how to do so effectively. And you can follow along with the code at this URL here. I'll just give you a second to copy it down if you want And a little about me. I've had many years experience as a software engineer. I've had the good fortune to work at some cool companies. I'm a maintainer of the Pants open source build system. I'm the co-founder of Toolchain, which is a startup in the build system space, and I'm a longtime user of Django, over 10 years now at this point.

Speaker 1: And a little about the structure of this workshop. We're going to cover five main areas. First, we'll talk about the motivation for the whole thing, that is, why Uh you should want to know how to run multiple Django services. Then we'll discuss why you would benefit from having those services coexist in a single repo. Then we'll talk about pants, which I've already mentioned. It's a build tool with a lot of features that make it a great fit for monorepos in general and Django-centric ones in particular Then we'll do a deep dive into the example code. First we'll give an overview of that example and then we'll get into all of the gritty details. So let's jump right in to motivation.

Speaker 1: So I don't need to tell this audience how useful and well-designed Django is. But Django kind of ends at a single service or in the Django terminology a project. If you follow the tutorial or the documentation. There's really not much there about how to manage and develop multiple services. So you kind of end up just having multiple single standalone services and you just do the same thing multiple times. There isn't really much in the Django world for how multiple Django services coexist and interact. But in reality we often need multiple services. So, for example, um this often comes into play where you start out your project with a single service because it's easy and

Speaker 1: natural and it makes sense and why have a complicated architecture before you know that you need it, right? Don't optimize prematurely. However, if your project is successful, you start adding more and more functionality to it And eventually you end up with this big bloated service that can become hard to modify and hard to deploy and very hard to scale. Like, for example, how do you performance tune a server that's doing so many different things with so many different parameters and constraints? And I personally saw this in action at Twitter where famously For a long time the entire service was a single giant Ruby on Rails app, and just getting it to scale to the amount of traffic was a nightmare. And if you remember the fail whale from about ten years ago, that was a big part of it.

Speaker 1: and Twitter have have themselves written multiple blog posts about this. So naturally you end up wanting to break your monolith up into multiple services, so maybe the monolith just becomes a front end that routes requests to a bunch of back-end services. Alternatively, maybe you are all in on multiple services, uh say even microservices from early on, but then you often find yourself with each service living in its own repo. For example, uh both the standard Django layout and the standard Python tooling kind of push you in this direction, they tend to be top-down. And if different teams own different services, then it's tempting to give each team the independence of their own private code base. But breaking your code base up over multiple repos

Speaker 1: has some major costs that we will look at in a moment. So say we agree that we want multiple services. The naive way to do that is to have multiple Django services where each one is completely independent. Each service has is standalone with its own settings and no code sharing, whether that's in a single repo or multiple repos. and no easy way to reason about dependencies. And this is, I claim, what we don't want. It's hard to share code, it's hard to manage changes, these servers are supposed to interoperate. They are part of the same code base. uh in practice even if you try to treat them separately. So what do we want? We want services to share code and settings as appropriate

Speaker 1: We want to be flexible about how we deploy them, and we want our code-based architecture to be independent of our deployment architecture. Code-based architecture should be driven by best practices at deplo at development time, not by what you happen to be deploying. Whereas deployment architecture changes over time and it's hard to to to make changes uh when databases are involved. So we want to set ourselves up for success. And obviously we want to avoid things like a lot of copy pasting. It's really important for maintainability because once things start to drift, it's really hard to reconcile them and then you know suddenly you find that your services that are supposed to be mutually compatible aren't so much anymore. So we've talked about what

Speaker 1: we don't want and we've talked about what we do want. And so some of the benefits that we will demonstrate In this workshop are how your multiple services can have shared settings and shared dependencies, how they can have minimal boilerplate how they can have easy dependency management, and something that we will look at shortly is database first design, how to design around your data So having talked about the motivation, let's talk a little bit about monorepos. Because I've claimed that a single shared repo is the right way to have your multiple Django services coexist. And that brings me to a discussion about monorepos, and we'll just take a few minutes to look at what they are

Speaker 1: and why they're desirable for code in general, not just in the Django services context. So a common characteristic of codebases is that they grow over time. And when I say codebase I mean all your organization's code, regardless of whether it's in a single repo or multiple repos. It's all one code base. You have an organization that's responsible for a set of code. And codebases grow because each developer keeps adding code over time, but also if you're hiring, then you're also adding more developers who are each adding code over time. So the codase can grow uh more than linearly over time. But a common consequence of this growth

Speaker 1: is that builds get harder, they get slower, they get less manageable, and they get more fragile. Now, I can already hear an objection, like this is Python, it's not a compiled language, so what do you mean by builds? And I should emphasize that I mean builds in the general sense. Every step of the way from source code to deployable artifact. So for example, all the tools and processes you run in order to resolve and download external dependencies, to generate code, to run type checking, of course running tests is a huge build step. uh debugging, running linters and formatters, actually packaging your code into some deployable artifact. These are all build steps, even if there's no compiler in the Python sense. So these builds get harder, as I mentioned, and what do we do about this?

Speaker 1: As your organization and your code base grow, We have to choose how to manage that code base in a scalable way. And we have two architectural alternatives that I refer to as multi-repo versus monorepo Multi-repo means you split the code base into growing numbers of small repos, often along team or library or project boundaries. And this often happens because it's the path of least resistance. It's uh the you know the Python tooling kind of nudges you in that direction, but there is an alternative and that is monorepo. And by definition, a monorepo is a unified code base that contains code for multiple projects that share underlying dependencies and they share functionality and tooling

Speaker 1: and so on. And a monorepo may contain code in multiple languages and frameworks, even though today we're focusing on Django. Different subteams in your organization will work on different, but often overlapping parts of the Monorepo because they have shared dependencies. And as your organization grows and as your code base grows, your monorepo grows right along with them. Now I should emphasize these both have mono in the name, but Monorepo and Monolithic server are not the same. Monorepo refers to code-based architecture, whereas monolithic server is deployment architecture. If you remember earlier I mentioned those two should be completely separate. So a monorepo is agnostic to whether you deploy monoliths or multiple services.

Speaker 1: obviously what we're discussing today and microservices. And in fact I claim that monorepos are often great for microservices because they can greatly reduce uh boilerplate and repetition and we are going to see many examples of that later. Now I will admit that multi-repo does sound better at first when people are presented with these two options. They think, well, multi-repo is more decentralized, uh it um it's tempting because Every team can have autonomy over its own repo and I can put a barrier between, you know, my little repo and all those other barbarians out there. If they want to make changes to my code, they have to come through me. But it turns out that for some core problems in code-based management, multi-repo

Speaker 1: doesn't solve the problem, but it hides them, so it actually makes things worse. Whereas a monorepo makes these core problems explicit so that at least you can tackle them head on. Now The core problem, or one of the hardest problems in code-based management, is managing changes in the presence of dependencies. Each of those things is hard on its own, but that intersection of managing uh changes in the presence of dependencies is where a lot of code-based pain lives, and I suspect many of you know what I'm talking about. So let's look at how these challenges are handled in a multi-repo world. So in a multi-repo world

Speaker 1: A repo has no idea which other repos consume it. If you think about it, the dependency information lives on the side of the depender, I depend on A, not the dependent. There's no metadata on the dependee side to say who depends on it. And note that dependency here can mean either a direct code dependency, so literally some code in repo B imports code from repo A and so presumably Repo A publishes artifacts like a wheel into some internal repository. Or it could be a service level dependency where they still have to agree on an API and then communicate over the network and so there is still uh dependency here. And because a repo has no idea which other repos consume it, you can make your changes in that repo without really being aware of the impact.

Speaker 1: of those changes. So if you want to be a good citizen uh when you make changes in your repo, you have to somehow find all the repos that depend on my repo and remember there's no obvious way to do that. You have to fix those repos to be compatible with my changes. And then because you've changed other repos, you have to continue recursively. So I've changed all the repos that depend on my original change, but now what about the repos that depend on those repos I've changed? So my changes have to percolate uh recursively through possibly many, many of these multi-repos. And this is a lot of work So if good citizenship is a lot of work, then very often what we end up with is bad citizenship. And that is where people just say, well, I'm making changes in my repo.

Speaker 1: It's self-contained. There's versioning, there's a change log. I will let other people uh figure out how to upgrade uh to a newer version of my repo. But this can lead to all sorts of dependency hell problems of incompatible dependencies, especially when you have multiple dependency paths onto the repo that you changed it may not even be possible to find a single version that is compatible with all of the intermediate changes. So essentially you end up kicking the problem down the road for other people to deal with in the future. And this happens to some degree, has to happen to some degree with external dependencies, but we're talking about dependencies within your organization. So you are

Speaker 1: making life harder for your coworkers and that's not great. Um by the time they get around to trying to upgrade, they may not have the context on how to do that, you may have moved on This is, as I mentioned, bad citizenship. But contrast that with a mono repo. All of the consumers of your code, and therefore everything that's impacted by your change, is right there in the same repo. So you can easily find all the consumers of your code using RipGrep or whatever your tool of choice is. You can run all the tests in the repo or all the relevant tests to ensure that your changes are good. Breakages are immediately visible. You have, you know, dependency management with changes is a problem, and a monorepo

Speaker 1: makes it very visible. Now Of course, we're talking about multiple services and you still have to be careful when deploying so that services remain compatible with each other across upgrades But overall, the Monorepo codebase architecture enforces responsibility and good teamwork, and it makes those deployments more likely to go smoothly. So I claim that monorepos are more flexible, uh easier to refactor, easier to debug, easier uh on code reuse, you have a single unified change history, and Of course, as we showed in this extended example, much, much better for managing uh changes through dependencies. So even though it seems counterintuitive, monorepos are actually more flexible than the alternatives.

Speaker 1: So, before we move on, I just want to make one more point about code-based architecture and how it affects your organization. There are many cases where you want localized decisions and creative chaos, but an organization is a top is a bottom-up, not a bottom-up kind of thing. Like in a well-functioning engineering organization, priorities and decisions and effort allocation flow top down. And I claim that a code base is a reflection of the organizing principles of the organization that owns it. If you have a fragmented code base, you may end up with a fragmented organization. And this is why many large companies have adopted the Monorepo architecture.

Speaker 1: in order to keep their organizations unified with a single purpose, even at huge scale. And the obvious example here is Google, where most of their code is in a single huge humongous repo that tens of thousands of engineers collaborate on and have for the last twenty years or however long it's been. Um So I leave you with that thought about monorepos. And that lets me um segue into a short conversation about um build tools for monorepos. So we've talked about monorepos. uh and why they are desirable. But now the question is how do you work effectively in a monorepo? So The standard Python tooling is wonderful.

Speaker 1: There is a tool for everything. There is PyTest for running tests, and there is MyPy for doing type checking, and Pip for installing dependencies. and um m you know linters and formatters and so many great tools. But they are not designed with monorepos in mind. For the most part they expect to run On all of your code, and they expect that code to not be too big. So when you use those tools naively, they do a lot of repeated work small changes will trigger full rebuilds. And so as your code base grows, so do your build time, so does resource consumption. And um We, you know, as your code base grows, you really want to speed things up.

Speaker 1: So there are two main ways to speed things up. One is do less work, and the other is do more work at the same time. Now to do less work you need two features. You need fine-grained invalidation so that The build is smart about only doing the work that has to happen. And caching so that if any piece of work has been done before by you or ideally by anyone on your team, you don't have to do it again And then to do more work at the same time, you need a system that can reason about concurrency so it knows which uh pieces of work depend on each other and so can't run at the same time versus which are independent and can.

Speaker 1: And ideally you also want uh remote execution so that instead of being limited uh on concurrency to just the number of cores on your laptop uh or your CI machine, you could run potentially dozens or hundreds of units of work at the same time. Like imagine all your unit tests running at the same time because you can run them on some remote cluster with uh many cores. So To work effectively in a monorepo and speed up your builds, you need tooling that has those kinds of features. And so these are build systems that are designed for monorepos. Now these tools don't reinvent the wheel. They sit on top of existing standard tooling, like the tools that I mentioned, but they orchestrate them for you. They decide when

Speaker 1: to run them And so there are a few tools in this category, but the one I'm going to focus on is the one I'm most familiar with, which as I mentioned is pants. We're going to give a very quick taste of it just enough to understand some of the examples that we're going to be looking at soon. So um Pants um command line supports requesting what we call goals on specific inputs So you don't say run a pie test, you say, I want to see the results of uh these tests. Um So there are all these generic goals like test and lint and package, and the system translates those requests into the necessary execution of underlying tools like PyTest

Speaker 1: or setup tools or Pylint or whatever And this is important because of invalidation and caching. We need some layer between what the user wants to achieve and the expensive part which is actually running processes. So these are some examples of Pants command lines. You can run on a specific file, you can run on some glob over multiple files, and you can even run on say just the files that have changed in Git, which as you can imagine is pretty useful. Now Monorepo build systems like Pants need to know about dependencies in your code Some tools require a lot of explicit metadata. Pants is able to mostly infer those from looking at your import statements. Occasionally you have to add manually

Speaker 1: add dependencies. uh when things can't be inferred from imports. And in Django that's actually uh pretty common uh if you think about it because so many dependencies in Django are kind of implied uh by naming conventions and so on. Now apart from code dependencies, um build systems uh like pants also maintain task dependencies, which is um Understanding what units of work exist and what the data dependencies are between them. And this is important that you don't rely on file system side effects like tools like Make do Instead you have this explicit uh contained mapping of outputs to inputs and the system maintains this uh

Speaker 1: what we call a rule graph, which is understands in order to produce this output I need this uh these inputs and you can form a graph that um maps uh outputs to inputs. Um So, uh and of course you can customize this and add your own logic because so many uh teams and organizations have custom build logic. So once you have code dependencies and these uh task dependencies, you can at runtime construct a workflow, and the workflow actually runs through all the rules applying initial inputs through to intermediate outputs all the way to final outputs which are what the user requested. And the important point here again is that this workflow is all side effect free.

Speaker 1: You uh can't rely on like magic files existing at certain locations and so on. Everything is explicitly mapped. And it's this explicit modeling that gives us those four features, which, as I mentioned earlier, is what makes builds scale with your code base. Now Remote caching and execution is a hard problem and that is uh parenthetically what my company Toolchain is working on providing as a service. Uh the capabilities here are all available in the uh Pants the open source system. But to actually run a remote execution cluster, that is a separate service. So we've set up a lot of background. Let's start looking at our example.

Speaker 1: So first let's give an overview. We are building the world's most elaborate, over-engineered uh hello world system. And again, here is the link to the code in case you missed it the first time. Um yeah, with this is by way of example, so obviously it's an extremely contrived example. Um what does it do? So basically the system will pick a greeting from a database based on the time of day, so you know, good morning, good evening, good night. It will translate the greeting into the language the user chooses. It will apply the greeting to a person. We have a database with some Sherlock Holmes characters in it, and you pick one

Speaker 1: And then it renders this greeting sentence, so something like, uh good morning, Sherlock Holmes, or Good evening, John Watson. Now, in reality, of course, this is extremely trivial, and you would never have multiple services to do something this dumb, but this is all for demonstration, so of course we've made it massively overcomplicated. Um I mentioned earlier a database first architecture, and um this is pretty important. We believe that you want to treat your databases as the first class thing that are not owned by any app or any service. Um and the slogan is that code is for now, but data is forever.

Speaker 1: And what I mean by that is that migrating databases is significantly harder and riskier than modifying stateless services. So it is n probably not good to think of a database as sometimes people do as an implementation detail of a of some functionality or of some service where, you know, the first class thing is the services API and the database is an implementation detail. I tend to believe in the opposite, that the database is the first-class entity and you mold your services around that, because the services may change over time, but the data does not, or at least not easily. So you start with designing your data model and how you distribute it among databases and you go from there. And so For our example, we're going to demonstrate multiple databases.

Speaker 1: We're going to have two here. One is the user's database, which contains user-related data And the other is the greetings database, which contains greeting-related data, like those different uh greetings and their translations. And This makes sense because the user-related data may be reusable in other contexts for other functionality. And It's you may often find yourself at a large enough service wanting multiple databases, uh, for example. They're easier to manage and provision. They might have different replication needs, different scalability concerns, certainly different tuning for you know different access patterns. And working with multiple databases in Django

Speaker 1: um can be done. Uh it requires a little trickery and we'll demonstrate some of that trickery. It's not completely straightforward, but it is certainly possible, and um if you think your service is going to get more complicated and elaborate and need to scale over time, then I recommend starting out with multiple logical databases because it is kind of hard to tack on later So we've talked about databases. Now we're going to assume that we need separate services. There are kind of three main pieces of functionality in our example. One is handling users , the second is selecting and translating greetings, and the third is rendering the UI to the user

Speaker 1: And so oops, the um architecture is uh you can see it here, the user hits um a front-end service And the front end queries the uh welcome and users backend services, which in turn talk to the databases to do their work. There's also the admin service here, which is the just the regular Django admin service. We will talk about that later, but we recommend deploying that as a separate service, not tacked on to an existing one. Um and having these multiple services, you know, again, you may have different deployment cadences for different functionality. Uh maybe UI engineers want to be able to iterate on the front end without uh having to redeploy

Speaker 1: the data services. So basically all the usual reasons why you might need multiple services. I assume since you're attending this workshop that you're already somewhat convinced that you need multiple services. So again, this is kind of roughly what the architecture looks like at the service level. And again, services, I'm using the word services for what Django sometimes refers to as a project. So basically a thing that listens on a port and has a settings. py, etc. And we we'll get into that uh shortly. But as we know, Django services are composed of apps. So in our example, what apps uh do we have So we have four apps, and the apps here are in the ovals. The welcome backend

Speaker 1: has the greet app, which manages the possible greetings and their association with the time of day We have the Translate app which knows how to translate greetings into various languages. Then for the user's back end, we have the person app which manages selecting a person to greet And then the front end has the UI app uh which manages uh rendering. And of course uh not uh shown here are all the standard Django apps like uh content types and auth and so on and we'll see uh where those slot in soon. So that has been an overview of uh this complicated overcomplicated system. for doing a simple thing. We've talked about the databases and the services and the apps, which are kind of the building blocks of uh Django

Speaker 1: architecture. And now let's get right into some of the grittiest uh details. Like we know what we're doing, we know why, we know what tooling we want to use, and we know the overview of the system we're putting together. So now we're going to look at some code snippets and see various design choices and tips and tricks that we've found make sense for managing multiple services. Now I should emphasize that none of this is rocket science, and you could probably derive these ideas yourself if you took the time. But sometimes it's nuanced, sometimes it's tricky. And when you put all these ideas together, you end up with something useful and powerful where you can really scale a system on top of Django.

Speaker 1: Now we'll be looking at simplified examples, but you always have the option to make things more complicated if you need to, because it's all just Python code. So let's start by looking at code structure. So you can see here kind of the um a couple of levels of uh just the uh repo structure. Um the lines in blue represent apps, so that's just you know Django code that is disconnected from any service And then the code in brown are the four services that I uh talked about. And this code structure makes it clear that apps aren't owned by services They aren't, you know, they don't live in a directory that also has service level things like a manage.

Speaker 1: py and a settings. py. And so this is a little bit different from uh what often happens in in common Django conventions. And it makes it clearer that you can reuse apps in different services if you need to, although we're actually not doing that in this simple example. But it creates a the just the code structure itself enforces this distinction between an app and a service which uh pulls in the apps that it needs, but it doesn't own them. It is not that they they don't live underneath it So let's look at apps. So here's one of the apps, the Greet app. These are the files in it. So that looks like a normal Django app, right? There's not much to see here yet. And each of the other apps looks like this as well.

Speaker 1: One thing to notice here is conftest. py, which we will talk about in a little bit, how we use that. So this is what an app, an example of one of our apps looks like. And this is what one of our services looks like. So these are the files that have to do with running and managing a service. And of course that all starts with settings. py. Now remember that Services um we have multiple databases uh and uh the manage. py which you know there are various database related uh commands uh there are at the service level. And so it's important to remember that some managed.

Speaker 1: py commands like say create superuser and migrate and db shell will require you to specify the database name with the dash dash database command line flag And that's actually a good thing so that you are always aware of which database you're connected to. Because as we'll see, we are not going to rely on the default database. We're going to require things to be explicit. So We've talked about the services in general, and I'd mentioned earlier that we have a separate service just to run the Django admin. And this is because we uh may want to lock that server down extra tight from a security perspective, because like that server will have access to all the databases. And it registers all the apps and it needs a bunch of middleware and other apps that the admin, the Django admin site requires.

Speaker 1: So the admin server is a little bit different. You can see it like sets up all the databases. We'll get into kind of how this database setup works in a little bit. And um it also does define a default database which it just points to the user's database and the reason for that is uh that the admin site expects to authenticate against the uh default database and we can't really change that. So uh that's what we got. So, let's start with the beating heart of any Django app, which is settings. py. Now The easy thing to do is for each set service to have its just a complete standalone settings file.

Speaker 1: But that's obviously not ideal because of all the repetition. And that gets unmanageable pretty quickly when you have to copy-paste things across multiple settings files and keep them all in sync. But settings are just Python code, so you can import them. So we like to do this by having a settingsbase. py that lives kind of somewhere near the root. And that has most of the settings that are kind of repeated across all of our services. And then in an individual service, we import all of the settings from setting base and then we add the ones that are specific to this service. I know this is not super innovative, like this trick is commonly used to uh have different settings for like development versus staging versus production, but we're leveraging it very aggressively here.

Speaker 1: Now, note that you have a lot of freedom to decide what goes in the settings base versus what goes in the individual settings for each service For example, maybe some installed apps are shared across all services and it makes sense that they go in the base. That's not the direction we went in here. Another thing to notice is that Many list-valued settings such as installed apps and middleware order matters in those lists. So there is a little bit of nuance here. So the per service settings. py, you know, um here we just append some middleware um and and but You might have to prepend or insert in the middle. So you might decide that's not worth the complexity, and instead you'll remove some elements from the base and repeat them in each service's settings to

Speaker 1: kind of avoid error So you have a lot of choices here. But the important thing to note here is that in this settings. py we import most of the settings from Settings Base and we just set the important things that are specific to our service, such as the root URL conf and the actual apps. Now, uh we saw a little example earlier of uh setting up databases for the admin. Let's return to that. The base settings file doesn't actually contain database definitions at all, as you can see here. Instead, it contains a function that the derived settings can use to set up databases The one thing is we do set up a default database here, even though it just points to an empty dictionary, because Django requires that to exist, it will crash if there is no key

Speaker 1: uh named default in the database's dictionary. Now In your derived settings, you call this setup database function and with a database name and it does the setup for you that gives you access to that database Now this is very simplified and we're only using SQLite , but you can probably imagine how to extend this to support Postgres say or whatever database you're using. and how to distinguish between development and staging and production databases and so on. So yeah, in this specific service, you know, you just call setup database and you give it the logical name of the database. And now this lets the service opt in to the logical database with that name.

Speaker 1: And so what that does is It creates this logical database named greetings in this case, we talked about it earlier, and it wires it up to a physical one, which is that SQLite file in this case. But how do database operations know which database to write to? So one way to do it is that every query set can call the using method. And you give it the name of the database, but that is laborious and easy to forget. And if you do forget, then it will try and access the default database and that doesn't exist. So bad things will happen So instead we use this really neat Django feature called database routing. So in Settings Base we set the database routers setting

Speaker 1: And database routing is an advanced Django feature that automates the decision of which database to use for each operation. It delegates to a list of objects that implement various functions, so you can read about it in the documentation. It's a bit too detailed to get into the specifics here, but you can see the code for our database router in that GitHub repo. And I will just mention that getting database routing right is is trickier than it looks, and so you have to write your router classes with care. And what we've done To keep things not too fragile, is to create a router that makes decisions on a per app basis. That is, it maps each app to a specific logical database And all queries involving the models in that app always go to that database.

Speaker 1: And that you can see that mapping here, right? Some of the um apps. go to the users database and others go to the greetings uh database. Now of course if you want to join across models from different apps because say you have a foreign key For example, we have a foreign key from one of the translation model into the greeting model, those have to live in the same logical database. And the router will enforce that And also note that we have to map every app, not just our ones, but even the standard uh Django ones like content types and sessions. Now, in this simple case, this means that an app can only map to a single logical database globally. But you may need an app to write to different databases in different servers

Speaker 1: because you're reusing the app. So in that case your database routers need to be specific to each server, or there needs to be some other logic to differentiate So it things can be more complicated than this example. Now, if you need the same app to write to different databases in the same server, then I would highly recommend that you not do that and just change your architecture to avoid that. That is a world of pain. So we've talked a little about how we work with multiple databases in a fairly streamlined way. The next thing we want to talk about is how do we avoid a ton of boilerplate? Every service usually needs a bunch of boilerplate for like for running management commands, which is manage. py and for running a production server through WizG, um

Speaker 1: say you're using uh GUInicorn or whatever. But very often the only difference uh If if you actually kind of look at this boilerplate that Django generates for you, very often the only difference is the settings module, which is determined by the Django settings module environment variable. And so we can easily minimize this boilerplate with a helper class. So we have this class called service. Again, you can see its source code in the repo. And It understands how to set settings and then run the underlying command that you requested. So note that we initialize this service object with the path to the current file And then inside service in its constructor, there is this logic to go from whatever that file was

Speaker 1: to the module name for the service, which we then store And now we can turn that module name into a reference to uh a settings uh module. Uh so in this the run manage function, which is uh you know part of that service object, all we do is set up Django settings module to point to uh the settings for uh that service and then we just call the into the regular Django uh management uh which is this is just a minor variation on what the standard manage. py does except the standard one hard codes the setting module uh when you generated it and here we use a little bit of trickery to deduce it and so we reduce all this boilerplate

Speaker 1: And we use a similar trick to run the production service. We point Unicorn at a generic Django wizg. py file, but first we uh set the settings Now we also pointed a generic Unicorn config file as you can see here. And of course if you need per-service unicorn config then you can add some logic here to you know detect that file and point to it if it exists. or fall back to a generic one. If it doesn't, then you know you obviously this is simplified and you have the freedom to be more complicated if you need to. Next thing I want to talk about is development ports. We mentioned manage. py, and as you know, we commonly use manage. py to run the development server using the run server command. And RunServer uses port 8000

Speaker 1: by default, but we have multiple services and we may need to run all of them while developing, so they can't all use the same port. Now you could manually provide the port number as a command line argument to run server, but A that's a hassle. And B, the services still need to find each other. So um Right, because we have the UI service is uh the the front-end service needs to uh talk to the back-end services. So just kind of ad hoc inventing port numbers is not uh for at development time is not gonna work So we introduced the concept of uh development ports where each service is allocated a different port. You can see it in the code here. And obviously we make sure these ports don't collide. And now we can use this get dev port

Speaker 1: in two uh different uh places. So The first is in uh the service, uh, we have a little hackery here to um detect when you're trying to run a run server with no argument. and we add the dev port argument for you. So this is the same run manage function we saw before. I've just added a couple of lines here to set the dev port for you. So that's super convenient because it means you can just do run server and the right port gets selected. Um and then um the other place you can use um the dev port is to locate the back ends.

Speaker 1: So this obviously only works in development mode. You can see that we throw an exception if you're not in development mode. But of course you need something similar for production mode that would be highly dependent on how you deploy and it's probably not going to use port numbers in a simple way So we've talked a little bit about development. Let's talk about testing. Now we want to be able to test logic in our apps independently of any service, which means independently of any settings files. So, right, we don't want to use the settings files which are a service level thing in tests for apps, which are an app level thing. So we need to be able to generate minimal settings just for use in tests. Now this is quite different from standard Django testing using the the test

Speaker 1: runner, because that thing really wants to use your services settings and it's it's not very self-contained. But remember that we use pants as our build system and it uses PyTest to run tests and PyTest is very powerful so we leverage that. And the way we do that is by using PyTest Django, which is a PyTest plugin that provides support for Django and in particular for setting up databases. And I mentioned earlier that we would come back to that conftest. py file, and here it is. Conftest. py is just a standard file that PyTest uses for test setup. It can contain hooks. And we use that to do a little bit of setup by just calling configure settings, which is a helper function we created

Speaker 1: that will set up minimal Django settings. for uh running tests. And here's what configure settings looks like as you can see It just sets up um minimal um per app uh settings for the tests uh in that app And of course if specific apps need extra settings then you would have to pass them into this function. We don't have an example of this here, but you could easily see how you could pass in extra settings and then pass them through to settings configure. One nice thing to notice here is this concept of an execution slot. So Pants runs tests in parallel. If your machine has eight cores, you might be running eight tests at the same time.

Speaker 1: And if you don't want them accessing the same database, typically. So Pads will set an environment variable with a slot number, and then you can grab that here. and use it to modify the database that the test accesses so that each test has its own database. And also note here that we just use the default database because and we're not bothering with database routers. Because the unit test involves just a single app, or maybe the apps it has foreign keys to, but we know that all those live in a single database. So we may as well just keep things simple when we test at the app level In the test file itself, we annotate the test functions that need database access, we use this mark, you know, PyTestmark Django DB, and this sets up the database and runs migrations before the test runs.

Speaker 1: And this allows you to just use these nice PyTest style functions instead of the unit test modules test class, because which means you can use all of PyTest's wonderful advanced features like fixture injection and the magic asserts and so on. And so to run the test you use this uh pants command, just pants test colon colon, and colon colon means run over the entire repo. But of course you can run on specific files or just directories if you want to. And this is running the tests in parallel, each one with its own conftest. py and its own database completely isolated from each other. You can do a similar thing with linters, since I'm demonstrating pants commands, you can run multiple linters concurrently. You can see here that it ran black and docformatter and flagate

Speaker 1: and iSort all at the same time. So finally let's talk about packaging. We've talked about developing and testing, but we need to deploy. So we deploy each service as a PEX file, and PEX stands for Python Executable. And a PEX file is a standalone executable file that contains all the Python code necessary to run the service uh both the internal code from your repo and all the third-party requirements. And Pants is smart enough to bundle just the necessary code using the dependency graph information. So you don't end up with every requirement from your requirements. txt in the PEX file, just the ones that are actually needed. So you can think of a PEX as like a smart, compact virtual env in a single file.

Speaker 1: And the only thing that isn't bundled into it is the Python interpreter itself. But when you execute the PEX, it knows how to find a suitable interpreter on the system that it's running in. The nice thing about PEX files is that deployment can be as simple as just copying the file to the server, assuming there's a Python interpreter on that server Or if you deploy as Docker images, then the images can all be identical except for this one file. So building These multiple Docker images is really quick because only that final layer, that one file, is different and creating that layer is just copying the file into the image. And PANS knows how to build these PEX files efficiently using the package command. So you can have this uniform tooling for everything from resolving requirements to generating code to running tests to running linters and formatters and to doing type checking.

Speaker 1: all the way through to packaging a deployable server that you can just say which is a single file you can execute. So to sum up We've I hope demonstrated that multiple Django services are often a good deployment architecture, that keeping them all in a single repo, single repo is often a good code-based architecture. and that there exist powerful tools and best practices to make this not just straightforward but even delightful. Thank you so much for attending. I'll be very happy to take questions And you can also find us for on this URL. There will be links to from there to our Slack channels. And we welcome questions and comments and concerns over there. Thank you again

Speaker 2: Benji, thanks for the for the demo and for the workshop and and for uh learning us a little bit more about multi and mono repos. Um initially I thought we were dealing

Speaker 3: so

Speaker 2: you you can understand me, right?

Speaker 3: You can hear me but you don't have it live on the platform.

Speaker 1: Yeah, yeah, I can hear you.

Speaker 2: Oh great, okay, great, okay. Um

Speaker 3: Initially I thought we were talking about a kind of uh monolithic application. Um

Speaker 2: but we are talking about a multi kind of yeah, multi-service

Speaker 3: Um and I would say uh

Speaker 2: that changed my perspective a little bit and I yeah heartily

Speaker 3: agree that

Speaker 2: keeping track of

Speaker 3: generate them Yeah

Speaker 2: like synchronizing features across your your field of applications and your yeah maybe maybe your your your core your your core processing uh uh architecture is extremely important

Speaker 3: and um yeah we always call it the lasagna you know

Speaker 2: like you have all these layers that have to talk to each other and You have to pin it with a uh with a pencil at times to make sure that all the features match up with everything below

Speaker 3: and especially once

Speaker 2: if you have to release something uh from a mono or from a multi uh repo

Speaker 3: then

Speaker 2: you have to extremely coordinate it extremely well. But I think in a in a multi-sort like a service-oriented architecture I think uh this is a very wise uh thing to do and

Speaker 3: let's see what we

Speaker 2: our company doesn't have a lot of uh uh small services uh out there yet. But at least we are on the tipping point of of of m like breaking up a uh kind of old style monolith and um first services are created. So we are in a perfect moment in time to to do it the correct way. So thank you very much for the presentation.

Speaker 1: I thank you. Yeah. I think the wider point I can't tell another organization if they need multiple services or one I think it evolves over the lifetime of a project and over the lifetime of a company. But what I would say is that I think it is important to distinguish between code-based architecture and deployment architecture. And you should pick the code-based architecture that works for your team. And

Speaker 4: kind of a big part of my point here was there is there is tooling out there for you.

Speaker 2: Yeah.

Speaker 4: Kind of if you want to make that distinction, there is tooling out there that will let you make the right code base architecture choice. It will allow you to evolve your deployment architecture without you having to make massive disruptive changes in your code base. So if you have a model, if then you Is it two and three or whatever? Like there's you know one of the things I work on in my day-to-day is tooling to make that um tractable and and sensible to do. Um but yeah everybody has their own kind of where there are money making where a single monolithic server is actually the right choice.

Speaker 3: Quite some examples

Speaker 2: like in these days that uh to be real honest if you're kind of a self-employed person or you are uh uh one yeah one senior developer in a company that does that, then there's no sense to to to split things up that much.

Speaker 3: We see when you get a larger team where

Speaker 2: for instance our front end goes so much faster than the backend uh yeah you need to split that up otherwise you are reverse merging uh like like there is no tomorrow. But um but I I now understand it clicked a little bit uh home when you made the uh the the presentation on the the small services, so which is actually all kind of a a big collection of backend uh back end things but then yeah a little bit better easily managed uh better for scaling better for uh outbalancing or yeah whatever you want to

Speaker 4: and also different deployment cadences like my Every organizer every every kind of use case is different, but you very often find that like the like some some UI generating front end needs to be deployed every day, but certain backends may only need to be deployed once every three weeks or a month.

Speaker 2: And then

Speaker 4: but they're actually harder to deploy for various reasons because maybe they're stateful in some way. So having um Having a monolith basically means you have the the you have to do the hardest thing the most frequent number of times. So

Speaker 2: Yeah, that's true.

Speaker 3: Can you send me that?

Speaker 2: Thank you for the presentation. That's what I wanted to say.

Speaker 4: I have to do it. Yeah, I'm a big fan of Django. I've been using Django for like 10, 11 years now. Um since 1. 6, I think, or something like that. So go ahead and go back.

Speaker 2: 1. 4, 16? Yeah. Something like that. Yeah.

Speaker 4: Yeah. Um especially, I mean obviously for the demo I used uh SQLite because it's a demo, but uh I use Postgres almost exclusively everywhere else and it's been

Speaker 2: The the best thing for proof of concept uh for SQLite is that you can you can commit data. That's what I uh that's what I like. With SQLite very quickly. For a small proof concept or something like that. Uh yeah. Okay. Uh thank you very much. I'm um

Speaker 4: thanks for stopping by.

Speaker 2: Yeah, you're welcome. Bye

Speaker 4: back.