Layered Django project structure for large-scale collaboration

Well, hello everyone. It's good to see you all here. I hope you had a great coffee break. First uh I'd like to uh start with a quick introduction My name is Chill. You can find me on uh GitHub, Twitter, and Mastodon. Uh I 'm a lead backend engineer in Kraken. I also have some other affiliations that I want to mention. I'm currently serving as the vice president in the Django Software Foundation. Uh I'm a co-organizer of London Django Meetup uh and uh uh PyCon Turkey. Uh I'm a proud member of PSF and PiLadies

and I also uh help organize over uh 10 Jungle Girls uh workshops uh since 2015. So if you want to talk about any of those after the talk, please come and find me. And let's come to today's agenda. Today we are going to talk about large scale collaboration and how to best structure your Django codebase to enable it. I will talk you through the architectural style that we use in Kraken and how it helps us achieve large-scale collaboration with an architect team. I will explain some of the details and some of the implementation. While doing this, we will talk about what problems it solves.

And I'll mention uh several uh pros and cons of using layer architecture. I'll demonstrate the concepts and ideas mentioned uh with some uh dummy coding examples. So uh To tell a bit more about Kraken, Kraken provides an advanced software solution for energy and provider pro uh utility providers worldwide, integrating renewable energy management. customer service and automation. The Kraken platform helps optimize operations , reduce costs and enhance service quality, supporting major global utilities.

We help them transition to decentralized and decarbonize energy systems On the other side, in our tech team, we have a pretty big code base with about thousand developers working on the same monolith. With an average of just over 150 deployments a day, we have a super fast iteration cycle. While doing this, our main focus is robustness and efficiency of the systems we are building. Today uh we're going to focus on uh how

uh uh Django and architectural architectural style help us. So let's let's take a take a look at how we utilize uh layered architecture to enable these and I hope you can take home uh few learnings today. I'm sure you have been there. Have you ever worked on a project that started small? And then got really large. Like you throw a party for five and hundred came. Everything imports from everything else

and whenever you change one thing it's connected to ten million other things You have no idea sometimes where to put stuff and you don't know where to find stuff Without the formal architecture in place, it's more likely that as your project grows, as your team grows you'll end up with a big ball of mott. In other words, an application that's tightly coupled Fragile, difficult to change and lack a clear reason or direction So let's uh explain a few uh

things before we uh start. Let's start with uh what's an architectural style. An architecture style describes the macro structure of a system, helps you put a formal architecture in place for your software, and helps define the basic characteristics. and behavior of an application. Different architectural styles offer different approaches to designing software systems, each with its own set of principles, benefits and trade-offs. Choosing the right architecture style depends on factors such as um Scalability requirements, team expertise, and

the nature of the application being developed. You should choose the one that meets your specific business needs and goals We're not going to talk about different architectural styles today, but I would love to hear that talk from one of us one day. In software architecture, layers refer to a way of organizing and structuring software components into distinct groups, each with a specific responsibility. Which brings us to layered architecture. This approach is known as layered architecture, or sometimes called as N

tier architecture. Each layer Has a well-defined role and interacts with adjacent layers in a controlled manner. It promotes separation of concerns, modularity, and maintainability. This particular style is layering by technical area. Like you can think of it as front end, back end. Uh but that's not the only option. You can layer in other ways uh for example like in the clean architecture or hexagonal architecture which we're not going to go into detail today.

So let's look at the Django. Django uses a modified version of the ModelViv uh MEC architecture called ModelViv template, MET. In Django wheels depend uh on the models and templates depend on the wheels. For instance, if you went to if you want to add something to a template It needs to come from the VMs. And similarly, if the VM needs data, you need to put it, uh, you need to store it in the models. We can say uh wheels and templates

form a distinct group That is responsible for displaying information to the user and interpreting user comments. Whereas models forms an adjacent layer, and we can call this data layer , it manages the storage and retrieval of data from the database. Django's model view template forms a layered architecture. How we choose ours. We use in Kraken we use Python and Django for our big monolith application. And that's why picking layered architecture as our architectural style

was a natural choice. One of the powerful features of the layered architecture style is the separation of concerns among components. Each layer is dedicated to handling logic relevant only to its specific technical area. For instance, components in the presentation layer which you can read at uh re- read as front-end also Focus solely on presentation logic and nothing else. For example, it's uh responsible from displaying the user data. While those in the business layer exclusively

manage business layer, I have to collect that user information It doesn't it doesn't uh it's not responsible and it doesn't care how to display it. Such categorization facilities the creation of efficient role and responsible to models within our architecture and uh enables different teams and groups to govern the areas that their domain or area expert. This also streamlines the development, testing, and maintenance of packages and modules, particularly when well-defined component component interfaces and contracts are established between layers. So next let's look at the key components of uh

our layered uh jungle project in Kraken So we have a interfaces layer, uh which is entry points plus all presentational concerns, like all. Below it we have application layer which is responsible from orchestration logic like coordination of everything Domain layer is responsible from business logic and data is where her persystem logic is located, in particular relating to databases. And uh dependencies uh flows uh downwards here. Let's look at uh

the layers uh in a bit of detail here Uh so the data layer we said where we locate persistent logic. This is a very Django heavy uh layer. We have Django models, managers, and migration files defined here. And one principle we are trying to follow here , which I think you're really good at , models should be kept thin. We avoid putting business logic on them. A bit uh not super uh jungle stick. Uh Djangonic uh to follow Pythonic. Uh but yeah. We keep models super

thin. I think we have around uh Uh 600 uh modules in this layer, and uh yeah, compared to other layers, it's a thin one. Uh here 's an example uh folder structure. Uh You can see it's a quite also flat a flat layer. We have different components under it. So on top of data layer We have domain layer. Domain layers is where core reusable business logic lives. So that's where we put the business logic out in.

Kraken Core has the concept of uh two kinds of procedure governed by uh different rules. Uh first is operations change uh operations change state in some way, for example by writing to uh a database or causing an external side uh side effect Most often uh these would be uh called uh comments, not operations, but we call them operations. We find it more uh intuitive. And uh second type is uh queries. Uh queries are read-only procedures that have no side effects. So domain layer is is a large layer in Kraken. It's our fattest layer We make sure the modules in domain layer is discoverable.

That's why we organize packages by domain category and subcategory, as you can see here And we locate uh queries in modules uh called queries and operations in modules called operations Uh let's look at a uh dummy coding example. Uh so uh this uh this function This uh this domain layer query function fetches an account user record by ID. What it does is it goes uh to the RM, fetches the data It serializes the data and turns it back. Simple. Let's continue. Let's next look at the application layer.

We call the entry points in this layer use cases. Each use case is responsible from one action. Interfaces may only change anything in the database through a use case. So all our interfaces All our VIOs, all our APIs can only change data through a use case. And once use cases are completed uh changes will persist. These are uh the main principles. But really It's a again super thin layer like the data layer and its pure responsibility is the coordination of operations and queries

from one or more domains. Database transactions, locks, queuing infrastructure, and event logging And to note , this layer is sometimes called the service layer for those who are more familiar with the layered architecture. So let's look at uh a coding example. Uh so this is what uh this is a dummy. a potential use case uh from our codebase. It updates the user preferences. So what it does it uh goes to domain layer It queries the domain layer for

the current preferences of the user. It makes a second call, an operation call, to the domain layer to update the preferences. And once we uh make the changes, uh it publishes an external event to those uh for those channels to act upon uh those signals and then uh it does uh logging It's also responsible from the transaction. You can see the uh atomic uh decorator at the top. And that's that's it. People clean. And uh super easy to read, test, maintain and change.

Uh Let's go back sh uh historically a bit. We introduced the application layer because uh It was challenging to determine which functions in the domain layer were the right ones to call Domain layer contained a mix of uh low-level and high-level functions, which caused confusion. To address this, we moved the high-level functions uh to the application layer. This change uh Among other things, improved our ability to test and manage these high-level uh functions. In the application layer We have uh we enforce strict rules

and we use linters to ensure quality. Specifically uh we enforce uh certain doc string and testing patterns at this layer which uh I uh mentioned uh in uh one of my other talks about coding conventions. Yeah. So let's move to the next layer, interfaces layer. So interfaces layer is where uh uh okay Uh the role of the interface layer is uh to translate requests from different protocols, such as uh but not limited to HTTP requests, into domain requests and conversely translates uh domain responses into

appropriate response uh domain uh responses into appropriate uh form. For example convert uh converting an uh application uh exception into uh an error response. So it has Django URLs Django wheels, Django forms, uh Django REST frameworks, serializers, GraphQL queries, GraphQL mutations, uh salary tasks. Because they're also entry points to our applications and uh Django management comments and uh templates because templates are are also a presentational concern. The main advantage of this layer that it locks out, separates

all presentational concerns from the application and domain layers. This allows us to build powerful domain models without the distraction of uh presentation-related issues. Let's look at uh one example uh from the interface layer. So uh this is a WiiO. A dummy view uh that creates an account. Uh what it does uh Remember we are the uh interface layer, the bottom layer is application layer. It makes a call to application layer to the account use cases to create the account

with the parameters, handles the exception, and uh on successful operation it returns a success uh message uh and renders the response. We have many other interfaces like API sites, consumer sites. I mentioned seller task, system uh task, con jobs. And this one is a support site example. Uh uh it's the back office platform we have We use the uh same use cases uh in uh in the other interfaces like we call the same use case

from our GraphQL API and uh when when we are migrating uh clients uh from within our uh management comments. So this gives uh us uh uh great uh power of reusability So uh that's the end of diving to uh the layers. Let's talk a bit uh about import lay linter. Next. We use Import Linter to enforce our architectural style. Import Linter is a uh open si uh open open source uh Django package Authored by David Sedham. And I think maybe we already

listened talk about it in one of the Django Con Europe. It's a tool that verifies your project's architecture by analyzing module imports and checking them against uh rules defined in a configuration file. Uh the configuration file contains one or more contracts. You see one contract here. Each contract has a specific type which determines the rules it will apply. For example, here uh if you can uh take a look, uh the type is uh layers. The layers contract type allows you to check dependency direction from high to low ensuring uh proper layering. Uh it has other uh types like

uh uh Yeah, I'm not going to go into detail now. So we have we have different uh we have several configuration files. Uh with different contracts, each tailored to different aspects of our architecture. And Import Linter is integrat integrated into our uh deployment pipeline. So uh any code that does not follow the architectural rules defined by those contracts will fail the test. Next, uh I'd like to briefly talk about uh another uh principle of uh layered architecture, uh open Closed layers. In layered architecture, layers can be either open or closed.

What does that mean? A closed layer means that uh as the request moves from uh layer to layer, it must it must go through uh the layer right below it to get to the next layer and uh the layer below that after that I mentioned about uh how uh like uh the historical story about our application layer, how it's born out of the domain layer. That hypothetical version of our architecture was an example of all layers being closed layers. A request originating from all uh interfaces layer must first go through the domain layer and uh to the data layer in this version.

And when an open layer is present, requests may pass through that layer to the next layer below that one. In our case, application layer is an open layer. I mentioned that interfaces may only act through use cases in applications layer. uh and interface uh code can also call directly into the domain layer, skipping application layer to weave only for weave only uh requests and uh this is uh at application layer level uh is our open layer contract within

closed layers So let's look at uh another example. Uh this time uh we are looking into uh a wheel that doesn't do any uh Database change. So it skips the application layer and goes directly into domain layer and uses a domain query to fetch the data required. render and display. We talk about some advantages of our layered architecture. But before finishing, I

also briefly talk to you on what is not working so well. We have an old code base, almost nine years old, and it's growing We didn't introduce all these rules from the start and even though we have the architectural style set , You can still find a lot of uh leg uh legacy code in the code base Previously we looked at one of the contracts from our import linter configuration.

What we haven't looked at is the ignore checks in that contract and the list. It's not exhaustive. But it's not one line, I promise. We are trying to actively burn those down. But We from time to time introduce also new ones, unfortunately. In a large-scale collaboration setup like ours, you need more than conventions, you need more than agreements to enforce your architecture style. The layers, those are defined to be close, uh

are not always so close. There are a lot of violations as I mentioned and uh import linter unfortunately doesn't support closed layers yet So uh you're as good and as complete as your trolling, it's safe to say An improvement we want to implement is having real closed layers instead of open layers and all over closed layers. It wouldn't be incorrect to say that we sometimes feel like we are playing against Django.

Django makes things so much easier that we tend to compromise. We still have uh Django ORM objects throughout the entire co-code base and uh it's uh it's uh everyday debate uh for uh most of us But it's not uh fair to put it all on Django, as uh something uh I haven't mentioned today uh is the domain-driven design. Uh And we also find it difficult uh to follow a DVD approach uh within our current architectural style. So um

Last but not least, we also like to have uh clear contracts within the layers. uh vertically uh as we do uh between layers uh which is a uh uh often problem at the moment. Which brings me uh to my last slide. So uh to sum it off, the structure and organization provided by our uh layered architecture Facilitates collaboration by promoting modularity, clear separation of concerns, defined interfaces, and standardization across the code base. The hierarchical and loosely coupled nature of it provides the flexibility

and scalability needed to accommodate growth and changing requirements over time. I'd like to thank you all today uh for listening uh the talk and like to finish by saying uh before questions if if you want to uh uh take one of those lovely kraken fleshies home and to ask more questions. You can find me in our boot. I can answer those questions uh in person right after our draw for these plushies and One o'clock and thank you