Modernizing CRUD Operations in Django

Thank you. Thank you, thank you. Um Yeah, so let's start. Uh as you said uh it's about trying to modernize uh even more uh application using Django Ninja. So let's start about how uh I'm gonna present it. So first uh we are going to see uh Django Ninja. If you don't know either uh yet uh what are the challenges uh that I faced with uh large scale uh applications then my vision and kind of potential solution spoiler Django and detailed examples also how I tried also to not only improve the developer experience

uh in order to develop endpoints but also the testing experience. And then finally the future directions, conclusion and maybe final uh thoughts. So first uh who am I who am I? So is it working? Yeah. Uh so first yeah, I grew up in uh Normandy in France, so on the north. Uh and then uh I moved uh uh uh at Toulouse at the south. for my studies and I stay there uh in order to work at Pixelia. It's a little startup uh with a couple of friends where we are developing AI tools for computer vision. And in my free time sort of playing yeah football, music painting uh and developing of course and used uh web or mobile applications

and recently open source so you see with uh two packages and for now yeah it's been uh five years that I've deployed uh I've been using Junk and Django Ninja for two years. Um yeah, so next introduction to Django Ninja. If you didn't see it uh two years ago, uh Vitalikus, the creator of Django Ninja made in introduction but it was remotely since uh because of the pandemic and stuff. Uh so a little uh summary about it. Uh I was all of you maybe know like it's uh kind of fast API developer experience where you have easy the um uh serialization uh using pyden tick schema and an open API documentation like

automatically. It's not a big deal since you can do it with uh other frameworks too, uh like Django Respermock. But it's kind of cool to have it. And let's start the fight. No, I mean uh it's a joke because uh it really like what I'm trying to say it really depends on you. I mean each one of you c can have some preference It's not like a framework is better than anyone else. Okay? So for example, what do I personally love about Django Ninja? It's the abstraction level, like taking the serialization and deserialization layer just a little bit higher. So you don't have to do it in your uh G function. So yeah, it's done in a middleware layer. So yeah, I really like it.

You just have to define yeah, just like this. Example I don't know if you can see code uh really at the backstage. But yeah, just a example so it's visual and yeah, I'm not just talking like that. Um so yeah, so what are then if the you know the developer experience is quite increased with the Django Ninja. What are the problems that I faced? Okay , just like any frameworks you would be using, you have at a certain moment a lot of models, a lot of repetitions in in the services that are using it, updating it, and also at the endpoints. So for the services, service layer it's quite easy. I mean it just function

and you are doing quite everything you want with it. But for endpoints I found it quite difficult. I mean, even though I was refactoring my code all the way I wanted with the services, I had a lot of repetition on the endpoint layer too. So duplicate duplicated logic Okay, sorry. Um increases of course the risk of inconsistencies. So my next question was how can I fix all of those problems? How can I minimize as I've written? play code and repetitive endpoints. How can I do it in order to yeah just be less frustrating when you are developing all of the stuff and when I don't know

uh salespeople are asking you a new vertical in your app application you have to repeat a lot of stuff and how to not just uh rush quick. Um so yeah a lot of questions that I try to basically try to respond it. And all of this started one year ago when I was vacationing Palma de Mallorca, so it's kind of fun that it was in Spain and now I'm back here. Um so yeah. I was trying to think about how could I fix all those problems. So one of those key was um okay there's some components some uh sorry some views that was uh repeated a lot all over my application. Sometimes it was for example a concrete one

uh the read operation I had a lot of uh get functions over an object that was repeated for different schemas. Sometimes you just want a little bit of in information, sometimes more, sometimes only IDs and listing IDs of uh specific objects. You have a lot of prohibitions and existing solutions like viewsets in uh Django Rasp Primox, for example, didn't offer the flexibility of defining several um crowd actions, for example. Um and you had with inheritance only one uh possible uh impl implementation of each view. So I was thinking about how to how can I treat them as components, like

a notion, a word, What you obviously often see in front-end development, but we are only taking the good side of it, all right, not JavaScript. Um and the key specifications uh include uh yeah, so class-based view components, even though it doesn't exist in Django. ninja. And that's not exactly what I'm trying to do or implement like expose uh class based views for every people using Django Ninja, but it's uh just that I need it in order to declare my views as model of as I ever wanted to. And yeah, so easy of creation of course the goal is to ease the way off uh the way for people to create their own um modular views. It depends on your specific um

yeah uh your specific uh business logic. It's not always about uh repetitive uh crowd endpoints uh everyone ever saw. So first how to create this potential core objects. So As you may know, uh yeah, the basic way of creating a gate operation, for example in order to read let's say uh department object, it's really a basic object, right? Just have an ID, an autofill, and a title. You wanna get it with uh Simple schema, you just get the object. Okay. From this, if you check the code of the get method on the router, you see that it just really about uh API operation with methods get. Okay. And from it, if you read the code of Django Ninja, you see

See that it's just wrapping up the method add API operation in a decorator. So from it, okay, I could create an object called API view , make the method path, all the attributes in it, and abstract away the view function that I could just return with a key method called as as operation, as a dict, and use it when calling router ad API operation with the return. results as a dictionary. So we have a method to a way to define an object, serialize it and use it in order to add the operation. So we just need the router, the object, and we can add

view. But then how to compose it in what I called API view sets. How to manage them and group them with I don't know, maybe and surely, with uh uh I don't know, yeah. similar properties like model of course. Um so yeah first I had a look at init subclass. It's kind of init for instances but for classes. So as soon as you're inheriting this API view set Well, I can access all the subclass, uh all the class attributes of of it, and especially the instance, the subclasses of API view. So you just have to inner it from API view, put it as a class attribute to API view. set

and then I can just iterate over it to change of course the view function name to because it's gonna be generic of course uh to the class attribute name so at the same time I'm dodging like the the conflict possible with uh view function Name. I'm then accessing the API view set from each module of views. So you can access uh similar like same properties on the API view set in the module of views and then call the function uh the method. add view to that is doing what I just talked about previously, like uh serializing your API view in as operation and call the router with it. But then it was a little bit too abstract. could not uh expose this to every one of you, uh everyone

every users and just say, Okay, let's go, you can create now your own API views. I had to uh take some really fundamentals that were were both repetitive and common to everyone. And thanks uh there's like some invoice called Create Read Update and Delete the Crud that was first uh introduced in uh nineteen eighty three by James Martin. Hope you maybe read the book, but yeah. Though was like I found with those the exact uh kind of endpoints I needed in order to showcase how to declare your views are modular alike as components. So yeah, it was kind of the introduction of the package. So I could expose it to everyone, make it understanding

understand and see exactly directly how is it um how you're gonna how you could have the declarative and iterative syntax to kind of every endpoints. You see easily with CRUD and you can easily think about it for your repetitive and common logic inside your proper business logic. So Yeah, just like that you uh I have defined like five classical uh classic views and be able to compose my view set as I ever wanted, even for example use the list views sever several times as Like I said, for example you can list uh a department um with a simple schema and an extended one with uh related

um objects and a simple one with just the IDs you can do whatever you want but uh you inde identified correctly that uh those endpoints were kind of the same and could be possibly uh refactored at the top level, I mean at the definition. Okay. A little bit too abstract so bit of code. Well it's um directly uh doing what we are writing it, you just see that I've leveraged the inheritance of values in the API view sets. So you just have For example, you s you know that the view set link link to department object is going to have views that are going to use the models.

So if I access it and access the default uh schemas, I could easily Lead create list, create, read, and update, delete, use. So just like that, in several lines of code, you just abstract away kind of the boilerplate code in order to expose a new model that you just integrated in your application. Um any a key part is the reuse the same in uh same views in the same view sets. So for example if you if you have a migration of a schema, for example, the head of the exposition of a new field or rename a field for Certain clients, you could just add a V2 path and just expose the same schema without the need to rewrite another endpoint.

So yeah. Again, a lot of abstract. talks so we need concrete examples like what does it really look like. Um but first some vanny uh Django ninja code. So you see when you are reading writing uh a classic uh lead For example, I'm going to do exactly the same for all the five endpoints. You have the basic one, uh let's say it's your um proper uh business logic uh endpoint, or right, not a cred endpoint, you have to identify around all the use of it. Um what's really similar in order to find a perfect interface. Like it's a general problem, okay? You have to define your own function. I could not do that but I

kind of done it for basic uh thread operations. So yeah, once you found the basic request components, maybe some uh method around it, pagination class, of course, even though I put a default like uh Django Ninja has done with limit offset paginations. So Yeah you could easily write some complex um endpoints but also really easy to read. For example here I'm uh easily defining a list employees endpoint. to uh departments. Yeah, I didn't introduce the Django models, but you can easily um uh know that employee has a foreign key to uh department object. So yeah from

uh direct path where I know the I ID is from the model defined that is in the view set and in the view um that yeah the ID is the one from department and then you can easily define your schemas and have your handpoint. So we declarative and kind of really usable but one time again I'm really like defining the tools to define it. You are free to not use the once I've um like the crowd uh views, my implementations of CRUD views, you can extend them really easily The key here is that you just have to define them as class attributes. So if you are creating your own custom view, it's gonna be another class. You don't have to change the API view set, you just have to add it as a class attribute.

And just like that you have your um you compose your view set as I said. And yeah, you can do the same for the other one, like create what's changing like init model. Maybe you're asking why, because sometimes you could have like with so with the listing employees, um some path uh parameters like okay if you specifying ID and then something else or even name or even any path parameters you ever wanted, there's gonna be some path par managers and maybe you are going to use it in your in the model. For example, even the request uh in order to link uh I don't know if you are storing the created by user um yeah and uh other uh yeah pre-save and post save

Like I prefer to do it and you maybe should do it uh here in the endpoint or in the service layer instead of uh the signals and uh joking inside the the models. But yeah, since it was kind of a little bit opinated to um force users to have the full clean method. I've put it in a pre save that you can easily overwrite if you wanna use directly my uh implementations of uh yeah uh my views here create view. Uh and one time again yeah it's It's uh in the blue one are always uh written like the request components. But yeah. So once again you can easily define some complex but same view, you just have to define a little bit more parameters. So yeah, as I said in the init

model method, I don't know if uh uh re far from the screen you can see the code, but um I I'll share the the slides uh right after. Uh the init model you just have to um re re root the Passparators ID in the department ID in order to, as I said, override the in if model so you have everything you can want to. Um and we can do the same. Well yeah. I'll skip this part because uh I think you and understood the idea. You just identify what could be customized in order to be used real largely and minimize the code wherever it's possible. So I've done the same for updates. And then you have it. And yeah, something uh I just talked about uh previously

was the possibility of defining just a new schema for the same operation really easily by just specifying the path. Uh like just path ID V two and you have it. Just link it to the right schemas. And finally, yeah, delete kind of the same. Uh I expose it uh pre delete and positive operations and a get model that I never never overwritten. But I'm just exposing exposing it in order to say that yeah I'm allowing any kind of uh customization, but you're free to do whatever you ever wanted. And yeah, and maybe not more yeah. And let's get How yeah, some

key component in uh the package is how can I uh abstract away the link and know which parameters it uh is of what type. Um it's really easy when you have the package. Path in the model you can easily um extract all the strings from your path, like the path monitor's name, and linked it with your model in order to know uh exactly which field in it is it. So you have kind of uh access get fields from the models meta arguments and then you have it. You can dynamically generate sorry at runtime your identity schema And yeah, the key parts, so as I said, you can easily uh

add your custom views because it's all it is about like creating some component like views for you proper to your business logic, not only Crowd operations. So yeah, I've talked about it about how to do it. Um yeah, uh again a lot of code. I I hope really far away from the screen is okay. Uh as I said, you just have to inherit from uh the API view class. Uh, Um where you define the interface you you want it to so it's really it's proper to your needs. Um but I've done an example using the self delete operations where you could do it with an update but it's not an update. And sometimes you wanna yeah just uh define your repetitive operations and you just have to define an inner

view function. Uh um and yeah, it's just easy at that and you can reuse it with uh one line everywhere uh you you want. across several uh view sets or um inside the same one. So you know if you are migrating schemas as I said. So yeah, really easy to define your own one. Just use API view set and your custom view and it's okay. So yeah, now that I have showcased uh all the features kind of of uh the package, what's coming after, uh of course uh a little bit of fix because I have one issue that abstract uh too much the API view class. Uh I I've made it too uh opinionated around the CRUD operations. Like I've really made it uh

with uh as inputs only uh three parameters, uh the request one, path query body, when you could potentially have file header or cookie. Like it's um kind of a lie to say that today we can uh compose any custom uh view but it's gonna be uh real soon like yeah matter of uh weeks before I got this and Um yeah, of course implement the most ask feature like uh showcase how to create some uh async uh views uh to yeah make people uh happy about it and you know just uh be on the trend. And then I could maybe Maybe think about a V one that way I'm waiting since uh one year and maybe provide some other common crowd operations, even though it's maybe not the goal of this package

is not to showcase how to do it, but not like use all the tools I will pride in the future. And maybe uh yeah, extend the vision of the framework. Uh what does that mean is at first I really wanted to uh make this available as I said it it does not depend on the frameworks. So maybe adapt It's easily to Django REST framework or uh FAST API or anything, you know, just you know to define some uh compostable uh approach. Um but I don't know yet. I I just see the V one and then we are going to But yeah, my uh experience uh here didn't stop to trying to enhance the developer experience uh on the specific task of defining endpoints.

I also tried to tackle the testing experience. experience was a little bit harder uh because yeah it was a lot of also repetitive. At first I was uh really naive because I tried to kind of uh uh r repeat the same pattern like having some component like uh test objects. Um while it kind of sounded quite right at first, one year after I would never do that again. Why? Because it was just hidden. Uh it was Just yeah, some code was hidden. You were not probably aware of what you were testing all over your applications, you're just using a random test component like uh object

and it will not take the responsibility of saying to you that okay, you are really testing everything. If you put this in your test, it's okay. Uh I was not really convinced and it was really bringing a lot of more problems there. Solutions. That's what kind of the thing I try to do like link the what? Oh 33 actually. Um yeah, but I try to link all the tests together like that. with the names it was kind of working but as I said kind of lack of flexibility and kind of really rigid. Um so yeah it was not no generalization was possible because I was through constraining

the the status code and stuff. Yeah, I I just skip it to you because it was really a mess. The important part was uh is that I s successfully like flattened all the errors I've made and just throw them away. uh with uh an object called scenario where I'm uh just it's really just a black box with inputs, uh request uh components and expected uh uh uh statement on the response. For now it's really basic, it's just Straight away the need to do the HTTP call uh and the use of the client. Um but yeah uh I found this interface a way more um easy in order to manage all your scaled uh tests over your big

application. And yeah, I I really like the the way it goes. J only one test for specific endpoint where we you are testing a lot of scenarios together. And the key here is that you can test it for example with several users with uh different permissions and on different objects and every scenario uh uh completely isol isolated uh with a database fullback as I've uh written. And yeah it was uh so after the separation and the complete uh rewrite of it and separation of Django Ninja it was just depending on Django. So I was like, okay, maybe it's it does not have a spot like in this package. So I separated it even though it's kind

I say a little bit small but yeah, it's uh the little brother of uh Django ninja crit. I've named it uh Django rest testing and yeah. Um take a look. So yeah, that's a conclusion. Yeah it was uh really time consuming to do all of those uh the two packages, the maintenance and the think about the features uh kind of alone so it was really hard and time consuming, even though it's uh really valuable uh learning experience. Um yeah, I've improved the developer experience uh for me, so why not for everyone? I hope it's gonna work. So yeah. Uh I just wanna thank uh all the stage contributions.

And sponsors uh uh of the packages. Uh really great uh without you it was it would have been a lot of uh more difficult. Uh of course, uh thanks uh to GitHub, uh README and JadeBrains for the open source license. Even though GitHub It's kind of natural but we are for kind of not aware of all the advantages of uh Gifab. And of course uh thank you Jang Rukan organizers for