Building a custom model field from the ground up

Speaker 1: And so now I have the pleasure of introducing uh Dimitri De Gallo.

Speaker 2: That's right.

Speaker 1: Yeah, okay, great. Building a custom model field from the ground up.

Speaker 2: Hello, everybody. Can you hear me? Cool. So I'm Dmitry and I'm working at kiwi. com as a technical team lead. I'm located in Prague, Czech Republic. I do Python from 2010. I command a couple of open source projects like Junk Money and I love open source and traveling. So Uh who use custom model fields? Could you raise your hands? Nice. Who tried to build your own? Cool Okay, so basically why do we need it? First, use case is when we need to map some custom database types into Python objects.

Speaker 2: And the other way around, store some complex Python objects in the database. So it's two ways. interaction and the there is a use case for it. Uh the project is called Django Money. It provides you money and currency objects and integrates uh everything into Django RM Also it has forms, admin integration, template tags, jungle race framework integration, currency race, stuff like this. This project is kind of mature. It was uh started in 2011 as a fork of another project. I started committing to it in 2013 and I used it in my previous job.

Speaker 2: I started to maintain it a couple of if a couple of years ago and the interesting fact is that there are two maintainers here on the conference. And I first time meet Benjamin here after six years of uh working together on the project. A little overview of the topics that we will cover today. First of all, we will inspect the storage level, how to map money to model, what are the queries underneath How descriptors could help you. Then we will make some queries. We'll try a lookup API and introduce some expressions.

Speaker 2: And some extras like migrations, serialization, validation, stuff like this. So let's go for storage. In our domain we have money. Basically it consists of two things amount and currency and we can use it like this. So we have money, amount currency, we have decimal and string Also we can use some arithmetics like addition, subtraction, some other operation. Also we can localize the representation depending on the locale uh we want to use and I have a question for you who use decimal or numeric type for storing money

Speaker 2: Floating point types. Couple of integers. Okay Uh I believe that the preferred way is to use decimal or numeric uh for storing because otherwise you could face uh precision loss is in most of the cases. So we need to map these to our beautiful model. For example we have item to to buy and we have Simple name and our new money field. Sure. I would recommend to start always with designing the interface or how you would use your new field. So basically we need some basic operations, create, get, update.

Speaker 2: It could be done in different ways, but there are some examples in the code. So save get some F with save and refreshing. No complex queries yet. And I really recommend you to make the test cases from from these expectations from your interface. just just like this or you can use built-in jungle test cases or use py test whatever you think will fit here So on the storage level we can uh we can approach in different ways. For example, we can store data in different fields as we do in Django Money, which is

Speaker 2: Not really good in all cases. So it's standard SQL and unfortunately uh managing two fields at once is not trivial in all cases. I will show you some examples of the consequences of such decisions. Okay, how can we connect it to the database? We have our money and we need to store amount in a decimal field and currency in char field and underlying data types in the database are numeric and varchar. Sir, and how we would do it on the Python level. There is a nice nice thing called descriptors

Speaker 2: and basically it allows you to customize attribute access So get, set, or delete, you can customize these things actually. And In our case, we need to build a money instance when we're getting something from the field And we need to set another field when we are setting something to this price field. Sir For this purpose we can utilize control uh contribute to class and add also this currency fuel there. But does it seem hacky to you? Who think that is hacky

Speaker 2: approach? Yes There is a nice alternative called structure type. It's from SQL 1999, I believe. And it implies the usual way of implementing custom fields because it's much easier to map one field from database to one field in Django. In Postgres you can, for example, create type Django Money as amount and currency of certain types uh and the queries will be a bit different So you need to utilize brackets a little bit more. But surprisingly it's not supported by MySQL. I believe 20 years is not enough.

Speaker 2: So and also you would have some overhead for attribute access because you need to get the uh tuple with the data and then uh extract uh some feel from the uh structure type but however it could be mitigated with using indexes in the worst case for example in sequential scan we will have to uh evaluate the condition for each row but for index-only scan we'll get everything from the index. So there are some ways to mitigate it. And how we can implement these two ways communication. My um

Speaker 2: in Django fields provide you with a couple of different uh methods and We need basically two of them from db value and here we need to construct the money instance from some dB level string or other object and get prep value we need to construct database level instances. And a little disclaimer, uh some examples are a little bit sloppy because There are no like corner cases and maybe some error validation, but I would like to emphasize that It's only like core actions that you need to make and also there are many other places to extend.

Speaker 2: These are the basic ones So for the first part the summary is design the interface first and use it as a test cases. Discover your database queries because you need to map Python to database. You need to know both sides Decide on your database support. If you really need to be database agnostic, or maybe you work on Postgres and you're happy with it, up to you And one field is simpler than multiple fields on on Django level at least Try structure types for composite values. It's a cool feature. Okay, let's let's make some queries.

Speaker 2: Basically, we can make some lookups, transforms, use some expressions, like we need to have some items that price is greater than 10 euros. or only Danish crowns as a currency, or maybe only amount more than 100. Or price is equal to some other field. Define your behavior unambiguously, first of all, and inspect your database queries again. So basically these lookups and transforms could be could look like this on the database level. You will need this transformation a bit um later So

Speaker 2: uh I believe from Django 1. 8 uh there is a lookup API that's really really cool thing. Um you need to basically define your lookup and register it. with register lookup here. So basically what you need to do is to uh emit some SQL and some parameters for your uh Queries on the database level. There are a lot of existing lookups that you can reuse. However, the big thing is that you need some SQL as the output. So you need to construct left side, right side, and parameters. For transforms, like

Speaker 2: extracting some certain subfields from the structure type you can utilize a similar thing called transform you need to again create some uh SQL as an output and have some output field for amount it's decimal field and for currency it's char field basically If you will go this way, uh basic expression should work out of the box. For example, here we have some number of items and values and annotate works just out of the box, maybe some order by and some

Speaker 2: other simple um expressions But if you need to use something more sophisticated with F expressions for example, you need to extend your domain because it's not aware of Django at all So uh here with money we have magic methods for addition, subtraction, and so on. So we need to add a knowledge of expressions from Django Like it is in example. So also you need to adapt your lookup implementation because it's also not aware yet about the uh expressions so you need to handle it as well and for

Speaker 2: structure types you could create for example your own expressions like subcolumn or something similar and use it like this to make some custom queries So create custom expressions if needed. The summary for the uh querying part is Pretty much the same. You need to define your lookups, transforms, and behavior unambiguously. You need to know your database queries and map these Python things to database things. Use Lookup API to build desired queries. It's really nice too. We started the project when it was like

Speaker 2: Django 1. 034 and there are a lot of different hacks to uh like uh have the same behavior So I really recommend to use lookup API instead. Extend magic methods on your domain entities to uh for uh f -expression support and create custom expressions for your structured types if you need it. Hey, extras. At some point you might need to migrate your data. So migrations. For this case you need to first of all extend the construct

Speaker 2: method on the class of your field. In this case you will have your custom options in migrations. Also, your domain entities are not aware about migrations. You could use the constructible decorator to add this support. So if you want to have some default value like this, you need to wrap your domain entities in this decorator. Serialization. So maybe you would like to use fixtures or something like this. You need to define You need to have a module with two things, a serializer class and

Speaker 2: the serializer callable And for example, this implementation hooks in value from field. And for our custom structure type, we emit a dictionary that contains these nested fields. And yeah, here is the result. Nested field. Okay For deserialization support we need to update our get prep value because in this case we will have a dictionary. So update it and it will work. After it, we need to register our serial serialization module. uh and the good place is um

Speaker 2: appconfig so basically you can define default appconfig And in this case, ready will be uh fired after uh in in some time when uh Django initializes. At some point you would like to maybe you would like to validate your fields somehow. Like uh set some minimum values, some maximum for different currencies, something like this. And Django provides you with a lot of tools already built like validators for minimum values, maximum values, you need to extend it a little bit to work with your domain instances

Speaker 2: In this case we have like different uh boundaries for uh amount and for certain uh currencies. So you can define anything you would you want. A bad example of um consequences of uh some decisions like having two different fields and working through decorate uh through Huh. Descriptors. Descriptors, right. Some modules we need to update um

Speaker 2: some functions inside in runtime. It's really really hacky and like I would not recommend to do it ever. But it was like six years ago we didn't have like many options like lookup API and stuff like this. But for structure types, it will work out of the box. Okay, so the summary for the third part Extend existing tools from Django, there are a lot of them really. Validators, serializers, migrations, everything is there, but you need to extend it in certain points. Use application config to register to register your uh extensions. Think about possible use cases for your field.

Speaker 2: Maybe you don't need serialization or Something like this. Okay, and summary for the talk. Always start with the interface design first. You need to know what you would like what what you expect from your uh custom field and use it for your test. Explore underlying database queries because it's really important because you need to map two worlds, Python databases Experiment with your implementation, choose the most simple and unambiguous. And try to evaluate possible consequences of chosen approach because we have like

Speaker 2: maybe seventy percent of the code that works only for uh supporting new things like uh lookup API and similar behavior that is in Django from some point but we went another way and we need to have a lot of hacks for it So Django provides you with a lot of extendable tools for mapping your domain entities to the database. Use them It's all I have. Thank you very much. Questions.

Speaker 1: Awesome. We have about six minutes for questions. That means we can do about three, four questions. Just a reminder that you can do it online. uh with DjangoCon QA uh hashtag and the IRC channel

Speaker 3: great thought thanks um Uh structured types uh is new to me. Um I'm sorry if I missed it, but uh is the what was the uh approach you took for um Creating a structured type via Django migration.

Speaker 2: Once again, please.

Speaker 3: Creating a structured type via Django migration.

Speaker 2: Yeah, yeah, it's possible or you need to run a skill like uh I had an example, create type name of the type as some structure. It would work.

Speaker 3: So you put that in a run SQL micro. Yes. Okay.

Speaker 2: Maybe in some newer Django versions it could be supported. I don't know.

Speaker 4: Another question about migrations. You mentioned the construct that is needed for creating the field type But did you also need to make uh special migration operations to add sp specific migration operations to support your field type? And if so, uh how did you go about that?

Speaker 2: Okay. Basically it depends on the database. I believe that in Postgres you still have alter type stuff like this. However, I'm not sure about the others, but in general case, yes, you need to define your extension for some alter field or something like this. Basically it's probably the most uh frequent use case I believe. So yes.

Speaker 4: And uh and when you di do define these sort of uh operations then the Django um auto migration creator will not use them will not detect it.

Speaker 2: They need to use it manually. As far as I know maybe something changed. Um

Speaker 5: hi.

Speaker 2: Hello.

Speaker 5: I didn't know about this that you can register register a custom uh your serializer with Django. Do you know if it's also picked up by the Django REST framework?

Speaker 2: Uh I don't know It if it's in Django and currently for example in Django Money we uh basically we replace JSON serializer with our own to add this support. So it should be used because it's in Django But for Jung Rest framework you probably need a little bit more, uh like not regarding serial serialization but like having support for this type of fields in serializers, for example

Speaker 5: Okay, thanks.

Speaker 2: Any other questions, please?

Speaker 1: Questions online?

Speaker 6: Um hi. What do you think about um using a JSON field if you have more than like Two things to store.

Speaker 2: Yes, uh JSON is a bit different, however I think it could be used. Uh there is no decimal in JSON standard, right? It's only floating point So for this exact use case, I wouldn't recommend to use it because of the precision loss by default. Or you can use string maybe and wrap it somehow. So it could be used, yes, as an option. But I don't know about the performance issues or stuff like this. It could be similar, most probably.

Speaker 7: Hey. Um I'm wondering if you can use like um the aggregate functions like sum and average and all of that uh using like the fields Does it work like

Speaker 2: uh I believe only like count stuff. It it will work out of the box, but for average you still need to extract Amount part out of it. And in plain SQL it will be average of the square brackets dot amount, for example. So you need to compile this SQL So basically you need to extend a little bit these aggregates to have a support for it.

Speaker 1: We have time for about one more question.

Speaker 8: Isn't it a little bit uh redundant? Normally you have one row in a database and everything is in the same currency. So isn't it a bit redundant to always keep that? And that's another answer to the question before if you do aggregates. You don't want to sum up let's say Danish krona and uh Euros together because it doesn't make any sense.

Speaker 2: It it depends on the use case. For example, uh I usually work with card transactions for example for these payments and we store different currencies and we need to aggregate by different currency. So uh I would say uh it more in implementation-wise because uh having two fields is a bit more complex than having one field. It's only always an option, it's a trade-off. You have a little bit of simpler implementation but you have like limited database support and maybe some performance downgrade so it's things to consider for your use case You always need to consider all these factors.

Speaker 2: But in some cases it could help to reduce some complexity from your code. In some cases

Speaker 8: not. So in your case you have sometimes uh a row with a lot of different uh amounts and different currencies?

Speaker 2: Uh I would say that we have different amounts on the same row. Yeah, yeah, y yes, I mean like different monies.

Speaker 8: Okay.

Speaker 2: Yes.

Speaker 1: Thank you so much. Uh