Django's GeneratedField by example with Paolo Melchiorre

Thank you, thank you very much. So here we are speaking today about uh jungle generated fields. I'll try to do some examples to better understand what they are. uh how we can use it. They've been introduced in Django 5. 0, so they're around for a long time now. uh but they are still not so largely used and we are trying to see if if they can use more and we can start directly seeing what they look like This is uh an example you can find in Django 5. 0 release notes. There is a small class, a square, and we calculate the area of the square.

directly in the database. Unfortunately it's the only example you can find in the world documentation. I know that uh the jungle domination is quite huge, it's impossible to have a sample for everything. Uh it will be uh very very uh big so but I l like learning by example and doing things. In fact what I like A lot is the war jungle girls workshop. This is a picture of uh I took during the Django Girls workshop in Pescara, the first one I organized in 2022. And I every time I as I I'm amazed as young girl learn doing things

uh directly and not uh listening someone explain things but directly doing things and this is has been for a long time also my way to learn things trying to experiment something and do indirectly and then maybe go back to the theory and deep down. But before going ahead on learning from the best speaker than me, I want to say ciao to everyone And if nobody is someone no don't know me, I'm Paolo Mecchiore. I come from Italy, as you can uh realize uh I've been a Python developer for a long time now and I'm active in the Django community right now I'm in the board

of the DSF and I contribute in some different way. I organized PyCon Italia and I organized some Django Girl workshop. I contributed as a navigator to the Django Node space. But what I like the most is developing with Python and using Django, of course, in particular working in the database level. And there are a lot of interesting things in things in the documentation of Django and one of the things is Uh the the section for performance and optimization. We can read very interesting uh things about the lower level of the computation.

apart from other things it suggests us to work at the database level to improve our performance. And this is something I like doing because I like database, of course, but also because I found in my career, in my project, every time some solution that improved so much the performance of my project. In fact, my first ever contribution to the Django code has been a specific function for the Django Postgres module. I was working in a project with a lot of instances of model and I was forced to add a new UAD column.

In doing that, working with the Python UAD module, I got frustrated what our slow was. Searching for new way to do the it I found that Postgres already supported it. So after using it for months in my project, I I decided to bring back that directly to Django. And it worked quite fast, seconds for updating the value of UID in a columns for millions of records. So in bringing back this uh functionality directly in Postgres in Django has been not so straightforward as using my local project

because of course you have two hards Yeah, more test documentation, you have to polish a bit your codes. And Marcus here and Mark helped me in Django Con Europe 2017 during the sprint. It has been uh a a very very effective way to learn how to contribute back. In fact I encourage you to participate in the sprint at the end of this this conference What I learned the most was to be aware of all the other database backends that we have in Django , not rely only on your database. And Starting from that point with the confidence of and the help of the other core developer , I started thinking new features to add in Django.

In particular, I was thinking uh to bring back in Jango something that I learned in another Python uh web framework, in particular the one I used at the beginning of my career, Zoop. It's quite old. I don't know how many of you know about it. Good. And this is a picture of me at the Plunk Conference in Naples in 2007. And uh starting from the points I is I think I start thinking how of this feature I can be able to bring back in jungle. One of these was uh some computated value. So uh during the pandemic period in particular which is in Europe has been very hard. I think also

here in the isolation of lockdown. I thought a new feature to Hadid Django and I searched researched a bit. For computated value I found that there have been already something like that, there was generated columns. They are they they are computated by the database. They are are always updates based on other fields. So when you modify a column and these generated columns update based on that without uh the needs of any trigger or any Python code at all. So quite promising but based on the lesson I learned from the sprints

I did some research so I So search if the support for this generated column was already in all the database became we support. We support in Jungle. And luckily that was the case. So I created a small proof of concept of the code. based on the array field I already use in Postgres and I tried to share this idea with other uh contributor of Django and the opportunity was in Python DE in Berlin. I met again likely Markus and Mark. also and I share with them some of my ideas and this one was um accepted and they encouraged me to share with the wider community.

Back in the day we were using the mailing list. the Django developer mailing list. So I sent an email proposing uh this idea and sharing the reference to the support database but also uh proposing uh to implement it So we had a long discussion of the mini lists and one uh realized that we had already an issue opened based on the topic what was a bit stale. So with my contribution , we started again talking about that and Jeremy showed up and proposed a pull request for implementing it. Which was great because at the time I had the time and the capacity to open myself

a pull request. I was very happy to see other people creating a pull request based on my suggestion As you can see from the number of comments, it took a long time to discuss about it. And this was not merged because some issue particularly if I remember correctly on Oracle. But luckily Lily showed up and completed and fixed those or this issue. As you can see in this case we had more comments And I'm responsible for a lot of them because uh I wasn't able to contribute with the code, but I tried to contribute in testing manually uh the um the code

and I created a list of example edge cases or use case for the utilization of the generated feed And in fact, I collected all this example I used for testing this pull request. And after the release, I shared a series of articles based on on to generate the columns uh using SQLite, Postgres, and GeoJungo too And actually we are going to see some of this example, maybe with some addition, and we'll see how to use it more effectively in our code. But before seeing some example, I want to

share with you what we have now in in Django itself. So uh the generator field now um are a new type of field that is computed by the database uh without any extra codes and they are totally managed by the database and are always in sync because the database upgrades it for you You have to specify an expression which can be deterministic only. We'll see some examples of when this does not work. And you can't reference another generated field in another generated field. And there are some restrictions based on the database you choose.

In fact, we're going to see SQLite and Postgres in this talk. You have to specify an output field, and it's more or less is the uh the type of uh of the generated value you are going to to create. And lastly you have to specify if you want to store if the column will be stored or virtual. But it depends more on the support you do of your database. For example, in Postgres you can also only have uh stored column and not virtual one. And for example in our in Oracle it's the opposite. You can't have stored column but only virtual one But let's go and see

using uh as in the first example the SQLite, which is the light database we can use and which is the default of course And let's start also with seeing some beautiful images from Hitley. So this is the Pantheon Dome in Rome. I don't know if you ever visited. It's a very beautiful place and it's still there after. thousand years. Romans liked a lot circles in their buildings. And in fact the circles is the class which we are going to to create. So it's a basic model we have only one field which is the radius and we want to calculate the area directly using database function

It's not a complex uh calculation, but it's interesting to see that we already have and uh Django query set in the RM all the function and the constants that we need to uh to calculate the area. And I put in all the model example a string representation that can help us to see the starting value and the final calculated value. If we see the SQL code is also immediate, we can see that a similar name for database function are Show here in the migration of the our model. And we can see this in action. So

we create our circle instance, we define a radius with number, and we see that it calculates in the um representation the area the number. We can also see what happened under the hoods of the in the SQL code. uh printing the latest SQL uh queries uh our connection performed. In our case we see that we passed a value for the radius And what we are in returning is the ID which is calculated for us and also the area. And this is the way we have the representation in the top of the slide here. Okay, let's go to see something different from geometric figures.

This is another picture from Venice. It's chocolate. Peace or cake and uh we'd see a price here and we maybe we decided that uh they're delicious if we want more than one for For example, so we need uh a class, an I think class with price and a quantity. And The calculation here is way basic than what we did before, but there is a slight a difference here. In the quantity field we they defined a database default, which is also a new feature in Django 5. 0. And this is can help us to have different uh behavior

for example we can generate this item and decide it to uh specifying only the price because we saw that the default is to have one piece of the of this one uh quantity of this item The calculation have an output field which is similar to the price one and in the presentation we we see the formula here. The SQL code of the migration is straightforward, but what we have difference between now and before is the default in the quantity column, which is one in this case And of course we can see that in action we can use in both ways creating an item without specifying any quantity and having

the results and or specifying at the beginning the quantity fields and this is what what we get So let's move to another city. This is Bologna, it's the city where we organized the last PyCon Italia conference, and we are going to organize her again at the end of May. Everyone are invited to join us. And in particular, this is a traffic like under the two towers, which is more or less in the center of the city. uh every time I associated the traffic light to uh affinity state machine so in this case we are going to create Something like that. We have an order class with two date time fields.

One is mandatory and the other one is nullable. And based on the present of these two that time field we are going to uh create um essentiate the status fields using some conditional expression I like using this type of condition directly in the database, and here we are going to check if the payment the time field is nullable or is null or not and based on that we uh set the status um text fields to one value or another. The default value is created. Of course we can add more condition here, but the space of the slide is is not so great.

And two states is is enough. And we can represent it having the status and also the one of these two the time fields for for representing it. What happened in the migration in the SQL code of the migration is that we have more or less the same. These are this condition and based on the payment uh presence of the value we can have the paid on the created status. As before we Can essentiate the order in two different ways, specifying only one, the mandatory creation that some fields, and we have as you can see the created state or specifying both. that the time field in this case we know that the order is already paid and we can print also the the

time of this but we can try to wreck things to see what what happens For example, we decided that we instantiate the order, and when we represent it, we see the created state. And a certain point in the flow we um we want to change and we want to add the payment date and we can also save the instance in the database. Unfortunately as you can see the status didn't change. Because at the moment when we save something, there is no returning automatically the generated field. What we have to do is to remember to refresh the instance from the

database. In this case we can use the refresh from DB or maybe getting again distance from it is something we have to remember when we work with generated fields. So if since we want to break things more harder, we can also would generate some error here. For example, we decided we don't want to instantiate directly the the order, but we only create it and without saving HIT in the database, we try to access the representation. Since we customize their representation and we try to access the status. But it can be reads from the database because it's not being saved.

The model has not been saved And in this case we can do something like that. For example, create a property that has a fallback from the status when it is written and the instances not have already been saved there. Okay, to do more complex thing we move to another database backend, of course Postgres. And after instantiating the new database engine, and we are going also to activate the country Postgres module to see something different. We go back to Venice. This is the Aqualta bookstore, which

means high water. And if you go inside you see a pile of books like that. If you are lucky you can have to search to find some very interesting and whole books, otherwise you uh you you buy something very uh bad stuff. It's like searching the JSON in the JSON uh field sometimes which is can be uh uh contain a lot of data in it it's complicated to extract interesting things. Um I I created a package model uh with only a name and uh json field which contain the data. What we want to do is to specify

the name of the package and bring back from PyPy uh some information on the packages and put it directly in the data and extract automatically the version. We are accessing only two level of keys in this data. We see here the SQL code specific for Postgres and also the syntax that is using Postgres to extract information from the JSON B column. So I tried here to use only Python uh the Python standard library and to try to uh read the data from uh the Python a PyPy repository and after putting directly the data, the JSON data

um in the data fields, what we have is the representation of the Django and with the latest version. We was able to do that also in the past. For example, without having the generated field, we was able to have a property, for example, to extract directly. uh diversion from from here. But and this is what we we get if we do that. We in getting all JSON data with a query, we retrieve all the JSON fields. that they can be also very heavy to to retrieve from the database. So we can do something more slightly different. We can defer the data column which is can be heavy

to retrieve and we can annotate it uh with the same expression we use in the generated field in this way or as we shown we can simply get the new creator generated uh field a version referring the data column And what why it's it's better to use the generated field for performance of course. So I tried locally with some ingesting some packages here with 1000 and 10,000 packages. and uh it's quite better to use the general field because you retrieve uh you spend only milliseconds on it and not not so many so many time. This is a representation of the numbers.

This are logarithmic scale graphic, but it gave you the idea So let's go back to Venice. This is a sculpture for Lorenzo Queen. Give me the idea of connecting things between borders. And we what I tried to do was connecting. in a person class the two the two field that is very easy the first and the last name to create the the full name. In this case we are using the concatenation expression uh to generate a new a new chart field. The SQL code we are using the concatenation operation. And what we have at the end is

the full name of the uh of the person we we generated. I'm showing you this It's not so complex, but in the first release of Django five point zero gener for the generative feed this was not possible to achieve because of a issue of uh Postgres database. So Now we can do that. So let's use it. But we can do also more complex things in concatenating together text field. For example, we can generate an object, a JSON object directly In this case we are using the JSON objects function and we can apply some transformation to our fields or concatenating the initial of the first name and last name and what and then we can bring back

represent the JSON we created In SQL is slightly similar. We define our keys that are slightly different from the value, and we concatenate the two. the two initial and what we get already calculated for us is the JSON representation we want to to get You can have more than one, you can also decide that this is not the best way to use it, but is a give an example of how flexible can be using generative fields. Let's go to Milan. This is the Brera Public Library. It 's very beautiful inside, but searching on this amount of uh text

can be intimidating. So for searching in texts we use the full text search that we already have in the Postgres module, for example. If we have a quotes model with author and text, we can generate automatically a search vector field for us. And we can see this in action. So let's see, we have a quote from Plato in this case. We instantiate the quote And what we can see with the value list is the vector generated automatically for us. It's interesting to inspect what it contains. And We can see in action that filtering for the plural

version of meanings. We we bring back the text that contains the singular form of it. Så this is a full text version. In action. If we want to do this more complicated, for example, let's see we have quotes in different languages. We can have a new chart-field languages that contain the language of it, and in the expression we can Use the languages directly to calculate and to have a different type of vectors. This is the migration that Django provides for us and it fails if we try to migrate it. It failed because as we said at the at the beginning, it's F to be

immutable function. In this case They are not because you don't know in advance which language you are going to use in your TS vector. But we can try to do something slightly different. In this case we still have a chart field for our languages and we decided to use the expression condition that we already used in the past. and decided to copy and paste more time the same expression. But the good point is that we can also have a fallback, a default using uh the fallback the basic uh dictionary in the full text search in Postgres is the simple one. The SQL code in this case is longer and but it contains w everything we we we need

and we saw also the fallbacks at the end To see this in action we have a quote from Cartesius And as you can see the search field contains only the vector of the lexem in English High and ham are removed. If we pass the quote with the languages in Italian, we see that it removes also the articles and how full text search works. And if we see the last one example in which we are using Latin. Since Latin is not supported by the Postgres database for full text search it falls back to the simple

extraction and so it extract the the word as they they are. So it's an effective way to work around when you have to use not emotable function. And we are going to the Como Lake. It's a beautiful lake, and this is a grand hotel tremendously. But if we want to book here, we have to specify the the range of data when we want to stay. And in this case we have a booking class with a start and hands uh date time and we can generate automatically uh date range field. Um to do that

unfortunately we don't have already the data range function So in this case we are going to create a new one on our home using the date range And the SQL code that generated are very straightforward because in instead Postgres already have the date range in it. And we can see that specifying in the start and end date time the dates here um bring back and we have the representation of the that range which is quite useful to specify a data to see if overlaps with your uh period of uh the data you want to to put it Okay, let's go to 3S.

This is a world map in the Miramare cluster, and I show you this image because we are going to use PostGIS backends and Geojle. of course. Working with Geo Django is can be simple or complex, it depends from what you want to do achieve. With this map from Modena, this is a continual world map, is one of the first maps with geographical reference on it, one of the first with the Brazil border. One common thing is it's very common to have with uh with map is to have the hash of your points. And in this case we have a class, a city class with the name of a

point. What we can calculate automatically can be also can be the hash of it. And it's very straightforward because in GeoJangle you already have this function, the GeoSh function. And under the hood, the SQL code for PostGIS can't use the GeoSh function too. In this case, I'm putting in the my the city my own town, which this point, and what we get is the the hash of it But we can we can also do something more complex. For example, we can calculate directly the G or J Son of our city model. And we already have luckily in GeoJango the the function to do hits.

We can represent directly the GeoJSON fields. And this is what we get if when instantiates the same city and we w bring back the GeoJSON. It can be effective you have to uh get all these points. without calculating again hits. So this is uh not in Italy at all. It's in Australia, but I wanted to find an example of very long straight route. This is a 90 mile straight route, which is more than uh for me mile is strange to think about, but it's uh quite uh 150 kilometers

long a straight route. And so to represent it, I created a root class with a name and a line that represented the hits. And we can calculate automatic automatically the length of this line in kilometers, of course. And this is the function we have directly in PostJS. And when we Instantiate the route, we see the calculation directly in kilometers for us. This is I think the last picture. Who among you know these mountains? Nobody. So it's it's US, it's Colorado. This is a landscape from Rocky Mountain National Park.

Why are we talking about Colorado? Because we we are going to represent directly the area of uh uh state in this case we have the name of and a polygon and we want to calculate automatically the area of it and this is what they look in SQL and why I'm showing you this because Colorado is more or less a rectangle on the map, which is strange, strange for me. And in searching for For border I found that more or less is like that. So I finished my my time and I show you more motivation to use generative fields.

Before going away, I want to thank all the organizers, speakers, sponsor, and volunteers here in the conference. You can use this slide and you can download it from my personal website. And the last picture, of course, is from my last trip in Italy. This is the Lake of Carezia, which is means Carrez in English and it is in the Dolma in northern Italy. And if you come and join us in the PyCon Italia, you can visit place like this. So thank you very much.