Bringing JSONField into Wagtail Core - Sage Abdullah

Speaker 1: Hello everyone, I'm Sage and today I'm gonna be talking about bringing Jason Field into Wagtail Core. So yeah, I'm Sage. My pronouns are he, him, I'm from Jakarta, Indonesia. and I'm currently working with Torchbox as a Wagtail developer. Previously I participated in the Google Summer of Code program with Django back in 2019 when I was still in university I implemented the cross-database JSON field that you can use starting from Django 3. 1. You can find me on GitHub and Twitter at LaymanH. So, JSON field. What is it? From the Django docs, it's a field for storing JSON-encoded data.

Speaker 1: In Python, the data is represented as dictionaries, lists, and the basic types that we see in Python. Alright, it says JSON encoded data, but what is it? Before that, let's talk a bit about JSON. If you are not familiar with JSON, though I'm guessing you are, here's an example of a JSON object It's a key value pair where the key is a string and the value can be a string, Boolean, integer, float. array or an object and arrays can have mixed types

Speaker 1: It also has null. So it's really similar to Python's dictionaries and lists, right? Now for JSON encoded data, it's basically a string that contains valid JSON data And since we are talking about data, let's talk a bit about databases. When we are talking about databases in Django, we usually talk about the models. For example, I have a profile model here. It has a one -to-one field to the user model, a status field, a last scene field, and a config field. The config field is a JSON field. This model represents a table in the database.

Speaker 1: So the table would be something like this. The table name is derived from the app name and the model name, and the model fields correspond to the table columns. That means for JSON field, the columns Would contain JSON data. It's not a basic type like um string or text. So it's it's a complex JSON object Though the database doesn't really store it in such a pretty format. It st uh yeah the data is stored as something that looks more like this

Speaker 1: Okay, so let's go back to JSON field. So how does it work? Let's say I have the profile model here, but I'm just gonna focus on the config. json field and let's say I have a dictionary I can create a profile object with the dictionary as the config. Then later on, if I retrieve the object from the database, I can check that the config is equal to the dictionary that I have. And if I access . config, we can see that it's a dictionary. That means if I change

Speaker 1: the values of for one of the keys, for example here I'm changing the font size. Yeah, it would just work. It's just like any other dictionary And if I save the model object and I retrieve it back from the database, it will have the new font size. But how does it work in the background? Let's go back a bit here and instead of using objects. create, I'm going to split this into two parts first by instantiating the profile object and then I will call dot save. What happens when we call . save So in the background, it turns the config

Speaker 1: into JSON encoded data. And when Django executes the insert statement, it's going to pass that JSON encoded data to the database. Later on when we retrieve the object. Django will execute a select query and the query will return the value as a string. the JSON encoded data. However, when we access . config, it's already turned into a dictionary. So how does it do that? Thankfully, Python has a built-in JSON library that we can use. To use it, we can do import

Speaker 1: JSON. Then if I have a dictionary, for example this one I can call json. lumps and pass that dictionary and I will get the json encoded data. And if I call json. loads And pass the JSON-encoded data, I will get a dictionary. And that dictionary is equal to the original dictionary that I have. Okay, so we have covered the way JSON field stored , the way JSON field stores and loads JSON data. And in addition to that, JSON Field also has its own custom lookups and transforms for querying the data.

Speaker 1: Let's say I have this JSON object here as the value of a JSON field. I can query with objects. filter some JSON field double underscore name equals sage and it will return all objects that have some JSON field with the key name and the value of Sage. It can also differentiate between missing case and keys with null value. So To query for missing keys we use is null equals true and to query for keys with null values we use equals none And we can chain the double underscores and keys as deep as we want, depending on how deep we want to query the data.

Speaker 1: And there are also the containment lookups. The first one is the contains lookup that allows you to query JSON field based on a subset of the JSON data In this example, the query is going to match the above object because the query value here is the subset of the expected object So the next one is the contained by lookup. It's the inverse of the contains lookup, so instead of a subset we can use a superset. In this example, I'm using a superset of the expected value and the query will match that object.

Speaker 1: And lastly, there are the key existent lookups. The key existence lookups consist of three lookups. The first one is the hack. key lookup that lets you query based on whether the object has a specific key and the second one is the task keys lookup which lets you query for objects that have all of the keys that you specify here. And the last one is the has any keys lookup, which lets you query for objects that have At least one of the following keys that you specify here. Okay, so yeah, that was JSON filth. But how do all of these things fit into WagTail?

Speaker 1: Well there are a few parts in Wagtail where we can use JSON field. And I'm going to start with the easy wins. The first one is page revision. It's used by Wagtail to store page revisions. The content of the page is JSON serialized and stored in the content underscore JSON field. However, if we look closely here at the database fields, the content underscore JSON field is actually a text field. So it's stored as text. And if we take a look into the source code, it's true, it actually uses text fill.

Speaker 1: So we can change this to use JSON field instead. Just swap text with JSON. And we might as well rename the field from content underscore JSON to just content And I saw that there are some places where we use the Django JSON encoder. um in the json loop dumps calls for this field at least. So let's put it here. And now our page revision model uses JSON field. Now the other thing that we have to do is to remove the use of json. loads and json. dumps every time we want to access and save

Speaker 1: the content. because it's already handled by JSON field. And if it's not needed anymore, we can just remove the import JSON. Okay, the next one is the base lock entry. It's an abstract base class that represents a record of an action performed on an object. For now there's only one concrete class. It's the page log entry. And Yeah, it records actions performed on a page. And the same thing happens here. So it has a data underscore JSON field that's actually a text field. But it also has

Speaker 1: a property named data that provides deserialized data. We'll look into that Okay, so yeah the baselog entry actually uses text build for data. json so we can just do the same thing as previous As in the previous one, just replace text with JSON and just rename the field to data because it's already obvious that it's JSON field. And also since we allow blank values, let's set an empty dictionary as the default. And I don't see that we use Django JSON encoder anywhere for this field, so let's skip it.

Speaker 1: So yeah, now we just need to remove the calls to json. dumps and json. loads And since the field itself is already deserialized by JSON field when you access . data, we no longer have to keep the property here. We can just remove it. And yeah, that PR closes an issue from last year. Someone made this issue from last year requesting to use JSON field for a baselock entry. Okay, now the not so easy wins Streamfield.

Speaker 1: So someone made this issue back in 2019 The title is a bit inaccurate because blocks are just Wagtails version of model fields, but they are not really model fields because The model field is just stream field. So the title should probably be JSON field for stream field. Anyway, this person wrote, I'm under the impression that block's data are actually JSON, but stringified and stored in text fields in database. It allows blocks to be database agnostic. And yeah, that's true.

Speaker 1: I would like to propose a configuration option to use JSON field instead of text field which will allow the data to be stored as JSON directly. It will also enable user to lever to leverage query options available for the JSON field. And somebody also replied, this might be easier once a ticket in Django is implemented to provide a built-in JSON field for all Django database backends And the funny thing is this issue was created on 4 May 2019, which was actually two days before the announcement that I got accepted for Google Summer of

Speaker 1: Code And yeah, so two days after this issue was made, I would work on this ticket for the rest of the summer. It's funny how things have come full circle for me. Okay, uh anyway, the next one is somebody just added a comment last month They said, I would like to pull this up since Wagtail 2. 16 has dropped Django 3. 0 and 3. 1 and can now offer JSON field for all supported databases And that's true actually because uh JSON field w for all databases

Speaker 1: It was released in Django 3. 1, but Wagtail was not able to immediately use that in the code base because we still have to support the older Django versions. Since now we have dropped the support, we can just go ahead and bring JSONField into Wagtail Core. Okay, so for StreamField it's quite different because previously we were just changing a field inside of a model. But now we need to modify the field itself. If we look at the code, string field is a subclass of the base field class, but its getInternalType method returns text field

Speaker 1: From the Django docs, it says that if the database storage for our custom field is similar in type to some other field we can use that other field's logic to create the right column. So does that mean We can just change this to JSON field. Wrong. Unfortunately, it's not as simple as that. The problem is we cannot change the base class of a custom field because Django won't detect the change and make a migration for it. And it gives an example of using a custom field based on chart

Speaker 1: field and trying to change the base class to text field. And since Django won't detect the changes, they recommend uh to create a new model field. and change the model definitions to use that field instead. And even though for stream field the case is a bit different because we are not going to change the base class. Um yeah the Streamfield base class is field, the base field class. And Streamfield already has its own way of managing the serialization and the deserialization process for JSON data. So

Speaker 1: it's already it already has the same logic although a bit more complicated than JSON field inside the model definel field definition. So we just need to change the get internal type return value But the pro the same problem happens here because Django won't detect any changes. Because if you are using Streamfield in your project and we change the get internal type of stream field, Django will not detect any changes because it sees your model definitions as the same. So, what's the solution? Should we create something like JSON

Speaker 1: thream field? Honestly, I'm not really in favor of this because that means we're going to have two different string field definitions. And what if in the future we want the JSON field-based stream field to be the standard? We don't want to be stuck with JSON Stream Field forever And yeah, it's going to be tricky to if we, for example, want to rename it back to Streamfield. So I talked a bit with the other Wacto developers and they suggested something clever. Use JSON field. But how does it work?

Speaker 1: So well it's pretty simple really. We just add a new keyword argument use json field to the stream field constructor and set that value to the field as an attribute. Also we need to include that in the deconstruct method The method is used by Django to create migrations. So that Django knows how to reconstruct the field from the migrations. Now that we have a new keyword argument, we include that in the one of the return values of deconstruct. And users will set the keyword argument in their model definitions.

Speaker 1: So Django will detect that there is a change. because you added something in the constructor, you added use json field in your model definitions for string fields. And Django will make the migrations. And now we can dynamically return get internal type based on self. usejson field. If it's true, we use json field. Otherwise we use text field And to enable JSON field lookups and transforms, we override the getLookup and getTransform methods. to call the methods from JSON field if self. usejson field is true

Speaker 1: And just to unify the configuration for JSON field, let's put that in a property and use it. And lastly, let's add a warning if the user has not explicitly set the use JSONField argument to either true or false. The reason why we set the devil to none is that because we want Django to create a migration that records the explicit true or false value of used JSON field. So in the future, if we for example want to make use JSON field equals through as the default, it's going to correctly

Speaker 1: It's going to correctly generate or not generate new migrations for the user. If we set the default value now to either true or false, Django will only generate the migration if you explicitly set it to use something other than the default. So for now, we require this argument to be explicitly set to Boolean. If you haven't set it , We are going to erase a warning that shows up if you run with Python-W but it's not going to break your project. Anyway, yeah, this is where we are at now.

Speaker 1: I've made the PR and it's been reviewed a few times It should be merged pretty soon I think. Hopefully we have it in Wagtail 2. 70. And yeah, the tests are passing. I have duplicated the stream field test to run with both use json field false and true. So what could this bring for us as WhiteTail users? The first one is improved RAID performance. for models that use stream fields or maybe for the page revisions and the baseline entries I put an asterisk asterisk there because some database systems like SQLite

Speaker 1: don't really have a dedicated data type for JSON, so it's still going to be saved as text. And you might not get any performance improvement. However, for Postgres and MySQL, it's likely that there would be some performance improvements as they store the data as binary And the next one is smarter search with JSON field lookups and transforms. So yeah, you can utilize JSON field lookups and lookups and transforms now which should be much better than just using text field lookups And maybe from this we can implement new features for StreamField and maybe blocks.

Speaker 1: Or maybe something else, you tell me. So yeah, that's it for me. Thank you for watching. The slides are available on these links. There are also links to the PRs. I'll post them on Slack as well. If you have any questions, feel free to ask and I'll try my best. best to answer. Before I go, here's a picture of my cats. Cheers.

Speaker 2: Thank you so much, Sage. We have one minute left. I think that's enough time for maybe one question. Do we have any? I haven't seen any in the chat, but are there any in the room? Here we go, we have one.

Speaker 3: Um something that I have troubled with in the past is doing data migrations for streaming. fields. How will there be uh will more structure from having a JSON field make those kind of things easier or harder?

Speaker 2: So one thing that this WagTail user has struggled with in the past is doing data migrations and Streamfield. And will this feature make it easier or harder?

Speaker 1: Okay, uh thank you for the question. Um honestly I don't know enough about the Wagtail code base to um answer that but uh if I were to guess uh from my uh from what I have known um it's probably not going to uh like directly change how it's going to be. It's it's not going to directly influence like how easy it is or not because I think the main work for that to be done is in Light Tell itself to first uh provide a like a system to track the changes for stream field uh just like how Django does for models.

Speaker 1: And once we have that in place, we can probably um I think that's where the JSON field-based stream field can be useful because now because if if we use JSON field-based stream field, the data will be stored as JSON in the database. And some databases like Postgres actually have native data types for those. And And they actually have uh like functions that can modify the data more uh in a more advanced ways. So uh WhiteTel could probably translate the uh like the

Speaker 1: versioning for stream field. So think of it like a like migrations but for stream field Once Wagtail has that system in place, it could probably translate like the operations done on the stream first, like maybe add block, remove block. into instructions that utilize the native JSON functions on the database.

Speaker 2: Awesome. Thank you so much for the answering those questions. I appreciate that. And give one more round of applause for Steve.