How to abuse Wagtail's StreamFields as much as you want

Ooh, funky. All right. So I'm going to do the the risky thing and do a live demo there. The concept of the talk is how to abuse the stream field as much as you want , given that I'm not going to represent the same thing, but it's the same as we've seen this morning. You know, a lot of us I think are using uh design systems that translate into blocks that go into stream fields and that allow to have uh huge flexibility and that's an amazing feature of uh Wagtail that really you know allows us to do anything we want. But the the only drawback there is that um if you are not careful with what you do

or for whichever reason you you need to adapt uh different stuff and so on You quickly end up with uh many many different blocks that reuse um or like different kinds of pages and so on. So for example, here is my uh file full of blocks You see this uh thousands of lines long and there's uh lots of uh definitions of lots of kinds of blocks etc etc. And here um for example I have um like standard uh button colors okay that uh that are in my UX. So suppose my designer says now I want uh to have um a blue button okay So I say blue here. So I just added one option, right? So I'm going to make a migration because otherwise Django is

going to complain. So I'll make tick Two two to two all right, no, not this. There we go. I'll make the migrations. Um so You might be wondering why it's it has not returned uh yet and I will ask uh in between a question to the audience what do you think is going to be the size of the migration file Um so when I ran it earlier it took forty seconds to generate So let's wait a little bit. I can do a dance in between. Um but basically since you know it's nested inside many pages, many stream fields and so on, like all those fields individually have been affected

And you don't change just this property inside this field, you change the whole field because it's uh the granularity of Django. So it does not go inside the field. So Let's see now what happened there. So you see that this thing has been um you know appearing and it's 5. 1 megabytes. Um so that's completely insane and if you look at that um well you see there's many lines, yes. So Pretty impractical in the end. And um we initially we did not even realize this, but it's just we were running the migrations on the Docker

container that has like two gigabytes of RAM or I don't know three And it was running out of memory and we were like, what the fuck? You know, when you start the when you run server, it's uh for hundred megabytes of memory at start or whatever, but then it was exploding. So Basically just loading the migration literals into RAM was exploding the thing and the migration folder was 500 megabytes. It was completely insane. So um I had to do something about it um that is uh well don't call the exorcist please um that is basically I created uh Like I created a little command here that uh is uh migration packing that is basically going through the migrations and looking for those huge literals.

um that are very big and um removes them from the file and transforms them into lazy loaded objects. So you'll see in a minute what I mean. is that here all the stream fields got separated into different uh little files that are um containing the same thing okay Uh but it's just uh all condensed, there's no um you know uh uh indentation or anything. It's uh just all the same. So if you look at the the size of the thing like Um we went from five megabytes to 280 kilobytes for the whole uh migration, um which is already a huge improvement. Like your Git is going to be happy about it

And on top of that it's less things to load, but on top of that, it's also you don't have to load it if you don't need it because now what migration looks like is this. Um it is just having a lazy stream block here in the parameter of uh the stream field. Um And that's it. And I am giving here the name of uh of uh the um uh the heavy literal and I'm using um what's the name of the module? Proxy uh proxy lazy object. So basically it's a Python module that makes a lazy object. So it it it will it's indistinguishable from your real object. uh from any other part of the system, but it will just load the object when uh uh you access

try to access a property or do something with it. And in that regard it allows you to load all the migrations without loading the literals of all the stream fields and uh without uh overrunning the RAM. And so We went from uh using uh gigabytes of RAM to random migrations to to a lot less, and we also decreased a lot the size of the migrations uh folder. So that's um little bit hacky of course uh because there's so unfortunately it's not open source because it's really disgusting what I did there, but uh it works Um you can have a look. I'll show you. Must be somewhere. Tick No, not this one.

Well back and about to put it here. So basically what I do is that I have uh this uh whole script that is You'll see running through the like it's really hard-coded uh not nicely. That is running through the AET of Python and that is just uh dumping the AST into different files. And then I have this uh static bit of uh code that is uh you know generating the lazy stream block uh based on the stuff and so it's lazy object proxy that I'm using. So yeah, basically that's it. But uh I must say it has been very important for us to to have this because at this scale of uh dementia on uh migration size sizes we we we needed something like this. I

don't know if you have questions.