Dynamic models without dynamic models

Speaker 1: Hey everyone. Next talk is about to start. Jacob is gonna be talking about dynamic models without dynamic models. Please give a round of applause and thank you, Jacob.

Speaker 2: Thanks for that warm welcome and thanks for coming to my talk. And thanks to all the organizers. It's been a fabulous week. My talk, Dynamic Models Without Dynamic Models. I'm going to talk a little bit about what I mean by that, you know, what even is a dynamic model and why was I satisfied with just approximating one. But first I want to give you a couple of words about myself. That's me on GitHub. My background's in music composition and technology. Like a lot of composers, I started dabbling in programming to get my arms around electronic music. Before I knew it, I was learning Python. Blink an eye, you're learning Django. Blink two eyes, you're on the triage and review team. It could happen to you. But seriously, to coastline on Sarah's point from yesterday, please join us. We'd love for you to lend your time and talent But today I'm going to be talking about work that I did on the Arches project, which is a project that wraps Django that's used in the cultural heritage community.

Speaker 2: I've been working on that project since uh a little under two years as part of my work at Fairlon Geographics, which is a Django and GIS consultancy based in the US. So I'm going to start with my use case. I'm going to tell you a little bit about arches, uh, just so you can see the kinds of users I'm developing for and why I might need something like a dynamic model. I work on this open source framework called Arches that wraps Django. It provides interoperable components for building cultural heritage websites. So some implementers include Historic England, Historic Places Los Angeles, among others. Here's a sample search interface showing some search results for cultural heritage data that happens to have a both a spatial extent and a temporal extent. These are the kinds of components that we make, you know, that come with arches when you just pip install arches.

Speaker 2: Uh the framework is generic enough that if you have some cultural heritage data and you want to stand up a website, you spawn an Arches project just like you'd spawn a Django project, you get generic Django models that power generic reports like the ones I just showed you, generic forms, generic maps. As well as a GUI interface you can use to build schemas describing your own data in its specificity. But as a developer on the Arches framework, I have no idea what you're going to do with our tool. Here's a demo project with about a dozen custom schemas created by a cultural heritage domain expert to model specific things like historical areas, maritime vessels, historical activities, that sort of thing. Once again, we don't know what they're doing with our framework. Now here's a user who's fleshed out all the form fields that will collect the data about, for instance, a historical area.

Speaker 2: Down the left-hand side you see basic stuff like location, parents, images. Nothing too fancy, nestable units of forms. Our users call these models. They call their historical area a model, right? The form fields for a maritime vessel. They call that the maritime vessel model. Then they ask us to do custom development against this. I kind of want a jangle model for maritime vessel, right? I want to be able to do maritime vessel. objects. all when I'm coding against it. But I don't have that because they're just building with arches. They're building with a generic framework. Instead, the data belonging to the instances from all those 12 models I just showed are commingled in generic tables More on the specific shape of those models in a second, so you can get a flavor of the kinds of transforms I'm kind of saddled with.

Speaker 2: But what I want to point out is all the information that shapes those 12 models I just showed are not stored where the data is stored. They're stored where the schemas are stored. Any queries on my generic data have to be schema aware. I have to go get information about the schema dynamically to do any sort of useful query on my generic data. Okay. So do I want a dynamic model to help me with this? Do I want to run some command that spits out a model that spits out a class maritime vessel so I can code against it and do maritime vessel. objects. all? What if I don't really need to wire it up with the admin? What if I actually don't want to wire it up with the migrations framework? What if I actually don't want to touch anything in the database? I don't want to create custom tables and custom views. I just want to leave the generic Django

Speaker 2: models alone But I still have desires, right? What if I still want something that acts like a Django model, right? Um, that can mutate in sync with the changes the users make to those models. What I realized I wanted was just the ability to have something that acted like a query set, so I could pass it to a hook, for instance, in Django Rest framework that expects you to be able to provide a return value from get query set. Even though what I was really doing was I needed a product of different models, right? I needed a way to express inside a query set a schema-aware query on some other table. So I wanted something that felt like a dynamic model, or was maybe duct-typed kind of like a dynamic model, but didn't literally involve code generation to spit out class maritime vessel. So this is an admittedly pretty meta

Speaker 2: use case, but meta topics tempt us every day in programming. So I'm hoping you'll find something recognizable in the kinds of problems I'm talking about. If so, grab me after I'd love to chat. Okay, so to give you a sense of the kinds of normalizing and denormal the kind of two-step I needed to do when doing these schema-aware queries, I just want to give a quick overview of what an entity attribute value database looks like. This is kind of what our generic tables look like Now there's definitely an aura of don't do this at home around these kinds of databases because they are pretty much an insane choice unless unless your data is as generic and unknowable as mine. If you know you've got a customers, if you've got customers and you got orders, you should just have a customers table and an orders table, right? But I don't know what I'm modeling. So here's the kind of data I was dealing with.

Speaker 2: You want to record some facts. Here's me. I spoke at DjangoCon. There's your entity, there's your attribute or your predicate, and then there's the value, DjangoCon. This is the kind of generic data we're recording And often our users are researchers that are interested in like making that attribute like pickable from a controlled list, right? Um so that their data is more interoperable. You don't want just people free texting in whatever for their attributes. Good stuff Okay, so my data ends up kind of looking like this. Now it's not long before you want to start collecting data about DjangoCon too. So you want to lift up DjangoCon and make DjangoCon another entity. And then you want to flesh out your attribute with a little more metadata. What is spoke ad anyway? Well it's a relation that relates two entities to each other.

Speaker 2: Now my values table is pointing to all of that information, and my values table is just a sea of pointers. You start scaling, your data starts looking like this. You start scaling more, your data looks like a Sudoku puzzle. Then you start asking questions. How did I get here? What are the seven stages of feelings I might have about this? But then you realize that many intelligent developers who came before you realize that this was, you know, an important way and a flexible way of designing totally or designing uh developer interfaces around extremely generic and fundamentally unknowable data. In fact, the rest of this talk might be just thought of as a sophisticated expression of bargaining with this fact. Before I show you the solution I landed on, I just want to acknowledge some of the precedents in the Django community around several overlapping classes of problems here.

Speaker 2: For example, there are several projects dealing with polymorphic models. with defining mutations for GraphQL clients, you know, which allows, for instance, the front end to just select a portion of data that it's interested in. And projects that specifically focus on EAV databases, entity attribute value databases. And all of these have their own use cases. Finally, there's the plain old type keyword. If you this is built-in in Python to let you create a class on the fly But I realized I didn't really need a class on the fly. I just needed some, I just needed a query set that I could hand off to Django Rest framework, for instance, that would accomplish that kind of product of those two tables. Really, I was just interested in the most minimal thing I could do to express my business logic as a query set. Okay, so here was the fever

Speaker 2: dream. If you've used query set annotations, the dot annotate method, you know that they allow you to calculate something in the database and then invent that result. Uh as an attribute on your model instance, right? It's not a formal attribute on your model, it's an intermediate result you've calculated in the database, and then it just becomes a static attribute on your model. Emphasis on static. They're not wired up with anything. They're dead. You can't use them to perform an update. You can't alter it and save it back. But that's what I really wanted to be able to do. I wanted to be able to do Jacob. spoke at, push some more conferences into that array, and save it back. So I realized I was in the market for some kind of smart annotation so that save just worked. And then ultimately

Speaker 2: You know, the goal, as I just mentioned, was to be able to wire it up with some kind of API client, uh, with the goal of delegating all the untransformation I was having to do from human-readable data like dot spoke at you know, one element array DjangoCon and then take the untransformation to get it back into the Sudoku puzzle. I didn't want to be doing that all over the place, either in my views. or handling it on the front end before sending the data over the wire. I wanted to delegate all of that to some central tool. So I landed on something like this. I have a custom query set that I link to my generic model, which here is just called entity. Uh it's got an entry point called with attributes that if you pass it, the name of a user-defined model, like in this case, let's use presenter from my last example.

Speaker 2: This with attributes method knows how to do a schema aware query, right? It'll figure out what are the relevant attributes for the presenter model by querying all of the schema tables. Annotate them onto my model instances that I get back from this query. As long as the queries weren't metadata lossy, as long as I didn't lose this information that I was passing to with attributes, I should be in business. So that allows me to write code like this. Now notice entity is totally generic Right? So I've just assigned the result of essentially a get call, and it's giving me an instance of my generic entity model. So I can't do is instance on it and find out, you know, what kind of entity are you? Are you a presenter?

Speaker 2: I have to rely on something else. So I'll get onto that get into that in a second. A slight detour before I show you some Django. As part of preparing this talk, I was thrilled to find this example of a developer working through the same problem in their own framework. DBT, which is a data analytics framework built on Jinja templates. You can tell from the upside down smiley here that this developer went through the same seven stages that I did. She shows her SQL that you know you would just write if you were just gonna do this by hand. And then she asks the obvious question, hold up, how did I know what SQL to write? She explains, well, I wrote some SQL before I wrote some SQL, right? Yet she read she explains the need to have a schema aware query of the generic data. Do a query to get the schema, use that to query your data.

Speaker 2: And then the next obvious question is how can I do this in the most idiomatic way for DBT? Um it was an interesting post and it reminded me like that's exactly what I was trying to do in Django, right? How can I accomplish this All I want to do is create a query set. I don't really need a dynamic model. How can I do this in the most idiomatic way for Django? Okay, an alternative my team also explored. In a sort of 180 from this, we imagined uh creating well not just imagined, we implemented this. Uh we uh spun up some SQL views and triggers and created a sort of wrapper function that would just code gen off all the views and triggers for you. So you had one entry point to just generate those database entities. And they would handle selects and updates. And then we wired them up with unmanaged models in Django. So In a Django model, you set managed equals false.

Speaker 2: Django knows not to create any tables for you. You can just tell Django where to look for the information. You set db table equals your SQL view, and then you hard code out your attributes like spoke at equals one-to-one field. Now you're in business. Now you can wire this up with Django Rest framework. But notice I've traded away a lot of dynamism here, right? If this model changes out from under me tomorrow because somebody went into the GUI and deleted the spoke at field, I've got a problem. So there's drift there. I need to do this in every single project. And I've also forked my own CRUD operations, right? I'm working on arches. Arches has been around for a while. There's battle-tested paths for saving instances and Now if you're doing this, you're jumping off the ORM and into Pure SQL. And finally there goes the ability to, as I described, just define a query set and hand it off to a third-party package, right?

Speaker 2: One of the advantages of the Django ecosystem is that Many packages sling the same lingo, right? They all have hooks similar to get query set that kind of assume that your business logic is going to be implemented as a query set. So those were my goals. Just like the DBT developer, I wanted to do something that was generic or could apply the same pattern in multiple projects, but I didn't want to take snapshots, you know, of my data. I wanted it to be resilient against change. I didn't want to create SQL views and triggers. I wanted it to feel like a Django model, even if I didn't have a literal class presenter. And, you know, I wanted to avoid raw SQL. I wanted it to be overridable and unit testable as Python code. So if your game, I'm going to just show you some slides where I show you what the Python looked like.

Speaker 2: And I'll just go layer by layer. Uh so you've already seen this model layer. I implemented this method with attributes on a custom query set. If you haven't seen that approach, what's nice about it is it's sort of guides you toward putting more of your model sorry, more of your logic in your models instead of in your views. You may have heard this catchphrase fat models, thin views. It keeps things kind of oriented around self. Right? Self being an instance of a model that might have already had some related instances fetched on it. If you're always accessing self dot related set It might have already been fetched and you're not fetching it again, which is an easy mistake to make if you're always starting new queries and custom views. So some of the responsibilities of this helper method were to make sure that I was Queering the schema as efficiently as possible, make

Speaker 2: sure that I was prefetching all of the values as carefully as I could, handling The sort of smart annotation I was talking about, like what exactly is the exact database expression, right, that will drill into that Sudoku puzzle and get me just the data I'm interested in. and what name you know dot spoke at, right? What name does it get assigned to on the Python instance? And then preserving some information about what was uh queried, right? If I've got a model save method and it receives a Jacob object and it's got a spoke at attribute, but it's not it's just a generic entity How is that save method going to know what to do with the value of spoke at unless I preserve some information about what was fetched? And then I wanted to wire it up with an API client. You could use anything. You don't have to use REST framework, but that's what I was familiar with, and that's what I decided to uh you know use for this first iteration.

Speaker 2: So the main things I needed to do were just a mix-in that would allow me to compose different API endpoints with this functionality. and map my schema types to serializer fields. So I use um Spokad as an example and I showed you how that might map to a one-to-one field. So I needed to maintain some sort of mapping from what are the kinds of data I'm collecting. Boolean obviously maps to Boolean, but what does entity to entity map to? You know, one-to-one, something else? And if for it to be fully dynamic or resilient against kind of any model that our users might build, I found myself overriding hooks like GetFields in France to be able to Just like a model serializer can infer all the fields that you need, or sorry, can infer all the serializer fields that you need from the shape of your

Speaker 2: literal Django model, I was doing something similar, right? I was trying to provide functionality for Arch's developers to not have to hard code out every single field in the maritime vessel model, but just have that inferred by overriding get fields. One thing that was really nice was exposing metadata. So I just use that example of you know spoke at. Okay. Is that a required field? Does it have a default value? All of those pieces of metadata were already stored in my um uh schema tables. But a fabulous thing about what ships with Django Rest framework is you can issue an options request to an endpoint and it the serializer will describe itself. It'll show you what the shape of the response would be, what the fields would be, which is especially important for me given how dynamic this is, right?

Speaker 2: To be able to issue that request and find out what the fields are going to be. And the front end could use that to build a generic form at runtime. So that was a really uh yeah good find when when uh doing this project. The challenge though is just that I was working on a mature application framework, right? We we already had APIs. We weren't using REST framework for pretty much precisely this reason. We didn't have a clean way of integrating with it So it just made it a little more challenging to reason about things like validation and type coercion because not all of our traffic was entering from the same point. Okay, uh in a little more detail I can look at those same three layers. So you've already seen this code. This was the dream to just be able to ask a generic factory, give me some presenters. Um Uh and then alter the value and save it back.

Speaker 2: Then I did have to confront the problem of name collisions. So let's say your generic entity model has a few columns on it. I don't know, name, created at, basic stuff. What if your user goes off and creates a maritime vessel model that has fields like name and created at? You have to decide whether you want to guard against that somehow or just separate out that namespace. So one tip I have if you're in a similar situation is that Python ships with this cool tool called Simple Namespace that simply lets me dot into attributes and and uh pretty print the content. So basically, yeah, I just created a separation between actual Django model fields on my Django instance, sorry, on my model instance and these manufactured or virtual fields that were being created through my annotations.

Speaker 2: If you want to look at the model in a little more detail, if you've never seen a syntax for attaching a custom query set to your model, just look up asManager. That's how you do it. Now your dot objects will have your additional methods. It's potentially distracting detail, but I left it in here just to point out that it's possible. I was supporting multiple versions of Arches with this tool where there was drift between the definitions of the models, and I basically threw up my hands and decided I needed to proxy the models for now. So the reason I left this in is just to show that the Django ORM is extremely extensible, even when uh, or extremely flexible, I should say. That if you find yourself needing to adjust the behavior of a model, don't be scared. Start small. Proxy the model if you need to. As I just mentioned, I needed to handle name collisions, so I needed to know if any incoming attributes were actual fields on the generic model, like name

Speaker 2: or updated at, or if they were um Yeah, if they were uh schema keyword arguments like spoke at. Um great, and then I put it in my simple namespace. Uh what I want to point out about uh this right here is uh just I add I have this helper that tries to generate a list of inserts, updates, and deletes. Because you can imagine an incoming generic resource might have thousands of values, and you know it's you want to start from a pattern that's gonna scale. Um But to do that, to generate that list of inserts, updates, and deletes, I needed to preserve the information that happened on the initial fetch, right? What data was I fetching? I was fetching presenters. Presenters have attributes like spoke at. I needed to have that information if I was going to be able to handle safe

Speaker 2: back. Okay, so where did that information come from? Well it came from the query. So here's my custom query set. For hygiene, I've just you know tried to declare what my private attributes I'm gonna use are by putting them in an init. And then I've exposed only one public entry point with attributes. That's the thing that's actually going to do the smart annotations. And then I found overriding a couple of Django hook Django private methods to be necessary. Um if you need to do a more manual annotation, like something comes back from the database, but you need to whatever, hydrate it with just a little bit more data. If you need to hook into the lifecycle of a query set, you can override underscore fetch all and add whatever special sauce you need to. Finally, since I was trying to remember things that were passed to the query set, like I was trying to remember what keywords were passed to it, I found that they were just disappearing.

Speaker 2: It's because I didn't override underscore clone. Overriding underscore clone allowed me to preserve the private attributes that I was trying to save through the life of a query set as it got copied. So in a little more detail, that's what was happening. I had private attributes like Spoke at goes into a list of queried attributes. As representation, are we going straight to JSON or are we still, you know, producing some sort of Python structure like a data class? I had my entry point, it could take a bunch of arguments. I was kind of riffing on the query set API, which allows you to pass defer and only and fancy stuff like that. And then, as I mentioned a minute ago, this is where I was trying to enforce that there was a relatively efficient query for the schema. Save off the information to a private attribute so I can rely on it later.

Speaker 2: And then generate all the f expressions that were going to allow me to drill into the Sudoku table and get exactly what I want from the values. And then if you haven't seen the syntax, if you're um creating a custom query set method, you just change stuff on self. Self is already a query set. It might have already been filtered. Doesn't matter where it came from, you can chain more stuff to it. So I just change my custom annotations onto it. And then I needed to support recursion at kind of unlimited depth, so I just made sure that my models that handled the values were going to fetch their own child values recursively and this is a relatively involved call to models. prefetch but it's it's documented well. It's fan a fantastic way to control the behavior of prefetch related if you need to customize it.

Speaker 2: I mentioned hooking into fetch all. I'm just iterating over the results, short circuiting if it's a values query. Because there's nothing to annotate if you don't have model instances. You've got to have model instances. And then here's where I preserve the information about the fetch, right? I was querying attributes like spoke at. Now I've put them on the entity that I fetched from the database so that I know what I'm doing when I try to save it back. And then also here if I need to do a more manual annotation, like I'm not satisfied with what Postgres gave me, I need to hydrate a little more data, you just do it there That's a good example, right? You might have a flag. Do I need to hydrate any more data or not? And you already saw underscore clone. Okay, uh just to wrap up, like the last layer was DRF, and like I said, this was not as you know, you can do this with the API client of your choice.

Speaker 2: Um The basic thing I needed to do was just have a map from what kinds of attributes I had and what kinds of serializer fields they mapped to. And then I needed to just override get fields and make a serializer on the fly. Just manipulate the field map, push some serializers into a dictionary. Really not more involved than that Making the value serializers on the fly was a little tricky. Here's where I did help myself to Python's type keyword down here on uh line 36. Because as you know, or if you are a user of Django Rest framework, you understand that serializers are classes. But this was not so much to surmount either. This is where I did help myself to creating classes on the fly.

Speaker 2: Um and then finally, as I mentioned it a few minutes ago, all I wanted to do was just be able to provide a sort of well-behaved query set to Get query set. So I have a mix-in that allows me to use this functionality with any kind of endpoints. Are you starting from entities? Are you starting from values? List detail. Parse some arguments that are common to all, and then pass them down to my little with attributes method that handled all of that complexity. Now I had like a well-behaved query set that was now interoperable with basically the rest of the Django ecosystem Yeah. The see how I'm doing on time. Should try to wrap up. So I'm going to go very briefly over this example about trees. But I found myself in the situation of needing to have recursive fetching of the values.

Speaker 2: And I realized, you know, the best tool for that is a recursive CTE, but it's not supported out of the box in Django. So if you're curious about it, have a look on the Django forum. There's a post on there about recursive prefetching with a very tantalizing gist from Simon about how you can do it without any third-party packages. To wrap up, um this was about having a net reduction in complexity, if you can believe it. The number of ways we were doing tr un sort of untransformations on save inside custom views, in custom projects, sometimes before even sending the request. did not always share the same patterns, sometimes did or did not take responsibility for things like you know different HTTP semantics or validation or you know different philosophies of validation or error handling or internationalization of the error messages

Speaker 2: You name it, right? Having instead something that could support almost any project and make the untransformations easier kind of clarified where the responsibilities lie uh or laid for all of those uh all of those use cases. So it was a bit like creating an internal ORM, but it did come off like a net reduction in in complexity Of course, it does vary with your taste for overriding private functionality, right? Somebody tells you to overwrite underscore fetch all, you might run kicking and screaming, right? And like many decisions in programming, it comes down to how much dynamism you really want to buy. Thanks. Happy to take questions.

Speaker 3: Hey. Uh great talk and really impressive project. Uh not just the bit you talked about, the entire Arches project's really impressive. I'm just wondering if at any point you became tempted to use maybe a document database or JSON field in Postgres or, you know, something like this. Uh

Speaker 2: we're we're slinging a lot of JSON, yeah. Some of those, okay, my our Sudoku puzzle is basically a JSON field, right? But uh with uh It's not schemaless, right? Because we've got other tables where we've got all the schema. So that's why I kind of described it as a product of the two tables. It's like we have a relatively flexible data storage format by heavily using JSON fields, but all the keys correspond to real rows and real relational tables where you know the users have defined exactly what's what is the child attribute of what. And because they're you know, foreign keys, they're indexed, and so the lookups and all that are fast. Um yeah.

Speaker 3: Thanks.

Speaker 2: Yep. Uh

Speaker 4: you mentioned that these uh pseudo dynamic models are used for your custom code. Uh so But you actually generate query sets that are richly annotated and and so uh did you find uh limitations in the code that you write later, like um aggregating over them was was that like ever needed and if so what were the problems

Speaker 2: Yeah, great question. Um So you know this tool is only necessary if something about your models is stable, right? If the s identifier of your presenter model is not stable, you have no chance of looking it up. So yes, this is a tool w For yeah, development tasks where some things are stable, but resilient against most things changing. Your question was about aggregation. I guess I found that by avoiding sort of custom views, queries always starting from you know related tables, orienting on self. Help me leverage the ORM better, right? And s and uh no, I haven't found anything that the ORM has been unable to express other than recursive CTEs, but that was my little workaround for that

Speaker 1: Okay, thank you, Jacob.

Speaker 2: Thank you.

Speaker 1: Yep.