Winemaking with Mutable Event Sourcing in Django with Chris Muthig

Speaker 1: Uh so thank you for coming out today. Uh my presentation is going to be about an immutable event sourcing or a mutable event sourcing system that my team at Invent has implemented over the last year. What we're going to cover briefly at the start is what event sourcing is, why my team found it compelling. We'll move on from that to discuss the challenges that we faced in using more standard event sourcing. and why we made the change to mutability. Um I'll discuss the the paradigm shift that came with that conceptually uh talk about some of the challenges that that comes with that paradigm shift. And finally we'll look at some of the code and what it looks like to work in the system on a day-to-day basis. I won't have time to go really deep into the internals of how we made it happen, but um the code will be available for everyone to review afterwards, so you can always look into it.

Speaker 1: First a little bit about myself. I've been writing software for about 15 years now. This is my first DjangoCon though. You can find me on GitHub. I do a little bit of open source work. I made my first Django contribution this year. It was pretty exciting stuff. Um, I've been immersed in the wine industry for nearly as long as I've been making wine. It started about 10 years ago. I started pouring tastings at the winery where my wife worked as a winemaker. From there I started helping out during harvest. I started helping out during the bottling season. I did everything from picking fruit to hitting punch downs to pressings all the way to that bottling phase And it's it's been a really a passion for me for that whole time. I've really enjoyed the process of actually making wine.

Speaker 1: Um and so these days I I make a little bit of traditional method sparkling cider at home. uh go through the whole process on my own and it's it's usually uh drinkable at least. Uh about six years ago I started as a tech lead at Inevent. Uh at Innovent our mission is to empower wine producers to produce to operate and advance healthy, thriving businesses. We provide a wide range of products that help them go from picking their fruit all the way to managing bottles that are ready for sale or being sold. Um, we do all of this at a level of granularity that was never previously available to winemakers, and so they can track composition, cogs. uh regulatory issues, auditability issues, tax issues, all of that comes together for them in our software.

Speaker 1: And all of this is built on Django powered APIs. So we have Angular front ends, but everything on our back end, all of our data store, all of our uh complex calculations, that's all done by Django. So, starting off with a very brief conversation of what event sourcing is. Event sourcing is a design pattern that captures all changes to a system. as a stream of immutable events on an append only log. And so what that means is that in a normal world, say like a Django ORM world, if we wanted to track a bank account, which is the most common example of an event source system usually. What we might have is a table, a model called account.

Speaker 1: And on that model, we might have a single field called balance that tracks what the balance of that account is. And so if someone uh deposits $10, that balance field now says $10. If someone withdraws $4, that balance now says $6. In an event source system, you don't store that single model. Instead, you store all of the things that have happened to that model. And so in the case of a bank account, we might have an event stream that looks like this where we have one event that says money deposited, $10 , another event that says money withdrawn, one, and another that says money withdrawn five. If a user wanted to look at their bank account, the way we would accomplish that is we would pull all of the events linked to that bank account

Speaker 1: and we would replay them onto an empty model. And so that balance would start with 10 plus 10, minus 1, minus 5, and we'd be able to see that their current bank account balance is 4. Uh the event sourcing paradigm comes with some really great benefits that our team found compelling, the reasons that we really wanted to take this on. The first was auditability. In a winemaking world, especially in the US, there are a a lot of government auditability concerns. You have to be able to track the entire life cycle of alcohol in the United States. And so by accepting the event sourcing pattern, you get auditability for free. Every event is your auditability.

Speaker 1: built in for you. And so that was really powerful for us. It was very important for us. The second feature that we really liked about an event sourcing system was the idea of temporal queries. And so when we say temporal queries, we just say we want to look at the state of our system at a particular point in time. And so, you know, from an accounting perspective, they don't really care what it looks like right now. They care what it looked like at the end of the fiscal year. So maybe their fiscal year was December 31st. Maybe their fiscal year ended on February. 28th. They get to choose that time. They want to see how much wine did I have at that time? How many financial assets did that represent? How much did the wine cost me at that point in time? They can replay all of the events in their system

Speaker 1: up to that particular date and they can get a view into their system at that point in time. Again, that's a really powerful feature for us if you're reporting to the government. You're not usually reporting right now. You're reporting on what it looked like four days ago. And in the last four days, you have moved thousands of gallons of wine. You've bottled thousands of gallons of wine and already sold it. It's gone. So if you look at what it is now, you cannot report to the government. Uh it's it's really important to also say that event sourcing is not a silver bullet. You shouldn't take the ideas here and just go wild with them. There's a lot of overhead, obviously, even just getting the state of the model becomes a query that you have to replay a lot of events for

Speaker 1: We use these event sourcing systems when it's important. And so if we see, hey, this aggregate, this idea of an account. needs to be temporarily queried or I need to have a strong audit trail, we use event sourcing. For everything else, we use standard Django models. It's all just there. It's easy to use and that hybrid approach gives us the right tool for the right job. So let's get to winemaking now. How is wine made? It's a pretty easy process, right? You take some fruit off of a vine, you crush it, you step on it, however you want to get that juice out. Throw it into a tank or barrel with some yeast and let it ferment. And then maybe you let it sit in a barrel for a while and age. Maybe you throw it right in a bottle and drink it.

Speaker 1: It's a pretty straightforward process. Anyone with a little bit of yeast and a gallon of apple cider at home can make wine. I do it all the time, and it's it's pretty fun. But if it's really that easy, then why do we actually need software to accomplish this? And the answer to that is it's it's not that easy. What you can see here are three workflow diagrams from one of my wife's wineries. The first two, left to right, represent the first 10 hours of every truck of fruit they receive. And they will process those first two workflows 24 hours a day for eight days or for eight weeks every year. during harvest. Super stressful time for them. That's the only time they can pick the fruit.

Speaker 1: It's the only time it's ready. And if they miss this opportunity, it's over. The third workflow is their fermentation, and that could run maybe ten days total. But all of this is happening constantly. They receive trucks every hour of every day for their entire harvest and they just cycle through this. And as soon as that fermentation is over, they have a secondary fermentation. As soon as the secondary fermentation is over, they have stabilization. As soon as stabilization is over, they start doing fining. After that, they start thinking about how they're going to blend. It's a very complex process, and that concept of having one shot to get it right puts a lot of pressure on winemakers and wine producers. to really make sure that they reduce their errors and they understand everything that's happened in their system through the whole life cycle.

Speaker 1: So for our examples today, we're going to simplify everything down and we're really just going to focus on blending and bottling because with just these four steps we can hit a lot of edge cases and a lot of uh not fun concepts. And so we've got four user behaviors we're gonna focus on today. First is receiving wine. We're gonna say we receive that in volume and gallons, because this is the US. And that's maybe coming in on a tanker truck. Maybe you had it pressed somewhere else and it's just coming in. That's how we get volume into the system. From there, we have a measure concept. This is really important and often not known about in a winemaking world, but if you're working with volumes, especially alcohol, and you're working with porous containers like wooden barrels or concrete tanks or clay amphora.

Speaker 1: The the tanks will absorb some of your liquid. It will disappear on you while it sits there Uh and even worse, if you're using barrels, some will just evaporate right through the barrel. We call the the volume loss to the container the devil's share and the volume loss to the air the angel's share. It happens all the time. And on top of that Whenever you start moving wine around, it isn't a lossless concept. Just because you said you moved 500 gallons doesn't mean 500 gallons made it from one tank to the next tank. The operation of a pump is a bit of an art and not not exactly a science. And so it's really easy to lose wine along the way, just moving it from one place to another, trying to blend things together. And so Before you do anything, before you take any action in a winery, the first thing you always do is measure

Speaker 1: how much wine do I have. I said 500 six months ago. Is it still 500? It's probably not. It's probably a little bit less than that. The next behavior we have is blending. This is a complex process in most winemaking procedures. This is where the winemaker's vision really comes through. You take uh any number of lots from Two to I've seen up to two thousand and you blend them all together to create a a comprehensive palette of what you want your wine drinker to experience. And so this is where a lot of the complexity in wine comes from, because that blending is now a point-in-time concept and critical to understand what is the state of this one wine lot that I have created.

Speaker 1: You can see we have a loop back to the measure state because normally in blending you don't get it right the first time. You might have to do it a couple of times, or if you just have 200 lots, it might just take a few tries to get them all together into one tank So we go back to measuring again and then we keep blending until we're ready and we bottle it down. In our system, bottling is where volume leaves our system and we stop thinking about it. So, with all of that in mind, we can talk about the challenges that my team faced with event sourcing. So the first challenge we faced was delayed recording. In many technical systems, the technical system is reality. You're, you know, doing thought work and everything you do with your technical system, that's that's the truth.

Speaker 1: In our space, we're working more in a manufacturing setting. And so there are people accomplishing tasks, they're re-measuring tanks, they're receiving volume, they're blending lots in an actual winery somewhere. Uh wineries are not known for having great data, and so some of them might have Wi Fi. You might be, you know, thirty feet underground working in a cellar all day, and if you don't have Wi Fi strewn throughout your entire cave system You know, you'll come back up for air 10 hours later and then you'll have to record everything you just did. And so we always have this idea of a delayed recording system. Some people might be able to record their behaviors quicker, others might take 12 hours, they might take three days. A bottling run can take up to three days depending on how large it is.

Speaker 1: And so our users might start it on Monday. finish it on Wednesday and still want to record it as if it happened on Monday, just because that's when they started it. That's when it mapped to their conceptual understanding of their world. The second issue we faced was correcting mistakes. Like I said, harvest is a very stressful time. Most wineries are running 24 hours a day. uh picking fruit at one o'clock in the morning, receiving the first truck at three o'clock in the morning. You have oftentimes people who are working their first harvest or college interns working their first job ever, who are on their sixth day, on their tenth hour. It's five o'clock in the morning, and they're processing

Speaker 1: $20,000 worth of fruit and just trying to record information to the system, we received 2,000 gallons. Maybe they meant to hit 20,000 gallons. Maybe they meant to 200 gallons. It's hard to say. Usually in our system, there's the people working in the cellar and then there's someone like a winemaker doing checks and corrections. and altering the plan for the day proactively as this digital work order flow goes through. And so they're seeing that and they're like, did you really mean to receive 200? Because we were expecting a 2,000 gallon truck today. And they'll need to fix that mistake. In an event sourcing system where it's an immutable concept, that can get kind of hard, especially when we add it to our last concept, which is the implication of ordering.

Speaker 1: So, like I said, that blend action is our main challenge. We can see here we have two user behaviors, a bottling and a blend in these charts. They're using lots A and lot B. Lot B is 100% grapes from Sonoma Valley. And lot A is 100% grapes from Napa Valley. This is important because whenever I bottle wine, there are US regulations that say what I can put on that label. If I want to label this as Sonoma or Napa, I have to be able to prove 85% of my grapes in that bottle came from that region. If I can't prove that, then I can't put it on the label. So this is very important for our winemakers. So we can see here we take 200 gallons of uh Lat B from Sonoma and we bottle down 100 gallons.

Speaker 1: That wine is finished, it's gone. I'm gonna slap a label on that. We take the rest of that lot and we blend it with lot A. So we have a you know one -third, two-thirds split between lot A and lot B. And so my final blend percentage is about 33% Napa and 67% SNOMA. Now let's look at those same actions reversed. If I did that blend first, then the result of my blend 50 gallons into 200 gallons is that I have a lot that is 80% SNOMA and 20% Napa. I then bottle down 100 gallons, and I've just lost a lot of money. Because the final label that I can say for this finished wine now is probably like

Speaker 1: North Coast. It's still a nice, nice region. It's great. It's in California. People recognize it sometimes. But You know, compared to a bottle from Napa, yeah, probably sell it for a quarter of the value. So if that was a hundred dollar bottle of Napa wine, I'm gonna sell that for $25 now. I just lost a lot of money. And more importantly, I didn't really match reality. If the government came in and wanted to audit me and say, cool, like you said this bottle was 80% Sonoma, well, if the bottling actually came before that blend, I need to be able to record it that way And like I said earlier, it can take a long time for some bottling runs to finish, and our users might record the next blend in the behavior way before they record the bottling.

Speaker 1: And so they need to be able to handle that concept that, yeah, I'm recording things out of the order that they actually happened in reality. And I need my system of record to match that. Usually in an event sourcing system, you might record like a corrective action where it's like, hey, you know, uh I I have 2,000 gallons right now. It was supposed to be 200, so I'll just record a corrective action of minus 1800. Not okay. The government is not going to be chill with that. You can't just say 1800 gallons disappeared. You can, but you have to tell them when, why, and how. Like it is very strictly recorded. They are strict about alcohol in the US. Outside of the US, they're a way more chill. But in the US, if you don't have this information, you're in a bad spot.

Speaker 1: So, those are the challenges, and from there we can start talking about how we actually solutioned on this. Hitting some core concepts here. Uh the first core concept is that our aggregates, those are our core domain models in our instance of wine, that's a wine lot. Uh they are Django models. So first place we cheated. I told you that event sourcing systems don't track the current state of an aggregate, they build it up. We recognize as a team that 90% of the time we actually do want the current state. That's where most people are looking right now. They have specific times where they need to look at a point in a time in the past, and we want to support that. But we want to make sure that use case of I want to see my current state is really snappy. And so we cheated

Speaker 1: and we did both. We have all of the events behind the scenes, but we also store all of our aggregates as standard Django models. You can query them with the ORM. you can get them back super fast, you can filter, you can aggregate, everything like that is available to us. The second core concept is that our event streams are also Django models. So again, Django engineers can be really comfortable here. We use our Postgres database as our event store, and we can filter, aggregate, everything we want to do using these Django models against each of our event streams that represent a different aggregate type. From there, we encapsulated the majority of the actual event sourcing behaviors in our system.

Speaker 1: We did this mostly using a concept that we call an aggregate repository It's a singleton class that implements a unit of work pattern. So really similar to how a lot of ORMs work where you just kind of instead of using transactions, they kind of hold in memory all of the things you want to do. And then they push it to the database at the end of the unit of work. We do a similar concept here where we track what are the aggregates that have been updated What are the new events that need to go into the system? We're gonna hold onto those within this. Uh we use it as a decorator or a context manager And at the end of that decorator, as it exits, it persists all of those aggregate updates into our database for us. It persists all of those events into our database for us. And it does a couple of other syntactic

Speaker 1: sugar things that I won't have a chance to cover today, but it's it's pretty nice. Uh and the fourth core concept is that we map our users requests our those complex behaviors remeasure blend uh bottle we map those as event sourced aggregate And so their request is not some ephemeral thing that came over an API or came in a form. It's an actual Django model that we have captured the state of as an event source concept. That means that our users can look at the past versions of their user behaviors, they can make edits to those user behaviors, and it's the core concept that allows us to create the mutability that we need. So digging in into those core concepts, we've got the persistent actions.

Speaker 1: So these are the aggregates. You can see that we have one for each of the user behaviors we defined originally: receive, blend, remeasure, and bottle. They all come with an ID. This is a monotonic ID. So we can't say that they're going to be sequential, but I can say that if you create an action before another one, the second one's going to have a larger ID. We have one pattern to do it. The example code I have here today uses a ULID pattern to create a monotonic string representation ID. They also have an effective at field. And this is the field that we allow our users to define to say when one of these behaviors occurred. This is what allows for that edit capability. This is what allows a lot of stronger functions for us.

Speaker 1: And you can also see that each one of these actions, our domain logic processes into a set of events. Not always one-to-one. So our receive action will have one event with it and that's volume received. But our blend action could have multiple action or multiple events. So volume removed, that would be for any of the lots that are being blended into another one. We're removing volume from those lots. And so we need to represent that as a loss in volume. And volume blended is a more complex action for the one that's receiving all of those. We need to say how much volume did you receive of each lot specifically? Because that's gonna be what defines that composition later, right?

Speaker 1: So again, these are all saved in the database. They are event sourced in a much more true sense. These are not editable in the same way. They they handle they act as an append-only immutable set. Once you create an action, we have a action created event. If you update that action, we have an action updated event that alters the data in it. And so that's how we get that versioned aspect here. But there's you conceptually it doesn't make sense to go back and edit an edit on your behavior. You just come up with a new behavior. So after we had all of that together, we found three scenarios that our system needed to be able to support.

Speaker 1: And those three scenarios were editing, deleting, and inserting an action. And each one of these is really what started making the event sourcing difficult for us. So we're going to walk through some diagrams here to kind of see how it flows at a conceptual level. So editing and action, which is where we're going to start. This is the most complex of them. And so after this, it just goes downhill. But so let's say we have an event stream here where we have volume received, bottled, and bottled all on a single lot, lot A. And they all can presumably come from actions. Well, our users look at the system and they say, we actually made a mistake on that bottling action in the middle on September 3rd. We need to change it so that instead of saying it was 75 gallons bottled, we bottled 50 gallons.

Speaker 1: Someone just made a typo, right? So the first thing that they're gonna do is edit that action to give us the new state that they want to. put onto the world. We call this the intent in our system internally. What is your intent in this action? Well, I want to remove 50 gallons. So once they have told us that, we need to rebuild the state of our aggregates up to that point in time. We do that by replaying those events onto an empty version of it. So we can see lot A had one event up to that point. It received 150 gallons. So our state right now is 150 gallons. We then delete the old version of the event. We completely remove it from the event stream. This is all handled by that aggregate repository.

Speaker 1: And we create a new version of the event that represents the user's current intent. So they said they wanted to actually bottle 50 gallons. So we're going to create a new version of that event and put it in there and say 50 gallons. And from there, we need to rebuild the current state. So, like I said, we have our lot A aggregate all the way on the uh far right there. That is representing the current state. It's a Django model. It's still there and we need to update it with the correct information. So we're going to replay the events that come after this one onto the same aggregate. So now we're at 100 gallons because we changed our action. And the next step is to remove the 75 from the final bottling. And when we do that, we get our new lot aggregate state of 25 gallons. Our aggregate aggregate repository, that decorator, is going to handle all of the information for us from

Speaker 1: okay, I need to delete the old version of that event. I need to persist the new version of that event, and I need to update that aggregate that now has a new state on it. All of that is handled more or less behind the scenes. So the next thing we had to do is delete an action. This is pretty much the same steps as editing an action, except you don't have to put anything new into the system, right? So the user tells us I want to delete this bottling that said I bottled 50 gallons. The first thing we do is we mark it as deleted. Uh in an auditable system, we want to make sure that user behaviors that were once recorded are not lost. And so we don't delete the actions. We don't delete the user behaviors from our system. We mark them as deleted and we go with a soft deletion behavior.

Speaker 1: This way If someone made a mistake or if someone was new and just wasn't sure how to handle it, uh a winemaker, a you know, someone in the finance team can look back and say, what happened? I really thought we bottled that day. They can find that deleted action and understand, okay, well, maybe we inadvertently deleted this. So we mark it and we use a timestamp to say when it was deleted. So nullable field, if it's null, it's deleted. If it's not null. uh then it's or sorry if it's null then it isn't deleted and if it's not null then it's deleted we know. So we mark deleted And then we build up that state again. So we said we received 200 gallons this time into the system. So that's our state at the start of this action that we're removing. We mark it as deleted, the actual event, and remove it from the system.

Speaker 1: And then we rebuild down from there. So there's nothing to insert this time. We just remove something. So we get our state as 200 gallons still after this action that's been deleted. We apply the 75 gallons bottled after it as the next event, and we get our new state of our aggregate as 125 gallons. The final state that we had to really support here was inserting an action. With our users, we call this backdating because inserting an action is a very odd concept. to to stell people, I guess. And so the words just work better to say we're backdating it. You're setting it at a particular time. And so here we have an event stream. Where we received 200 gallons, we bottled 50, and then we bottled 75. So our final state is that we have 75 gallons.

Speaker 1: Well, someone wants to go in and backdate a blending before that last bottling Because they have changed the composition before that bottling. And based on the diagrams we saw earlier, this is important. They need to be able to go backwards and say This is where that blending occurred. And so that last 75 gallons that was bottled has a different composition than the first 50 gallons. So they give us an action. You can see here that it has an effective at that is before the last effective at in our event stream. And it has an action ID that is larger than our last effective app in there. This is how we order our system. And we do give some leeway to our users. They're allowed to backdate two actions to the exact same time.

Speaker 1: They can say they did two things on January 1st at 9 a. m. And in order to support that, we implicitly take the idea of the action as a tiebreaker in all of our systems. And so whichever action was actually recorded to the system first, we say came first. And we do tell our users, if you need it to be more specific than that, then you know offset it by a minute or so, if you really want to be clear about this. But we use that combination of the ID of the action and the effective at of the action to create that ordering in our event sourcing system, where usually that would just be whenever it came into the event stream, that's where it fits We handle it a little bit differently. So

Speaker 1: our users give us this action, this blend action set at September 3rd, 150 gallons. We recreate the state of our lot up to that point. And so we received 200 gallons. We bottled 50. So we're at 150 at this point in time. We insert the new event into the stream at that exact place using that ordering. Then we rebuild the state after that event. Now we have 300 gallons because we just blended in 150. And we replayed that downstream. So we're taking 75 out of 300 now. And instead of ending up with zero, we end up with 225 gallons in our system. So all of this is great, but it did come up with a strong edge case from an

Speaker 1: event sourcing perspective. Usually with an event source system, you validate the intent and context of each of the events before they go into the stream. So I already discussed intent. Intent is what a user wants to do. Context is everything around what they want to do And so in the case of a winery system, they might want to drain 50 gallons. The context would be, do I have 50 gallons? Do I have 100 gallons? Is this an empty lot that you just try to remove 50 gallons from? So usually you would validate that at the time of the event creation and you would say, yeah, I have I want to remove 50 gallons, I have 100. This is a valid operation Well, because in our system you can edit that stream, we can't count on that anymore.

Speaker 1: In a normal event sourcing system, once an event is in the stream, it's always valid. In our event sourcing system, that is no longer the case. And so looking here at this stream, we received 100 gallons, we bottled 75, we have 25 left. We want to insert backdate a bottling action before the final bottling action, saying we bottled 50 gallons. That's fine. We go back, insert this event Build the aggregate state up to that point and then start replaying down. Now, as we hit those replays, we have to do that contextual validation again. And so within our code, this is one of the only difficult things and mental overheads right now is you have to, whenever you're writing the validation for your actions.

Speaker 1: You have to understand what are my validations that are contextual and what are my validations that are just standard. You know, maybe in a bottling action you tried to report that you bottled negative 50 gallons. That doesn't make sense, but it's also not contextual. So we can validate that you gave us a good integer to say, did you bottle a real number? But we can't necessarily validate that you had 50 gallons. Those are two different concepts. And so Again, our aggregate repository handles this for us. As we replay those downstream events, we separate out contextual and non-contextual validations. And we reapply those contextual validations for our users as they go. And if there's an issue downstream, we bubble that back up to them and tell them like, hey, there's an action down here, it's this one.

Speaker 1: You need to change something about it for this edit to work. Or maybe it just gives them an idea like actually I made a mistake and I picked the wrong lot. I was supposed to be bottling lot B, not lot A. But all of that check for them is critical because it keeps them within regulatory compliance and it keeps our system consistent. Alright, so let's look at some code. Let's see if I can do this. Awesome. Can everybody see that? Great. So we're gonna just high level look at some code here. Um So we use a kind of a service layer. We call them use cases. This is where we put our core domain logic.

Speaker 1: And so you can see here we're going to look at our behaviors of receiving volume. The first step here is that store aggregate changes decorator. This is our aggregate repository decorator. If we dig into this concept. . . You can see that we create an instance of our singleton and just let the response do what it wants to do. And At the end of it, we will either persist all of the changes into our system or we'll clear them out if there was a failure somewhere along the way. Because this is a singleton, each one of our aggregates can access it in the same way. And behind the scenes, they're pushing their updated state into this aggregate repository for us. so that we're really not having to think about it too much.

Speaker 1: So that's where we enter. From there we do some, you know, high-level checks. We're gonna skip the effective app for right now uh because we'll do that in a second phase. Um so they give us a a wine lots ID. We want to pull that wine lot, that aggregate out of our database. Again, we have Our current state of aggregate saved as Django models, so we can pull them right out. We don't need to do anything fancy, and they're not backdating this action, so we know they're applying this on the current state. So we just pull out the current state real quick, make sure it's there, and then generate an action. Those actions are aggregates. I'm not going to dig into that part too much because uh limited time. But then we take that action and we process it.

Speaker 1: We apply it onto the world and we apply it onto each of the relevant aggregates in this system. And so in this case, it's just a really small function. We take the action, the volume received stated on that action, and we call that on our lot aggregate. We will dig into this one. So you can see here it does a little bit of that validation. It checks, is my lot deleted? I don't want to record actions on a deleted lot. And from there it creates an event. So you can see here this is a volume received an event. These are um Pydantic models under the hood for us way down here. We found that super helpful. We use Pydantic to help us serialize and deserialize in and out of JSON fields in our event

Speaker 1: stream. But we create that, we add all of the relevant information, the aggregate ID, the actions ID, when it occurred, and the volume. And then we apply that event onto our aggregate. This apply function is part of the syntactic sugar that we've built out with our core aggregate model class. It will look for a function based on the name of the event. So in this case we have a volume received event. It's gonna look for a volume, apply volume received. Function that we can see here. And it's simple stuff. We're not doing anything crazy right now. When a volume received event is applied onto our aggregate, we update the volume of that aggregate with the new volume.

Speaker 1: So you can see this is how it is updating itself internally based on the state of events. And the other thing that apply function does is it does that contextual validation for us. And it is what actually puts those events into our aggregate repository. And so again, we don't want our developers thinking too hard about the actual event stream. We want them thinking about the domain concept behind the event stream. And so the supply function covers a lot of that up, makes it easy for us to work on a day-to-day basis. So we process the action to generate all the events. And we return it back out. Done, right? We have that state. We can return it out. This might go back up to for us a API layer, either DRF or Django

Speaker 1: Ninja, and we'll send out the data about the action that was created. Now, going a little step deeper, we can look at backdating that event or that action. So let's say a user wanted to backdate this to last Monday, not to today. We do again some high-level checks. You know, is the effective at time that we're trying to put on this event actually in the past? Did you accidentally put it in the future? We we don't want to support that. So we do some checks on that. And then what we do here is something special and more encapsulation. If we're working with a backdated aggregate, if we want to backdate this action. we need to grab the state of that aggregate at the appropriate time. And so we have a nice helper function, load editable aggregates at time.

Speaker 1: It's a mouthful, but guess what? We can do worse. It is very descriptive though, and that's what was important to our team. And we hand this a list of aggregates, all of the same type, preferably, but behind the scenes, it will handle that for you. You say, hey, get me this list of aggregates at a particular point in time. It goes into the event stream, pulls out all of the related events, and builds up the state of those aggregates for you automatically. We don't have to worry too much about n plus one queries here because it does it per aggregate type. And so if I had vessels and lots all coming together at the same time that I needed to work with. It would say, all right, well, I'm going to make one query to the database to get the events for the lots, and one database query to get the events for the vessels.

Speaker 1: Those are held in two different streams, and so we'll have two queries to pull that off. So we get the backdated state, and then you can see it looks pretty similar after that, right? Like we create the action on the aggregates and we process it. The process is the same function. We don't have to change the domain logic of what it means to process an action. And then the only different step here is that we do have to reapply those downstream events. So we do have again another helper function. Where you take the aggregates that you've updated, you hand it off to this helper and say, hey, here's this new state of this thing at this particular time, at this action, in fact. I need you to take everything past this and replay it and give me a check. Just make sure everything's fine

Speaker 1: and update my aggregate for me. And this does this all for us automatically. At the end of this, our aggregate repository has what aggregates have been updated, what events have been created, puts it all in there for us, does all of our checks, and then we return back out the action Uh and the other op possibility that I wanted to walk through with you all is editing that receive volume action. So this one is a little more complex. Uh we step into that aggregate repository again. We do a few more checks to make sure that the action actually exists, make sure it's the right type of action that we're trying to edit. We grab out that wine lot that was referenced in the request and then we do a little something different because this edit

Speaker 1: might not be changing how much volume I received. It might be changing what volume I received. It might be changing the lot on my action. And so even though this action references a single lot, I might be dereferencing the old one and adding a reference for the new one. And so we have to be able to handle that. We check to see like, is there a different lot in the request than there is in the action? If so, load the state of both of those. And you can see we have another helper function here, I told you I could do worst. Worse, load editable aggregates at time and point So this one actually takes both the time and what we call a sequence number. We use our action IDs as that monotonic sequence number. And it loads the aggregates at that point in time based on that action tiebreaker.

Speaker 1: It also does one other thing for us. It finds any events related to those aggregates from that action ID and it marks them as deleted. That's where that helper comes in place. That's how our aggregate repository knows. These are the actions or the events that I'm going to get rid of. I'm going to replace them with something else this time. So from there, we uh update the action aggregate, give it the new data, and then It should look familiar. We process the same action back onto whatever lot we're working with. And finally we reapply those events downstream. This is one place where we do have some room for improvement. Right now we're reapplying those, and this does create a bit of an M plus one situation for reasons that I can't talk about too much, but we have under the hood an optimistic locking pattern as well.

Speaker 1: That makes it a little more difficult to bulk update and bulk insert. We have some ideas on how to fix that, and we just haven't had a chance to really dig into it yet. Um But even for us when we've looked at, you know, 200 to 400 aggregates at a time, this isn't really an issue. It's super snappy, super fast to do these checks, these validations, and update those models. Um and I just have like one or two more minutes. And so I did want to um Show uh again this code is available afterwards, so please dig into it if you're curious. If you want to see where it starts getting hairy, we do have a use case in here to calculate the composition of a lot Understanding that you might want the lot now and it might have been blended into

Speaker 1: 50 times. It might have been cyclically blended into lot A into B into C back into A, back into B, back into A And doing all these calculations is much more complex. In reality, we handle this asynchronously. We push it onto a task Q to create this composition. In this case, I have shown a function that iteratively goes through, determines all the downstream calculations or the upstream calculations to get the composition, and then replays those events for you as a single stream. I highly recommend taking a look at this. It's it's some fun code. Um Awesome. So all the code is available on GitHub. You can also find me on GitHub if you want.

Speaker 1: You can also find me on the DjangoCon Slack. I'm at ChrisMuthig. If you have any questions, please come find me, send me a message on Slack, open a discussion on the GitHub repo. It's the discussions are open for you. Uh and finally, we're hiring. We're looking for a new senior back-end engineer to join our team. So if all of this sounds super exciting for you and if you want to really dive into winemaking. Uh we're a product first team, so you will learn a lot about wine that you never thought you needed to know. Uh come talk to me. Thank you.

Speaker 2: Thanks, Chris. We have a minute One question?

Speaker 1: I can try taking a question, sure.

Speaker 2: All right. Ryan's gonna ask a question. It's gonna take five minutes to answer. Uh

Speaker 3: thank you so much. That's super interesting. Um I was wondering if you could talk a little bit more about The cyclic events, did you take any inspiration from Git and how they handle making the graph a cyclic? And then the other one was um Do you lock the uh aggregate table uh in order to sort of like or I guess the question is what do you do about two users trying to change uh events that would Uh have an outcome in the aggregate?

Speaker 1: Uh no, I did not look at GitHub's implementation of the cyclic graph. Maybe I should have, but it might actually be more complex than even ours. As for uh conflicts, we under the hood do have an optimistic locking system built into all of this. Um if you look up the GitHub repo, I I talk about it a little bit in the README and link to where it's implemented uh but we essentially have a version on each one of the aggregates and so we do a check on that version before we persist. As we pull it out, we do a check and then before we persist we also do a check to make sure that we don't have conflicts going on.

Speaker 2: Cool, awesome. Another hand for Chris and uh sorry we're at time. We'll have to ask questions in the hall.