Tarun Garg - Design Django models such that your future self will thank you 👋

I'm so excited to be here. It's been a dream to speak at DjangoCon since the day I started working with Django. First, I'd like to start by thanking everyone involved in making this Conference happened the complete Jango Khan organizing team, the volunteers team, and each one of you present here. You folks is the reason why we are having the virtual conference in the first place. Make no mistake, virus is still creating havoc out there, but as there is dialogue in Bollywood that the show must go on and you folks are ensuring that show is going on. So kudos for that Talking about myself, my name is Tarun. I work as an engineering manager at a startup named ScottStack. We are a startup based out of US and India, where our primary objective is to optimize Big ticket B2C sales funnel by using a combination of data science and technology.

Without further ado, let's get started with the talk. The title of my talk is Design Django Models Such That Your Future Self Will Thank You I know that title might sound a little bit clickbaity, but I hope that the content that is coming your way will make it worthy. And if not, feel free to leave me a note on Twitter. My Twitter hand is in the bottom right. Goes by tarun underscore garb to I'll take that as a positive feedback. And if you feel like that I made an impact on your learning positively, then also leave me a note on Twitter that will just make my day. Moving forward, let's decompress what's coming up in this talk. First, we are going to be discussing what this talk is about and why should you care. In this section, I'll try to explain what is the purpose of this talk, who is the intended audience.

Etc. etc. And hopefully by the end of this section, I'll be able to convince you to stick around for a little bit longer. Then we'll set up a premise. So, how the rest of this talk will happen is that we'll set up a premise, and that premise will be building a product and we'll go through the product development lifecycle of this app and see how at different points different learnings happen with respect to Django models The idea again, the objective again would be to ensure that when you leave, you have a clear-cut idea as to when and how to implement those set themes. And the all five themes that we are going to be discussing about in this talk are auditing your models, timestamping your fields, soft deleting your entries, use of choices, and denominalization. Those are the top five key themes On which all of our learnings will be directed towards, and each one of them will be discussing when to use them and not to use them.

Each in each one of them will discuss nitty-gritties of each of those key themes or areas And then we'll end this talk with key takeaways and noting down the insights. First thing first, what is this talk is about? I've been working with Django since 2016, and past five years I made a lot of mistakes with respect to Django And Django modeling. So this talk basically encompasses some of those learnings and mistakes I made in past five years. While I was building those asked product features, these were not so obvious mistakes, but it is in the hindsight only that those turned out to be mistakes. So my objective here will be to ensure that you don't make those mistakes again in your Django while you're designing your Django models and you make new mistakes and we all learn and grow as a community.

And you design your jungle model such that your future self will thank you for. These are essentially the things that I wish I knew I started. So I'm just telling you so that you know when you are starting. Although this talk is relevant for All programming levels, be it novice programmers, experienced or intermediate, everyone should enjoy this talk and I hope you'll take out something good from it. So let's get started with setting up the premise of this talk. So for setting up the premise, I thought what would be the common ground that we all as an audience can relate to, and I thought let's build a messaging app We all have used a m one messaging app or the other in our life till now, be it Slack, WhatsApp or IRC or

you know uh Discord or Microsoft Teams or you name it every organization has a different s form of You know, IRC chat implemented. So I thought let's build in Yama. Let's call it Yama. Yama is yet another messaging application. Why am I? Because why not? We have a lot of messaging app in our ecosystem. So we need to build a unified app That encompasses features of all of them and that encompasses cons of each one of them individually. Obviously, it's a joke, and the only reason why I start why I choose Jamma or building a messaging app is due to the fact that I thought we all can, you know, connect on that particular thesis and all the incidents that I will I'll take you through in this talk

are completely uh you know made up and any resemblance with the real events is purely coincidental It's better to clarify that. So far so good. We are gonna be building Yama. So first let's look at what would be the high-level database design of Yama So in Yama, we would first have an organization. You can call it organization or customer or client or whatever you want to call it. Those are the basically buyers or Of your product. So basically you would have an organization. Each organization would contain different channels in them, and each organization would have a set of users And then users will post messages onto those channels and users will also react to those messages. What do I mean by reacting is Users will post reactions like emoticons or reactions like thumbs up or thumbs down or spiral notepad

or something like that So, this would be the high-level database design of Yama would look like. I've also tried to depict what would many-to-many and one-to-one relationship would look like. Although it might not be 100% accurate, but I hope it just gives you some idea. So moving forward, so we are building Yama, let's get together to the adventures of Yama Now, suppose product team comes up and said that we need to add a support for adding a new user via user interface as well as CSV, CSV is comma separated values A developer in me said, sure, we already have a user model for that. I'll just add the support for import soon. And what that user model used to look like back then was Use classic Django stuff where user is inheriting from models.

model and user would have some some fields like username, profile picture, and other user-related fields. This would be the user model would look like. And I said, okay I just add a support for imports on top of it and everything should be good. Three hours later product team comes up and said, hey, can you tell me why does this user exist in our system? A developer in me said, sure, allow me some time to debug. And exactly 10 or 11 minutes later, I realized. That in order to debug why this particular user exists in our system, I need to know when this user was created. And I realized that I don't have that information in my schema because that was not the Asked product feature. The product feature is first just to implement the support for

user. They did not ask that you might need when this particular user was created in the future, you might need when this particular user was updated in the future etc so that that that was my first learning was and that's why the first theme that we are gonna be discussing today which is audit your models So, first let's discuss what is audit, and as with everything we do in programming as programmers, let's Google the definition According to Google, an audit is an official examination of present state of something. Again, repeating, an audit is an official examination of present state of something. You might think, how does this, but how does this relate to Django or programming in general? So auditing your models means that have some metadata fields which will help help you audit your models later. Speaking more practically

Have some fields like created at or updated at in your model that will help you audit your model later. That when this particular model entry was created, when this particular model entry was updated This is the most basic form of auditing that you might have in your Django model by having created it. Where created that means when this particular model entry was created. Updated at means where this when this particular Model entry was updated, and I enabled both of them by using two Django constructs, which is autonow add and auto now. So, this is the most basic form of auditing that you can have in your model Then you can also have more advanced form of auditing by having things like updated by, who updated it, created by who created it, deleted by, who deleted it, and raw JSON.

Raw JSON can be thought of suppose you are receiving a JSON from a third party and then you are transforming the JSON and then you're storing the transformed thing in your model, right? Uh as separate Values or properties, right? So in that case, if you will need to debug something in future that why this particular model entry looks like the way it looks like You might need what was the raw JSON that was sent to me by this third party. In that case, in those cases, raw Jason might come handy to you So this is the most advanced form, more advanced form of auditing that you can achieve by updated ad, created by, deleted by Rogerson, deleted ad. And the most basic form can be achieved by created ad and updated ad So this is auditing, but then there are some things that I learned from experience.

First thing that I learned was what is the difference between autonow add is equals to two and default times dot now I realize that autonow add is just a Django level construct while default time zone. now is a database level construct. What do I mean by that is When you define an auto now add, so afford the uninitiated, what is autonow add? So when you do auto add, autonow add on a Model property in Django. What Django does is when you are trying to create a new model entry, the fields with auto now add will get auto-populated with the current timestamp. Okay So but this is I realized later that that this is strictly a Django level construct. It is not database level construct. What the what do I mean by that is it's Django

sends the current timestamp value to the database, database does not generate it automatically. While having default times out. now means that if you don't supply the value of the this particular field, right, my default value is gonna be the current time. So default times out. now is a database level construct while Autonov it is a Django level construct and what difference it makes to our lives is for example you are trying to control your model or suppose you are trying to insert In your model through a raw SQL, and that Raw SQL is done by either some third party or some Lambda code or some code in your application only. For some reason, you cannot use Django objects. model. objects. creator model. save and you are trying to insert by ROS equal by writing SQL statement, insert into etc

etc. So in that case the fields which have auto now had, you will have to manually supply the value or the database will either throw an error or just stores the no value. While in the later case, if you have default Times out on down for that field, the database will automatically take the value as current time. So this is one difference it makes. Another difference it makes is why testing So, in testing, if you're trying to create an instance of the past, right, for example, for some taste test, I want to create an instance which was 30 days earlier for some reason, right? Then it is That difficult to do with authonoard, but with default times authonow, it is pretty easy because you can just supply the value and database will say, Oh, you have supplied the value, I'll ignore the default and your value, I'll consider it as truth value

While in case of autonow ad Django will try to supply the value no matter if you have already supplied it or not. So this is the some things that I learned and I realized that auto You know, default times out to now might just be a better choice than AutoNow add for my use cases. Another thing that I learned from experience was the performance implications of having raw JSON in your model And why those performance implications come about in the first place? First is due to the reason that Django defaults to fetching all the fields. But what does that mean? That means that if you have 10 fields in your model And you say Django model dot objects dot filter, then all those ten fields would be fetched For the selected rows, right?

And think if you have a JSON field in as one of the fields in your model, what that would do is if a JSON has, for example, in size 2 Kb or 5 Kb or 1 Mb for that matter Your database requirement will increase. Your database memory requirement will increase and your application memory requirement will also increase because you are now also having the draw json in your memory, right? And it is not only the memory increase due to that application performance or compute also takes a hit. The second reason of having performance implications while having raw JSON is due to the way Postgres handles oversize attributes. So Postgres has a concept called as toast Where every table you create in Postgres, Postgres also creates a complementary toast table where every oversized row, what is an oversized row?

A row which is not fitting into The minimum size, minimum pay, maximum pace size in uh Postgres, which is I think uh of 8 kb out of the app I think two or four kb is used for storing metadata around what is the next pointer, what is the previous pointer, header, checksum, etc. etc. And the rest of the 4 kb is the actual data So any row which is exceeding that 4 kb, what Postgres will do is first Postgres will try to fit in using compression or encoding or something of that sort. If it still does not push, Postgres will break this data down and will also store that particular information that this row has been broken down into multiple pieces. in your toast table so whenever you're trying to fetch it it will also go via toast table to fetch you know uh to collate all those different

All the all all the rows data that is contained in the different blocks in your disk. So this is the two these are the two reasons why having JSON in your model Is considered to be a non-performant way. So, what is the solution? I have already told you the problem. What is the solution? First solution I realized that evaluate for a use case Whether you really need raw JSON, right? Because after all it is a good to have. It is a good to good to have in the sense that you might need to debug it later, right? So evaluate for a use case whether you really need or maybe you can just go away with you know logging something in some other database or an ELK stack or something of that sort. That might also be solution If you really need raw JSON, then consider having that JSON as a one-on-one field in some other model. So you can create another model which will just have that JSON field, and this model is related

In as a 100 1 to 1 relationship with each row of this model, right? So, what would that would do is that this main model is sane, is does not does not have any JSON value. And if you need to know the JSON, you can anytime go to this this particular Related model and get the JSON right. So that way the first problem that we had, which was Django defaults to fetching all the fields and due to which the performance implications came in, is also removed Another way to remove that performance implication is maybe you can override the Django manager for that particular model, and you can override that manager to defer fetching the JSON So that when you need JSON, only then you fetch it, you don't fetch it needlessly. Both are valid solutions. I have preferred in my experience to go with this because this looks like a clean way to go about this.

So these are the things some of the things that I learned from experience around audit your models. So with all of those learnings, now we can alter our Yama user model with We added created at updated at another orating fields in our you know uh user model. You you notice here that created had is a default of time zone. now I think time zone has a spelling mistake. Apologies for that, but Created at a default value of time zot now and updated at is being populated by auto now and auto now. Sorry. So this is First learning that we had, which is which was around auditor models. While I close out this, I'll also leave you with something that you might have already heard of, something called as Yagani. Yagani is a software engineering principal

Which is a acronym for you aren't gonna need it, but I'll just throw under a different term which will go about you will not regret it. So auditing is such a easy thing to do in Django or many modern programming languages. Either this provide a native native support or there is some library or the other. who sub who is able to who can enable you to do so. So it is very easy to do, but the benefits It has are tremendous. So you will not regret having auditing your models. If you can't do any you know fancy auditing, just do very simple thing which is created at and updated at and see basic your use case if you need more. So, this was our first theme, which was auditor models. Again, let's go back to Adventures of Yama.

Now, the product team came up with the requirement that we need to add a support for bookmarking a message in a channel. What is a bookmarking? Bookmarking means pinning a message if people familiar with Slack or saving a message or something like you know uh Permanently pinning that message onto that channel. Bookmarking is can be thought of something like that. So the developer in me said, sure, we can just add is bookmarked flag in the message and uh whichever. Message has is bookmarked as true. That particular message is a bookmarked message, saved message, or a pinned message. And by default, all of those messages will have this value as false All good till now.

Three weeks later, uh this time nothing dramatic happens, thankfully, but I realized that Boolean field can be swapped with something better here. What is that something better? Is basically what is our second theme is about, which is timestamp your fields. So, first let's figure out what is timestamp. As with everything else, let's Google the definition A timestamp is a digital record of time of occurrence of a particular event. Again, repeating, a timestamp is a digital record of time of occurrence of a particular event. But how does this relate to here? How does this relate to the incident that we talked about? So, in the incident we talked about, we created an S bookmark field which has two values, true and false.

Value of that flag is true, that means the value of bookmark is true. If the value is false, it is false, right The what timestamp your field says that instead of having these Boolean flags, what you can do is you can have this date time. Fields instead, right? What that would do is if bookmarked at has a date time value, it means this particular message is bookmarked. That means it has a truth value If bookmarked field, bookmarked add field has a null value, that means this particular message is not bookmarked, that means it is a false value. As you can see here All I wanted to know is whether the particular message is bookmarked or not, right? I am able to do so using both the solution by having is

bookmarked also, by having bookmarked that also, right? But The benefits that you have with Bookmark Get is that you get more data, more information about your data structure, about your underlying model, about your underlying data manipulation without any extra cost Right. So the essence almost remains the same, but having timestamp field just provides you more with more data. It's like fee of cost. More detail, right? So this is basically what is meant by time stamping your fields, which is means that instead of having Boolean fields, maybe you can swap them with dead time fields The essence remains the same, but the benefits is usually much greater. There are things that I learned from experience. So when I first placed this idea to my team, my team came up with a

I would say question or doubt that you know it it might very well be that storage cost will increase due to that. Sure, it might increase. I'm not sure how that particular Database that we are using deals with Boolean fuels and data. Definitely storage cost might increase, but my point here is the storage cost is almost negligible Especially when compared to today's world where stories is getting cheaper day by day, storage is getting more and more commoditized Right. So storage is getting more cheaper day by day. And if there is a comparison between storage and the you know computation or developer experience, I'll always prioritize Computation and developer experience because of the same reason that storage is getting cheaper day by day. So, storage cost is almost negligible as compared to readability and other benefits it provides

Another realization that we had is that it is almost impossible to replicate it for Nulbulin fields because Nulbulin fields has three states to maintain, and you cannot maintain three states by having just one variable. Bullen field used to have just two states, which is true or false. You can maintain two states by just having one variable, but it is impossible to maintain three states by having one variable. You will again need to have more than one variable In order for this particular theme to work out. Another realization that we had is that it is not a divide between timestamp or Boolean field. Sometimes you got to have both of them also. One example that I can think of is Suppose you have a toggle which is constantly toggling between on value and off value. In that case

, well you you can just have you know where was this toggle turned on value and whether if that value exists that means The toggle is on. If that value does not exist, the toggle is off. But you might also want to know when the toggle was turned off. In that case, you will have another date time field, right? So And there are certain legit cases where you know you got to have both of them also. So again reiterating it is not a debate between either this or this Both can be complementary to each other also. So with these learnings, now we can change our message model schema with Again, having created that updated at all those auditing fields that we needed, and we can now have bookmarked at flag Which is a date time field which null is

within null is equals to two, where null volume one value would mean that this particular message is not bookmarked. A date time field there would mean this particular message is bookmarked Again, throw in around what I said in theme, you will not regret it. Right. Moving forward to the next Yama adventure. Product team again came up and said, Hey, now our app is getting traction. Now we need to add functionality for letting a user delete a message A developer in me said, sure, that sounds pretty easy, right? We can just add deleted it in message, not is deleted, because we have already had those learnings in the past that you should not have is deleted. you should have deleted it to give you more visibility around things right so i said sure we can just add deleted and ad in the message

So I added a deleted at uh field in that message model. So in future what happened was every query that involved message We had to make sure that we incorporate deleted at is null condition in that particular query. And as the team grew As the number of business analysts, as the number of data analysts, as the number of operation analysts grew, it became a pain to pass on that context every time. Right? It became a common source of error. Right? Sure, you can just Add a view, Postgres view on top of it and say that okay, just query this view and this what this view is doing, this view is just fetching all the rows with deleted it is null automatically We can have that, but then this view just becomes another thing to maintain, right?

Another thing to take care of. Another thing that we realized was that for organization where deletion was a frequent operation, meaning that the number of deleted rows is greater than 15% in number. Then auto also it became a performance overhead to maintain. It was not performance issues, it will became a performance overhead to maintain why Because now your database contains many dead rows which nobody is using. They're just occupying space in the in your database In that table space, right? So every time first problem is already there that every time you are creating any index or creating any query, you have to make sure to add deleted at is null is equals to flag uh deleted at is null condition. The second is it just became a performance overhead to maintain.

With all of this, we come to our next theme, which is soft delete or not. So, what is soft delete? But this time we are not gonna use Google, we are gonna use DuckDuckGo. According to DuckDuckGo Soft delete in database lingo means that you set a flag on existing table which indicates that a record has been deleted instead of actually deleting that record. That's what we were doing in the past, right? We had a deleted at property in our message model that used to denote Whether this record has been deleted or not instead of actually deleting that record. So basically, we were doing soft delete. So a developer in me had this curiosity Next, why should I do soft delete? Or should I just go ahead and do a hard delete

and store the deleted data in some archived form How should I how should I make that decision? Right? Is there a case where I should just go buzzer and do hard delete and store deleted data in archive form form? And with that curiosity, I reached out to my mentor And as with any experienced programmer would say, this add it depends. It depends on what That's what that was my next question. So they said it depends on will you need to reference that data? That data means the deleted data for frequent use cases. The main point here is that will you need to You might want to, but will you need to reference that for frequent use cases? If you don't need to reference the data for frequent use cases, it might be very well that you can just hard delete it from the main table and store it in some

archive table So that if you need in future for any infrequent use cases, you can just you know uh take from that archive table. That was the first Case that this said first condition that it depends on. The second thing she said depends on was what will be the ratio of deleted entries and live entries If they you think the deleted entries will go above 10% or 20% of the live entries Then it might make sense to you know have that deleted data in an archive table instead of having it on main table because having it in main table will just Make your performance worse and worse, and it will be just another performance overhead to maintain.

Another thing that she said was, What will be your query pattern look like? Identify what would be that query pattern would look like. If your query pattern would always be that we are, I'm always gonna be fetching live entries, then probably it indicates already that you are you should be better off with having a hard delete and uh storing it in some archive form. And if you think that your query pattern would inc might increase often You know, uh referencing that deleted data, then you might have to think about it whether you should do soft delete or not. So with that She finally concluded that it depends on these three factors. And on the basis of these three factors, I came to a conclusion that in our case of Yama, we really need to fetch the

deleted messages Except for compliance audit or something of that sort, which is a once-in-a-lifetime operation or once in a year operation or something of that sort. So, with that particular information, I created a deleted message model Which had exactly the same properties and attributes as the original message. Again, you need to evaluate for you your use case whether you should do self delete or not. Don't go by blindly that you should always do soft delete or you should always do hard delete. Remember that there are both these options available and evaluate basis your use case Coming back to adventures of Yama again, now product team came up with a another product requirement which goes by that we need to define roles for user and those roles can be

admin and normal user now at this point I had around two years of Django experience so I knew there exist something like choices in Django So I said sure we can just add role field with choices in user model and with that I implemented that and this is what the implementation looked like where I would have choices, and choices are usually list of tuples where the first value of the tuple is what actually gets stored in the database, and second value is the what is shown on the model form level. And I defined a positive integer field with choices as these. User role choices and what it used to store in databases uh that here you see a user with test for username

Get stored with role 0, 0 refers to normal. Test 5 username gets stored with role 2, which refers to archived, and anytime somebody wants to know Who are all the admin users in my system? They would just go and query, select star from user where role is equal to one. So I implemented this choices thing. Things were good until it wasn't Again it became a context problem to remember all the numerical mapping for roles. Consider you have only three mappings as of now, admin, normal and archived And in future, suppose that 3 increases to 10. Now as a developer analyst or data analyst or business analyst or operation analyst You will have to remember all that numerical mapping in order to make sense of the data that you receive from your database, right?

So remembering all that numerical became pain and as the number of fields with choices increased it all became more and more painful. Another thing that we realized was that Django choices are just a model form level constraint check, that is, it does not ensure constraint and quality check at the lowest level, which is the database. What that means is that if a developer comes in and says that okay user dot role is equals to XYZ and that XYZ role does not exist And say user. save, this that will work. Why? Because you're not interacting with the model form anywhere. It Django choices are just model form level constraint check. It is not a constraint check on the database level. So if anyone tries to directly manipulate the database state, he or she will be able to do so.

So it was another learning. So with that we jump to theme 4 which is use of choices So things I learned from experience in user choices are first problem that we had if you remember was that remembering all that numerical mapping was becoming a pain So, this was the first learning that we had, which is basically keep strings instead of numbers for values. The benefit outweighs the disadvantages. The only disadvantage I can think of this approach is storage, the storage might increase But keeping string instead of number also reduces the contextual overhead for data analyst, business analyst, operation analyst and developers also. Now, if they need to know what all the normal users in the system are, they can just go select start from user where role is equal to

normal. Right? They don't need to Go and check what is the numerical mapping looks like and then do all of that uh fancy stuff. They can just uh Search via role is equal to normal. So it reduces that overhead and again the benefit outweighs the disadvantages that you might have Another problem that we had or started having was as the project grew, right? The number of fields with choices increased a lot. And as the number of fields increased, it became harder and harder to maintain those choices in the code, right? Because those choices need to exist in the code also somewhere, right? So, this is the another thing that I learned from experiences to keep choices as constant variables and as namespaced as possible, right?

So, consider here in this example, I created a user role class Which is inheriting from models. txt choices. Models. txt choices is the recent feature released by Django, which helps you namespace your choices. So I have namespaced all my choices under a class name user role, right So anytime I will need to reference normal user role, I can just reference it by user role. normal. Right? If anytime I need to reference all the choices, I can always reference it by user role. choices So this is the benefit it provides as your project grows as the number of developers you add in a team continuously keeps continuously increasing Having the thing, having these choices as as namespaced as possible provides many advantages.

Another problem that we, if you remember, we had was Django was not enforcing those constraints at the lowest level, which was the database. So we realize that okay we can have those check constraints on database level by having something like this where we say that okay Before inserting into database, check if the value lies in between these values and if it lies, only then it let it insert. If not, throw with throw an integrity error. So basically this is a constraint that you can create on database level. With that, we can move to our next adventure of Yamaha. So till now we have covered auditing, we have covered soft delete, we have covered timestamping fields, we have covered use of choices. So next adventure of Yama again product team came up with a requirement that we need an hourly report to get the number of messages sent in each organization.

A developer in me said, sure that looks like an easy requirement to implement, consider it done. And I created a Cron job or a schedule job that used to run the following school query every hour where the SQL query used to work like that. It used to Fetch the messages from the message table, used to join on user, then used to join on organization to reach an organization because message is related to a user and the user is related to organization. So to reach organization, we have to have a join. And then you I used to group by one one means org. name, right? So this was the SQL query that I used to run every hour to get a report of uh number of messages sent in each organization. Things were good in Citadel until one Friday our database server crashed due to this query.

And on debugging, we figured out what was the reason for crashing of database server. It was the expensive joint cost between user and message for some of the organizations So that led to our last and final theme for this talk, which is denormalization. So when talking about denormalization, let's first discuss what is denormalization. And this time let's use Bing as a search engine. Uh so according to Bing, denormalization is the process of adding pre-computed redundant. Data to an otherwise normalized relational table to improve the read performance of the database. This part is important. The process of adding pre-computed redundant data to an otherwise normalized relational database to improve the read performance of the database.

So this is denominalization and we thought maybe denominalization could help us here, but we were always taught in academics that denominalization is bad. Normalization is the holy grail solution of everything in database. If you do denormalization, you will have problems like update anomalies, insertion anomalies, selection anomalies. Make no mistakes, those principles also are true, but it is okay to denormalize. In fact, denormalization is One of the top reasons why new age databases are able to scale up. This is the first acceptance that we needed to have as an engineer organization that it is okay to denormalize. It is okay to deno do denormalization. So with that realization, let's discuss what does the before and after scenario

look like. So before denormalization, what are uh Yama Database model used to be look like is an organization is related to a user and a user is related to a message. After the realization of denomination, we thought, hey, maybe One option that we could have is directly relate a message to organization so that the joint between that user and message which was causing that problem does not happen at all. We remove that completely out of that equation. That was the first option that we considered. And if you think about it, that also makes sense, right? You might you will rarely find a case where you will need to change an organization altogether for a message And if there would be some cases, it would be pretty rare. Another option that we can think we thought of

was if we are gonna have denominalization Why not just store messages count along with organization? So anytime a message gets inserted into the message table, I also increase the message count in the organization. Right. It has some implications on right performance that we are gonna discussing we are gonna be discussing about that in the next few slides. But we thought oh maybe that could also be one of the solutions and that is the solution that we went ahead later on that's storing message count along with the organization. So, this is how denominalization helped us solve that problem. But as with everything else, there are some things that I learned here also from experience. First thing I have learned was everything done in excess is bad. And the same is true for both normalization as well as denormalization.

Again, repeating everything done in excess is bad. The same is true for both normalization and denormalization. Normalization, if done in excess Affects your read query performance because then you will have to have complex joints, complicated joints, etc. But denomination if done in excess Will affect your database integrity if not done well, will affect your database constraint if not done well. Because Now you need to maintain that integrity by updating at two places. What is being told to us in academics, right? That is what update anomaly is about. That if you update one thing, you will need to remember to update another thing That is the same problem with denominalization happens, right? But if done in excess is when it is bad, if done in moderate amount, it is okay to denominalize

Another thing that I realized was everything done in software optimization is a trade-off, and here the trade-off is between faster reads and slower writes, which is the case of denominalization versus Faster writes and slower reads, which is the case of normalization. Right in normalization, you have faster writes because you are doing atomic rides, but reads are slower because of joints In denormalization reads are faster because you have removed joints, but writes are slower because now you need to maintain constraints, integrity, etc. etc. manually while writing to your denormalized tables. So it is also a trade-off. So it is important to realize that denominalization is okay, but done in excess is bad, and it is a trade-off between faster reads and slower writes versus faster writes and slower reads.

With that, we are good to close our talk, and it is time to bring down the curtains. Before I you know tell you about key takeaways from this talk. I just wanted to explicitly state it out that don't follow any advice blindly from me or anyone without evaluating for you your user. I have made these mistakes multiple times in my career where I said, oh, X person is saying that we should do this, let's do that. Then for my use case, I realized, oh This particular thing was bad for my use case due to XYZ reasons, right? So evaluate for your use case, see if it really makes sense. See if you see any disadvantages. See if you Delve deep into each of these concepts after going home and see if some of those make sense for you or not, and then only try to implement it.

Don't follow anyone blindly for that matter, me also So, with that, let's note down key takeaways from this session. First was auditing your models can go a long way in making debuggability easy. And most simple form of or simplest form of auditing can be achieved by adding created data and updated data in your models. Another learning that we had was when tempted to use Boolean fields for something, evaluate whether you can use timestamp fields instead. It helps you kill two birds with one stone. I do not recommend killing birds, but it's just an idiom. Another learning that we had was soft delete is good, but it can be root of problems also. So evaluate. I at this point in my career of a believer that we should do so

Seriously think about hard deleting and if there is no strong reason for soft delete then don't do soft delete because soft delete can be root of your performance problems Use Django choices wisely and maintain consistency using check constraint or something of that sort. And the last denormalization is okay and easy trade-off. So, with that, I am good to close this talk and I am open to any feedback or questions that you might have My Twitter handle is Tharundanderscore Gag2. If you have any positive or negative feedback, feel free to leave me a note on Twitter. I'd like to hear from you And if you are more active on LinkedIn, below is my LinkedIn handle, Tarungark 546. Feel free to reach me out there also.

With that, uh it's time to say goodbye, goodbye, take care, stay safe, get vaccinated, and I hope we'll be able to see each other in person next year around. Take care, bye-bye