The Pit of Success for Per Object Permissions in Django

Speaker 1: Hello everyone, my name is Madeline Boyd, and today I'll be talking about how to handle permissions correctly by removing the need to worry about checking them. If you work on a system which involves permissions or sharing, I hope you'll learn something from this talk. The title comes from the expression to fall into the pit of success. That is, to have the easiest default be the correct course. For example, if you're designing an API or framework You want developers to fall into the pit of success, whereas by not trying, they end up doing the correct thing anyway. In this case, I wanted to avoid permissions bugs by having permission checking be built into the ORM itself. When BusinessLogic queries the database, it should only return results the requesting user has the permission to see.

Speaker 1: Model object edits or deletes should only be allowed if the requesting user has permission to make those changes and return an error otherwise. Permission checking for standard CRUD operations, so create, read, update, and delete, should be built into the API itself. So quick outline, spoiler alert, this talk fits into the I had a problem and I couldn't find the right solution, so I built a solution archetype. My problem was the risk of permission bugs and human error. I wanted to automate them away. So I'll talk about the solution I built. In addition, I'll discuss um some other details about it, so both checking for at the model level, like whether someone has permissions to compute, to read or update a particular model, but also fields on that model as well.

Speaker 1: I'll talk a little bit about related fields, how to handle permissions on those And also how we implemented ACLs for specific allow listing of permissions on particular objects Um and I'll also, towards the end, discuss some considerations for using this of solution like the one I have discussed, um, when it may be overkill, uh, when it might Like and how to mitigate potential issues with it, such as um it be being slow if used in the wrong way. So I'll before we before I go further, I'll talk a little bit more about the problem statement and the motivating factors for why I went into all this hassle. So I'm trying to build a website where it's really easy to both share and lock down data

Speaker 1: working at bit. io. So think Google Docs, but with Postgres. So you can Upload data, get a relational database really quickly, um work in that data, but more importantly, share it with people, collaborate with friends or colleagues. and allow them to at have access to your data. But also having being able to have some somewhat finely grade permissions that you can give read access to some people, write access to others. um admin permission to trusted colleagues. You can make your data public so the entire world can see it. And you can also prevent bad actors from having access to your data. So only the people you allow and you want should be able to have access to your data.

Speaker 1: So that being said, uh databases are boring, so my forthcoming examples will use cake instead. So, I looked at the existing solutions for managing permissions in Django, and I'll give a quick overview of what's out there. So I in particular needed a system which would support subject verb object permissions, which are not supported in Django by default. So you can say that Alice has the eat cake permission, but you can't say that Alice has permission to eat Bob's cake but not Charlie's You could programmatically declare new permissions for each permission object pair, like eat bobcake, but that quickly gets unwieldy, especially if you support dynamically created or user-created objects

Speaker 1: or Um god forbid there's naming overlap or permission naming overlap. Um so this on the left is how the Django built-in permission system checks things. You have a user, which is the request dot request. user, and you can you can return whether or not they have a particular permission defined as app label and then permission name. But there's no way you can't pass an object into this permission check. So if you want to do something like that, like on the right, then you have to use something like Django Rules or Django Guardian. So Jinko Guardian is the most popular solution for this for having object level permissions. It's great for the ACL case because for

Speaker 2: If you want to say that Alice has permission to eat Bob's cake, you would call some code that just allows Alice as having the E cake permission on Bob's cake.

Speaker 1: Similarly to check permissions, you would explicitly check in the code, does Alice have eat cake permission on Bob's cheek? Yes? Alright, let her eat cake. It's popular, it's been around, uh, it's stable, um, but the f you have to do these manual permissions diff definitions and checking. And I wanted something that was more automatic. Django Rules is more in the direction of what I in particular needed. It's fast, it's in memory, and it allows you to define predicates to say whether or not a given subject has a given permission on a given object. And those predicates are short functions that you can define and map to that permission name on a given model class. But you still have to remember to manually check these permissions at runtime.

Speaker 1: So you if you If you give a list, if you want to fetch a list of cakes and only return the ones that um You that a particular user has permission to see, you have to remember to explicitly filter out the ones in which someone doesn't have permission Um so for an I'll give a quick example of how you might add permissions in Django rules. So let's say I have some delicious cake. And I want to make sure that anyone who is either my friend or likes cake can have some. Except for Adam, he knows why. It would look something like this. So I define my cake model, I have my rules permissions dictionary, and then I specify the permission name as the key in that dictionary

Speaker 1: on the classes on the model classes metaclass. and specify the predicates. So with Django rule as predicates, you can string them together with Boolean operators. You can do ands or ors in line, which is nice. And so in this case I would define the permissions for eating cake as such. You cannot be Adam. And you um if you're my like I can actually define relations as part of the permission. So Is a requester um in like my user objects friends many-to-many array. So it's kind of cool. Um but what happens here if you it can be bad if you forget to check this permission and you accidentally let someone who

Speaker 1: is not my friend or somehow like Adam gets through and because I forget to check and so he gets some cake, like that would be terrible. Um or but You could not only forget the permission check, but you could also check the wrong permission. Or you can like have it someone new who just joined who doesn't know which permission to check. Um so wouldn't it be nice? You could try to catch all these things at code review, but you know, because every bug ever gets caught at code review. Um but wouldn't it be nice if the system did the right thing and only the returned objects and details If the system only returned objects and details which the currently active requester had permission to see. So instead of something here at the top, which is how you might filter out only the cakes that I have permission to see um in Jake rules if we had something like the bottom.

Speaker 1: So a more if not all of y'all are building cake gallery sites Another example might be user profiles. So let's say I'm going to a website and I want to make sure that like people have different different user accounts and different people have their profiles.

Speaker 2: But you only want to allow allow listed connections to see your profile. So you're not exposing the profile to the entire world.

Speaker 1: Maybe you want to lock it down to only authenticated users or um only users where you've explicitly have some kind of connection, whether it's like friend or LinkedIn connection or you're collaborating in a particular document. That might be a case where you would need to do something like this where you want to filter out a list of View of models that only a given person can see. So what I'm going towards and would be what I would like to have is something like the bottom where I just have a In this case, it would be a custom model manager that gives me the list of just the per just the model objects that the request. user has permission to see. So now I'll talk a little bit. So that's the existing solutions that are out there. Now I'll talk a little bit about what I built and how it worked for me.

Speaker 1: And specifically, I'll start off by focusing on permission checks at the whole model layer, and then I'll talk a little bit more later on about how to do field-specific permission checks. So this is how you can bake these permission checks into the ORM to get those nice pre-filtered queries. So here's how you might go about it. So the first step is to store the active requester, request. user in most cases as a thread local variable. So this is so it's in scope, so you can have it, even if you're not passing it in as an argument, towards um Like for here, we're there's filtered objects, does not take a requester as an ob as an argument. You can still reference the current requesting user when you're making these permission checks and computing these queries.

Speaker 1: I say request. user in most cases because we also, for bit. io, we support logged out users. The idea was we wanted to allow people to try out the functionality of the website without having to sign up for an account. So we have a different kind of requester that does not map to request. user because in those cases request. user is just anonymous, but we wanted to have a unique user model object, which we keyed off of the session key. If you have more questions about that, talk to me after. We also, to implement this, need to override some model and query set of manager methods. This essentially requires that We have a custom base, a custom subclass of query set, a custom subclass of model, and a custom subclass of manager

Speaker 1: that we can use to String through all the permission checks and make sure the right permissions are checked for the right actions. And then finally, raise permission denied if the requester doesn't have permission to take a particular action, or filter out model objects the requester does not have permission to see. So here's how we stash their current requester so we can access it globally, but it's still scoped by thread. We also have middleware to To like take the request and make sure we're setting the requester object to be request. user or the current logged out use the current logged out user. um before the request and make sure we unset it at the end of the request so we don't have

Speaker 1: weaking of requester across requests because that that can lead to some really interesting privacy bugs. And this also, this method also makes it easier to access the current requester by just calling requester. get. Um so for here's what you would here's the other methods that you will need to override. Uh so fetch all query set fetch all is the recourse of the Django ORM. This is the method that takes a query set, fetches that data from the database, and returns it and hydrates Python objects that you can then manipulate. So if you override this You can get pretty far. And what in our case, what we do is we override

Speaker 1: fetch all, um, take the result set and filter out Filter out the objects that the given requester would not have permission to see based on their current permissions Also need to override manager. queries get query set to pass in our custom query set model, which has the overwritten fetch all. And also we need to override model. save and model. delete so we can enforce permissions on creates, updates, and deletes. So here is how you m this is a summary of how we override Fetch All. This excludes how we handle for related permissions or

Speaker 1: reverse relations, which I'll get to later in this talk. But basically Call superfetch all, it populates a result cache, filter out everything in the result cache that doesn't pass permissions. Um the permission check is Basically by default, it allows for filtering out like the the default permission here in the perm check list is going to be view model. But if you want to have custom permissions, um you can add them in here, and that's how you that's how you can add them to the query set and then still have the query set respect those permissions. This is also how we handle release reverse permissions on related models.

Speaker 1: So I'll talk a little bit about how we handle per-field privacy, but before I get into it, I want to give a quick overview of fields and descriptors. This is kinda like if this were live, I would poll you guys and ask like how many of you know what descriptors are, but I'll just go through it. You can skip and fast forward if you know this stuff. Um so fields and descriptors, what are they? A field is um in Django it Is the it's a property on a model class that maps to a column in the underlying table that stores all the information about that model in your database So if you have a cake model, that will correspond to a cake table somewhere in your backing database. And each row on that table will be mapped to a cake instance, and each column

Speaker 1: that table will map to a cake field. Um so but you have fields defined on the class, but then you have instances of that class, but which also have Properties with the same name as the field. So what's how does that map? How does that the field attribute map to the class, their instance attribute? Well, uh Django uses something called descriptors. And descriptors are a fun magic Python design pattern where you have a class that pretends like it's a property and has this magic method underscore underscore get underscore underscore and underscore underscore set underscore underscore which

Speaker 1: allow this class object to pretend like it's a property on another class, like it's like a weird symbiotic relationship. So here's a pretty minimal example. The benefit of descriptors is that you can have dynamically computed property lookups. So here you have a class called time, and time is a descriptor here, and it has a method called underscore underscore get underscore underscore which returns the current time. Then you have another class called A with two properties, one of which is a standard property called 5 that always just returns 5, and a second property called now, which will return the current time if you call it. So you have you create an instance of A and you call a dot five, returns five, great, awesome.

Speaker 1: But then you have a. now. And if you call a. now, even though now is def as a property defined on the A class, if you call it now on an instance of A, then you will turn the current time. And It's weird because a. now is actually a descriptor, but it's returning a value, and that's because of the underscore underscore get magic, which basically says If this is a property on something and you're calling get, return this value. Don't return the actual instance itself. So, uh how are descriptors used in Django? Well, descriptors are what the Django RM attaches to model

Speaker 1: instances. To return whatever the val that whatever the value of that field is for that instance. So you can think of fields as being defined on classes and descriptors as being defined on instances. It's tricky because their descriptors are hidden from you. You won't really know they're there unless you do some introspecting. And that's because in this case, if you have an instance of the cake class and it has a flavor field and you call cake. flavor, it won't say descriptor, it'll just say vanilla. as you can see in this a little bit more elaborate example. And so how does so in this example, uh time the descriptor was defined on the class, but

Speaker 1: In Django, you have um you define fields on the class, not descriptors. So how is the descriptor getting in there? Well, um what Django does is each field has a method called contribute to class. And all that pretty much does is it iterates through the fields that are defined on it, and then it creates descriptors and then also sets them on that class. So that subsequent instances will get those descriptors. It's just it's kind of like blur plate, I guess. So in this case, if you have a flavor descriptor where underscore underscore get returns vanilla This is how all birthday cakes that have a flavor field will always have vanilla as their flavor. So if you've worked with the Django RM, you've probably customized some of these fields.

Speaker 1: This is how that works under the hood So, now getting back to how we are talking about per-field privacy. So, um, if you will permit me to use an ice cream metaphor in lieu of cake for a slide. I'll use an example from my childhood to describe how you might implement field level privacy, which is the permission to get or set a particular field on a model class that may or may not be different from the permissions in the model as a whole. So for instance, like I maybe I'm viewing a user profile, um, but like there is an email address attached to that, but like I don't have permission to see that email, only the person who has that profile does. That's like one example. So I can see the user, but I can't see the email field because of custom permissions

Speaker 1: So, um ice cream metaphor. Growing up, there was a restaurant chain called Friendlies that had these ice cream Sundays that had a secret surprise of a mystery candy at the bottom. So it could be gummy bears or M M s Skittles, you know. Um so you could only access that secret surprise at the bottom if you had completely eaten the whole Sunday, which was definitely incentive to eat the Sunday quickly and get a tummy ache. Um and you can only set that secret surprise or configure that secret surprise if you are friendly 's employee, because if you were a random stranger across the street uh or just walking along and you tried to sneak candy into the bottom of ice cream that was going to be handed to children, you would probably get arrested.

Speaker 1: So if I were to represent the Friendlies Ice Cream Sunday as a model class, it would have a secret surprise field with custom permissions. So I may be able to view and eat the whole Sunday, but I can only view and eat, I can only know what the secret surprise is or that it even exists once I've completely eaten the whole Sunday. Because there was every once in a while there's a very sad day where there's no secret surprise and it's it's terrible. Um to set the secret surprise, you must be a friendlies employee. So you could define the predicates like this. Um so that's an example of a use case of perfect privacy. Um but to implement it. When declaring a field on a model, you need to use a custom field class.

Speaker 1: And you must also use a custom descriptor, which you have to thread through. And then the descriptor in its underscore underscore get method will check the permission that you've defined on that field and then say, does the requester have this permission? If so, return the value. If not. um then raise permission denied. So how you do that plumbing is in your overridden field class, you implement the contribute to class method and then you override that. So this is what you might do with it. You call super to add all the other fields, but then you also Overwrite the descriptor that it's added with your own custom topping descriptor. So if you want to have a permission on like

Speaker 1: a particular This is also how you pass in the permission names. So you can define the permission names on the field itself. So in this case, in this topping field, I've defined the view perm name to be view surprise and the change perm name to be change surprise. So you define the field, you add the permission names in the init method. You plumb them through in the descriptor underscore underscore get, you check those permissions or underscore underscore set for changing them. And that's how you implement permissions on a field. That works for your standard primitive fields like text field, in field, etc. But for related fields, you need to do something else. So related fields are a way to have fields that point to other model classes.

Speaker 1: And you can have one-to-one fields, you can have foreign key, which is just one-to-many, or you can have many-to-many fields. For those, you must, in addition to implementing created contribute to class, you must also implement contribute to related class and set your custom descriptor in that. So that's because for related fields there is also the reverse permission. So if I have a say there's like class A and class B and I have some field like a dot friends. And I want to also say, okay, if I I don't have permission to see A's friends, and B is one of A's friends, and then if I do B dot friends, I want to make sure A is excluded from that list.

Speaker 1: So this is where that's like one example of making sure that you do the related permission check as well. You also have to override something called related manager class. And that's because when you do something like a. friends or cake. toppings Cake. toppings is actually a related manager. So it's both a descriptor and a related manager. So you need to override that. The problem is that that is set at runtime. So what you need to do to override the related manager class is you need to override a related manager class call super, take that class, dynamically subclass that with your own class, and then

Speaker 1: pass in the permission checks there. So it looks a little bit like this So related manager class is a method that gets caught at runtime, computes what the related manager class should be for a. friends or cake. toppings. And then what we do is we override that related manager, override get query set, pass in our own special query set, or add our own permission checks. to make sure that we're checking the reverse permission as well and then return that class. So there's a lot of overriding and diamondism because Python is cool, so why not? Alright, so important side note. So again, I would ask like how many people are familiar with default manager versus base manager. It comes in handy when you're overriding your related managers with custom stuff like this.

Speaker 1: So the default manager you can have to be whatever you want. Most people said it's just be objects, but you can doesn't have to be objects. It can be anything anything else you want, like filtered objects, etc. And then that's what gets used as for the default manager in this related manager class. There's also something called base manager, which is you wanted to It's very rare that you actually want to override this because this is what Django uses under the hood to compute certain kinds of relations and joins, and overriding it can break Django. So just Make sure your default manager is your overridden manager subclass so the relations pick up the right thing, but the base manager should be the original Django manager class. This is just some text in the documentation about why this is important.

Speaker 1: If you go to the Django docs managers page, you can find this. Alright, so Accles. I'll talk a little bit quickly about this. So this is if all you care about is Ackles, then something like Django Guardian will probably work great for you.

Speaker 2: where you are explicitly allowing or setting or explicitly setting permissions that people have permissions in a particular thing.

Speaker 1: So if you are like serving a birthday cake at a party and you want to explicitly say that like Timmy, like Johnny, whoever is invited, that they you can create ACLs for them and that they are on this list. And that maybe there's like a special like birthday girl and she has a special birthday girl role so she gets to eat the cake first and blow out the candles and everyone else can like eat it. But like So you're explicitly configuring which objects have which permission on the or which subjects have a given permission or which particular permissions on a particular object. Here's how we implement it. Um we have an ACL

Speaker 1: model, and um originally we were using generic foreign keys. Uh so Foreign keys or most Jingo relations can only point to one other type of class. So what happens when you want to have a normal user and um a You want to have an Ackle 's point to both your logged in user and maybe a logged-out user class, or two different classes, but they should both be able to be the accessor on an ACL. Um and like also a resource, you may want to have the same aqua class mapped to different resources. So maybe cakes, maybe pies, maybe in our case, um When we're sharing databases, it could be like a repository or a schema permission or a table within that schema or a column permission.

Speaker 1: So different types of objects we want to set Accles for. One way you can do it is with generic foreign keys, which is what we did initially. We moved away from it because it made a few things tricky. It by breaking the Django database model abstraction where relations are links between tables or making it tricky. It made it difficult to compute certain performance optimizations, and it made migrations and um the ability to do certain kinds of intro introspection very difficult. So we moved to a pattern of delegates where we have an access everything The ACL always points to an accessor delegate class, that accessor delegate or resource delegate classes have a series of one-to-one

Speaker 1: nullable blankable fields only one of which will actually ever be non -none. That points to whatever the access word for that should be. So if we have a logged in user class and a logged out user class um the accessor delegate will have um a logged in user field and a logged out user field. It will only one of those will be valid. And if you do accessordelegate. value or accessordelegate. owner, it will point to the one that is not valid. or the one that is not none. So that's how we we do that instead of generic foreign keys and it makes it a bit easier to do things like prefetching, um, where you're prefetching across different fields. Or compute other kinds of fields and filters. We also have different roles.

Speaker 1: So you can have, for example, an admin role, a reader role, a writer role. Um, and one giant enum and the same type of aquals to be used for all of those. So that's the overview of how this works, and I'll talk about a few considerations or Things to keep in mind before jumping in and using a solution like this. So the main consideration to be aware of here is performance. If you are making database calls in the predicates where you're checking permissions, or you're doing a lot of redundant permission checks, it can be slow The best way to get around this is to pre-fetch the fields that you know you will need in making those permission checks. So essentially pre-fetching all the fields that you will need when you fetch the field for the first time.

Speaker 1: for any permission sharks on that on that model so that you don't have to re-return to the database to fetch them again. So we wrote some code to raise an exception if a database call is happening in a predicate, which helped us catch these things. I recommend that.

Speaker 2: We're also looking into an easier solution for this, aside from is to

Speaker 1: either allow list

Speaker 2: or write a function which is permission names I need for prefetching and then just always prefetch those.

Speaker 1: Um we're looking into maybe writing a tool that can infer what things need to be fetched. Prefetched at runtime, but we haven't had the time to do that yet. So if you're curious about that or you want a nice technical challenge, talk to me after this. Um so another way to speed up permission checks is to check once explicitly what you need to check and then override permission checks to the next chunk of your business logic. If you're doing this all the time, then Django Rules or Django Guardian will work for you because you're always explicitly checking permissions. What like I find that's mostly useful in um more custom cases. So as an example, so on the left, um don't do this. Uh this is doing This is conducting permission checks without prefetching the right fields.

Speaker 1: On the right, this is nice. This is what I would recommend because this is prefetching all the fields that you would need to do as permission checks. So they are all fetched from the database the first time you go there. And then for prefetch related, they're joined in Python so that you don't have to recompute. Select related is joins all of the SQL layer. So another consideration, this solution may be overkill if you don't have that many, if your permission checks are infrequent, or you don't have that many dynamically generated objects, or your permission checking doesn't follow a crud pattern. Like you're not checking permissions mostly on views, creates, updates, deletes. Or maybe you don't want your permissions checks to be explicit because you don't trust magic generally.

Speaker 1: And you'd rather be explicit what's going on in the code than don't do something like this. And then finally, um this solution, it's not really a framework, it's but it's not a library. Library I think of like I can plug in, I can call its methods, it does what I want. In this case, you have to override your the base model and manager classes. So it's essentially a drop-in replacement for the Django ORM. In general, I think we should use fewer frameworks and more libraries, but this is the best way I could find to accomplish this particular goal. And it's it's not as invasive as it could be. So that is the overview. Thank you all so much for coming.

Speaker 1: So I've built this and I could open source it. I just haven't yet because I'm not sure who else would need it. If you think this is cool or you'd want to work on it or you would like to use this in your own project, let me know, and I will happily I can put in the work to do the open sourcing. So reach out to me. I'm Madeline Boyd at Twitter. That is a very weird spelling of Madeline. There is a second A in it. My mother thought it was the more normal spelling of the name. Whatever, um, or you can reach me at void at bit. io. It's fewer letters and easier to type. Um pick your favorite communication medium and let me know And once again, thank you so much for coming and watching this talk. And thank you for, I hope you learned something. Thank you to the Django Conference Organize

Speaker 1: DjangoCon organizers who put in the work to put this all together. If you have any feedback for me or you think that my talk uh anything I could have clarified more or just feedback in general, again reach out to me here. But thank you and happy programming. I'd love to thank my team at BitDodayo who made the system happen. We're hiring backend software engineers just like you. If you'd like to join us or what we're working on sounds interesting to you, reach out to me at void at bit. io. Thank you!