Leveraging the ORM to enforce object level permissions

Hi everyone, this is Jeremy Strutch presenting for DjangoCon 2021. Today I'll be talking about implementing object-based permissions in Django. So let's go ahead and jump in. First off, a little bit about myself. I'm a former network engineer, turned to software developer, so I originally got my start, my career in traditional network infrastructure and engineering. Uh and over the past few years have transitioned into uh full-time software development, currently at uh NS1. Uh I have been working with Django, however, since uh around early 2008, I think, uh when I originally had uh for um you know just a personnel blog Uh these days though I am the uh best probably best known as the founder and lead maintainer of Netbox.

So what is Netbox? It's an application for uh basically modeling network infrastructure. So if you think about traditional uh data center infrastructure and IP address management uh performs all those uh tasks as well as some other roles. Uh it was originally conceived at DigitalOcean back when I was a network engineer on the infrastructure team there. And we released it as an open source uh project under the Apache 2 license back in 2016. So it's completely built on Django. We've got about uh 70 or so complex models, lots of relationships among them. It's a pretty advanced application and it's gotten really popular, especially in recent years. If it sounds like something you're interested, by all means check us out on GitHub.

Um but in developing Netbox we've uh noticed a few things about our users. Um we have a very, very wide diverse user base , many of them have very, very large networks to the tunes of tens or even hundreds of thousands of nodes on a network or addresses on the network. They also have unpredictable use cases, meaning uh you know a a financial company might use something different than a service provider, which would be different than like higher education, and so on. So we see very widely differing use cases among industries and different organizational sizes and depending on even what they're using it for. different organizational structures as well. Some people are some or some organizations rather are one or two person shops. Some have huge teams running their infrastructure and every step along

the way between those. And so we've seen plenty of unique approaches around working uh working around limitations. So for example, if an organization wants to model something that Netbox doesn't support natively, uh they have some clever workaround sometimes to uh to make it do what they want it to. This leads to some interesting challenges specifically around permissions. So a couple years back, we started looking at uh Doing or seeing what we could do to to provide more robust permissions in Netbox beyond what Django provides natively as a framework. And really what the question boiled down to is just this. It's how do we grant permissions to users and or groups for subsets of objects within that box on a dynamic

basis? meaning we want them to apply uh we want the permissions to apply even to objects that haven't been created yet So that's what I'm going to be talking about today. And first we'll start with a quick dive in Django permissions as they exist in the framework today To provide some scope and context to this conversation, if you're familiar with the acronym AAA, that's authentication, authorization, and accounting, pretty uh standard term in security. Basically we're going to be looking at that second point right there. So authentication is who you are, authorization is what you can do, and accounting is what you did. Permissions is uh pertinent to the second point, which is authorization. So we're not going to be talking about authentication at all, just the permissions framework. The default permissions backend in uh in Django is the model backend that's provided under contrib.

auth. backends. Uh permissions are created automatically every time you create a model and uh we get four four default permissions out of the box for each model. Those are the uh what we'll refer to sometimes as crud actions, being um create, read, update, and delete. In Django vernacular, those are going to be view, add, change, and delete. So the four basic options you can, or operations rather, that you can do with an instance. Permission names uh are followed the structure below here. This is applabel. action underscore name of the model Here's an example. We have a model called Book. It's an application called Library. It's got some fields on there, and under the Meta class, we have permissions defined. So here we're actually we are defining a uh custom permission as well so Django is is very flexible and that lets you do this as well pretty easily.

The permission here is publish underscore book So this model will result in five ultimate uh permissions being created whenever you run the migration to introduce that model within the Django uh application. So you got viewbook, add book, change book, delete book, and our custom permission, which is publishbook. When we evaluate these permissions, if we're doing it in the view, we're typically going to call user. hasperm or hasperms with an S on the end for evaluating multiple permissions Within a template, a couple ways we can do that. Typically we're going to do a perm 's dot and then the permission name or look for the permission name as a string within the perm 's context variable So none of this should be new. Hopefully, this is just a quick refresher for folks who may not

have uh much use to mess with permissions on a regular basis. This is really where the kind of the crux of the matter. Everything we just talked about is for evaluating permissions on a model basis. So for example, we saw the book model, right? This applies to all instances, all books that we might create in our Django application. What if we want to enforce permissions for a specific object? Well, let's take a look at the hasperm uh method. Remember we called user. hasperm to evaluate that. Uh within the doc string of this, we see a note at the end. Uh so first off, has perm is going to just retreat return true if the user has the specified permission. So if I call hasperm view underscore book or sorry uh library. view underscore book, if they have that permission, it's gonna return it. You'll note though that this the method signature has this option to pass an object as well.

By default it's gonna be none. And uh this line at the end of the doc string here is kind of interesting. It says if an object is provided, check permissions for that object. Now the documentation actually doesn't make any mention of this. So this is pretty interesting and encouraging. So alright, maybe maybe this is already kind of built in. Let's see what we can do. So starting from the Hasper method here on the user instance, we uh did a little bit of digging and uh Turns out the rabbit hole goes pretty far. Uh the hasperm method actually calls a utility function, which references a method on the uh model backend. uh which referen in turn r uh references its parent class of base backend and it goes down a few more levels. But ultimately what you'll find is that uh it will return false, hasperm, the the initial method, will return false because

Uh model backend. get permissions is saying if you pass any object regardless of what's passed there, if it's not none, it's going to return an empty set because I don't know how me being um model backend is saying, hey, I don't know how to evaluate permissions for a specific object, so I'm just gonna say no, you don't have any. Right, it's just failing secure in that in that manner, which is great. It's a good design choice. Unfortunately it's it's uh not gonna um obviously give us anything to work with uh out of the box. Alright, so what solutions are available if Django if uh it's not native in the framework? Uh the first package we looked at, who you might be familiar with, is called Django Guardian, and it provides basically object-based permissions. through a direct assignment of objects to specific users or groups.

So you can see from the code here what we're doing is basically importing this user object permission and then we're going to assign this permission. to uh a user and and a specific object and then we can evaluate that for a specific object and that's great. They've overridden that has per method This is great if you have what I all call owner-centric models. So if you think about like content management systems, blogs, where the author of a piece of content should typically be the owner. That's a great use case. Unfortunately, it doesn't quite work for us because we want to avoid those explicit relationships just because of the scale and breadth of which uh of resources that Netbox models, uh it doesn't really scale for us. Another package we looked at was Django

rules. So this is really cool in that it allows the developer to define predicates in their terminology. which uh can then be referenced to enforce permissions, or referenced by rules rather to enforce permissions. So here's an example of a predicate being set, uh is book author. Uh which is basically some piece of logic that will determine whether or not a the given user is an author of a book. Now obviously that's a very uh kind of trite example but you can imagine the the flexibility that this affords. And then you have rules here that can be set and evaluated Tons of flexibility, very cool. Unfortunately it does rely on static and predetermined rules which weren't quite suit suitable for our case because as I alluded to earlier, we don't really know how users are going to use NetBox

sometimes. um or how they're going to uh set up permissions and what kind of assignments they're actually going to be interested in evaluating. So unfortunately that didn't quite work out for us either. Can we build our own? Nothing out there, at least nothing at the time, suited our needs, so we set about trying to do it on our own. Before we did that, we'll be outlined some design goals. So first off, we wanted to make sure we retained compatibility with the stock model-based permissions that you get out of Django out of the box. Second, we wanted to ensure that granting permissions could be done based on arbitrary object attributes, meaning we can say, okay, you have you there's uh you know the concept of a site in Netbox. Say uh you can assign users to just have access to sites in the Europe region or sites that are have a status of active, for example.

Um we want to make it that kind of uh to to afford that degree of flexibility We also want to provide a user-friendly interface and an API to make those assignments, meaning that we don't want users to have to go and edit code somewhere. to to set up their their their rules for permission assignments. We want that all to be done through the user interface so that no one has to be a developer just to assign permissions. And finally we want to make sure permissions apply immediately upon changes and that's true in Django today. The aha m so we we we thought some some we thought for some time about how best to do this and I think the aha moment really came about and we realized This mechanism for filtering objects by attribute is really all already exists in the form of the Django

ORM. That does exactly what we want. Think about if we want to find if I want to find sites that are in Europe, I can do that. I can filter by the region to which the site is assigned. filter by its status or any number of other attributes. I can uh traverse related objects too. So all that functionality is kind of there. And we said basically we just want a way to capture that flexibility of the ORM during permissions enforcement. So the idea was let's let's this uh declare query set filters in JSON and then just store them in the database. So we're basically just storing query set filters. Here's an example of that. So we've got JSON data there. This is just a string at top where we're filtering both region name is Europe and its status is active.

So this is a query set filter that applies to the site model. So what we're going to do is take that string, which is just raw JSON , load it using the load string functionality of the JSON library into a dictionary called params. This is now a Python dictionary. Then we're going to pass that dictionary as keyword filters to the or I'm sorry, keyword arguments to the filter method for the site objects query set. So pretty straightforward To do this though, we need a way to save that JSON data in the database. So the answer there, of course, was to create a new model. This is our object permission model, or rather a very simplified version of it. You can see some s some fields that you probably would expect. First off, it has a name as a Boolean to indicate whether or not it's enabled.

And then we have a bunch of many-to-many fields. So basically We're we're mapping uh object types, groups, and users uh as a many to many as many to many relationships. And then we have an array field on which we're we're um recording a set of actions uh that apply to this permission and we'll talk about that in a second. And then finally at the bottom we see a JSON field called constraints and that is where we are uh in the previous example here, the the uh params, that's where we're saving that in the database. Uh so basically this model is correlating uh content types or or models with users and or groups and uh actions. So actions again going back to the view, add, change, delete that are are provided by default from Django.

Uh and we also you can specify your uh custom actions if needed. And they're stored as just an array of strings. Uh the constraints are optional. If you don't provide constraints, it essentially replicates the built-in functionality of the Django permissions as they exist today, just in a different manner. But with constraints there, basically we're doing a logical AND when combining them within an instance. So for our previous example had region and status, so it's going to be region is Europe and status is active. because they're defined as one uh they're all defined as one set of constraints in a specific object permission instance. Whereas if we have multiple instances, each with their own constraints, we're going to be doing a logical OR. So if they were two different permissions, we would say you can, you know, you have access to sites in Europe

or a site, any site that is active. It is important to note that this model really supplants uh Django's stock permission model. So we're not we're no longer using that And here they are side by side to give you a better idea just of the differences. So it's important to call out the stock permission model has one instance per content type per action Whereas we kind of bundle all those up together in object permission to make things a little bit more streamlined. Typically within Netbox, you know, we we see most Most of the write actions, so um add, change, delete, are all going to be bundled together. Um not always the case, but where it is, and that's going to be 90 plus percent of the time. it's a little more efficient to do it in that manner.

It's also important to call out that the groups and users many-to-many fields are actually on the permission object here, whereas in the stock model they're on the the user and the group objects Okay, so we have our model here. We have a way of conveying uh query set parameters, uh or filters rather. How do we actually apply them to a query set? So there's a few steps. First, once we have these defined, we're going to retrieve all of the object permissions for a given user. And we do that by filtering on both the user or a group of which the user is a member. for the specific action that we care about. And that double underscore contains is referencing the view action. So here we care about view. And of course we need to make sure they are enabled. That'll give us all instances of object permissions that apply for this user

for the action that we care about Second, we're going to build a query set uh that's filtered by all these constraints using again that OR logic. So we start with a Q object and that allows us to do complex filtering on the ORM and basically iterate through these permissions that we retrieved from the database and build this constraints dictionary. So for every constraint that we find, every every object permission instance rather, we are ORing its constraints. So that all becomes this one big Q object that we can then apply directly to the filter method on a query set. So model. objects. filter and then just passing in the constraints, which again is a Q object. It's not a dictionary. So we are just passing it directly. Once we have that, if we need to check for a specific object, so that's going to give us all objects. If we wanted to check for a specific object, we just filter by the primary key.

So we're filtering by both primary key and the assigned permissions. So whether or not that primary key exists, it's only going to return if um if the permissions actually apply to it. So basically we can check we can call it. first or dot exists um to check whether those permissions are in fact valid Obviously that's all a little bit unwieldy, so we uh created um a query set manager called restricted query set. All this is doing is providing a. restrict method to just allow passing the user and this desired action in when doing a query. Obviously simplifies that quite a bit. And the uh permissions retrieval is actually being done uh behind the scenes, or it's being cached.

Okay, so um that kind of makes sense. Hopeful or hopefully it's starting to make sense. Um what about user. hasperm, right? That's kind of where we started with all this. What is what does that look like with this uh with this uh implementation So it became apparent pretty quickly that we would need a custom authentication backend to support the uh the built-in permissions evaluation logic. Uh so what we did was extend Django 's model backend. being the name of the class. And then override. Oh, so we did two things. First we overrode the hasperm method. And we overrid um Get all permissions. And this was the one if you're if you remember from earlier. This is the one that was actually responsible for returning an empty set whenever an object was passed. So obviously we needed to remove that check.

But yeah, the meat of it is under the hasperm method. So this method is really going to check a few things. First off, if the user is inactive, we're going to return false. If it's a super user, we're going to return true. And then we're going to look for any model level permission assigned to the user. Because if you don't have a model level level permission, we don't need this the next step. uh which is to check for uh the assignment of specific constraints with within those permissions So basically we're checking model level first. If you don't have, for example, if there are no permissions that I'll allow a user to modify to um to modify any site, then we can go ahead and return false. We don't have to check for the constraints. But if we do find one or more or more, then we'll go ahead and check those constraints.

It's also important to call out that the query is being done to object state in the database, not in the instance. Um that's the uh sorry, that's the uh attributes of the object being evaluated. So if I'm looking at a site, for example, and applying this uh using this backend for object permissions It's going to be looking at the database representation of the site, not whatever might be instantiated in memory at the moment. Cool, so what about my so that's querying objects, what about modifying them? Querying is fine if you just need to view, obviously, or delete, because we know uh delete what deleting will do to an object However, it doesn't address

modifying objects. So when we go to modify an object, we need to look at both the pre-modification, so the initi uh uh state So what it looks like before we've done anything, and then we need to look at the post-modification state. So after we've made our changes, check that it's still in a permitted permitted state. So for example, if let's say I have access to I can change any active site, but then I go to an active site and I change it to be inactive or reserved or you know some other status, uh that that end condition then no longer uh is permitted. So the the stance that we've taken is basically You can't modify an object into a non-permitted state. Um, because obviously then you would no longer be able to modify the object, and that probably disagrees with the reason why you weren't allowed to do that in the first place

So how do we how do we do that? Once a user has changed something, how do we go and figure out whether the modified version of that object agrees with the with the permissions? The answer we came up with was to use atomic transactions. So if you're not familiar, just a real quick primer on atomic transactions, basically we are going to wrap a an underlying uh database transaction into an atomic action. So we're uh here we've invoked the context manager with transaction. atomic. Then we're gonna uh obviously do some modify object in in some way and then call save on it. And then we're looking for a specific exception. If something bad happens, then we can that that transaction gets aborted.

So no changes being written to the database permanently, and then we can handle whatever exception we need to. This is this isn't anything to do with permission specifically. This is again just a general uh primer on atomic transactions. So the idea is you can make a trans you can make a temporary modification of the database. Run some other logic and then if something went wrong go ahead and import that. So here's how we're using atomic transactions for permissions enforcement First we're going to check the pre-mo pre-modified object state. Alright, so this is us using that restrict method on the query set. So first we're going to find um the object so we're gonna filter the query set but using using restrict by user and uh the desired action and buy a primary key and then get that object. So if it doesn't exist, we're done.

There's no more going off from here. But assuming we do find that valid object, we move on to the second step. So then we're going to modify it. So let's say I change the object status to something bad, something that I'm not allowed to do, and then I try to save it within an atomic transaction. After we call save, we then try that query again. So we're making another database query to the to retrieve the now modified state of that object. So we're running the same query again and just checking whether or not it exists. So it's a very lightweight query, but basically we're saying if it no longer exists, we know that whatever happened to that object during this during the previous step altered some attribute to a state that is not permitted by the assigned permissions. So now I can just say raise permissions violation and then in doing so

undo the uh the transaction. This ensures that the object is no longer modify it remains unmodified in the database. We do use a custom exception here to avoid accidentally catching it or catching um unrelated uh exceptions from you know for other reasons. Um we do have to wrap and uh you know handle that exception. So we have to do something with that exception once it's been raised ideally Um or depending on what you're doing specifically, you might just raise in uh catch that and then in turn raise permission denied for API views and so forth. Some other considerations to be aware of. We have to protect all avenues for object modification.

So this is pretty straightforward when you're just looking at like the user interface and traditional views that are serving a human. Um but when you bring in something like Jenko Rust Framework or Graphene for GraphQL, all those API views potentially uh uh provide an avenue for users to modify objects. So all of those have to be have to be diligently protected in the same manner. As well, anywhere that basically anywhere you're calling save, bulk, update, etc. Any method that can modify the database has to be has to be um controlled and secured in some way. This approach does also work for bulk uh um creating and editing by the way Uh we just we you know our examples have dealt with a single object, but you can do it in bulk just as well. The only catch is that uh

reporting errors can be a little bit tricky if you're trying to update ten things at once and then only like maybe two of the ten. um have uh are no longer valid for for the you know per the permissions policy, it can be kind of tricky to figure out, okay, well how do I do it do I call out those specific two, for example, or do I just say, hey, something you did went wrong Unfortunately right now we're doing the latter, basically saying, hey, you you know one of those thousand objects that you that you just modified doesn't really doesn't really agree with the policy, so try that again. And the way in doing that is just using filter and then passing in a list of primary keys versus a single primary key. There is plenty of room for improvement. Of course, on the current implementation, and that's something we're we're working on today.

So as I mentioned, we're working on Basically taking our implementation inside Netbox, uh it's been it was introduced back in Netbox uh 2. 10, I believe. Uh so about a year or so ago. And uh it's been working pretty well. But uh now we're actually starting to work on um extracting that implementation, kind of pretty it up a bit and making it available as a standalone package. So you know decoupling it from the Netbox implementation into something that can stand on its own Uh we 're it's still a work in progress. Uh we're shooting for end of 2021, early 2022 And this is going to be available as a repo there you see the URL Django Object Permissions under NS1

Labs Uh commun any you know community help with development and testing would be greatly appreciated. Uh we're pretty excited to have this split out from Netbox and to be able to make it accessible for the you know the greater Django community So if you're interested in doing that, by all means, you know, check it out, stop by, uh, help us test it, document it, um, anything else you think should be addressed or however we can extend it to make it you know more usable, more durable, uh is is obviously of of uh great interest to us. So if you're interested, please uh check us out. Alright, with that said, uh that should be wrapping up the the talk. Uh again, my name is Jeremy Stretch. Thank you everyone so much for attending my virtual presentation. I really appreciate it. I'll be available in chat and

I think a few other avenues. here to answer questions, any follow-up questions, or again you can always uh just catch me on uh Twitter or Slack and um I'll see you around. Thanks again