Solving Problems With Django Forms by Kirt Gittens

Speaker 1: Come on, no. Uh

Speaker 2: so before I get started, just to raise hands, uh how many people have heard of or worked with crispy forms? Okay, so there's a lot of people. I want to like see how many people are going to be upset at me if I like say things about crispy forms before I get started. So uh quick just to introduce myself, uh I'm Kurt Gittens. Uh like you said, I'm a software engineer at DealerTrack And at DealerTrack, we work a lot with Django Forms. We have a lot of situations where we need really complicated data entry, a lot of dynamic validations and dynamic forms. And I'll kind of get into what I mean by dynamic forms. But this talk comes from a lot of what we've been building at DealerTrack and what we've learned. in terms of how to kind of use Django Forms to extend its capabilities

Speaker 2: to be dynamic. But before I go into the problems that we actually solved I want to kind of back up and talk about what Django Forms actually is. And uh this is kind of parallel to what's in the Django documentation. Um but basically like Django Forms offers you Abstractions over three kind of core things, right? So you get an abstraction over the structuring of a form, which tells you like what fields are going to be on the form when you write your form class. Uh you define these class level variables that are objects that are your fields. So the structure of your form is basically that, like what fields are going to be on the form? Um then there's the rendering of a form, which is like Django takes care of actually creating the HTML elements that your users need to actually put data into.

Speaker 2: And so Django takes care of that for you. There's a template tag, all you need to do is use the form tag, and it renders the form. So Django Forms gives you functionality for that. And the last part is validating and processing the data And um that is where you define a set of rules for when the data that the user enters is going to be accepted. So um You have your the basic Jenga validations that you get from like validators that these are things that Jenga adds like when you create uh like an integer or fields that capture like numeric data and you set a max and min value You have those kind of validations and then you have more complicated validations that you can write in your form clean or your individual field clean. So these are kind of the three Main things that Django gives you. And so

Speaker 2: at a lot of the situations that we're having at DealerTrack, we needed to create dynamic, uh, we need to basically make all three of these things dynamic. And so starting with structuring, what I mean by creating a dynamic form structure is that we needed, we had a situation where we needed a form where the fields on it might change. depending on certain pieces of user context. And this is like probably a problem that multiple it's not just unique to us, right? A lot of people might have to deal with something like this. So like the reason why you might need to do something like this is like say you have a really basic form that captures some data, but um then you need to introduce a piece of context to change the form, right? So you can do that like this, right? You can add an if condition uh that adds a field to the form, right?

Speaker 2: Self. fields is a dictionary. You can put a field into it dynamically when you instantiate the form. And everything works. The problem with this is that uh oh hold on. I think that's yeah, that's the right slide. So the problem with this is it doesn't scale when you have a lot of conditions. So when you start getting really complicated layouts that have a ton of conditions and a ton of fields that need to change, this solution doesn't work too well because you'll have a really messy form in it that like has a ton of if conditions and a bunch of custom business logic. And what you want to do is separate the two of those things, right? You want to be able to separate your logic that determines what you see as part of the form structure from those rules. And so the way that we chose to solve this problem is by treating our fields as data.

Speaker 2: And what that means is like the way that you define forms or fields with Django right now is you use like it's an object, right? Um and you set some quarks on an object when you instantiate it and that determines what uh Django renders and how Django validates it But all that information um you could basically represent as a dictionary, right, instead of an object. And working at it working with it as a dictionary makes it a lot easier to kind of uh move the layouts around, uh your data becomes a lot more malleable. So um you we start off with a solution that looks something like this, right? So your fields now become like entries in an ordered dictionary. And your field structure is no longer directly tied to the form. Instead of having a class definition that says, here's the fields, here's the order.

Speaker 2: You have this, which basically takes in your field structure object, um, which is, you know, basically the same as what self-dot fields ultimately builds. But what you get now is you have a layout that's separate. So if you want to have a different layout, you define a different variable with a new order dictionary. And you can actually take this a step further and move the actual field objects out of it, or rather these field variables with the data, and kind of replace them With uh the way this solution works is so you now instead of having a order dictionary that contains the actual names, you create a list of strings and you grab those names from some sort of module. Uh in this Uh example, I'm using a class, but you could swap that out for anything, right?

Speaker 2: So you have uh a list now that defines your field layout. And it's completely separate from your Django form. So what you can do, and what we actually do at DealerTrack, is we have an API call that determines what fields are on the form at runtime. So our API takes care of the context and all of the specific business rules that determine what fields need to be on the form. And then it spits out just a list of strings. And this knows what to do after it has that list of strings, right? It does a get attribute on the class that contains the field. Uh so we have this kind of class that or that will contain all the possible fields that could be on the form. Uh since you need to have a superset, you need those fields definitions somewhere. Um so we grab an attribute from that class.

Speaker 2: Uh we basically like instantiate the actual Django field object, whether it's a character field or you know um like an integer field or something like that. And then we just stick it into self. fields, because we can add entries to that really simply. So that's all you need to do to have a dynamic field layout. But the other piece of it is actually rendering this. So we need to get HTML from this dynamic field layout. So we know about the Django form tag, right? You just place it into your template and it renders all the fields on your form for you. One of the problems with this though is if you need to have any custom HTML or CSS instead of what Django Django generates for you, it becomes difficult because you're not exactly sure what elements are going to be on the page.

Speaker 2: Uh so for a while we had a solution that worked with Django Crispy Forms. Um but uh I think for stuff like this it's probably I would advise staying away from Django Crispy Forms. Because it forces you to tie your particular form to a particular layout. So Crispy Forms gives you abstractions over like the actual HTML elements that you'll see on the page. But if you have a dynamic layout, you don't really want to tie that up directly to your form definition, right? You could use that layout in multiple forms, and you don't want to be tied to one particular rendering of that. So uh that's kind of how the dynamic field structure problem works. So one of the other problems that we had to solve was dynamic form validation.

Speaker 2: Um kind of an example of why you would want to do this, uh kind of a really trivial one, is like if you have an address field in your application. Um you might want to call out to a third-party service to validate that your address is actually correct, right? But the problem is that third-party service has errors, your Django form has errors, and you want the two of those to really act as one Um because it's just interesting to talk about, our actual use case uh at DealerTrack was that we have uh third-party users who write validations in a language that we've created. That influence our Django form. So, and then those validations that they write are evaluated by a microservice that we have. And that all that aspect of it is actually Pretty complicated, but Django

Speaker 2: allows us to kind of not worry about dealing with those errors that once we actually get them back from the API. So all you need to really do to integrate external errors that you get from another service With the internal errors of your form, it's just call add error. So uh like I wouldn't actually call clean or call an API directly in clean like this. But basically the approach is kind of the same, right? You make a call out to an API, you map up the errors that you get back from the API with the fields that you have in your form. And then you just call add error and Django takes care of the rest. So what's good about this is like if you have an existing infrastructure for displaying errors to the user, like I'm sure a lot of web applications do, right?

Speaker 2: You need your user to see in a lot of cases, what's wrong on the form. This allows you to integrate those external errors as if they came from Django and you don't have to worry about doing any of the extra work. So you just addError takes a field and an error message and it displays it on that particular field. And or add it, it adds it to the field in the Django form, and then however your display works, uh, it's going to continue to work the same way. So one last problem that we looked at at DealerTrack was having user-driven fields. And basically what I mean by this is like Um you give the user ability to add like a field to your Django form. So kind of to look at an example of this, right? Okay, so this is in the middle of the animation, but basically

Speaker 2: this demonstrates like say you have a list of potential optional inputs that you want the user to be able to add, the user can select one and add it to your Django form. So there might it seem like at first there might be a lot of problems because like uh we think of Django forms as like you have a static definition of the form. But you can actually handle this completely with our previous solution. Obviously, you need a little bit of JavaScript to kind of make sure that this works the way that it is supposed to work. Um but there's basically like three really simple pieces to the solution. So on the UI aspect, actually rendering the fields is taken care of by JavaScript. So Um which might seem a little strange because you'll have initially a mismatch between what's the user actually is seeing and data entering

Speaker 2: and what is actually on your Django form when you render it, but you allow JavaScript to um basically implement the functionality to allow the user to add the fields to the form, and then when you save Your dynamic field structure part that we talked about before comes into play because now you can change, you can add those additional fields that the user added to the form. So the JavaScript part is uh not that complicated. It looks kind of like this is like a really trivial implementation of it. Um And basically the key part is that you need to pass data from your template context in uh when you're rendering the Django template. To JavaScript. So like here I have um the drop-down contents, which is like a list of all the potential fields I want to allow the user to add.

Speaker 2: And then save drop-down fields is a dictionary of The field um if the user has saved data for it already and the value that the user has saved for it. The reason why I need both of those things is because like I don't want to wipe out or I don't want the user to lose any data, right? So if they save something When they reload the form, JavaScript needs to create the fields that they've already saved. So all this does is it loops through the drop-down contents, and if the user saved data for it, it creates like a static field that looks like everything else. If the user hasn't, it adds it to the drop-down. And then um what actually drives a drop down is just like a really simple JavaScript that checks for a change and adds a field in the same way that this might. Um the Django form side of it is even simpler

Speaker 2: because we have our base fields, which are what you see when you render the form. And then you have the drop-down fields. And now when you're saving, you can just add the drop-down fields into your field structure. Because you can change it at any point. You can have a different , you can load with a different form than you save with. And now your validations work even though those fields weren't in your form when the user loaded the page. Uh so one last thing I kind of kind of want to talk about a little bit is um So a lot of these problems ended up leading us to the decision to move away from the architecture that we currently had, which was Django forms and actually just regular HTML Django templates. And we moved to um React. js and Redux.

Speaker 2: And so, but the actual backend is pretty much the exact same in React. js and Redux. You still have We still use our dynamic field structure and that actually helps us more when we move to React. js because what happens is We implement React components that mirror the Django fields. And then what happens is you send your whole field structure instead of letting Django render it. You can dump it as a JSON and treat it as data once again and send it to the front end and let React render it, and then React takes kind of treats it, you can treat your forms then as like pages in your single page application, or yeah, single page application framework with React. And then what we use Redux for is to kind of um imitate

Speaker 2: the sort of like state storage and save that you would get with like doing regular post requests. So Redux is just um like a plugin that you can use that allows you to store state. Uh React just handles the actual rendering So Django Form spits out a field structure, the same thing that we had before. React renders it because you have the same React components that mirror the Django fields, and then Redux just takes care of making sure that the data that the user enters. is saved and then um when the user you know wants to post back to the form they can do that via Ajax. Uh so that's really it. Um any questions? I think I have some time for questions and answer. So

Speaker 3: Sorry but in the last part about using start using React and Redux.

Speaker 2: Yeah.

Speaker 3: How do you handle the server side validation? Right. Do you usually get messages from the backend? How do you present those into the React UI?

Speaker 2: So the way that this actually works is we're kind of uh and this is like kind of this solution is not in the best stages that it currently could be. But um our back end Uh basically instantiates a Django form that because we have that field structure which represents a Django form. So the backend instantiated Django form with the data that it gets via Ajax from React. Um we get form errors back, and then we just send those form errors back as JSON to React. js. And then React renders it the same way that we would before. Does that answer your question? Yeah.

Speaker 3: Thanks.

Speaker 4: Uh do you want to go? Uh thanks for the talk. That was really interesting. In fact, uh I work with uh Django based CMS uh called Wagtail that's very similar uh in the way that it's handling its own forms in Sadmin. Do you uh have any use cases where you have Like saved the state of the form at time that it rendered. Because if the form is just data, then you could save that JSON string of of that dictionary and then access that form in the exact state as it was during that time. And then you kind of have this sort of Django like the the migration state uh type thing with your forms.

Speaker 2: So um with Redux, that actually kind of is what you get. So Redux tracks the state of the form. So as soon as you render the page, get initial populates the field values and then That also populates Redux 's state. So at that point you have a state that basically represents all of the data that you have on the page. And um That state stays updated. So if the user changes something on the UI element, it updates in the Redux state. And then that's kind of how We don't do anything else to like capture the post data. Redux just stays updated with the state. And then when you save or you know when you post back to the server via Ajax, um it's the same state. And whatever the user updated gets updated when you send it back.

Speaker 5: Um so I really love the idea of uh uh sending the uh form structure to React as JSON and having uh React components that mirror the Django form structure. So my question is just um do you are your React components for forms open source or do you know of any available uh projects that have uh fields that mirror the Django form structure?

Speaker 2: That is a good question because we really should and I think that is like an ultimate goal of ours is to open source the solution once it becomes like detangled from all the weird special case stuff that we have written in it. Um so that's probab that's probably something that's gonna happen in the future future because I can see that being useful to a lot of people who want to integrate Django and React is if you want to go that route, we already have a solution built tonight. wanna be able to say that it's going to be open source at some point in the near future. I just can't tell you exactly when that would be. So I I don't know if there are other projects that had that sort of thing. I know we can't be the only people integrating Django and React. And I don't know. I think there's another talk even happening about that integration. So I'd be curious to see

Speaker 2: what other libraries are out there. I don't really know.

Speaker 5: Great. Thanks so much. Uh

Speaker 6: hey, you mentioned um that you have sometimes third party services where you do validation or on certain user inputs.

Speaker 2: Yeah.

Speaker 6: And that you don't want to do the actual uh appending of the errors in the forms clean method. Yeah. So where do you actually do that or what's your approach there?

Speaker 2: Um So I meant like doing the service call in the actual form clean. Um what I would just do is like move that out into a method so you don't have this huge because you're gonna have a whole bunch of stuff in your clean if you have any sort of complex requirements as far as the data that can be entered. So yeah, in terms of like where you would add the third-party errors, I would just do it in a another method on your form that gets called and clean. But make sure that your clean doesn't become some unmanageable thing is kind of what I was getting at with that.

Speaker 4: Hello, thank you for your report. I have a question. So uh with your approach, um front-end controls the structure of the form Right? So are there some security issues like cross-site scripting? Like uh from JavaScript I can post some unwanted fields that shouldn't be there.

Speaker 2: Um so I guess content the content does not control the structure of the form, right? The data, I guess you're talking about more of the user-driven fields thing. And it even that, there is a super, there always has to be a superset of what is allowable because you can't let the user define custom fields. Or if you do, you kind of have to hack around it and do something where like the custom field that the user is defining is actually some, it's a concrete field on the back end. You can never let the user modify your actual form structure So what we have is like we have a superset of all the possible things the user could do. We don't let them add their own dynamic things that aren't part of our superset, because that, you're right, that would have security issues.

Speaker 7: Um I mean so like there is for example a field like uh credit card number and you expect it only in some circumstances like user should fill in his first and last name or some checkbox But I can construct um JSON that contains credit credit card without those fields and I can send it So do you have some additional server-side validation for such cases?

Speaker 2: Yeah, so the field structure, the form that renders Um it has to match or rather the form that the user enters data into has to match the backend. They're not two separate structures So if I send you if I render a field structure, but then the user enters an additional, I guess in that case Um that doesn't really hit the server side. If I understand what you're suggesting, is that like somebody scripts an additional field that isn't really on the form? Uh it

Speaker 7: it is on the form, but it appears only in some cases.

Speaker 2: Right. And right, in that case, your field structure on the back end Would know that that's not there. And so that data, that data would you would get some sort of validation error on the back end when you try and post a field that isn't there, that would cause a problem. So

Speaker 7: Okay, thanks.

Speaker 1: Have you tried to allow a user to get halfway through a um kind of long form um and save their state and then come back to it and pick up where they left off, maybe in the middle of one particular form?

Speaker 2: Yeah. So we've gone back and forth about that, it seems like. Like um but yeah do you have some other s like specific questions about like doing that?

Speaker 1: Just just like yeah have you had challenges Since you have tried it have you had challenges integrating that kind of work doing that with this workbook.

Speaker 2: Yeah. Um the problems with that, we have had a lot of challenges with that. The problems with that tend to be like validation Like um we have all these problems about like uh what kind of validation some validations are not important all the time. Uh for instance like in that case if you want the user to be able to save partial data and come back later Uh you don't want all you don't want to always run your required validation. But your required validation is important. So uh we've gone I we've gone back and forth about like what you actually do in those circumstances. But I think it's definitely with this structure it's not a super difficult problem to solve. I think you can go back and forth pretty easily on what you actually want the end user behavior to be. All right. So uh I think that's it

Speaker 2: then.