A different Form of navigation

Speaker 1: So today I'm going to talk about a different form of navigation. We're going to talk about forms. Forms are awesome. Everybody used forms before, right? We use contact pages, Django Admins, site signups. You use a form. 50% of CRUD create and update. Usually those use more forms. They're great for shuttling data into databases. I'm not going to talk about any of that. It's great, not going to talk about it. Today I'm actually going to show you a way to use forms, not for data collection, but actually as a core element for user navigation and interaction. Most of us have lots of data. Data we get, data we get from other systems, data that users give us. There are people that we're writing software for that want to understand that data, as we just learned.

Speaker 1: They could use AI and machine learning, but they're not always technical. They don't have the money for it, or they just say, hey, I have a developer. I'll have them do it. So they ask us to build an interface so they can understand their data, they can inspect their data, they can just say, hey, I'm interested in something, let me take a look at it. You can use the techniques that I'm going to show you to build a power search, an advanced search, and build other tooling on top of that. We have Python. We have SQL. Those are great at dealing with big data. Big data, lots of data, data analysis. We just need tools. to tell the database, to tell Django, Python, what we want to do with that. And these are not the tools we want.

Speaker 1: We need a better tool to let our analysts tell the database what they want to do. I work in healthcare and The the people who want to look at the data, their old model was they'd call a developer and say, hey, I want to understand this data. I want numerators defined in this way, denominators defined in this way. Okay, let's build a contract to build out that one report. Five months later the contract's done, it's all signed. And then Two weeks, three weeks, four weeks later, whenever somebody found some time, here's a report. Five minutes later, that's not what I wanted. Alright, let's do it again. So maybe once or twice a year you get the report that you want So I'm going to use a case study to give you an example and then we're going to build it out

Speaker 1: that I built to handle this. So the case study is called Riscape. That's the name of an application. It's got about 150 gigabytes of data on the back end. About 130 million records, and um each record has 47 fields. A lot of those are null, a lot of those carryover, but there's 47 fields in them. And that's an image image of the front page of Riskscape. Um this is a video. I didn't trust myself to do a live Demo. So we're gonna see if this works. Uh if not, I'll bring up VLC or something and do uh the demo that way. I'm gonna pause it and describe what you're seeing as we go. Um so front page of Riscape. Wonderful. So as we go. Awesome.

Speaker 1: So this should look like a form, maybe to people. This is using Bootstrap and a lot of jQuery. Happy to talk about that another time if anybody wants to know. But in fact there's two forms here. So you can see where it says outcome, that's one form, and inclusion criteria, that's another form. The data is broadly similar. It uses those same underlying 47 fields, but there are some differences. A and how it's displayed and B some of the fields that are there. But broadly speaking, for this application, outcome as a numerator, inclusion criteria, denominator for the graphs and things that our users, my users ultimately want to see And here we are just doing some selection. This is saying I want to see people with type 2 diabetes. And took me a minute to pick one.

Speaker 1: And I said they have uh unhealthy BMI. And I didn't change the inclusion criteria. So now I want to map it. And we're gonna get a map. So that used a form. Everybody understands that. That was a form we submitted it. And you can see up top our outcomes of interest, those were the outcomes and inclusion criteria. Those were the inclusion criteria. So far nothing too interesting. This is where I think things get interesting. I just jumped to another tab, and this is not a single-page application that actually made a request to the backend. But it kept the context of the form. You can see above the outcome and the inclusion criteria exactly the same. So that form

Speaker 1: propagated through to a completely new page. Completely new view. The data is also different. Same underlying data, but on the last page it was stratified by zip code. Now it's stratified as you can see by age group. And we did it again. Um and that is sped up. I cut quite a few frames out. Um but it is still usable on the real data set. It takes two to three minutes and I want to speed it up, but the people who use this don't want to pay any more. They said two to three minutes is fine. So that's what it does. But again, the context of our form flowed through. Pretty cool And now one more and I think an even neater trick. We're going to go back to the dashboard.

Speaker 1: And I'm just going to make a selection. I'm going to click on a link. Another map. But if you notice at the top the outcome of interest and inclusion criteria, they're populated. That used a form. Nobody saw a form. I clicked on a link, but that actually did use a form. And we're going to talk about how I did both of those things and how you could too if you wanted to. Alright. So talk about form-based interaction in this way. So there's two broad models that you can handle. One, the context carries through from page to page. Uh and the other is that you can initiate a form context via a link, a button, some kind of widget. Um all the examples I use are Django

Speaker 1: 111. Uh broadly speaking, they've worked since 1. 4. Uh they 'll work in 2. 0 with some slight modifications or none. I haven't actually tried it yet. Um so some background, some tooling to understand. Uh Django Model Query Form, it's an app I wrote. There are a few different apps that you could plug in for this, but basically speaking, it generates a meta form based on the model's meta. When you use it, it builds a Q object. So it does an AND between fields and then an OR within a field. So you could say BMI, overweight, or obese, and type 2 diabetes, and age 20 and above. It then filters a query set of the model.

Speaker 1: So by default it just uses model. objects. all. You can also pass it a pre-filtered query set if you knew you wanted to do something like you always wanted to filter out under age 20 you could just do that and give it that query set and it handles it. It does a lot more. I could talk about it in the future but for the context of this that's enough. So then using it is pretty much identical as using Django forms. We have a simple model of a house, apartment, whatever, uh place of living. So choices for bedrooms, bathrooms, type of heating, whether it has a garage. And then we define the model form, the query form, which uses the model house and includes, just like you would in a form, the bed, the bath, heat, garage. And then a very simple view.

Speaker 1: I prefer function-based views. This works happily in class-based views, which is the way I roll. So we If it's a post, we read the form. And form. process process is the method that actually runs the query against your model. Else we instantiate the form. And we get a very simple look and feel. This has slight CSS on it to give a um a line view as opposed to straight down the page so I could fit it. But otherwise, this is what you get when you use Django model query form. Very simple Basic but complete. Does what you need it to do. Um so that was pretty cool. Um but That's a lot of boilerplate code to write on function after function after function.

Speaker 1: So we make a method for reuse, and we're gonna talk about the grayed out section later. Um but very simply Take the post or the get uh uh or the get, uh return a search form. And then in our view, instead of writing that we just call the search form. Um and nothing happened. Uh can anyone tell me why nothing happened? Because we didn't actually do anything, right? We just moved some logic into a utility function. Um But it's not completely true. We actually built a framework so that we can use it to carry our context from page to page So now we're going to define two new views, search, which actually functions for the search, and detail, so if you wanted to look at a single house. You can ignore the link equals none, we're going to use that later.

Speaker 1: But so we process the forms just like before and we render them awesome and you actually would probably do stuff with your query set result in the search, but I didn't The one important thing that we need to do here with the form is we need to render the form on all the pages we want to carry the context through. You actually do have to supply the form to the back end populated in order for it to carry its context through. It's okay if you hide it. You can submit hidden forms, I think, in my result, in my um. Example I use display none and it works just happily. And then the other is you need a small chunk of JavaScript. So when you submit your form via the widget that you want to submit with,

Speaker 1: you actually change the target of the URL, so you change its action and then submit it. So we would have on-click send form URL search, on-click, send form, URL details. And clicking each of those links. We get two different views. These are simple templates, and I'll explain them briefly. So we don't see the search form. Again, the search form is hidden What I'm showing here, um Django Model Query Form provides a ordered dict of the selections based on the clean data that then you can I call a pretty print query. Um so you can actually print what the query was and we see the query actually carried through the two pages without any interaction with the form other than clicking the link and I have slightly different text so that you can see it's on a different page.

Speaker 1: And I'm going very fast. So gonna have a lot of time for questions. So that's pretty cool. We can carry a context through. We could build it with just a function-based view with a utility like I showed. You can build it in a class-based view. I'm pretty sure you could build it as a mix-in, though I don't use them enough to be exactly sure how you do it. You could use a decorator. You could use middleware and context processors, bunch of ways to handle this depending on if you want it on every page or only some pages. But that's how you would carry a form context across your site. So pretty interesting. But the next and I think more interesting use case is initiating your forms via a link so the user doesn't actually know they use forms.

Speaker 1: And conceptually, how do you do this? So forms can be serialized, right? Very simply. And you can use that serialization to bootstrap your request-response cycle. with the form. So I have serialized them as just a dictionary, search links. You can serialize them to your database, use a JSON field and just store it there. And that's great. You can have a developer build it like this, but also now that you can store it to the database and the interaction to build these links is just the form, you can have your content creators building their own links. So if they see something that's interesting that their users want instead of calling me and you know paying too much to add one dictionary item, they could click on the form, build it out.

Speaker 1: Save that and say this is the link for well large homes or apartments or efficiencies or something else. So now we have a search link, which is the URL for the search page, but we have a target ID. One, two, three, in this case, you could store it as text, anything that you could look up in the database. And just by following those links, we instantiated the form on the back end via this little chunk of code. So if they provide a search ID, we look it up, we populate the data of the form And then very importantly, we bind it. That's probably the most important line in this entire talk.

Speaker 1: If you don't bind it, the model 's query form won't work because it works on clean data and you have to have a bound form to handle that. But we get two different views. We use the same underlying code and you can imagine in my case with 47 fields and in fact I only showed you the first page of that. uh dashboard there's actually three pages so right now they have 18 defined um and they 're you know eighteen lines of code effectively to drive that whole thing um So talking a little bit about this, how you could use it, other things you could do. So one thing that's really useful in my use case, the 150 gigabytes get updated once a month.

Speaker 1: Um so the data is pretty static. And Because of that, we can actually cache our results for long-running queries. Like I said, that time series view takes two to three minutes to run. But I can cache the results along with the query form and its indicator and so my users for their oft used views they get the result in five seconds and it only takes five seconds because I have a polling mechanism to get the result. On top of that, I can also pre-cache my results. So I know what I want for my dashboard, so when the data gets updated monthly I have my form serialized to the database. So I can just kick off jobs that cache those results.

Speaker 1: And more interesting I think is that we store um uses of different forms. So every time somebody puts something new into a form, we save it to the database as a new version of the form, not a new version of the form, but a different instance of a different form. data and we keep metrics on every time that's called so we automatically pre-cache all of the dashboard views But we can also pre-cache views that are being used a lot. And we build metrics on top of that. So I provide to our end users who are also Some of them administrators and say hey our users are looking at this. It's not something that you thought was interesting, but somebody thinks it does.

Speaker 1: Should we add it to the dashboard? Um should we go forward with it? I really flew through this talk It's first one and I'm low on sleep and high on coffee, so sorry about that. Um but um so a few resources I use so Django MotoQuery Form, it's up on GitHub. I'm always happy to talk about that. Um Riskscape, that's on Bitbucket. I'm happy to talk about that. It's a cool little application. It's not of much use because the 47 fields to to you know just uh random person not using uh my company's ecosystem because separately we have a huge healthcare database that we do um algorithmic detection of diseases and most of that data actually feeds Riscape So otherwise a vast number of the fields are meaningless.

Speaker 1: Um, but I'm happy to talk about it. It's pretty cool. Um like I said that was Bootstrap. I'm happy to talk about that too. And uh this talk is actually up on GitHub also if every anybody wanted to take a look at it, take a look at the slides. And yeah. That was really fast. Again, sorry about that. But we have plenty of time for questions and maybe a slightly longer break. Uh if anybody has questions. And thank you. In my defense it was like twenty-five minutes when I titled to myself in the mirror.

Speaker 2: Thanks for the talk. Got a question about the potential security implications here. Sure. There's at least two potential vectors that I could see this might be interesting. One is uh the denial of service either intentional or unintentional because you're giving someone complete access to the database to query. Sure. The second is uh Uh data leakage because you're exposing information that isn't something you can directly query, but in the act of querying you can establish whether it exists or not. Sure. You could query users and you could query the passwords, you could effectively extract the entire password table without actually exposing the password as something that was queryable. So as w what mechanisms does model query form have to either protect against that, is it active defense if you need

Speaker 2: to need to think about what you're exposing, hide the things you shouldn't, or sure. What what exists?

Speaker 1: Uh in some ways I'd say it's even worse than that. I didn't show it. Um but you can happily and easily let it follow relationships um to the nth degree. Uh I'd recommend not to because it's default configuration. It basically does a select field for the foreign keys or whatever and so that could take a long time. Um it's not something I handle um per se. Uh you know you have A fair amount of flexibility because you can pre-process any query set that you want to pass to the end user. Also, I didn't show it. So the default field um so by default it only has widgets and queue objects for non-text fields. This doesn't answer but because text fields like you could do a like but that's kind of useless whatever. But As a developer, you can define how the Q

Speaker 1: object is built either by individual field or also by field type. So it's um So this is a long way of saying it doesn't handle that. You can also, and I'll speak to how we do sort of in Riskscape, um, in your application using Django model query form, you can certainly do that. So so in the Riscape, um, at first blush, it's a really big problem, right? Because that's healthcare data. Um so there's a couple answers to that. First, there's a relatively massive um user access system in place in front of it, be not even in in Django that's just built in front of it. But the data that we provide to that has been defined. by the people who built it. Or sorry, by the people who sort of gathered and generated the data,

Speaker 1: by their lawyers and by me, because I'm also an attorney by training. as being non-identifiable even if you take the entire data set. And that includes such things. So we did limit model query form and how it runs in such ways. So when that map shows up or those bar graphs, if the number of hits falls below a threshold We just say insufficient data. We can't do it. So generally if it falls under um a hundred patients in the denominator or ten patients in the numerator for whatever the UV stratification is, we refuse to show the data. But you'd have to handle that outside of or sort of as you interact with model query form.

Speaker 3: Hi, thanks. You mentioned that it's not a single page app. but it got me to thinking that it might exhibit some of the issues that people sometimes complain about with single-page apps like Uh the back button doesn't behave as expected or some of the other things that people might like to do like bookmark a particular page and then come back to it later won't behave the way they expect. Have you encountered any of those issues or uh is that something you think about?

Speaker 1: Yeah. So this is a place where um working f for uh basically a consulting group that works with hospitals and working as a software developer who builds open source software among other things uh come into tension. Uh right, there's a lot of things that I want to build that nobody's willing to pay for uh because they're like, hey, it works for like I really want to speed up that three minutes and I have like seven different ideas on how to do it, but they're like, yeah, it works for us. Um so for the back button issue is an issue, uh certainly. I have tried to make take efforts to alleviate it. Um I don't think it showed up there, but one thing is that if you don't have a working form context, the buttons to jump to a different graph view, um disable themselves.

Speaker 1: They say, we don't have anything to work on, so we're going to disable ourselves. So that's not wonderful, but it at least doesn't let the user get themselves into an inconsistent state where it's like they think something should happen and it doesn't. Um you could certainly uh build it in such a way um that I shouldn't say certainly. Um The data itself, because the data isn't completely protected, I could propagate all the data forward to the page so that they could bookmark it. Um and in fact, because you're following links at least the link based context can be bookmarked. The form initiated based context it's harder. But again, because I am actually storing every one of those interactions to the database That

Speaker 1: ID does indicate something and you can link to it. I haven't tested it broadly. I'm pretty sure there are probably edge cases where that breaks.

Speaker 4: Okay. Happy to give you an early break. Okay, then we're a bit early for the coffee break. Let's hope the coffee is ready. Take your time to take a short break out to your left, talk to our sponsors, enjoy, and we'll back here on the stage at 11. 15. Thank you very much, Chaim.

Speaker 1: Thank you.