Fighting Homelessness with Django with Benjamin "Zags" Zagorsky

Speaker 1: Hello Django Khan. I'm here today to talk to you about fighting homelessness with Django. So in this talk, I'm going to talk about a project we did with Massachusetts Office of Housing. We built CHAMP, the online application for subsidized housing for the state of Massachusetts. So first who am I? I'm Zags. As you can see, that is a last name-based nickname if you haven't run into those. This is my fourth time speaking at DjangoCon. Thrilled to be back. I'm also the co-founder and chief technology officer at Zagoran Zagran's a Boston-based software consulting firm. We do full stack, web and mobile. Django is our primary but not exclusive backend. We do server management, the whole gambit. We also work across both the public and private sectors, and this is a great mix

Speaker 1: because private sector software pushes us on the technology front. People love to be on the cutting edge, and government software pushes us on the requirements front. There's often lots of regulation that we need to comply with that creates lots of interesting software challenges. A saying that I actually want to push back on is this notion of good enough for government work. Meaning you can do a slapdash job and as long as you check all the boxes, that's fine for a government project. This has not been true at all in my experience working in government software. Our goal when building a software system is to build an excellent system And government software often has incredible requirements that put pressure on that and force us to do very challenging things. So in our work on CHAMP, we have We're not talking Instagram scale here, but still plenty of user and data scale.

Speaker 1: We have over 10,000 monthly users on the site. Applicants, people applying for housing. They're in there all times a day. So we've got constant uptime requirements, staff users, people reviewing applications at housing authorities. There's about 250 housing authorities in Massachusetts. And so we have hundreds of staff users hitting that site very heavily throughout the workday as they're doing their jobs We've got terabytes of data, tables with millions of rows, and all of this is under the guise of four chapters of Massachusetts state regulation that describes Things this software needs to do. So this has created a lot of really interesting software challenges, and I'm gonna tell you about some of them. So, first, we'll start with just a quick overview on affordable housing in Massachusetts, what we built, how we did it, and we'll get to some of those interesting challenges that hopefully you can take and uh apply to your own work.

Speaker 1: Now, I have way more than will fit in 25 minutes, so I'm going to cheat using time travel. Throughout this talk, I have links to previous DjangoCon talks I've given that go into some of these topics in more depth. There'll be QR codes If you want to scan them, also my slides are in Slack for those people here in person. So first, affordable housing in Massachusetts. Big thanks to the Office of Housing and Livable Communities for giving me permission to actually talk about this project. They manage many, many affordable housing programs. We're gonna focus on these top two categories. There's actually several programs under these categories So public housing is where the state has funded construction of apartment buildings, where those units are leased for below market rent, and vouchers are where the state pays for Portions of people's rent for units that they're renting in the private market.

Speaker 1: But both of these are long-term rental assistance. These are the programs that we're covering in CHAP right now. And the goal for both of these categories is to have People able to afford housing in their area. These are for low-income families so they can get back on their feet, rebuild their lives. And both of these categories have a particular emphasis on homeless applicants. There are carve outs for emergency homeless applicants, people who are homeless or on the brink of homelessness for one of a number of reasons. They make it to the top of the waiting list so they can get housed sooner to remediate that situation. But what is a waiting list? So a big challenge with housing, in case you haven't heard, is there's not enough of it. So unlike a typical government benefit where we people apply, they get put in a pool, and case managers can just review them in parallel

Speaker 1: With housing, people have to apply to a waiting list, and only once a unit becomes available or a voucher becomes available is the housing authority screening applicants for eligibility. And when they do that, they take the top cut of their waiting list, sorted by priority, those homeless applicants at the top, and also when people applied. They screen them for eligibility, they make offers to the top eligible applicants. And this introduces a huge element of temporal complexity. Applicants are applying months, if not years, before they get pulled into screening. They have to keep their information current during that entire time interval. And then they have to get pulled into a complex screening workflow at an arbitrary point in the future when they're not necessarily paying attention, and this isn't top of mind. There's also data complexity because housing is local. Most housing authorities in Massachusetts manage housing for one town.

Speaker 1: And so applicants who quite reasonably are trying to maximize their chances of getting housed are applying to a whole number of housing authorities. In many cases, dozens. Pre-CHAMP, this worked very poorly because the way that people applied for housing was entirely on paper. Applicants had to send a separate paper application in person or by mail to every single housing authority they wanted to apply for. In many cases, dozens. When their information changed, and keep in mind they have to keep it current for that entire waiting period. If they moved addresses, had a kid, they had to send a paper update to every single one of those housing authorities they'd applied for. And then housing authorities, when they pulled applicants into screening, were sending out paper mail, that was the only way to elicit screening responses

Speaker 1: to applicants, many of whom were homeless and have no permanent address who had to receive this paper mail and respond on a very short timeframe to get approved for housing. On top of this, the Office of Housing had no visibility into what was going on in the system unless they wanted to survey every single housing authority. So the state legislature in 2014 said implement, this is their exact wording, a centralized internet website. And off of this mandate, we built CHAMP. So, sorry, what did we build? We built CHAMP, hey. Um, this is the common housing application for Massachusetts programs. We absolutely picked words to make a cool acronym. We digitized and centralized the process. Now the digital aspect of this should seem obvious. Um you can now apply online, you can apply on desktop, on your phone.

Speaker 1: Less obvious was we also digitized the screening process And we actually had to navigate around some state legislation to do this. Applicants can now receive screening entirely digitally if they opt out of paper mail. And they can respond to screening entirely digitally, take pictures of their documents with their phone. And then the the centralized aspect of this makes this easier also, not just for those applicants doing this online who not only have to do it once, but also for applicants still working on paper. We still support paper for access reasons. But applicants applying on paper only need to submit one application, and that gets entered into the system and is centrally available to every housing authority. And screening responses only need to be uploaded once. your proof of ID, your income documentation, and that's available to every housing authority you've applied to.

Speaker 1: Now high level, what did we build? There's many more features than this. These are the big ones. You apply for housing. This is a dynamic application. Questions change based on your responses to previous questions. We'll come back to that. That feeds into automatic weightless ranking. A housing authority has a unit or voucher available. We give them that top cut of applicants they should screen for them. We generate their screening letters. These are PDFs that we generate that they can mail or get emailed to applicants. We collect screening responses. We have document management for that We record housing authorities determinations for screening applicants. That then feeds back into the automatic waitlist ranking. We've got a lot of computed values in the system. We'll come back to that too. And then we track applicants all the way through we signing so that all of this is available for reports that the Office of Housing uses.

Speaker 1: The impact of this has been huge. Thousands of people are housed each year through CHAMP, including over a thousand emergency homeless applicants a year. And while these programs did exist before and people were getting housing through them, homeless people in particular had a very hard time applying for and completing the screening process because of the paper and paper mail requirements. So an all-digital experience has made it much easier for homeless people to get housing in Massachusetts. It's easier for applicants in general. They only need to apply once to apply to many housing authorities, upload each document once. It's less work per applicant for housing authorities themselves. Applicants are at least per applicant, applicants are doing a lot of their own data entry themselves. And we were able to centralize a big piece of the screening process that was previously redundant, screening four priorities, screening those homeless claims.

Speaker 1: is now done by a central agency instead of being done redundantly by each housing authority and champ made that possible. And finally the Office of Housing now has massive data visibility. They can now answer questions that they couldn't before such as how many homeless people get housed each year? They now don't they now they know. Now I bet you can guess the first technology on the list of what we use to build it. It's Django, of course. That's why I'm giving this talk here. Django didn't do it. We didn't do it with just Django, but while we do have view there on the front end, actually Django is powering most of our front end. We have a Django front end with Vue doing some of the front-end dynamism, and I will talk about that more later. For hosting, we've got Elastic Beanstalk, Amazon's platform as a service. And we also use Elastic Beanstalk workers for our task framework.

Speaker 1: What those do is they take asynchronous and schedule tasks, convert them to web requests. And then pass them over to Django. So essentially we can have a task framework implemented in Django. Kinda cool. If you want to know more about hosting and DevOps, check out my Django Con talk last year, hosting and DevOps in Django. There is a link for that. I love Django. The fact that I'm talking of DjangoCon four years running should probably be a hint at that. But for Champ in particular, it has given us so much benefit. Django has enabled us to do rapid prototyping. This was essential in us getting the project in the first place. Our first two weeks on this project using Django, we built a functional prototype of the applicant side of CHAMP. and use that to build stakeholder engagement enough for us to be able to continue working on this project.

Speaker 1: It's also useful for doing fast iteration of new features even now on the mature product that we have. Django's strong security model, its automatic injection protections and CSRF protections have gotten us through many security audits with minimal findings. Django's ORM Not only makes queries easy at baseline, but also has those advanced features for query tuning that we use at scale. Django migrations make it easy to change our data model, which we do with nearly every feature. And Django forms are so good at doing form rendering and validation, that's actually why we use Django to power most of our front end instead of relying on these front-end frameworks. Django also has native text translation. This is one of the projects where we have used that, and we translate the applicant side of CHAMP into six other languages with it. Django doesn't work alone. We've got lots of libraries in the mix.

Speaker 1: Here's at least four I want to highlight. Django storages for file storage lets us put files on S3 very easily. Django SES for email sending. PDF kit lets us take our Django templates. Translations at all and convert those into those PDF notices we generate and Django Crispy forms for form layout and styling. If you want to know more about Django tools and Django forms, check out my talk at DjangoCon three years ago on rapid prototyping in Django. I go into a lot more detail there. An area I didn't cover in that talk that I'm going to touch on now is Django and JavaScript. And we've got two tools that we use very heavily on Champ that we get a lot of mileage out of. One is data tables. This might be a blast from the past for some of you. Data tables is a library that takes an HTML table and does front-end

Speaker 1: paging, filtering, and sorting. And it works really well with Django. You can take just a Django template, make an HTML table with for loop and if conditions, throw one line of JavaScript in there, and boom, you have this magic front-end interactive table. And then Lest you think you're running yourself into a corner, Data Tables has all of the hooks you need to add extra functionality to this. So in the future, when you want to add custom sorts or custom filters or back-end paging, data tables can carry you through that as well. So very easy to get started and low regrets in the future, which that's a that's a good technology choice. Django and View, I mentioned we use these in conjunction. Now Again, we're using Django mostly to build our front end and just we're sprinkling some view on top.

Speaker 1: So how do we do this? And why do we do this? Well the why is because we have so many places, especially that dynamic application form, where we want Somebody when they answer one question to have another question show up depending on the answer or go away if they answer something different. Or you check a box, you get three extra questions, you uncheck it, they go away. If Django could do this natively, I'd use front-end frameworks half as much This is my top feature request, any Django maintainers in the audience. Okay. But for now, we're using Vue for this. And the reason we're using Vue is because it works With HTML tags, which you can just put in your Django template. View calls them directives. That's fine. We all know they're HTML tags So we just add these HTML tags that Vue cares about to our page, throw a Vue app in at the bottom of the page, and when we ship all of that over to the client, Vue wakes up and takes over the front-end

Speaker 1: control of our page. Now, there's a little bit of work we need to do to get data over to view. So we did make a uh custom view directive to pick up Django initial form data. using views before mount to read the data that's in those HTML inputs. And for structured data, we're using Django's JSON script template tag to pass over a JSON blob that where Vue can pick it up. Um now all of this is a little abstract, so let me show you what this looks like in practice. And this is not quite the vision I have for Django. This is kind of an intermediate step. But What we did is we made a custom Django template tag, and it renders Django form fields with those extra HTML tags that Vue cares about, like vmodel, and in the case of the second one, VIF. And the result of this is this first form field is a checkbox. When you check it, that second field shows up.

Speaker 1: When you uncheck it, it goes away. And all of this is done using a Django template. We just told Vue what to do. Now in order to do this we need our view JavaScript code to be in a place where Django can load it. We're using Vite for our compilation. We have a little bit of extra code in our Vite compilation to put To make per page JavaScript files, and then we just put those in Django's static files and Django's regular static tag can load them from there. All right, so this has gone great, right? What what could go wrong? Um aside from government procurement, that's a whole other topic. Um Five technical challenges that we ran into that I think are interesting. First, data migration. We did not have an easy time with this Because we didn't just have one source database, one target database, and could build a mapping.

Speaker 1: We were migrating initially data from 230 housing authorities, some of whom didn't have a database. They were working entirely on paper before. So we built two intermediate files, uh file formats, a CSV for those on digital systems. The reason for the CSV Was because most of these systems could export custom reports in CSV format. And then we had an Excel template for the housing authorities who are gonna be doing data entry off of paper, and that's so we could build in validation rules to that Excel file to assist the people doing data entry We did validation on upload, and for this validation, we returned all of the errors in the file, attempting to return just the first one and then stop. But then people have to upload, get an error, upload, get an error, upload, get an error. We give them all the errors on upload. And we tried to be very forgiving. Columns out of order, um, extra columns at the end.

Speaker 1: Uh Massachusetts has those leading zero zip codes, so we treated four digit numbers as a as a five-digit zip code with a zero in front. Right. Everything to try and get data into the system intelligibly. And then we had to do a massive duplicate reconciliation because. Applicants apply to multiple housing authorities and we were migrating data per housing authority. And we wanted to get these to that central per applicant record. So, duplicate management, challenge number two. This was a problem not just for the initial data migration, but also on an ongoing basis. And we had three layers that we took to this problem. First, preventative measures, automatic merging, and manual merging. Now preventative measures weren't going to help with the data migration. We knew we had duplicates coming in, but that certainly would help with an ongoing basis. So from the beginning, whenever

Speaker 1: either an applicant or a staff user entered an application, we had them check, is this person already in the system? If so, please stop and go work on the existing application. Now that was a good first check. We knew we needed to do more. So next we had the automatic merge. For automatic merging, we had a strict set of criteria. And if a set of applications matched on all of these criteria, we knew this cluster of applications was a duplicate. We could merge them automatically. We still do this every night. Then we had the likely duplicates. This was on a looser set of criteria. We had a whole bunch. If applications matched on, you know, some number of these criteria, we said, okay, this is a likely duplicate. Cluster. We put those into a queue for review. Somebody from the Office of Housing reviews those on a regular basis. Also, staff users as they're working through the system can flag, I think these applications are duplicates.

Speaker 1: They go into that queue. Now, once applications are slated for merging, either automatically or in that manual review process, we take that group, we identify one of them as canonical. There's a set of rules for this. We mark all the others as duplicate, take them out of circulation, link them to that canonical application, and then copy only the non-sensitive data. We copy what latelists they're on. You know, some timestamps, but not sensitive data like financial information because in the event of an erroneous merge, we don't want to have any data leaks. A third area that's been very interesting to to tackle has been performance. This could be its own talk. We'll do it in one slide. Fortunately, there have been talks this year covering some of these. Things. So um

Speaker 1: first for performance you need to know where your problems are. Uh for this you need application performance monitoring. We use Scout. A fun recommendation to give because I actually found out about Scout from a previous DjangoCon. So in full circle here. This is a paid product. Most of them are Um what Scout does for us is it shows us slow pages, it it has great breakdowns including time stack traces uh and approximate memory use. So it's like having Django debug toolbar, but in production, but only for us, not for the users. Now, here's my top five recommendations. These will cover a majority of performance problems, but not all of them. So the most common performance problem in a web application is just too many database calls. If you're making database calls like foreign key references in a loop, you've got an N plus one

Speaker 1: query problem. Pull those queries out to the top of the loop using select or prefetch related. If you're making too many updates in a loop, you want to pull those updates to the bottom of your loop and use bulk create or bulk update to do that in one shot. If you've got slow queries, I mean honestly it's too slow is the second hardest problem in software after it doesn't work. So a lot of stuff you can do about slow queries, but just three things to kind of reach for first: index, annotate, or Django's F queries. One that surprised me, repetitive template rendering. Rendering Django form is pretty fast. Turns out rendering several hundred of them, not as fast, that kind of breaks that second barrier, the one second barrier we're trying to stay under. So for example, if you have an inline form for every row in a table, not so great from a performance standpoint. Put those in a standalone page

Speaker 1: or use your front-end framework to render them. That's how we took care of it. And then finally, this is especially if you're doing some report or something that wants to read through your whole table. Instantiating a million plus Django models actually takes some time. And if all you want to do is read the data, you're not doing updates. Query set. values can give you like a 2x speed up on a run through an entire database table. Speaking of reports, the fourth challenge that we had was reporting. Now, reporting Doesn't have the same runtime constraints as web requests. You don't need to have your reports run in under a second. But there are memory constraints. And what we certainly didn't want was for our reports memory use to just grow unboundedly as our data grew. So what we did was a paged iteration of our database tables.

Speaker 1: Grab 2000, the next 2000, the next 2000. But when you're doing paging, you have to be careful because it's very tempting to use Django's slice syntax. It's just so easy. But what slicing does is it does a limit offset. And what limit offset does under the hood is well offset's a linear time operation. And if you're doing a linear time operation for each page, you've got a quadratic time iteration. Unless you think I'll be clever, I'll use Django's pageinator, that's using slice and which is limit offset under the hood. Page inator is meant for paging for a front-end table, not for page iteration for reports. So what you need to do is successive primary key filtering, say give me my first 2000 primary keys, my next 2000 primary keys, and so on, and that gives you that good linear time iteration.

Speaker 1: One other challenge we have under the reporting label is we have a data warehouse that we try and export, or that we do export, a whole bunch of data from our system to on a regular basis. Now for this we have an intermediate data schema. So Champ exports data into this schema, the data warehouse picks it up from the schema. And this is really important. Using this intermediate schema and not just having the data warehouse read from database snapshots directly for two big reasons. One is schema independence. As I mentioned, we change our schema on nearly every feature. And as long as we can do that in a way that is compatible with that intermediate schema, we can change it all day. And if we need to make a change that's bigger than that, then we release a new version of that intermediate schema and coordinate that rollout with the data warehouse team. The other is so that we can export those computed values that we have all over our system to the data warehouse without needing the data warehouse to duplicate our

Speaker 1: computing val our value computing logic, which if you do that. They almost instantly get out of sync and then you have data discrepancies and it's nightmare. The last challenge that we had to tackle At least that I'll cover in this talk. There's plenty more happy to tell you afterwards. Zero downtime deployments. We've got people using the system around the clock. Now, the easy thing to think about for zero downtime deployments is you always need to have a server running. To accept those requests. So for deployments, we have a of an in an infrastructure solution for this. Most managed hosting will do this these days. So we use Elastic BeanSock immutable deployments. That means we've got the load balancer, we've got servers in rotation with the load balancer. When we do a deployment, new servers get stuck into rotation. And then only once they're there and responsive, the old servers get taken away.

Speaker 1: So there's always a server there available to respond to requests. But that's not enough. Because you've got requests coming in. And remember, a user is not just going to make one request. They're going to be making a whole bunch of requests. And it would be very strange and potentially incorrect for users to be bouncing back and forth between an old and new copy of the application So for that we added sticky sessions to the load balancer, which means a user will keep getting sent to a server as long as that server exists. This is just a checkbox in AWS. And so that way users are getting a consistent back-end and largely front-end experience since Django is serving most of our front end. And then for the last part, for the JavaScript part of the front end, we're also serving JavaScript files right from those servers using Nginx. So that users are getting a consistent front-end JavaScript and CSS

Speaker 1: files to match the backend that they're currently hitting during this deployment window, leveraging those sticky sessions. And that would almost be it, except there's also a database, and you have to make sure that no matter which version of the application somebody's talking to, that's gonna actually work with your database. So for this, at least on our typical deploy, we use backwards compatible migrations. And that is a whole other topic, which I covered in detail at DjangoCon two years ago in my my talk Django Migrations, Pitfalls, and Solutions. So if you want to know more about Django Migrations, backwards compatible ones in particular, give that a look. All right, that is all I have for you today. Here are those links again if you wanted one and I went by too fast. Also, here's my email. I'm gonna be sticking around in the hallway afterwards for questions. But if you are not in person,

Speaker 1: or if you don't have a chance to grab me, or if you need software consulting work, please drop me a line. Thank you.

Speaker 2: Incredibles eggs. It's always interesting hearing about real-world implementations of of the systems that that we're here to talk about. So thank you. Uh please one one more round of applause for our last Live speaker today.