The promised Django Land; the tale of one team’s epic journey... by Nicole Zuckerman

Speaker 1: Greetings! Thank you for joining me today. I'm Nicole. Yep, that's me. That one. I'm gonna be telling you a story today This story takes place where I work, a company that's called Clover Health. We offer health insurance to Medicare eligible folks, i. e. people who are 65 plus or who have disabilities That qualify them sooner, trying to improve people's health outcomes and be kinda nice to interact with. We'll start with chapter one, Origins. This is our backstory wherein we meet our cast of characters We'll then explore what we plan to do versus what actually happened. We'll talk about some of the technical challenges, and then do an assessment of our work's success.

Speaker 1: Are you ready? Just so you know, this isn't going to be a why Django is better than Flask Talk. They're both useful tools, so if you were hoping to do some Flask bashing, I won't blame you if you leave now. Five, four, three. Okay, I'm going. Once upon a time, in a land far, far away called Clover Health, the engineering team built an API in Flask. This was the right decision for them at the time. They wanted the flexibility to build applications without an external set of opinions shaping. the code. They wanted the code to be easy to trace and understand without requiring skills with a third-party framework. And they wanted to use a set of tools consistent with their data pipelines and other systems.

Speaker 1: But over time, the team became very sad because there were too many ways of doing one thing with no established pattern for people to follow. And because people ended up rolling their own stuff from scratch a lot of times. They wrote boilerplate code day after day, and any new engineer who joined the team also had to learn Flask if they didn't already have experience using Flask using it in a production environment. Even worse, they had to learn how to do things from the SQL Alchemy documentation, which is like notoriously evil. And if there were going to be like a Thanos in this presentation, it would be SQL Alchemy documentation. It turned out that the APIs the team ended up building weren't so different from what Django could handle after all. So the engineers got together in small groups and talked about it, and there was much discussion and much much disagreement and then they talked about it some more.

Speaker 1: Does that sound like your engineering organization? And then one day Paul, who is starring as Nick Fury in this, came to the team with a proposal for how we could transition to Django and show them the path to Django land. He'd written a proof of concept for a library in Django to make it easier thought through an initial design of how to accommodate Flask while transitioning to Django, and looked into ways to adding schema support for Postgres. He did all this, but he couldn't make the decision alone. The team needed to pursue it of their own accord And there was much discussion on the team again. Would moving to Django actually encourage a consistent pattern for building Building APIs, or would we have the same struggles? Would the engineering time saved getting up to speed or writing endpoints using the REST framework

Speaker 1: outweigh the time investment it would take to actually port the existing existing code to Django. Would we have the support and the time to do it, or would we be interrupted midway through by some product direction shift? We decided as a team to go for it on apparently March 31st in 2017. This is the slide deck from that day. And to be honest, I love Django. I wasn't at first fully in support of this plan. I was like, we have a ton of patterns right now. We're just gonna have a ton of patterns then. Like what's this actually going to save us? But we agreed on a thing, so we were gonna do it. James Bennett starring as Vision in this and Myself as Scarlet Witch, not so Scarlet, but yeah, go with it, uh, used our Hack Week time

Speaker 1: to create a Django API repo in April of 2017. apparently, and started creating an MVP Django project that would serve new features in Django and legacy code from Flask. We were able to route requests through Django to Flask and set up a couple of rudimentary endpoints in Django to serve as an example of what pattern to follow in the future. But like any port, there was more work than could be completed completed in two and a half days. However, would we complete a task we set for ourselves? This is where you insert a commercial break. Our true our two intrepid heroes were rescued from the monumental effort by their many friends on the web applications team at Cloverhealth. Who created this Django Stewardship Group to push forward the great work? They met every week from April to September, breaking down problems, discussing solutions, volunteering for tasks that would make this project a success.

Speaker 1: They did these tasks in their non-project time while still in their normal workday and collaboratively in small breakout groups from the main stewardship group, which was just like all engineers who were interested in a APIs. But this wasn't enough, and we were afraid we'd never get over the initial development mountain to the land beyond where we'd be able to create new Django apps in peace. Two engineers in particular, Santiago who's playing Hawkeye in this, and Rohan in the role of Black Widow, were staffed on support engineering slash This thing called reliability support. I'm not sure if you all have this at your organizations, but this was um a project that engineers rotated onto for a period of time where they were responsible for triaging bugs, handling outages of upgrading packages, et cetera.

Speaker 1: So on their stint on support engineering, they decided to focus a two-week sprint completely on moving the Django project closer to its final goal, which was having Django in production, serving endpoints itself, and passing along any requests it couldn't handle to Flask. They got buy-in from the uh managers of the team and set out tackling one hurdle after another. I could go into all the struggles that they faced during those two weeks, but that would be like its own talk in itself. So suffice to say. They faced many struggles and setbacks, but they pushed the work forward, getting us like 90% of the way there. And when the sprint was done, they sat down and went And the stewardship group picked up the mantle again and got the project the final distance to production.

Speaker 1: So if that was a lot of talking with just some random Avengers uh photos for you. Here's the TLDR. We got the whole team discussing and got some alignment. We used Hack Week to get the ball rolling, a stewardship group to break the project down into bite-sized pieces and take some of those off. We got like 90% of the way there with a sprint by two engineers, which like it's not so bad. And then the stewardship group got the rest of those like odds and ends that were preventing us from getting production. Cool backstory? Good. Chapter 2. The Royal Plan and the Reality, or what we intended versus what we actually did. The plan was to port

Speaker 1: over time. We would find and share good patterns for incremental porting, any greenhouse, any uh greenfield projects in Django. Whenever possible. Small updates to the Flask API could stay in Flask, but major refactors should be porting to Django. If you touch the idea was that if you touched a legacy Flask app, port it if at all possible, and make sure to include that in the project scope of work We wanted to make Django flexible enough to handle the way that Flask works so that we're only really developing in Django and let Flask remain as it is and not sink further effort into developing it anymore. The reality poured over time. We didn't actually end up like finding good patterns and getting a system for sharing them.

Speaker 1: Mostly we settled on let's make a decision and stick with it. Which it's not quite the lofty goal we were going for, but like it's effective and practical. We do actually do any greenfield projects in Django for the most most part. Small-ish updates to Flask stay in Flask, although the definition of small is somewhat malleable. And refactors, large refactors we do port to Jank. We did, in fact, make Django flexible to handle the way the Flask works rather than the other way around, but we still had to make updates in Flask to some libraries that affected projects that we couldn't port.

Speaker 1: The plan, request routing. How will Django and Flask talk to each other? Initially, we'd consider having Django import our Flask application and call the views directly, sharing the same UWISGI process. As we thought it through though, it would be kind of tricky to get the SQL Alchemy database sessions playing nicely with the Django sessions for things like automatic rollbacks in the event of a failure of some kind. And we wanted to retain the atomicity of our transaction, regardless of whether everything happened in Django, everything in Flask, or some combination of the two Um latency was uh one of the things that we were kind of concerned about, but in the end it turned out that it kind of didn't matter which way we did it. The latency was like low enough for us.

Speaker 1: to just like go with whatever was most convenient. So Django calling Flask directly Maybe not. Ultimately what we ended up doing was uh standing both Django and Flask up independently, but having Django send requests to Flask to handle if Django doesn't have the app or endpoint yet um over the network. So and Flask would send a response back to Django, Django would serve it up to the front end So we ended up doing this over the network. And so it kind of looked like this. We have this front end. Sends requests to the Django API. Django API handles it if it can, if not, goes to Flask. Back to Django, back to the front end in the response.

Speaker 1: Um and the requests to Flask were only in the case that Django Django would otherwise be raising a 404. My grand vision was unidirectional flow. This was a sanity measure I felt was really important to make it easier for us all to understand. understand the relationship between Django and Flask. Django might ask for data from Flask when it needs it, but I didn't ever want Flask to call into Django to get info. That way to me lie madness and just constantly calling each other in this infinite recursion that sounded terrible. So rather than having Django asked for Flask and Flask asks for Django for info, which we imagine would get messy. I kind of railroaded this through our team

Speaker 1: and was like, Django asked Flask for info, Flask must remain isolated and not ask anything of Django. Django. Flask should only serve information to Django. That means for some tools that were used by both Flask and Django, we ported the tool but still had to maintain I felt like duplicated code was easier to reason about than have Flask and Django interdependent. What do you know? We actually stuck to something for the most part. It was something of a struggle when tools were written in general. And someone wanted to use it in Flask. But we stayed strong on this one with one exception. When I was doing my test run of this talk for folks at Clover to get feedback, one of my coworkers did a Mm. Well actually

Speaker 1: there is one place in Flash that calls Django. But it's only one and that's pretty good over like two and a half years. He felt it really important for me to caveat this saying Like we only did this because those that is like slated to be pulled out on its own in the near future. We work closely with like the authors of those services to ensure there was no circular dependency and like didn't make the decision lightly. He must have like seen the disappointment on my face. So in general this does mean that we still have tools that exist in both locations though. And we have to make sure that we have like feature parity between them. The plan was to use the same database. So there'd be no duplicated data. We still have one complete source of truth and there was less to stand up and maintain. In reality, we did end up doing that.

Speaker 1: We have no duplicate data, but we have to keep structure consistent across Django and Flask models. We have our one source of truth, but we have a multitude of ways to write to a table. and potentially locks. We have fewer databases to maintain, but it is a hell of a thing to manage it. Some of the challenges were, as I mentioned, keeping consistent models in both applications. Making Django schema aware, that was a thing. that we knew that we would have to do, but didn't realize it was going to be such a pain because our databases use schemas like this stuff comes from the public. Or like this stuff is um comes from data from our internal apps

Speaker 1: dot table. Um and so now we need a Django to support that as well Um fake migrations ended up being a huge pain. Um we and we had to like write pretty extensive documentation to explain it so that other engineers who were not us would like understand it when it came time to write a migration. So the first time we migrated, um the first time we wrote a migration for a table in Jen. we um ran it with like dash dash fake uh like manage. py safe migrate or like manage. py migrate migrate dash-fake to say like pretend like you actually ran this thing but the table's actually already in the structure so don't actually worry about it

Speaker 1: and then after that all migrations had to be handled in J So there was no ability to like manage back and forth. Another problem is this is kind of what not only our data well mostly what our database said. look like but also just our dev environments in general. This is what it really did. We would run Flask migrations And then we would run Django migrations. And then we switch back into Flask and run the fixtures. And then we'd switch back into Django to run those fixtures for things that were managed in Django. And then we would create fake data, and then you were done. But that only got more complicated over time. So here's a glimpse into our make

Speaker 1: file where on line 296 you see refreshdb to get our local dev database set up and contain fake data. We run migrations on the Flask app and then fixtures on Flask app and then switch to Django to run migrations on Django online 296. and then run fixtures in Django and then back to Flask where we create random data and then switch back to Django again to make more data and then stripe data and then add local users. User stuff. And so it's okay if everything works right, but so many things could go wrong and it's not really desirable. Plus, it requires leaving notes in the code like this. To note that like you can't use db init or create db commands when you're doing your local dev setup, that that is like specifically for circle CI.

Speaker 1: Um because you could totally end up in a state where you're like, oh, CreateDB, that looks like what I need to do and run that locally and be like, why the heck is this not working? I have no idea why I don't have fake data. Another part of the plan was just to have Django do authentication of the user and then construct and pass a Flask user object to the Flask process and use the same user table defining permissions and stuff like that. And this was great because like Django has like great authentication. It was going to be very easy to just do out of the box relatively little work. In reality, Flask needed the user in order to do authorization checks, and

Speaker 1: we needed consistent authentication across Django, Flask, and other internal tools and we didn't want to write to a shared user session table from both Django and Flask. So we ended up using JSON Web Tokens. It was supposed to be an intermediate measure that was like going to be made more robust as like a follow-on project, but lol. That doesn't always happen right away. Definitely didn't get the love and attention right away. I thought it should have. So Django and Flask have different session tables. And so we moved the session info into the JSON web token. telling Django to use our existing user table, which we had to modify in Flask to meet Django's needs. And now you see where the migration stuff was like

Speaker 1: So those were some that was like our grand vision and then the kind of chaos of what we actually did. Fortunately, we had some support for making all this stuff happen. We had a voluntary task force, that stewardship group, for the most part, that met weekly where our work was done as part of your normal workday, not like hero hours at mid midnight because I am like allergic to those. We had engineering support, uh sorry, engineering managers supporting it where like you could say in your stand-up, I'm spending half a day today porting XX thing to Django API and acknowledging with them that if they chose to say, please don't do that, we need you to work on this project thing, that like the work of porting was literally not gonna happen and we'd be stuck straddling these two things for a very long time.

Speaker 1: time. So we had support, but how do we encourage adoption? One of the first things we did was document how to how to use Django Rest framework the way that we plan to use it in our organization because there's tons of tutorials and how-to 's out there for how to use Django. how to use Django Rest framework. But there's like a huge gap between I'm gonna write my little polls app and this is my real life production code, how how do I stitch these together? We also had working sessions to um hammer out any like any problems that came up where you could like work with other people to do it and not just bur

Speaker 1: like share uh bear the burden by yourself. We also, this is kind of coincidental, we got as many people involved in the port as possible. So we ended up maximizing the number of people who were comfortable with the new world Very early on, so there weren't that many late adopters who were like, oh, I don't know how this works. Um, we had a Slack channel full of knowledgeable people who were happy to help or clear roadblocks. There was a GitHub group to review your code once it's ported or make suggestions and there were office hours where you could get help right away. Cool. So six months later Everything that's crossed out was done. Authentication was handled, albeit with this intermediate solution that wasn't as robust as we wanted. Um we taken care of event logging. There was There was an internal tool that we ported that was

Speaker 1: for like obtaining member um attributes that was like a core piece of the things that we needed support and some basic functionality for logging interactions with members. The one thing we didn't totally finish getting to during that six-month period that we had planned into was this internal library for tracking temporal data. It was like 90% done, just needed a few little tests here and there, but we all know that that last 10% is like takes forever. But still, that's pretty good. Okay, so this is where the plot thickens, right? The technical challenges, the meat and potatoes. The intermediate step of JSON web tokens was uh is is still kind of a pain. It took us like two years to actually do it the way that we wanted to do it, where we

Speaker 1: um had consistent authentication across all of our tools. Another problem was that some things worked okay as they were in Flasks, though we never ended up porting them. Because if it ain't broken Don't fix it. There's also things that were too big to port. So like you needed to make a specific project in order to do it if it's valuable enough to spend the time on. We also had to balance business needs expedients with the desire to write only in Django. We had more internal tools support still, and migrations are still a huge pain. But then plot twist, over the team, over time, the team generally arrived at the idea that microservices would make our lives better, and because there was so much tied up in one monolith, it made things too interdependent, made teams reliant.

Speaker 1: on each other before they could ship anything. And no, we weren't living living under a rock before then. We knew like microservices was a way that a lot of people were going. We just um hadn't quite figured out yet the shape of how we were going to differentiate ourselves from other health insurance plans that were targeting this audience. So it would be like a little preemptive to start breaking things up when we were like, what is what is our product really. So now that we'd have had a better idea of what it looked like, it made more sense to start breaking things out into these smaller separate services. But then what happens to our beautiful Django API? We don't want it to be sad. Well, it's actually alive and with us today. We have tons of rights to the repo and regular deploys. It spawned a lot of patterns that we began using when we broke out our little microservices.

Speaker 2: Our little baby Django service

Speaker 1: Some of my coworkers created this library called Temple Django, which naming, whatever. That's a project template. it now uh that helps ensure future microservices one use Django to follow some best practices and patterns finally that we've established for packaging for package installs and like Sphinx docks and stuff like that We don't definitely benefit from the speed of writing Django ORM queries instead of SQL Alchemy, or at least I do. I love the Django ORM. And we get all the confidence of Django like security right out of the box. Some of the drawbacks are that some things may always live in Flask, and this complicates our dev environments. We have to run like 8,000 services at a time. Our fixtures and our fake data are still complicated.

Speaker 1: Surprise, Django also has many ways to write views. And some patterns we still don't use the out-of-the-box Django way, like for unit tests, even though one of the reasons that we wanted to switch was like, oh, give us some principles. So where does this leave us now? We're still supporting both. We're having thorough documentations at uh sorry, thorough conversations about when it's the right time to refactor at least. And we're not complaining all the time about proliferation of patterns, so like that's a success. And we're spending, personally, I'm spending a lot less time Googling SQL Alchemy ORM documentation, which like meaningfully improves my quality of life. So if you're gonna take something away from this talk other than my very soothing storytime voice, let it be

Speaker 1: get by in from both ground level and management that might seem like a no No-brainer, but it will really hamstring your efforts if not everyone is on board. For us, rewriting everything in one push wasn't gonna work, but like have that honest conversation. where you talk about what the goals are and what the values are, which are not necessarily the same thing. So like know what you're getting into. And spread the knowledge around. The more people who are comfortable in your new framework while you port, the better, right? So part two, a new Django Where do we go from here? Do we call this a success? Personally, I think it's a success in progress. We're seeing meaningful change and the things that were bothering us from before. And we haven't wasted a lot of time or resources porting things that don't need to move

Speaker 1: or sinking more time into something that wasn't a good fit. We do have some added complications and wrinkles. like migrations, etc. But they're relatively small potatoes and are kind of livable tech debt. One important thing for me is that had we not set up the expectation Up front that this port will likely take us a few years. We're not going to front load at all. I might feel differently about the fact that we're two years in and still working on this. But knowing what the priorities were up front, which is like keep doing product work Maintain your work-life balance, do this sustainably. Precluded my worrying about not doing it fast enough. So the work is still going on. Maybe in a couple of years we'll regret some of these choices after all, but Maybe you can find out too. Come work with me. You too can shape the future of tech

Speaker 1: at Cloverhealth. And I mean work with me. You can't work with that dog. Her name's Chloe and the position has already been filled By me. We're hiring engineers in remote from all over the US, so come talk to me if that's something you're interested in. And thank you for attending my storytime.

Speaker 3: Uh hi. the APIs from Flask to Django. Um was it considered that uh you could have just added an Nginx layer on top of these two and then use the configuration to reroute the A Yeah, API and point to Flask as well as Django while slowly migrating APIs from Flask to Django? I was just wondering.

Speaker 1: Let me restate the question as I hear it. Um you're saying why didn't we just put Nginx over and have Nginx do the routing of like Disco This goes to Flask. Um we thought about it, I think this is like two and a half years ago, so I'm struggling to remember, but I think the reason was that we Um we wanted to keep knowledge of like what was ported and what wasn't kind of local rather than adding another place that has to figure out whether something is ported and then when you port something you have to let Nginx know as well. I think that was why. Mostly for simplicity of like how many places do I have to change something when I port it Hi

Speaker 4: Nicole.

Speaker 1: Hi.

Speaker 4: Given the dance you ended up having to do with JWTs and the fact that some things never moved off of Flask kind of as a result of the setup Looking back, do you still does do you and the team still feel good about having the kind of flask routing happen over the network as compared to a code base you and I used to work on where Django was kind of uh a wrapper around the old legacy code base?

Speaker 1: Um I think in general the team feels pretty good about the strategy that we ended up choosing It doesn't get in most people's way most of the time. There are definitely other methods that would probably have worked, but our general problem in engineering was like we could do this, we could do that. What are we going to do? And we just needed to like pick a thing. stick with it and we weren't really concerned about like buyer's remorse.

Speaker 5: All right. Thank you so much. Let's give Nicole another round of applause please.