Feature Flags: Deploy to some of the people all of the time, and all of the

Speaker 1: Thank you. Hello, thank you. Uh very happy to be here. Uh My first DjangoCon, my third day ever of DjangoCon Europe. I will be talking today about feature flags. A topic which has been interesting me over the last couple of years. So

Speaker 2: happy to come and share what I've been think what I've been thinking about with that. Thank you, MongoDB, for the socks. Very nice. Uh thank you to my employer, A Cernis, for paying my wa wage while I'm here and paying my flight to come and uh see you. I've put a few flags which relate to uh a Cernus you won't have heard of us. We're a small SAS. platform small startup based in Nantes with teams also in Porto and Gdansk. We have a five-person product development team, which is relevant to the feature flag discussion and how we use them. And we're building a SaaS platform for people doing stuff with telecom.

Speaker 2: Customers mainly in Germany, some in the UK, France, and Switzerland. And if you we're building it on Django and React and some of the other kind of key words There, which we can discuss if you're interested in these kind of fields, are point clouds, building information modeling, documentation, collaboration, sharing, file sharing, large file sharing. Uh that's me, some more flags which speak to me again. If any of those flags speak to you, I'll be happy to discuss Those things. I'm the lead developer at Cernis , which means I'm the Python lead, full stack developer, feature developer. and

Speaker 2: kind of team uh team manager. Uh before uh becoming a Python developer, I spent 20 years working in the construction industry doing specialist consulting. uh design and analysis, wind engineering, wind tunnel testing, which means I'm one of the people that Will was talking about this morning who are more afraid of web development than data science. uh data analysis. I'm involved in uh the local meetup and last year I became co-mantainer despite myself of the Django Waffle uh package and also some uh very very specialist libraries for point cloud

Speaker 2: You could find me at a few of the links on the bottom of the page there. So uh four stages for today's talk. Number one, what are feature flags? And maybe my definition of feature flags for today is not the same as your definition. Why do I think uh why do why do I use them and why do I think other people should start to use them in this same uh this same definition? So share something of my journey and our my team's journey over the last year adopting feature flags and give you some an idea of how to get started using feature flags with Django.

Speaker 2: So uh quick show of hands. Anybody who does use feature flags today. Yeah. And keep your hands up if you use them as a temporary tool for deployment. Okay, and put your hand back up if you use them for permanent customization for specific workspaces. Okay, yeah, so those are the two versions So feature flags are switches or flags to turn features on or off. They might be linked to a user or a workspace or a work group or a particular environment, development, production, whatever. They may be time-based, they might have a date for when they turn on and turn off, and some feature flag systems include that.

Speaker 2: It can be very simple. I'll show you a very, very simple implementation of feature flags in a second. Or you know, we're developers we can add as much complexity as you like to a system. And what I'm talking today is temporary tools for development. So to help a development team or a single developer. uh to uh develop and roll out software more more easily, more less stress. So a quick example. Uh let's take the simplest possible example. Hello world. I'm gonna try some live coding

Speaker 2: here. See how it goes. Okay Apologize apologies for my operating system. It's a function of the industry in which I operate You will have to trust me that the file greet. py contains exactly what I showed you just now. Okay. So wait, you haven't seen it all yet. Okay. So this is great. This is really good code, you agree, but I think it could be even better.

Speaker 2: I've got a brilliant idea, which is That it could maybe greet me personally. Not just say hello world, but maybe it could greet me. And the way I'm going to implement this Is again I try to find the simplest possible uh implementation of this, just adding a flag at the command line. So if I I add personal to my uh Python call. Then it will ask me my name. And if not, then it will just carry on as normal. So I make this update. Deploy it to production. In this case, production is a SharePoint folder where any of my colleagues can activate the uh

Speaker 2: the script. And I'm going to call this greet1. py and run it. My colleagues run it. Nothing has changed. Run it and add the flag. Personal. Ah what did I do I get a typo? Even worse, I have an Azerti keyboard here and it's in

Speaker 2: QWERTY layout. I say I'm called Django Khan. Hello Django Khan. Okay, this this is great. Okay, this is running in production, but My colleague Ida faces customers every day. She's not sure whether she's gonna like this. So I then ask her to try it out. Ida, can you try this out? Use the feature flag. So she tries it. Types. Ida. Ah, she loves it as well. Ida 's sold. My colleague Felix works in QA. He needs to approve this. You can test it directly in production. It's not going to break things for customers because they're still using the standard version.

Speaker 2: He tries at Felix. Hello Felix, okay. But he's QA, so he thought things. Mm. Let's try import OS. Uh print uh os dot in viron What's gonna happen here? Hello, hello, import OS, print OS to the environment. Okay, Felix is sold. He says, that's great, we're safe.

Speaker 2: You're not gonna leak in our environment variables. So let's put it to production. What do we do? Do we just l update the batch uh script, the bash script with this feature flag? What do you think? Yeah. No, stop, my friend. First we refactor, simplify, and remove the flag. Look. How much nicer is that? Compare that to that. Okay, we tidy up after the flag.

Speaker 2: No. We can run directly with the flag. Everyone has access to the new feature. Everybody's happy. Okay. Mm-hmm. Okay, so flags, as I'm showing them, are a tool like this for deploying and collaborating straight away with colleagues directly in the production environment without breaking things for uh uh customers. They are not as I'm showing them today, permanent customization for groups of users.

Speaker 2: I would suggest that maybe for that it should be part of the data model. If you have a customization or a settings or a configuration object, maybe that's the right way to handle customization. Depends on your context. They're also not needed. I would say maybe this little example I could have deployed it straight to production, but then Ida was really quite worried about how the customers were gonna react. So maybe it was quicker to just show her. in production that it was gonna be fine. And why why are they recommended then? Um so to speed up development, to speed up uh collaboration with colleagues. To deploy with less stress, okay, we already know it's working in production before we turn it on for users.

Speaker 2: And another advantage potentially is it can separate feature deployment from code deployment. We adopted it as part of moving to trunk-based development, so trying to get as much of our development done on the same uh trunk. branch as a team to have um uh more stable code to to ship fewer bugs because we're uh testing things in smaller bunches uh in something more like or or directly in the production environment. Um You could see uh I'm not the only one who says so. Trisha Ghee

Speaker 2: is a big fan of trunk-based development. Uh a chap called Simon Willison back in 2014 said that the return in of investment when he started using feature flags was ridiculous. Anybody which web framework did Simon Willison co-create 20 years ago? Yes. Okay, and why not? Well we saw that uh we temporarily added a lot of complexity to the code, three extra lines. uh to the code, an extra code path to test. Uh it can encourage over customization. Anybody being in a code base where there were too many feature flags.

Speaker 2: running in production. Yeah, okay. Takes time to implement, takes time to remove it. So hopefully you've already seen uh the uh the idea. I'll go through how we adopted them then uh again in a five-person dev team. uh product manager, uh lead developer, full stack uh developer, QA, uh data engineer. Using Django REST and React. We were experiencing some painful merges. We had an average release cycle of new code of One release every two weeks, maybe.

Speaker 2: Sometimes it took months working on a single branch or a single feature to get it into production. And our customers were using the app every week, roughly. And we wanted to match our release cycle with our customer use cycle so we could iterate at the same speed that the customers were using things. And our our first step, we were not yet ready to kind of commit to a lot of work, was to put some hard-coded flags into the front end, basically a TypeScript function. Which had a list of user IDs and said okay if this user has the flag, then show them this feature and if not then show them the boring old code.

Speaker 2: Uh got us going very quickly. It just add required us to add one uh TypeScript file, a few if else. In the code base. And we found that pretty quickly we were deploying more frequently. We were happier working on trunk. With a bit of encouragement, every one of the kind of five team members used these flags , merged quickly to trunk. And we usually remember to delete the flags afterwards. And the the downsides that we saw, well, act activating the flags then in that model

Speaker 2: requires deployment. You have to deploy a new TypeScript file every time you want to deploy a feature. And we ended up with long lists of user IDs in the code base, which is far from ideal. So, uh, what do you do if you're using a robust web framework and you want to add something new? You have looked for what exists in the ecosystem. And I found something called Django Waffle. Which looked really cool, built into the Django user model. So it directly gave you uh commands for deploying to all staff, deploying to superusers, deploying to groups defined using Django groups. We were able to customize the model. So

Speaker 2: using our customer workspace model, we could say, okay, deploy this flat just to this, this, and this workspace. No more need to redeploy to activate the flags. They're stored in the database. And you can manage your dev environment and your production environment and your pre-production separately. Downsides again, though it took took a bit of time to set that up. I was happy to have had six months experience of a messy solution before scoping how much work to do uh customizing Django waffle. The documentation at the time I was looking at it was out of date and wasn't building in CI

Speaker 2: and there were no tests for Python 312, I think. So it goes back to the question, okay, what uh libraries can we rely on in the ecosystem? I opened a couple of pull requests on the uh GitHub repo and immediately had lots of two or three different people responding to pull requests and updating the CI, updating Django, updating Python, everyone with strong opinions on how the pull rest request should be done and what uh uh uh uh etc etc. It's part of jazz band. So that gave me the confidence to say, okay, let's go ahead with Django Waffle So here's what I found in Django Waffle.

Speaker 2: It says it aims for a simple API, a kind of batteries-included approach, which works well with the Django built-in user model. It aims to be simple to install and manage, that's my experience. Robust enough for production. It's got uh it says it has aggressive caching. And I know someone had a couple of problems customizing their user model and then having to make make sure that worked well with the Django caching. Basically I could see it had more than enough features for what we wanted to do. It had flags so you could turn things on and off for different people. Switches, which are basically flags, but they apply to everybody.

Speaker 2: and samples which let you roll things out for a proportion of your users. You just say, okay, I want this for 10% of the users and then 50% and then 80%. It has management manage commands, so you create flags at the command line and uh template tags, including I think a separate integration with Jinja. There's a uh JSON endpoint which we use uh for the uh integration with the front end. We just pick up the flag state from the front end. And there's s support for CI testing. So how how do you get started? You start off with pip install Django Waffle. Then you need to add a bit of configuration.

Speaker 2: add waffle to your installed app and add the built-in middleware. All of that's provided by the package. And you do a manage migrate because there are some migrations included in the package. And then to get started, you just uh type manage waffle flag and then the name of your flag. Dash dash creates create a new one, dash dash user, dash dash staff, dash dash super user, dash dash everyone, whatever you need. There is a uh it's well documented. You can uh do a Dash help if you want to know all of the options in there

Speaker 2: There's a uh decorator or a simple function um approach in views. So You check whether the flag is active. If so, you put in your brilliant new feature. If not, you keep the old code. Uh there's a template tag, flag else, and flag. If you're using templates, I'm not, but And don't forget, when you when it's been deployed to everybody, when you're happy you're not gonna have to roll it back

Speaker 2: Then separate command waffle delete my flag to clear up behind yourself. And I like the idea someone was saying set a uh cron job, a reminder. If the flag has been active for everyone for one month, then raise an alert or um I just just delete the thing. A few alternatives alternatives that I came across in the ecosystem. Django flags seems to be maintained, uh, but uh less popular, less used, and uh it's maybe covering a different use case. Django waffle doesn't look right for you, then maybe Django flags

Speaker 2: is better. Gargoyle was a popular one, which Simon Willison referenced 10 years ago, but that's been archived as far as I can see And uh different platforms, I've never tried any of those uh kind of managed platforms for feature flags. That's another option that some people choose is uh a separate platform which lets you manage all of your flags and have a dashboard to uh to view them. Uh we had a talk about some of the worst software bugs in history the other day. A couple of them came from feature flags. uh a small one and a uh um

Speaker 2: kind of preparing for this talk I went to uh uh meetup at the start of the year and came across uh uh uh one developer who's just sworn off flags. He said, okay, yeah, we tried them. We won't do that again, because when the product managers found out we had we could deploy features on a case by case basis for different customers. They just sold that to everybody. And we had to and we couldn't we couldn't make any more progress because there were just if-else everywhere in our code base. Duplicated code. So he uh he made the kind of unilateral decision to remove the flags. Doesn't want to hear about feature flags again.

Speaker 2: And if you want to hear a worst example then a worse example then have a search for uh Uh DevOps caution retail or the $460 million mistake. Um Which was a company, a high frequency trading company, which used feature flags to uh activate different options in their uh high frequency trading algorithm. They reused an old feature flag which had been sat around unused for many years. They decided to reactivate it for a new feature. High frequency trading, then you're buying and selling uh stocks very, very, very quickly. They activated this new uh feature flag without realizing that it was already set differently in different production machines

Speaker 2: one of their machines suddenly started uh behaving very very strangely uh making uh lots of very bad kind of purchasing and sale decisions. No one understood where it came from because it was kind of Something to do with the environment of that one production machine. So they tried to roll everything back, but rolling it back meant that actually this feature was applied on all eight machines. Forty-five minutes later they worked it out, shut the whole machine down. But uh by that time they had bankrupted the largest at at the time, I think the largest high frequency fruit trading company in the world. So they

Speaker 2: then spent six months ro winding up the company and paying off all their uh creditors. So please do clean up your flags when you've done with them. You will be happier and less risk of nasty side effects. A few pointers if you want to go further with feature flags. Uh people use them for A B testing. for canary rollouts, kind of rolling out the a feature to a few customers to see if anyone panics or if anything breaks. And for experiments, experimenting with different uh features.

Speaker 2: And there's a nice article from Martin Fowler online. Which discusses these different types of flags and different uses in setting them out them out in terms of increasing duration and uh different ways of classifying Uh feature flags.

Speaker 3: Great, thank you. That was wonderful. Um I've got two questions, but I'm gonna go for the better one because we've only got time for that one. Uh cleaning up flags. Have you got any great advice on implementing system checks or any other heuristic to make sure that when you you want to get rid of a flag you f a flag, you find all the all the places where it's featured and make sure they've finally been cleaned it up.

Speaker 2: Yeah, I'd say, you know, what what do you do in your team to uh for for good housekeeping kind of adopt whether it's code review or um uh periodical checks. I want to write a cron job. Um I mean we use Huey running in uh our production system, so I just want to run a Huey task or maybe a Django startup task uh to check. Is there a flag? Is it set for everyone? And was the last modified date more than a month ago? And if so, then probably raise an uh alert in Sentry just because we're using Sentry

Speaker 2: for um for error monitoring and I think that's an error if There's a f a flag which is sitting there untouched.

Speaker 4: So we are running out of time again. So thank you for the talk.