Slow Food Digests Better - or how to maintain a 8.5 year old Python project without

Speaker 1: The next talk is again uh describing something that many of us have gone through in the past. If you've used Django for a bit, or if you've started on a Django project that has been written some time ago. You may have run into the issue of reading the Django documentation and seeing some solution to your problems and starting to use that solution only to notice that you were accidentally brow browsing the incredibly version Django documentation at the latest version, but the project is some ancient version from like two years ago or something And so this incredible solution won't work. Or you had to upgrade Django to the next version

Speaker 1: and while Django helps you a lot by providing a very detailed change log, this isn't exactly pleasant either. Christopher is going to tell us about a very old Django project and how they at Mosabell worked with it. Thank you

Speaker 2: Thank you. All righty. Um so hey, I'm Chris. Uh I'm here to talk about um a very very old Python project that originally was a PHP project, but let's start a little bit um with a introduction. Um so I'm a software developer for almost 10 years now. And During that time I worked with many smaller and bigger companies, um startups, fully grown companies and have been working with many, many code bases as probably um a few other people here have as well And notice that over time while also while adding new features

Speaker 2: and um implementing that new cool async stuff and other um things can be very exciting or and exhilarating. It's it I was more torn towards old, ancient big projects and had funny enough, lots of fun to tackle them, touch them, work with them, um create proper linting, create proper dependency management, um continuous integration tests and kind of trying to modernize those projects. One of the projects that I've been working on now since October 2015 on or at Mozilla is add-ons. mozilla.

Speaker 2: org Which is actually one of the most or one of the oldest websites at Mozilla and has been around for approximately 13 years now Add onsubMozilla. org has gone through a uh huge transition from a quickly hacked together PHP project um to a now almost modern Django application and will be heading towards Python 3 and Django 2 hopefully sometime in 2019 and maybe starting uh already in 2018. Um a few more numbers, add ons. mozzilla. org serves approximately three million unique users per week.

Speaker 2: And the code base is something around 250,000 lines of code, um, counting Python, JavaScript, CSS, and SQL migrations. So let's start with a little h uh history lesson. Um that is not complete at all because that would take far too long. Um so in 2005 Mozilla. org okay that screenshot is not good. Anyhow uh started with flat HTML files and sprinkled with HP with PHP code and a few data database queries here and there. So basically just Exactly like everyone told you even back then not to do it. Um

Speaker 2: it was a beta though, so nobody cared. Um It has been redesigned over the years uh many, many times. So here's a new fresh uh Mozilla design uh that kind of focused on the idea of being uh promoting um customization and custom uh custom yeah customization in firefox uh through firefox add-ons. The PHP application grew more and more complex over time, and so it has been rewritten um to be based on a PHP framework, cake cake PHP version 1. 1, which, as far as I researched, uh wasn't that mature. or mature back then and uh only version 1.

Speaker 2: 2 made it much easier. So um as it happens in life you settle on the wrong technology or it was probably a very good decision back then But it turns out later that it isn't. So you have to stick with it and have to work with it over time. Even using a more professional PHP web framework, um the whole application got more and more complex. Now we're in two in two thousand nine, the whole page is still using PHP, but after uh Django one point la one point zero got released in September 2008. Um a few people inside Mozilla

Speaker 2: started to promote Python and Django as the new technology to move m move forward and try to port many of our projects to Python and Django. It started with smaller websites and eventually got also towards projects like AMO. Um the transitioning time or the yeah, the transitioning time from the PHP version to a Python and Django version Took approximately 16 months, um plus a little bit of transitioning time, going live, going back and forth, um having lots of downtimes. and stuff like that. Um the whole 16 month were very stressful for the users, obviously, because the site was unreal

Speaker 2: reliable. They just wanted to search for an add-on, wanted to install it, wanted to be happy, but that didn't work. And there were bugs, um, maybe some translation errors, stuff like that. It was also very stressful for uh the QA team And management obviously because well issues arise, things take longer than you expect, because you can't just rewrite one code to the other and expect to just type it down and it works. You have to implement new features, you have to work with the data that you originally had in the database. And um you have to migrate to the new whole new system. So that is very error-prone and there's only a few ways to that you can use to actually test that

Speaker 2: and the best way to do that is manual testing and that's very error prone as well. So in January 2010 plus a few more months uh transitioning time the whole system got pushed into the live system. Um the big advantage of the rewrite to Django was that during that time or to Python and Django during that time. Lots of new unit tests got added, um which simply didn't exist before. Um it didn't have a hundred percent coverage, but it was close enough. And um yeah, lots of new features uh did happen and a few bug fixes and wrongly designed features in the PHP version were were also corrected as much as possible.

Speaker 2: So um almost six years before I joined Mozilla and uh Yeah, the whole porting process uh started and uh this whole uh s or the new version or the new Python and Django version uh was called Zimboni. Now Canadians were involved, Canadians love their ice hockey, so that's a zimboni for everyone who doesn't know about that what that is It's one of these ice machines that make sure the ice is uh perfectly clean and uh yeah, works like expected. So between 2013 and 2015, um A few more redesigns on the left hand side you see the result basically um

Speaker 2: after the website got ported to uh Python and Django. And um yeah Over those two years two or three redesigns happened, which is basically the one thing uh AMO is famous for every new year gets a new design uh because Why not? Um the whole site wasn't too special actually, so it was primarily using Django. It was sprinkled with lots of raw SQL as it was before. Um but it had some cool net new things like Celery and RabbitMQ for asynchronous um task management. And um Yeah. One problem dur or right after that part basically and between 2013 and 2015

Speaker 2: was that most of the development resources weren't working on AMO, so it kind of got paused. uh during the development um or the development got paused on AMO and it was left in that state that it was it was rewritten from scratch. Um maybe a few workarounds were there and essentially many of the old entries and the old schema of the database back from the PHP world. was still there. So as a site note, the the page was built around or the new page was built around the existing database. So that was cool, or not so much. That was because during that time Firefox Marketplace happened, uh

Speaker 2: which is now not. uh there or was discontinued. And many of the AMO resources were working on the Firefox Marketplace site. And making sure That AMO was ready for Firefox OS and their apps. So that uh yeah, Firefox OS could use the marketplace to Or users of Firefox S could use the marketplace to search for an app, install that app just like everyone knows. Android or iOS or any other mobile system works. And the interesting part here was that the Firefox Marketplace actually was derived from the AMO work. And initially the same code base tried to be both

Speaker 2: Firefox Marketplace and AMO. So we had funny hacks like there was a model uh from an add-on and that add-on model changed depending on a setting and it was either an app or an add-on depending on the site that it was running on Zimboni, on the other hand, so the system that's now running Firefox Marketplace, um got cleaned up a lot and um implemented a completely new front-end based on React. js. um adding lots and lots of new functionality and pushing more and more functionally towards Elasticsearch for performance reasons. We do that because our database clusters

Speaker 2: are very performant, can serve a lot of queries, but there are um there is a huge amount of just static data on our site and um yeah. Performance tests have shown that Elasticsearch can can serve the data that we need much faster uh than our database cluster, which is also used for some statistical data, um, so the data isn't stored. in the perfect way that we needed. Um now all that changed again in 2015 and Olympia was born. Also again a Zimboni. but a super cool, shiny new version of a Zamboni named Olympia. Around that time a new team got formed, um including myself, a few long-term AMO developers, marketplace developers, um

Speaker 2: And we all got faced with a very difficult task to use the old addons. mozilla. org um code and Re re kind of use that code and implement or reanimate that code basically and um had a huge list of features that we wanted to add and um Also needed to make sure that that code um will be living in the next few years. So the code back then was ancient. It was barely using Django 1. 6 and every attempt to port to a newer Django version

Speaker 2: before 1. 6 was kind of like It's installing 1. 6, making sure the tests run, making sure the site run, um, and that's basically push that to the live system as soon as possible. um and deprecated features or things that will be eventually uh removed in Django um largely w weren't touched. So It was running Django 1. 6 but many of the old and internal um APIs were still used Um we had essentially no unit tests for our JavaScript code and the site was very JavaScript heavy. We have super awesome shiny

Speaker 2: install buttons that have twenty something different states. Uh we have the whole add-on developer upload system that's running asynchronous uh JavaScript and accessing um some APIs And all that code was tested by hand manually, but didn't have any unit tests at all. So that was very error-prone Our CSS also was very massive. It had been through many, many redesigns over the time, and and we had Plain CSS, we had SCSS, we had some less code and some stylus code, just to name a few systems that we used and were actually running in production to style the page. So there was Painful to

Speaker 2: see. We had to make a few hard decisions whether or not we want to rewrite the whole thing completely from scratch. again um or uh if we want to improve incrementally. We essentially decided to improve uh to improve incrementally the uh the the code and system over time because rewriting from scratch for us was almost impossible. We were a very small team and unable to maintain two different versions at the same time because you have to write the new version, you have to maintain the existing version, and then again you have to transition between both versions back and forth. Also, the existing uh version had a huge amount of features that many of us knew but never had seen since quite some time in the past.

Speaker 2: uh collections support for Firefox, support for Thunderbirds, Sea Monkey, themes, different types of add-ons, and still some legacy code when the code was used for both Firefox marketplace and AMO. So a rewrite from scratch wasn't possible and um Yeah, we decided to incrementally improve the whole system. Um another factor was that we had thousands of unit tests um that may demand that made demand maintenance much easier um and at least got us some peaceful nights when we pushed some some code live and our tests were running or some of the existing tests uh didn't break.

Speaker 2: So over the time uh we had been tasked to port the whole system to Django 1. 8 um during 2016 to 2016 and 17. Uh removed old cult code where possible, restructured the whole repository. add a proper dependency management, used much more of Travis, moved to uh GitHub for our uh issue tracking involved the community much more, added more documentation, and also had to work towards the bigger goal of Mozilla to move towards the new add-on system, which is web extensions Um which is using um HTML5, JavaScript, and other technologies to build your extension or add-on

Speaker 2: in the browser. Um that happened and we also had to redesign because we do that every year. And uh that was a one of the bigger challenges back then and uh we kind of ended up with one new file that was called restyle. less, which essentially took all the p uh all the old styles Changed things back back and forth so that that the restyle looked good, but all that never ended up um actually touching the existing styles that we had in the co in the in the repository. because that would have been way too complex and error prone because we didn't properly know where they're used.

Speaker 2: So we ended up writing a completely new front end from scratch. based on React and as a single page app, um which is accessing uh the add-on server via a new REST API. Uh much more functionality got moved to Elasticsearch and APIs so that the front end could use them. And we Um started with a small experiment duer during uh one of our all hands um which with a yeah very small experiment that used React as a single page app uh with server-side rendering and um implemented a very small amount of our page, which which is the

Speaker 2: discovery pane that you see here when you go in your Firefox um browser uh to the add-ons page or about add-ons and see um Recommendations of add-ons that you can install simply by swiping that uh button to the right-hand side. And that worked out nicely. was pushed to production after a few months of development and kind of laid it or set the groundwork uh for future development of AMO and the front end i itself. Uh so we were we were We rewrote the whole front end from scratch as well. Um this work got released in November 2017

Speaker 2: and this is how the page is looking uh or was looking in in December. Um yeah, the whole work got released right before Firefox fifty seven or firefox quantum got released um and had the luxury of focusing entirely on web extensions and themes and a few other um pieces. And so essentially we did both with AMO. We incrementally improved the back end of our site and on server um over time because we had Lots of unit tests, but a very, very complex system to maintain and had the or made the the decision to uh completely rewrite our front end from scratch Because we didn't have the luxury of unit tests

Speaker 2: and other maintenance helpers for the front end and essentially needed something very clean. and ended up s uh writing something from scratch. So that Now leaves us with a um completely completely new s system and we're slowly uh deprecating features on the old side. Um for example when we uh launch new replacement sites for apps like Thunderbird, SeaMonkey, for example. Uh we can ri uh get rid of lots of new code which will make the maintenance of our back end much, much easier and potentially allow us to move towards more recent uh Django versions and dependencies

Speaker 2: and implement new features much much more quicker or much much quicker So what did we learn? We are heavy users of feature flags and switches and are using Django Waffle for that. We try or usually try to develop new features that are that sit behind a waffle flag so that we can push them to production. once new things are finished and we have a QA team that can test them on our development and staging systems. And have our UX and UI team look at the features that we implemented and have them give us the go. when things look good again uh so

Speaker 2: then we just have to flip the switch. Um so that makes it much easier to Push new code, make sure it runs on your development and staging systems, also push that code to production without putting it live, but also making sure that you didn't break any other systems that you rely on And that makes it much easier to revert any broken deployments, um obviously because you only have to flip back that switch. That can be or is usually very easy, but depends on your feature and if you need any database migrations, for example. So if you have that, you need to take some special care So

Speaker 2: another learning that we recently made or almost a year now, um or made since almost a year now, is third party dependencies can be a huge pain. Um have been always since we moved from Django 1. 6 to 1. 7 and 1. 8. um with a huge or very old date um code base as ours and using many many internal functionality of Django um that was a huge pain. And the move from 1. 6 to 1. 8 alone costs us around three months of more or less full-time development. Also, while dependencies and semi-automatic upgrades via PyUp or Greenkeeper are awesome, and they usually work on the unknown

Speaker 2: server, um we have Often the problem on our front end that we get swamped with JavaScript dependency upgrades every now and then and it's very hard to keep on top of them. So um Be aware of that. You don't have to be on the bleeding edge. Um but also make sure you're um subscribe to the correct mailing lists and project bug trackers so that you can be aware of any security issues or bug fixes that are potentially in fact impacting your site. Also, upgrading ancient dependencies can be very daunting and very time consuming, and it's often hard to find the right balance.

Speaker 2: between doing the upgrade and knowing how long that'll take. Experience obviously helps. Um but what also helps is a very time -boxed work window and experimental branch that you just start up, give you like two hours or three hours and work on that, see what breaks and make a list of things that you think will happen uh or must happen um so that the upgrade can happen. Um it also helps adding new dependencies To your continuous integration system while working on the system as well, so that you have both running at the same time.

Speaker 2: That makes sure, for example, if you're upgrading Django, that your code base can run, for example, both versions. And um then essentially you only have to s flip over to, for example, from Django 1. 8 to 1. 11 in your production. system and essentially can remove the old f um waffle flags or flags and and and and code um and workarounds from your 1. 8 times. That's something we didn't do for our upgrade from 1. 8 to 1. 11, and that bit us a bit. uh and yeah cost us many many months or actually a year of work uh during that upgrade because we ran into lots and lots of third-party dependencies uh

Speaker 2: that were broken, unmaintained, uh because we used many or we back in the days many of the dependencies that we used were Written by Mozilla, maintained by Mozilla, but over time the people changed positions and the projects are unmaintained now, and um we had to upgrade them step by step or get rid of them So that's very time consuming and daunting. So make sure the dependencies that you're using are up to date, uh are well maintained, or uh also make sure that your on top of them and yeah know how to how to handle them and if they're unmaintained make sure that you get rid of them or replace them um or contribute to them uh

Speaker 2: on time before you run into any problems. I actually can't stress this enough. A QI a QA team actually saves lives. Um While unit tests are awesome, um automated Selenium UI tests are awesome as well, but having a QA team that manually tests your system and also makes sure um thatch cases are covered, that the whole new features that you add are um working as expected. um is a huge time saver um because often

Speaker 2: if you're using for example mocks um you Or not often, but if you're using mocks and if you use them too much, for example, you can end up um not actually testing the right systems. So having a QA t team that has a clear list of things that need to be tested. helps pushing your features, upgrades, dependency upgrades, and even migrations makes all that work much, much easier. Also explaining what you did and how to test it to someone else makes it much easier to reason over the code and the functionality that you implemented Yeah, makes a reason about that or makes yourself reason about that and

Speaker 2: allows you to um See any per uh potential edge cases that you might have forgotten over time. Um implement features incrementally and flip the switch to that feature once they're ready. It's not often as simple as that, but if you can do that, do that because it helps having that code into pro in production. sooner rather than later and catch any per potential uh errors. On time Also, don't panic. Usually you're there for the long run and you have to make sure the project works for quite some time.

Speaker 2: If you see some code that really doesn't look that great, it usually there usually was a reason for the code back when it was written like that. and um no one actually wanted to do something bad. Um because yeah when you write your code you use your best judgment and uh go forth with that. Um so That was my talk. Thanks a lot for uh listening. You can find me on GitHub, Twitter, and there's also the awesome uh Mizilla. org add-ons blog. We can find a lot more technical information about AMO and the stuff that we do. And if you have any questions, just find me after

Speaker 2: the talk. And um yeah, thank you.