The windy path to fast, pain-free, reproducible developer environments

Thank you very much. And hello Porto. So today I'm gonna talk about developer requirements and I'm just gonna jump straight in because we've got quite a lot to cover in 30 minutes. Um I want you to imagine you're working on a Django site locally with RunServer. You've got a bug to solve and you're deep into debugging uh your Django view. You're in that iteration phase where you want to be constantly making small code changes to get to the root of the issue. So you have to change uh so you're gonna make a change to add a trusty old print line and you hit save And then you wait and you wait a little bit more and eighteen long seconds pass by By 16 seconds I'm mildly miffed.

By 12 seconds I'm counting how many times that Chrome animation is looped. By eight seconds I'm seriously questioning my life choices. And by four seconds, uh I've given up and I'm reading something else to be honest. So how do we get here? This was our situation on a daily basis. Instead of making five to ten edits a minute on a Django code base, you reduce to two if you're lucky and it's a super Super excruciating level of frustration for everyone. Uh makes day-to-day development super annoying. Um so how did we get here? Cross-platform developer tooling. And to understand why that matters, we first need to understand why we'd even need it in the first place.

Without any help from developer tooling on a cross-platform team, and by cross-platform I mean across OS, Windows, Mac, Linux. You'll see install issues due to different hardware, operating systems, previous installs and configuration, severe version mismatch across your colleagues' environments or worse production. And if you're in a leadership position such as I am, you're constantly needing to troubleshoot others' issues. And uh I'll go as far as saying that a decent uh developer setup is a prerequisite to hiring juniors Because otherwise you're deliberately setting them up to fail and it's just not fair. So historically, we and any other cross-platform team have looked to develop a tooling to help us out with this

Um a nostalgic slide for anyone that's been a developer for probably more than ten -fifteen years. Um There were the before times like pre-2010. Um we don't speak about the before times, but it's safe to say many joyless hours were wasted setting up developer environments uh with custom scripts or worse, nothing. Then in 2010, Mitchell Hashimoto, CEO of what became HashiCorp, came along and gave us Vagrant, which made working with VirtualBox VMs in a headless fashion via the command line much easier. It had its drawbacks though, and when Fig, um which was later renamed to Docker Compose when they bought it, came along in 2014. It wasn't long before I switched to it for all my projects

and so did a lot of our industry. For those of you that don't know Docker Compose, you can think of it as a tool for starting various Docker images with their own file system. And then optionally mounting your host file system so you can make changes uh live and that update in the actual Docker image without having to do a rebuild On Linux systems, this occurs natively due to Linux container support, but on Mac effectively starts a VM known as the hypervisor. And this means that the host file system must interact with the virtual file system, which is key to understanding where most of our problems lie. And so this is our actual stack for languages. We've got Python, JavaScript, TypeScript, and then a whole bunch of services that

back the product. As an industry, uh we like to call monoliths simple and they tend to be when compared to microservices, but that doesn't mean that setting them up on a per machine basis isn't a complete chore. You end up doing like a lot of puppeteering to bring things up. For example, can you imagine bringing up all these services one by one just to get your development environment going? Nope. So to give a little bit more background, um we're unijourno. com, we're a freelancer market. We have about ten thousand files in our repository that we need to care about for the purposes of this talk. Um it's about 2. 6 million lines of code, most of it JavaScript. This isn't a boast because this isn't even a big repository, uh, but it plays into where we're going.

And so that brings us into the Docker Compose stack we had. We were running the Python container and all services within the Docker Compose stack. um and the front end build sits outside it deliberately because we never once managed to successfully successfully isolate the front end build without serious performance issues And so when the slowdowns of the back-end container progressively got worse and worse, it was obvious where to look. And a simple experiment confirmed our thoughts. We simply ran um We simply ran the Python container without the file mount and changes such as like reloading the run server end up happening really quickly. So, how do we solve it? Uh we know the issue is related to the file sync

as the Python container with known mounts just runs fine. But to do local development we need the mount. And we need the mount because otherwise you're going to be redo rebuilding the Docker images after every change you make, which is just not scalable at all We also know it's related to the size of the repo, and we know we can't just delete our repo. Um and we know we don't want to go backwards to a while where we spend hours every week debugging various environment uh issues and versioning problems. So What are we working with? What are we trying to improve? Our run server reload was about 15 to 18 seconds on an M1 MacBook Air. any management command would just sit there for about ten to fifteen seconds before it even did anything

and I never got to the bottom of truly why this was, but to reset the test database via PyTest it took 14 to 15 minutes, which uh insane. Um oh Ahead of myself. So um the goal for us is at least a 50% reduction as part of this project to get us back to reasonable uh but not terrible levels And we had a look at all the way forwards. We went back to first principles to really see all the options available to us. And the first one is obviously there's always the option of doing absolutely nothing. Docker have shown a decent amount of interest in fixing the issues with their hypervisor on macOS. Um but the situation is really gnarly.

Uh this is the main GitHub issue about the problem. The line on the right is actually only half the thread, but if I made it any longer it would have wouldn't even appear on the screen. So this is opened in March 2020, has about 700 comments. It's the kind of GitHub thread you never want to find when Googling for a quick solution to your problem. But they have tried hard. They have tried hard. So um recently they've switched their plans from integrating mutagen, which I'll get to, to then GRP Fuse, and then again earlier this year with the experiment experimental uh Virtue FS. But none of them helped us unfortunately and um the introduction of M1 chips and us getting MacBooks, despite I asked my boss for new laptops, he gave them to us, the situation ended up worse

Serious airgorn phase moment. Here we go. So option two, and I'm sure a lot of you on Linux are screaming, why don't you just use Linux? Um it's not a realistic solution for our company to use. The entire company use MacBooks, our all of our IT infrastructure is Apple, security policies, security questionnaires, like the list is endless. Everything refers to Apple Um and I can't change that. I'm just one man. Being uh freelancer market, we also work with many freelancers and they bring their own kit and they use Windows Um and some of them use Linux and get the platform working fine. So yeah, um we need cross-platform support basically. And so the third option

is to try and speed up the sync. We know that when we like don't mount the file it's fine. So what if we have we could what if we can find a way to improve the performance of the sync between the hypervisor and the host machine? And we did it actually for a little bit There's a project called Mutagen, and you can think of this uh like a bi-directional rsync with a low latency firewatcher. It's written by one guy as far as I'm concerned. Don't have his name here unfortunately. It was actually designed to sync between your local machine and uh remote locations, whether that be Docker or otherwise. Perhaps you're doing development in the cloud and you want to sync your local file system. But there is a handy solution that specifically works with Compose by creating a volume and using the mutagen

protocol to sync that with your host machine instead. But we used it and it was in alpha. It did help for us for a long period. But then we had issues changing branches. We'd get like file uh stale file state especially on large branch changes. And random bits of code appearing in your PRs is not something you ever want to deal with on a daily basis. So we had to get rid of it. Having said that, it's moved on a lot since we used it a year ago. And if you are having like Docker speed issues on Mac or Windows Give this a go before you do anything else in this talk. So the fourth option, and the one we're going to talk about the most today, is going native. And so by native I mean running the Python project on the actual macOS

uh file system. And everything stopping us doing this is all the problems I've discussed already due to you need um You need a reproducible environment for developers. And the fifth option, which this talk was going to be more about, but it's not going to fit in the time. is to go fully remote. So take development to the cloud um and base the the cloud development um with Linux. There's a lot of services doing this and I think it will be I think it's the future, but we'll get to that. Um so yeah, there we were. Uh we had five choices. Three were dead ends, two remained. Um and that is when Nix entered the chat. So It changed everything for us in terms of local development

and um from my perspective potentially much further afield too once we uh spend more time with it. So I was first introduced to it by Repolit, which is like an online cloud development environment. In about 2018 they switched to using Nix to set up their 50 language environments um and they made a public bet on the tech. And then I saw at the start of pandemic, Shopi uh Shopify were switching all the development setup to it. It's hard to say Nix is on his hype cycle given how long it's been around Um but there certainly has been a shift in usage over the past few years. Um and with this hype I'm about to tell you Comes a big fat disclaimer. Nix is super complex. You should not trust anyone who says otherwise. It has a fairly steep leaning curve to truly master, and a developer experience

experience is still actively being worked on. So what is Nix? It's uh it's a package manager and an operating system and a language. And they really could have spent more time on name n naming, but alas. We are going to focus on the package manager bit I'm always wary of hype, so I'm going to try and explain how I sold it to myself by starting with what it promises, why the authors felt it needed to exist and then cover its core working mechanics. Uh hopefully without sounding like a fanvoy. The fact is um it's a tool that let us get back to building our app, which is what we want to do. We're not here to care about Nix, we're here to get our job done uh without wanting to poke our own eyes out. So Without further ado, um

Nix's first release was in 2003, followed by this paper which got released in 2004. Honestly, this came as quite a shock to me when researching, as I've only been reading about it since 2018. And I was at school in 2003 when this came out. The paper itself is a fantastic uh approachable read. I'd recommend anyone to get a deeper dive on the why and how uh Nix exists. Notably it calls the act of transferring an app to different environments a deceivingly hard problem. And I think we as Python developers can agree given what we've been through. So what does it promise? Reproducible builds. Given the same input, the working output should be the same on another machine.

Now you might be thinking here, this is what Docker offers, but Docker offers runtime reproducibility and not build time. When you build an image with Docker, the resulting image is portable and uh obviously we'll reproduce because of that. If you run the build again, say a day later, on the same Docker file, the resulting image may differ. This might be what you want. But it is not reproducibility. Um and that's the fact with Docker. So NYX is uh declarative. in that you describe aspects to construct the environment you want. Again, this is slightly different to Docker because with Docker files you build your world procedurally line by line in the Docker file. And anything can happen on those lines.

It promises us an explicit dependency graph. So Nick's belief is that every piece of software on your machine implicitly relies on each other and on the hardware it was built with. and the platform too. At its core, Nix is a graph generator to make all those links explicit and therefore aid in portability. If the promise Skip, next one. So another thing it promises is cross-platform build. So it has full support for Linux and Mac OS, which is a rare thing. And it also supports Windows if you're using the Windows subsystem for Linux. And then use IVIR Ubuntu instead. So this ticked all of our boxes because these are the three operating systems we work with on a daily basis.

Okay, cool, but how does it do that? Um so if you're like me, promises are not enough Until I can get a true understanding of how something works, uh I leave it in the high basket. So I'm going to try and explain uh the core blocks in a way which gives you just enough to get it but doesn't make you uh hate me. So next up we have uh the first building block of Nix is the store. So after you install Nix you'll be left with this directory on your machine This is where Nix keeps its stuff. You can think of it like a massive cupboard with a hundred drawers. If you take a look inside it, you'll see a mixture of directories and files all appended with a hash

Uh the hash is very important but just remember it exists for now. What you are looking at is NYX file system -based graph database. So each entry in this directory, whether it be a directory or a file, is a node in that graph. And the relationships between them are the edges of that graph. So some facts about the store that will help you understand how it acts. So only Nix can write to the store It is a special zone, untouchable by anything else. Once a node is written, it's immutable, so it can never change. The node will have the same name on any machine that it's built on. It just has to trust me for that for now. And if you want to see the Nix dependency graph in action, you can run this NixTalk command,

passing it any node. And then pipe the contents to graph is, which will generate this uh hideous looking beast. As you can see, I've used my active Python node here and it depends on OpenSSL, which in turn relies on Apple's core foundation, which in turn relies on LibObject DC. None of these concepts are new, it's just it's system level software. We haven't had a good story for that pretty much ever. Cool. So, how does Nix know what to write to the store? Earlier we said one of the rules of the store was that only Nix can write to the store. And the answer to that is derivations, which is a super scary word. They're a scary name for a file which exists as a node in the store itself.

And the responsibility. These are the DRV files you see before you, and their contents look like this. Now, thankfully you don't actually need to read this. This is a JSON representation rather than the actual plain text DRV format, but it gives an understanding of the data structure within. It's a simple key-value JSON-like object. The derivations contain all the nix storage edges, the paths, or the dependencies of the package in question And the hash of the contents of the file end up dictating the path the derivation is stored at in the next store we showed earlier This is important because if a dependency the package relies on changes, this would result in uh new derivation.

As I said, they're completely immutable. So what kind of uh stuff does is actually exists in this file? Um It literally defines how to build the package from source. That's a key part of Nix is that it's a flexible package manner. So while it has um a binary object cache that you can use. Everything can be built from source because you have everything you need to do that. So if we look through the arguments, um the platform is what operating system is on, macOS Linux The input DRVs, this is the other derivations that we depend on and that would be needed to do the build. The input sources is the other already built nodes that we depend on.

The builder is what program runs the build. Is it bash? Is it GCC? What is it? ARGS and N are arguments and environment variables for the build. And the outputs is like Which files get output into the next store due to running this build? Um okay cool. But what makes the derivations And this is where it gets a little bit more complex. The answer is Nix, the language itself This is probably the most controversial part of Nix, or at least the part I see question the most. So why why does Nix need its own language? Why can't I just use YAML? The answer is because Nix's goals are unique amongst all package managers.

Nix the language is both incredibly helpful to learn, but also the most useless for any other purpose. But that uselessness is uh is entirely deliberate. So it has two standout features which make it safe and good for this use case. Number one is that the entire language is lazily evaluated. So I think a lot of you will be aware of Django's lazy string translation or the lazy URL lookup. Um and well the idea here is exactly the same, it's exactly applied to the entire language. Anything you write is only evaluated when it is actually needed. And therefore when you're dealing with massive graphs of data that depend on certain scenarios It speeds everything up. And the second most important feature is that it's side effect free.

And by side effects I mean any input-output. So no network No like console log or anything like that. There is a bit of debug output, but there's no disk access and no user input. And that's and that's why it's useless language for other use cases. So except there is one exception uh to the no side effects rule. Um and in understanding what it is, you'll understand the whole purpose of a language. So when you call the derivation function this derivation function with parameters, Nix writes the derivation to the store The sole purpose of the Nix language is to create derivations. And so it's these uh it's these um Well, you write derivations with the Nix

language as part of uh working out what the dependencies are for your given package at any given time. And the the final building block Because everything I've said is useless if Nick cannot ensure the very Nick cannot ensure the various builds only pulled from the store is sandboxing. And it does this in two ways, by by patching by providing patch versions of compilers and linkers that don't look in the normal locations. And secondly, more importantly, by building each derivation within a sandbox that denies Access to anything it isn't supposed to get its uh grubby hands on. Um and this leads into how we take advantage of it um for developer environments

It all starts with a little tool that ships with Nix called Nix Shell. And when you run it, it looks for a file in the current directory called shell. nix. And in that file, there should be a derivation. um a defined definition and the output uh and basically what it does is it runs that derivation, puts everything it needs into the next store, creates a bash uh subshell, relinks everything so that that subshell only has access to what it actually should have. Um and from then onwards your shell um It's fine. So if any of you used to use uh make virtual env or any of the virtual wrapper stuff, you know about Shims and you know about how they

to change the um the shell paths to make sure we're working in an isolated environment. This isn't much different. All your binaries just end up pointing direct to the uh Nix store. Um and so you may be asking, do I have to build everything from source myself? No Because of the immutability and because of all the metadata which goes into creating a unique Nix node, that string um can be treated as safe to cache once that node is built. And so that's where Nix packages comes in. It's the official package repository for the Nix uh ecosystem and it's basically a giant repo on GitHub full of derivations that other people have uh supplied.

One thing oh no so Nix um so Nix packages has a giant cache of everything you need So you end up just downloading from the cache when installing because they they have a lot of machines running all these derivations and creating all different builds for all different platforms. So yeah, you don't have to build everything from source. Cool. Awesome. So this gets us to our little shell Nix, which sits in our platform repository. And I'm going to walk over here a little bit so I can see it. I'm going to talk through what this is, what the various parts do, but you don't really need to know about Lithinix language uh which this is. So at the top you have the imports and aliases. So that first line what it's doing is it says

get the Nix package tarball from GitHub at this exact commit And this exact commit maps to a channel which are basically like branches. So they have a one for stable, unstable, and then various operating system releases. The following three lines are inputs, and then the packages make shell function, that's our call which creates the the derivation for us. So then the native build inputs attribute, we just list what we want. Um and as you can see we set up uh we set up our our Python poetry pre-commit, node. js, yarn, everything is um defined in there. And then what we also do is we use it as a base layer for

uh still using yarn and poetry. We haven't gone full Nix, you can go full Nix if you want, but it's a bit Too much for now. We just wanted a base layer that we could actually always run poetry and never get a failure due to something not being on the system or whatever. And so for each package is that needs custom stuff, we just include the package there. And as you can see at the bottom, there's a little exception for uh Darwin systems, which is macOS. Yeah, so you need Xcode build, and for some reason no conva no canvas doesn't build unless you have that frameworks cortex thing. And then at the bottom you have the shell hook, which is It's basically a bash shell script which you can execute on entering into the next shell.

So one problem with this setup, which works flawlessly out of the box, which is really handy for us Um one problem we did have early on is it's a bit of a faff having to constantly remember to be inside the Nix shell when you enter a directory. So luckily um DRenv DRM. net has a native integration that makes this really easy and solves two problems in one go. One, you get the automatic Nix shell instantiation on entering a directory. And two, you get to use whatever shell you want with your own prompt rather than using Nix as uh bash subshell. And so what does it look like for us? We enter the directory

It's a video classic. Oh it is running, cool. So yeah. So you enter the directory and boom you get our little ASCII art thing. You get we output the exact Python version you're on, the exact node version you're on. We also have a little uh uh shim to basically say if your Python dependencies are up to date with the poetry install, and same again for Yarn. Um and it's just been super helpful to us. Um Another issue we had is that we still needed to run the services via Docker. Remember they were fine, so we left them in Docker. So without some form of solution, we'd now be opening uh three terminal windows to do local development. The back end, the front end, and Docker Compose itself. We'd actually had this problem of having no uh shared task language amongst the team uh for a while.

So we decided to go looking for a modern makefile alternative and came across tasfile, which is written in Golang. Um it's fantastic. is um a YAML-based definition. More YAML, I know, but it's good. You can run tasks in parallel, tasks can have dependencies of other tasks. You can pass um uh arguments straight from the command line through down into the task below. It's just a really nice API for getting stuff done. And this is what it looks like for us when we just run task. Gives us a nice documentation of all um All the scripts you can run on a repo basically. And for instance, once you're inside the shell, you can simply run task up to bring the entire stack up. It's just one command. And obviously by entering the directory, Nix has set up all its stuff and installed everything for you.

So to bring up our stack, it can just be the case of checking out the repo. installing Nix, going into the directory and then running that command, which is super great. So the results, I'm getting close to the end. Huge speed ups across the board Uh way more than the fifty percent we were hoping for. Um run server reload down to three and uh three to five seconds. I was speaking with the Kolo uh peeps outside about getting this down further because I still think it's Quite slow to be honest for a large project. Needless to say, it was a massive success. But the larger success was really discovering Nix. And that brings me on to um What's next?

So we could go full Nix if we wanted to, and by full Nix I mean that we would also uh declare all of our Python dependencies and all of our JavaScript dependencies within Nix itself. There are projects to help you do this. One of them is called Poetry to Nix and that can take a poetry file um and basically convert it into a nix uh derivation. We tried that. We used some weird packages. Um and no There's a lot of little hacks that Poetry to Nix do because the Python ecosystem like for example if you store install uh I don't know LXML from source. There's nothing in the Python ecosystem that tells you it needs specific things of GCC and other thing compilers on your system.

So yeah, so it's it's one for us to look at. More interestingly for me, um we still deploy Tohoku the old school way with build packs, but we're looking to move away And that will inevitably mean uh deploying Docker containers. So Nix is amazing at creating Docker images. Um what better way to build them in a reproducible fashion? There's literally a function uh which a couple of lines and you'll have from any given deriv derivation you'll have your image and you'll be ready to go. And Railway, they're a Heroku competitor, have just launched nixbax. com So if you kind of want to skip a lot of the inner bits of Nix and want to just get something running in a bunch of given languages, go and have a look at that.

And last slide. So the future. What does the future hold? I think I'm gonna have to skip this because I'm run out of time. But if you want to talk to me later about why I think the future uh of local development is actually in the cloud, then do you catch me afterwards. Um Having said all this is a strange point to end on given I've just told you about Nix. The thing is Nix is going to be useful in the cloud anyway. A lot of how those services work, like Gitpod, uh Octeto and some other ones, they basically run Docker images. And so what are you going to want? You're going to want reproducible Docker images. And there you go Thank you.