Development with Ansible and VMs

I'm Jeff. Um, I'm actually the CTO. I don't want to take my boss's job. So small clarification there. Um but yeah, so I'm super excited to be here. Thank you so much for for listening to me for being here I want to tell you a little bit about development with Ansible and VMs. This is based largely off of what we do at my company, Choose. And so I want to kind of go through how we've set things up and why we think it's pretty great. So let's take a look at what we're going to learn today. We're going to start by trying to understand what is a better dev environment. What are we actually looking for? What's our wish list in terms of what makes a really great dev environment?

We're going to look at some environment tools. How do we create the right environment for our code to live and run in? Then we're going to talk about config tools and how we set up that environment to run the code that we wanted to, to have all the packages we needed to. And then we'll wrap up and kind of check in on what we learned today. So I hope you can learn something that you can put into practice tomorrow with this. We've found Ansible and developing in VMs to be really, really great for us. So I hope you agree. So let's start with what is a better dev environment? What are we really looking for on that in building that dev environment in running our code?

So what's our wish list, right? We start uh with an environment that's standardized, right? If we have a known baseline That means there are no surprises. So we want to start there. And we want it to be repeatable. So if the standard environment is repeatable, this gives us a lot of power We can spin up a new dev environment. If we do something crazy to it and we don't like what we've done, like we installed an extra package by hand and We don't want to deal with that. You can toss it away and recreate it from scratch because it's standardized, it's repeatable. You can build two of them and run them side by side. If someone steals your laptop, you can just get a new laptop and with a few commands

be up and running again with your full dev environment just as you left it. Those two things together can be really powerful, but there's more. We want it to be isolated. So this means we have complete control over what packages are running in that environment You know, let's say that you want uh Django 1. 5 for a slightly older project that you're running, and then you want to run Django 1. 7 RC. whatever we're on today, uh for a personal passion project in a different environment. With with these isolated environments, you can run one in one, the other in the other. Uh but more than that, we also want to be able to Isolate system packages.

So let's say that one of your projects needs Postgres 8 series, 8. 4, and the other one needs 9. 3. Uh so you're able to accomplish that as well if it's really truly fully isolated, and that's what we're looking for here. Uh we want our dev environment to be as like production as absolutely humanly possible. Anyone who's dealt with a production only bug knows the pain that that causes. The best way to avoid that is you make sure that your uh dev environment is as absolutely picture perfect close to your production environment as as possible. And I'm a little bit fanatical about that myself. So that means what do we need?

We need total control over how our code's running. That means We need to control what OS it's running in. We need to control what system packages are available, what users are set up on that box, everything. So that's that's on our wish list. But even though you're running in a production-like environment, you want to use your tools, right? If for you to be productive, maybe you're a Mac person, you use Sublime. Maybe you like to run uh Ubuntu or Debian and Vim or Emacs, whatever it is, whatever your tool set is, you should be able to use that so that you can be fully productive. while still running your code in the most production-like environment possible, right? So that's on our wish list.

And finally If we have all of this, we also want it to be shareable, right? So we have a new person join the team, they're going to be working on the code too. They should benefit from all of this as well. So we want to be able to just hand them something and go off to the races. So that's our wish list. How are we going to do it? So let's talk a little bit about environment tools. So these are the tools that are going to kind of set up that baseline environment. You've probably all heard of pip. Pip is for installing Python packages. I'm not really going to go into it because that's not really the point of the talk, but we're going to use it and you should all keep using it. It's pretty great. Pip leads us to virtualenv, which you've uh also probably heard of

if you've uh been in the Python world for a little while. Um VirtualEnv isolates Python versions and Python packages for those versions. So this is what can allow you to run Python 2. 7 in one environment and Python 3. 3 in a different environment. to run Django 1. 5 in one place and one point uh and 1. 7 in another place. Um and that is really helpful and that is really great, but uh I would argue it doesn't go far enough. And that's where these VMs really come into play. So let's talk about what a VM is. If you're not familiar with it, a VM stands for virtual machine. And basically, it's a little tiny computer running

inside the computer. So it's all, you know, it's virtualizing an entire computer running inside your computer. Um I have one running right now on my computer. Um and so what this does for you is In that computer inside the computer, you can isolate everything, right? You can set it up to be its own OS, right? So we run our production environment is uh Ubuntu like and so our VM when I'm developing is Ubuntu but if you can see the top of my MacBook you know I'm running my stuff on Mac So it doesn't have to match. You get to set it up, set up the VM with your own whatever OS you need, whatever system level packages you need, all of that kind of stuff. And this is how you isolate uh

the entire box. So like I said, you get your Python packages isolated, you get your system packages isolated, and get it as close to production as possible. Now uh if you've never used a VM you might be wondering how the hell does this work? So uh a little bit about that, um you know, why am I able to run this other box and then still kind of seamlessly use uh my Mac system uh to edit things. Uh it sets up really easy networking between them so that I can open up a web browser and open up whatever's running on the guest uh on the little machine inside the machine um and uh it mirrors folders So you're able to say

this is my project folder, I want it exactly mirrored between my machine and the machine within the machine. And any changes made in one place are reflected instantly in the other back and forth, which is how when you're using your editor on your OS , those changes happen literally simultaneously. on the little tiny uh VM and you know if you're running Django there, Django sees those changes, reloads, it all happens automatically. It's very seamless. It's it's really fun. So that's VMs. Let me talk you through a little bit kind of what our setup is. There are a number of different ways to do this. I'm going to walk you through what we do. We use two tools here. One's called VirtualBox and the other is Vagrant.

They're both free and open source and really wonderful. And allow you to virtualize pretty much anything. Um VirtualBox is the thing that actually runs the virtual computer, the virtual machine. Uh and Vegro provides some really nice command line interfaces. uh to working with VirtualBox. So I'll show you a little bit what that looks like. So at a super basic level, this is How you would set up your kind of baby's first VM machine. You would download those two packages I showed you and get them installed. You can type Vagrant in it. And here we're specifying a price precise 64 version of Ubuntu. Vagrant actually hosts a bunch of these kind of basic boxes.

If you have a disk image that you want to use yourself, you can do that too, of course, but they have a bunch of kind of standard ones and it makes it really easy to get up and running with them. You type Vagrant up and it installs the machine, gets it booted up, sets up all the networking, uh all the basic stuff. And you have a running computer. And then that last one, bigger in SSH, allows you to SSH into that box and you have full access to it right from there. So it's really that simple. It's three lines. But of course, that's just the kind of basic starting point. So let me walk you through a little bit how we've got our VM set up. And I'll walk you through this line by line in a second. You don't have to kind of stare at the whole thing here. But this really is, I mean I made a couple of changes for you know

TV magic. But basically this is all that we really have at Choose to To configure our virtual machine. And this file, it's called a Vagrant file. It's what you use to tell Vagrant, kind of your configuration for that box. And this I recommend that you generally want to check into version control so that you can have one standardized thing, you can version it, you can share it with your team. So Let me walk you through this. So we start with uh convig vmbox equals that one that you saw. in the previous step. This is how you would specify it in kind of a more canonical way and share it with your team. Points to the right box if you don't have this installed, like if some

if a teammate just pulled down the code uh and is using Vagrant, it will know where to go and get that box and it'll install it for you. So that makes that really easy. Then we set up some networking, a host name and then a private network with a static IP. uh 100100 in this case. Um and that means that you can access that virtual machine at that IP address. And Vabrant does all that setup for you. So all that kind of networking, headache stuff, all handled for you. This last bit down here is Ansible, so we'll get into this a little bit more. uh in a few minutes. But basically you tell it which provisioner to use. You can use any of a number of them. In this case we're using Ansible to set up the box.

And you point it to a playbook, which is kind of Ansible terms for uh kind of the the entry point for for your Ansible configuration. Um so like I said we'll talk more about this but this is how uh on In your Vagram file, you basically tell it how to set up this machine. It's really that simple. That really is pretty much all we have. So let's talk a little bit about the config tools. So this is where we talk about configuration management. What is configuration management? If you've never used any of these tools before, basically what you're doing is you're defining how the system gets set up from top to bottom. So Vagrant and the VM setup kind of gets you bootstrapped with that OS.

You're running a box now. And configuration management is that last piece where you're saying, okay, from that OS, What do we need set up? We need Postgres. We need this version of Python. We need these users on the box. We need the, you know, all of that kind of stuff. This is the tool that you're going to use to do it. This is a big key for me. You write this all in code or GAML or some sort of a language like that. This is really important because it means it's documented and it's versioned. How to set up a new box to run a dev environment and everyone's running slightly different stuff, you have no guarantee that a bug is reproducible.

You have no guarantee that your teammate can help you out. you have no guarantee that even you could reproduce it uh under different circumstances. Uh but if everything's documented, everyone knows exactly what's going on. It's versions, so as you make changes to the code, you have new dependencies, those are all captured there in the history. It's a really beautiful thing. And like I said, it's also shareable, so now your teammates can all get in on this. You can share it really easily. So Let's talk about all of these tools in this ecosystem. To be clear, these are all different and sort of competing configuration management options. Chef and Puppet have been around a while. They're both written in Ruby.

Ansible and SaltStack are both written in Python. Uh Ansible's kind of the new kid on the block. Uh and bash just please don't. Uh it's uh it's better than nothing, I guess, but uh there are way better options and it's not that hard to get started on them. Bash scripting is gonna be a beast to maintain. Um so I can't recommend that. I'm not gonna go in uh to all of the reasons why we're using Ansible right now. Um That's kind of outside the scope of this talk will be my cop out for that. Um but if you want to talk about that after the talk or whatever, you can come find me. Um what I will say is Ansible has been Very good to us. It's written in Python, so if you ever need to dig into the internals for something, that's very easy to do.

I've just I've I've tried all of them and Ansible is is my choice. So we'll we'll leave it at that for now. But you all kind of came knowing that you were going to hear about Ansible. So let's get in and and go through kind of a basic example of what an Ansible uh setup would look like. Um so don't worry about reading this, you're not supposed to be able to yet. Um but this is uh the entirety of uh our Python setup using Ansible. Um so we've split our system into multiple different things, right? We have one of these files for Postgres, we have one for uh Redis, we have one for Supervisor, you know, all of these different kind of baseline things that we want set up

We have a different file for Enhansible and you kind of organize things that way. I want to walk you through our Python one, both because this is kind of a Python-y sort of place. uh and because it's uh I think a really great example of how uh kind of simple and straightforward this can be uh but also very powerful. Um so let's start looking at this. At a at a baseline, all Ansible is is a series of steps that you want it to do. That really like There's a lot of complication built around it, but really, at the end of the day, Ansible's just a series of steps that you've put into a YAML file And these names, these are just kind of human-readable definitions of what you're doing step by step. So these were all in that file that you you saw a moment ago.

I'm going to go through these one by one. But at a high level, what are we trying to do? We're going to install some basic system level packages. We're going to get pip and virtual env kind of bootstrapped on the system, get those up to the right versions. We're going to install everything in our requirements. txt file. And then for a little touch of luxury, we're going to add a couple of lines to our bash RC file that when we SSH into the box using Vagrant. that these will uh activate the virtual env and CD into the project directory so that we're all kind of ready to go in the right place uh every time that we SSH into the box. So let's look at these one by one.

So for installing the system packages, let's take a look at this. It starts with this line, APT package equals crazy curly stuff, and state installed. So Ansible, the way it works is it has all of these kind of uh understandings of what different system uh system level options are. So in this case it knows what APT is, it's a package manager, and it knows how to interact with it. So you have this really nice shorthand for saying, here 's the package I want and I want it installed. Uh just go do that, Ansible. Um and it does that in a way that is uh idempotent. Uh if you've Hung out with configuration management stuff a while. You've probably heard the word item potent is just a fancy word for saying

it'll do the thing once, and then if it tries to do it again but knows that it's already been done, it won't do the work twice. Right. Um so as with this, right, uh it'll make sure that the packages are installed, but if you go through and run the configuration again, it won't reinstall them or anything silly like that. APT already does that of course, but Lots of the other uh Ansible tools will handle that item potency for you. Um so what are these double curly braces? This is a really interesting thing and really powerful tool uh with Ansible if you look at this next line uh with items. So basically what you're doing is you're giving it a list. In this case Python, Python dev, Python pip, libxml, lib uh XLST. And you're saying I want to do that action, that APT thing, with each of these items.

And what it's doing is it's actually rendering that line as a little mini temp template, the APT line. And it looks a lot like Django's templating language because it acts a lot like Django's templating language. Technically it's actually using Jinja under the hood. But if you know Django's template language, you'll feel right at home. So basically it's taking each of those items one by one, plopping them in, and doing the APT install for each. And then we pseudo the command because that's what you gotta do. So that is how we're installing some standard system level packages with Ansible. Pretty straightforward. Step two, we're going to bootstrap pip and virtual env. As you can see, in the same way that Ansible understands what APT is, Ansible

also built in understands what PIP is and how to interact with PIP. You'll notice we installed the pip package in our last uh step. So there's already uh kind of the system, uh the system package or APT installed its version of pip. But we also, you know, pip can update itself and we want to be working with the latest version of pip. And we want uh uh a version of virtual env installed uh and we can use pip to do both of those things to kind of get a baseline setup on our system. Uh again, so here we've got two options. We're saying name equals curly braces again And we're saying version equals curly braces again. So this is using width items but in a slightly more advanced format. So here we're defining

dictionaries instead of just a straight list. So here we have a list of dictionaries. There's tons of power under the hood in terms of how with items works. There are lots of crazy things you can do This obviously makes it pretty clean to encapsulate pegging a certain package and a certain version and having it handle all of that. So once you've done this, we know that we have uh the version of pip that we expect to have, the version of virtual end that we expect to have. And we're on to the next step, which is to install our requirements. txt. So you can see we're using pip again, but this time in a slightly different way The same way that you know you can install with a requirements file or you can install a straight-up package using pip, Ansible understands those two functions as well.

So in this case, if we use pip, and specify a requirements file and a virtual end that we want to use. It knows how to do that. It does all the heavy lifting for us. Depending on how many packages you had off, obviously this step might take a little while. Um but it knows how to do that, it handles everything for you. So that step's pretty straightforward. Um The last step, as I said, a little bit of luxury, a little nice to have. We're going to add a couple of lines to our bash RC file. And we're going to use this new tool. Ansible also has a lot of different ways of working with files on the system, as you might might expect. This one's called line in file, which does pretty much what it sounds like. It makes sure that this line is in this file.

So we're going to give it a destination and say we're talking about this bash RC file. And once again we're using this curly brace item thing and we're going to tell it make sure that this these lines are in the file. And the items that we're doing, we're going to activate the virtual environment like I said and we're going to change directories into our project directory. So every time we Vagr an SSH into this box, these things happen automatically from the bash RC file. And like I said, the item potency applies here again. Line and file knows how to search through that file, make sure that it's not duplicating those lines. So it's not adding those every time you configure. It makes sure they're there. If they're not, it adds them. Pretty straightforward. So that's really all that is.

And I I hope you agree. I think that you know That is really simple, really readable, and also pretty powerful, right? To have that now documented in code, to have that shareable with the rest of the team. I think is a really, really powerful thing. And again, so we do this with Postgres, we do this with Redis, we do this with Solar, Supervisor, and a bunch of other things to get our whole system up uh up and running. So let's talk uh let's let's go back and see what we've learned. I want to first show you uh what kind of how magical I feel like this is and especially you know Onboarding a new developer into your team, working together with them, it's really important, at least to me, to make sure that they can get up and running fast.

They can feel ownership of the code on day one. And this makes that really possible So let's walk through just a day in the life, the first day of someone joining our team. They install VirtualBox Invagrant, that takes a few minutes, but no big deal. You get clone our repository, no big deal, and you type Vagrant up. And that's glossed over a little bit, sure, but That is really, really, really close to all you have to do. And you do wait a while on this step. That step, the vagrant up, it's doing a lot of stuff, and it does take a while. But it really is about as simple as that. And you're up and running with a totally working version of Choose. It does everything you want.

Like it literally is running. You can open it up in a browser. You can make changes to the code and see it immediately represented in your browser full working environment. It's pretty great. So let's go back to our dev uh wish list and see if we've hit all of our points. We want it to be standardized. So we've we've standardized our setup with code. We've written it all down in code, so it's all standardized across all of our team. It's repeatable. You just type Vagrant up and you've got a new box. It's isolated at the OS level, at the system package level, at the Python level, at every level. It's a totally isolated environment that we have complete control of It is as close to production as we can possibly make it, but it still lets you use your tools as you're working, your editor, your OS.

And you can share it with your team and get them up to speed in almost no time. Um so I think this is pretty good Uh I hope you agree. Uh again, I'm Jeff. Uh I really appreciate you taking the time to listen to all this. Uh I have to pimp choose real quick because they paid for me to be here. Uh this is this is my company. Uh we do lunches, so if you have lunches at your uh at your office and you want them to be better, we can help you do that, get in touch. Um but yeah, that's it. Thank you so much