Bootstrapping your Local Python Environment

Hello everyone, my name is Calvin Hendrix Parker and I am excited to be here for DjangoCon 2021 even if we can't be together. Should be exciting to all be joining and gathering online for this great event. And today I'll be talking about bootstrapping your local Python environment and why this is so dang important. Uh let me get kicked off with We probably should set an intention. Like if with any good you know act in life, we really should make sure you are on the right path to peace and harmony. And what better way than the Zen of Python? So if you're not familiar, there's an Easter egg in Python, if you type import uh this or Python minus M this. you'll get the Xenopython. And these are some great tenants to keep in mind as you are a programmer

or living your everyday life. Uh just great things to follow. A lot of times in my when I'm teaching new new people how to program in Python, I'll tell them to print these out and put them in a for nice frame and put them in their their bathroom while they're brushing their teeth in the evenings they can read through them and and really bask in that Xenopython. And a couple of these tenets of the Xenopython will become really clear and important to us as we move along in setting up our environment locally. I guess what is the the real issue here? Uh I think a lot of us can relate to this comic that came out. There's probably there's probably an XKCD. for just about any situation in life and especially this one is uh 100% appropriate. This is uh Randall Monroe posted X

this X XKCD in 2018 and admitted he had done some bad things to get to this point. But you can see you've you know there's lots of different you know OS installs, the Anaconda Python, some kind of pip, you've got a virtual environment sitting around over here Uh if you're on a Mac, you've got the the system OS version and frameworks that could be also installed. Python Environmental Protection Agency wants to seal it in a cement chamber is the uh comment on this specific com um Python uh XKCD cartoon. And I want to make sure we don't end up here, because this can be really frustrating, especially as a newcomer to Python and even as a seasoned veteran, when you're developing in Python, you want to make sure you are using the right Python for the job.

And Python can live in so many places. There are Pythons provided by the operating system, different ones for Mac OS, Red Hat, and Ubuntu. Those are You know, don't touch those Pythons. They are meant for the operating systems to use and not for you to use. You could install Python from an app store. Example the Microsoft store has Python in there. You can just literally go search for Python and install it You could download the Python installer from python. org. You could be getting it from a package manager such as apt, brew, chocolatey. If you're not familiar with uh brew and chocolatey or apt, those are all great package managers. Brew is for Mac OS and Linux, and Chocolatey is if you're on Windows. You could also be installing from a Python distribution. such as the ActiveState Python distribution or the Anaconda distribution.

And we'll kind of compare and contrast some of these options. I'm going to give you what I think is my recommended path forward for achieving that Zen of Python. Because why should you care about all of this? I know this sounds so maybe uh mundane, but this can be a huge roadblock to really being productive uh with Python or any programming language on your own system. So we want to make sure we stay zen. And these are three of those tenants I was mentioning earlier and that we would want to make sure we're following The beautiful is better than ugly, explicit is better than implicit, and simple is better than complex. I think in life as we're solving all kinds of problems, we want to try and strive for those simplest versions of those problems first Something that is reproducible, something that is beautiful and elegant and easy for us to follow every time we go to do development on our local machine.

But we're gonna start off with some rules. Uh let's kick this off with my number one rule, no pseudo. If you have to use root privilege privileges to do anything on your local development system or even on a remote server someplace, there's probably something wrong here. Like if it feels bad, you probably it probably is bad. And sudo is one of those number one rules where I don't think you should be using sudo because I don't care and no whining, you don't need root to use Python on these systems. You especially don't need root when you are installing Python packages into a Python install in your system. I think the main reason why systems like Ubuntu don't ship with pip out of the box for installing packages.

is because if you're using sudo to call pip and installing it into the global python interpreter, you've got bigger problems ahead and we don't want to end up in that super fund site situation. The second big rule here is I I think just really never use the system Python. That system Python is there for a reason, and that reason is for the system. Uh you are not the system. You are a developer who are building software who will be deployed to other systems, and those other systems may not have the same Python. Those other systems may not have the same dependencies installed into their Python, and you can't depend on that system Python including all the bits and pieces that you would normally expect to have. For example, I just mentioned that the Ubuntu system Python doesn't include pip out of the box.

The FreeBSD Python that installs even from packages doesn't include uh SQLite or a couple other kind of core out-of-the-box batteries included pieces, they expect you to install those uh separately. But it calls causes problems down the line, so don't use the system Python. for anything. I know what you're saying now. Okay, smart guy. What am I going to use for Python on my local system to be a developer? Because I wanna actually code Python. I don't care about being a system administrator. I want to be productive. And I am going to suggest that everyone takes a great look at pipinv, or sorry, not pipinv, pynv. And I'll talk about pipinv later. But pyenv is a great tool for managing your Python environments. This allows you to change uh and install multiple versions of Python.

So maybe you've got multiple uh versions required for different projects you're working on. Maybe you need 3. 7 for one project, 3. 8 for another, and 3. 9 for yet a third. How do you do that with a system Python that is gonna upgrade off underneath you as the system upgrades itself? You want to make sure you've got a stable and controlled environment where you are in charge of what Python you're using. You could also change the global Python that you use when you type Python at the command line. Pit PyIv allows you to set which Python gets executed very explicitly. So again that explicit is better than implicit and other than a Python we're using here. You can also override Python versions with environment variables And search for commands across multiple Python versions at a time. That comes in handy if you're using

doing testing using talks when you want to have multiple versions available. it's really handy to be able to have multiple versions installed on your system so you can actually run your unit tests across multiple systems. One caveat here though is that PyMv out of the box doesn't work on Windows If you're going to want to use it at Windows, you're going to want to check out this pyMv-win repository that I'm linking to here in this slide. And these slides will be available for download on my GitHub, so you can check those out. Alright, enough talk, more action. Let's actually see PyEnv in action. So I think it'll speak for itself as to why I love this tool. So we're gonna do a quick little uh demo. I've got py env installed and I installed that from homebrew, but you'll be able to uh do it yourself And I have a few versions of Python installed.

So you can see I've got uh nine396, 394, uh 391, all those different versions available to me to use uh for Python development. So we can use Pyamp to install another version of Python that we don't have. So we saw we have versions 396 installed, but we don't have 397. So let's actually install 397. 399. 7. It goes to python. org, downloads the 397 tarball, and then proceeds to install the standard options for Python right out of the box. So what we're going to do is while that's installing, I'm going to continue with a quick demo. If I do a pip freeze This is using my global Python interpreter.

So if I actually look and see what that is, if I do Python, you'll see it's Python 396. That is coming from PyEnv. If I look at PyEnv global. It'll tell me that Python 396 is my globally installed interpreter that I use anywhere on the system. If I want to use a specific interpreter for specific projects, so let's do that. Let's uh we're gonna make a project one. Directory and we'll make a pro well actually we'll make a project two directory as well. And we'll go into project one. And this project I want to use Python 3. 8. So we will, and we'll and I'm actually going to use a plugin here that's really handy for creating virtual environments. So we're going to do Pyimv.

Local, so we want to specify a specific local one. 3. 8. 6. So now I'm using Python 386. You can actually see my prompt is updated to show Python 386 is uh being in use. And now if I do py env and I do a make create a virtual env , We'll just call this one project one inv. Oh, I guess specify the version first. 3. 8. 6 This is going to create a virtual environment which is basically a little sandbox area for me to install whatever I want into. Now I'm going to use that here for my local. Instead of this 386, we'll use project one inv And you'll also notice my prompt is now showing me which virtual environment is activated.

If I do pip freeze again. it'll be empty. But if I do pip install requests, that should go off and install requests into this specific specific virtual environment, our little sandbox. We'll do a pip freeze again. So you go you see there's request 2. 26. 0 is installed. Let's go over to project 2. Now you notice my prompt changed. The version of Python went away and the virtual environment went away. Let's create a PyM virtual environment for this project. We're going to use 3. 396 Actually, I wonder if the oh 397 is available. Let's go ahead and use 397 now since we installed it and made it available

for project two inf And then we'll tell our project to use this as our local interpreter for this specific project. So pyM local proje2. Bingo. So if I do pip freeze, we shouldn't see the package requests because we haven't installed it here. Now we're using Python 3. 9. And maybe we need to use this very specific version of requests. Maybe we want version 2. 25, because maybe something in 2. 26 has broken our our packages somehow. So we'll do install and we'll do requests. Oops. Yeah. Pip install requests less than

two dot twenty-six. And now you'll see if I do pip freeze We have different packages altogether. If I cd back over into project one and do the same command You can now see that uh something changed between 2. 25 and 2. 26 is in 2. 25 we're using Chard A for some kind of character normalizer. We changed over to cart charset that normalizer. So there could be some incompatibilities with your current project. This is a great way to isolate those two environments, and I have not used the sudo command once in all of this whole process.

But I'm able to actually maintain two separate sandboxes, two different versions of Python, two different sets of installed packages. And that is a quick overview of Seeing what's uh Pyam can do for us. Now if we look at versions, you sh whoops, you'll see we did get 397 installed. You'll also see our virtual environments in here. You could create virtual environments that you're sharing across multiple projects because maybe they share the same dependencies, but generally they're pretty easy and lightweight that you can create a virtual inf for any project you would be working on. for every project I would be working on. I would create a virtual environment to sandbox off from the rest of my system. And that way I'm isolated from different changes. This helps make sure we have a r a very repeatable process for

using uh Python on our system. Let's go back and continue on with the presentation. So I did show you one of the plugins. If you wanted to install those plugins yourself, you do need to add them separate from pyim. But if you do a brew search pyinv, you'll see the other plugins that are available. So I'll do that here, Brew Search PyInv. And you'll see I've got the virtual env and virtual env wrapper. plugins installed. We just use the virtual virtual env one. VirtualInv wrapper will put your virtual environments into a different place and make it so you can switch between them easily. Here we're switching between them just by moving into different directories.

So if I go, you'll see Proj2. Oops, Proj 2. Right now I'm in Proj 1, so I'm in Project 1 directory. The Proj 1 environment 's there. If I move over in Proj 2, you'll see my virtual environment and Python have all changed. I didn't have to activate or deactivate, which is kind of the old school way. of moving between the virtual environments and I'll show an example of that so you can kind of understand how this all works uh a little clearer kind of the backstory here. Alright, we're back in. So virtual limb and virtual wrapper, again, I just showed those off. I won't go over that again. Now virtual environments are included out of the box. If you have simple needs, this is for you. And it's even common if I don't want to go through the whole process of installing PyInv onto a system

because I just need to have a specific tool available. I may use the VMv package included inside of Python, which is like a lighter weight version of VirtualMv. It's not the same package as VirtualInv, it's a different one that's included in the standard library. to create a virtual environment for me to use. So if I go back here and if I go into confdemo, you'll see I've got it's a very empty it's an empty directory. Uh it's using Python 396. If I type Python minus mvinv and create a vmv folder, that will create a folder right here called vmv that contains a Python 3. 9. 6 clean sandbox for me. You can see it right there. You can activate it just like you would a a virtual inf uh in the old school

virtual package. by doing sourcing the activate. Uh the you can see my prompt shows now I'm using the VINv s environment. And same options apply here. If I pip freeze there's nothing in there. If I pip install Like PyYAML Oops without two M's. Good thing no one was name squatting on that one. And if I do pip freeze. I only have PyAML installed into that one virtual environment. So that's kind of the traditional or batteries-included virtual environment, which could actually work out well for you if you do have those kinds of simple needs. Now this gets more exciting because we want to talk about I got tools I want to use. Maybe you want to use Black or you've got uh HTTP

or quick command line tools that are written in Python. Heck even Docker Compose uh is this way. If you install Pipex from Brew or whatever your package manager of choice may be and run pipex insure or insure path, it'll actually get you set up with Pipex, which is a kind of a combination of a couple tools. It's basically like uh virtual env and it does some manipulation of your path to make sure that any tool you install into these virtual ms, they're separate virtual ms themselves, but they all get added into your path correctly. So well it adds a simlink into your path So that you can actually access these commands without having to install them into your global interpreter. So this is probably the best way to make sure you don't install random stuff into your global Python interpreter. So if we did some damage here, I was doing a quick little install of HTTPy.

They will actually make these commands, HTTP and HTTPS, globally available to you. Let me show a quick demo of that as well, because I think it's uh quite impressive. I've got Pipex installed. If I do pipex list, it'll show you all the packages I've installed via pipex. So I've got black, which I use with PyCharm. I've got iSort using the PyCharm. If I wanted to install BPyTop, which if you've not checked out that tool, uh oops, I'll make sure I install it and not just try and call it. That will go and basically create a virtual environment, installs uh bpy top into it, and now if I run bpy top without having to activate any virtual environment or change anything about my path, I will get the BPyP package

running right here nicely alongside other things. If I wanted to use Markdown, let's do that real quick. PipEx install markdown. This will create a markdown and gives me a nice markdown under Py command that I can use. Now, Markdown has optional requirements to use pigments, and so I can actually inject Pigments into that specific virtual environment and have a separate install of pigments that is not related to the markdown environment. Oops, that's the other way around. The joy of live demoing.

So we'll take and inject pigments into Markdown. And now we can use the uh pigments highlight plugins for markdown as part of this package. So if I do Pipex list. We should see markdown. Yep, right here. If you wanted to have the Pigments script available, there's an additional command line flag you would add to it, and that would actually give you both those commands present on your system at once. So super easy and nice and simple way to keep that all uh up in in check basically. And not and we've never used sudo. We've not used our system Python. We've been using the PyM Pythons and VirtualMs, and now we use Pipex in the same way to get us some nice little sandbox tools that we can call anytime we want.

So you may be saying to yourself, well, I know there's this user scheme. If I wanted to install some package into my uh local home directory without using a root, that would get around the no sudo rule, but still allow me to use the system uh Python package. Can be done a little dangerous. It can run into the same problems as using the global Python. And also there's no in uninstall of a package from the user scheme. This basically puts all of your like it puts a Python environment underneath uh. local in your home directory or I can't remember where it is on Windows. But it's just much easier to use the the virtual environments to manage these because you can't have multiple user schemes. You can only have one user scheme. So if you have two projects that require two different versions of requests or numpy or some other package.

this user scheme will still fall down uh on you. Alright, what about Anaconda Python and Conda or ActiveState Python and PipIv and AD ASDF and Poetry and PDM and Pyproject. tal? Uh bugesh, there's lots of options and Python packaging and Python environments are definitely in a kind of a weird state because there are lots of options and it's hard as a new person to understand what to choose. I think Anaconda is is a good option if you are going to stick to just data science or the things that are provided by the conda package, but it adds an extra layer of complexity when you go to work with the rest of the world who may not be using the conda package manager Same thing kind of holds true for Active State Python. It's really going to be its own little ecosystem of

how you install packages and how you manage your dependencies. You're kind of dependent on their systems for having the right sets of packages installed or uh compiled um C extensions. Pipim we were really actually I thought it was quite promising. Um it ran into some issues and kinda fell out of repair. I believe it's it's back maintained again. It's underneath the um Python Packaging Authority under the Python Software Foundation. We'll see what happens there. It's a little slow compared to my preference for package like dependency management. The thing Pipim brought to the table was a lock file on hashes so you could really lock down your dependencies. But its dependency calculation mechanism is a little slow. ASDF is a multi-language

version of PyM. So if you want to manage your node, your Ruby, your Python, your Go. There are plugins for all those. I put a link here to the Python one specifically, so if you want to check that out My only downside that one is it's not it doesn't have the nice plugins like PyInv does for virtualinv, so managing those virtual environments and switching between them automatically doesn't just happen out of the box. Poetry is a other another framework for managing project dependencies for your libraries. Another issue I find with that one is you're going to be learning a whole nother set of systems and tools. to really operate inside the poetry world. If you're in there and you like it, it looks like really promising, but it just seemed like another bit of overhead when I just want to get work done. PDM's another one. And then Pyproject.

toml uh is also what's kind of coming down the line as far as a specification for how to um note what your dependencies are and and how to pin them to specific versions, keep an eye on that. There'll be more tools that are going to be I guess standardizing around the Py Project Toml specification, which is a good thing, but the tools aren't all quite there yet. I know that there's a build tool underneath the Python packaging authority. That it's in flux and we've had to basically roll back a couple times to make sure we can keep our projects moving forward, but that's the future. Uh we're just not in that future yet. But I want repeatability and simplicity. So how do I pin my versions for my projects and ensure that I get repeatability?

With a requirements file, let me show all that real quick Uh we are in uh let's go back to the conf demo here. Yes So in this virtual environment we've installed PyAML541, but how do I make sure I get 541 the next time I go to install this? So what a lot of people recommend would be saving this off into a requirements files. Like that. And now if I were to remove the virtual environment And I actually need to deactivate the virtual environment so it goes away. My pip freeze will not show PyAML anymore.

So I can recreate that. virtual environment and then I can take the contents of that requirements file and play it back on my virtual environment and have it ready to go. So if I do pip install mySr it goes and grabs the very specific version of PyYAML that's in the PyPy the Python package index The issue here is that you've got many different versions of that package out there that have different hashes. You can see this one is the Mini Linux wheel that got downloaded from PyPy. You could be installing a uh a a Mac

OS wheel, a Windows wheel, and you're going to want to ensure that no one is tampered with that and actually enable some a level of additional security in your own project. to say that I only accept packages that match specific hashes. So the the developers uploaded a package to PyPy, they've included a hash. I will at some point when I use that package grab that hash so that at some point in the future I want to reinstall it at the same exact version. If those hashes don't match, something funny is going on, and I can be assured now, or if they if they do match, I can be assured I've got the exact same package I installed. six months ago. And so that is where a tool like Pip Tools comes into play. We really like this. So PipTools has a compile functionality that allows us to compile

uh a hashes file. Actually I can probably show you how that works. Pip install piptools. So we'll use pip tools to get our hashes. So pip tools, but we don't have it activated yet. Let's uh activate this virtual environment. Well actually you know you don't have to activate virtual environments if you're using the old school virtual Ms. You can just call something in that path. Oh, I guess I don't have it in here. We'll activate and then we'll run Python minus M piptools and you'll see we've got the uh Oh of course I installed the wrong place. I didn't have a virtual virtual environment activated.

Now I do. Let's do this again. And I'll fix my other environment, which was in my home directory and not the root one. So I did not make a dire mistake, but it is not the most optimal thing. Here we go. So pip tools compile. So we can actually tell piptools compile to look at our requirements file and generate a set of hashes and commands. So if if we did Let's see here, compile Bingo. So now we can put this into our requirements file, and when we play it back, it's going to double check our package name against all one of the our packages should match one of these hashes that's here.

and if it doesn't it won't install it and it'll throw an error. So again that kind of extra safety net that PipTools gives us now. Let's go even a step further. I really want repeatable and I want it to be fully isolated and not even running technically, sometimes even on my own system. This is where we use Docker. We bring in Docker and use pip tools to compile so that every time we build an image, we get that exact same image. uh day in day out. And st I mean what's also nice with using Docker images is I can put them into a a Docker container repository And then pull it back out later and use that same image without having to go through the whole build process. Now I can give a quick demo here. I think we have enough time to do that.

I have got a Docker demo set up over here and I have my uh Docker set up or opened up inside of PyCharm because I want to show you actually how you can use a Docker container image As your Python interpreter for PyCharm. So if we look at this, we've got a simple Python file. It's actually the stock Python file that PyCharm puts in when you create a new Python project. If we click over here on our Docker file, I've even gone so far as to make sure I'm using a multi-stage build, which is a great way to make sure you don't have a whole bunch of cruft built up in your final image when you deploy this into production. And so we've got the Python 3. 9 uh slim bullseye uh image that we are gonna base our images off of

And we're going to add in an extra extra option here called develop so if we're low developing locally we're going to want certain tools installed whereas if we're deploying to production and maybe our CI tools are building the actual image that's going to go to production uh will have developed this default to know. I still install a virtual environment even inside of this container. The default containers from the Python Software Foundation are gonna have a user local bin with a Python in there that matches 3. 9. But I want to have a separate environment, a Python sandbox, that I can move over to my final image that I will build afterwards. So in this case, I'll just make a virtual environment inside of app, set my path, and so from here on out all the commands that run are going to be using the virtual environment's

pip install and not the uh system pip install. So even on a container I'm not using the system Python. We'll copy in our requirements. So I've got a requirements file over here that has actually nothing in it right now. So base. txt I'm going to show you how we'll compile that. We will, if it's development, we're going to install pip tools into our system so we can do the compile. Otherwise, we don't need pip tools to be present in our final output for production And then we'll install our base. txt file, not the IN file in this case. So now that our build is done, we're going to basically make our final image that we would use to deploy to production or use as we're developing. So again, we'll use the 39 symbolseye, set the path, we set our source uh directory, so we'll copy everything from our

project into source. We'll copy our from build, the virtual environment we created, and then copy our main. py inside so that we can actually run it using this version of Python. So how do we compile our pip tools to in from base. in, so we're going to install requests into this specific environment. So to make it easier, I've set up a make file. We use make for just a general task running. This is a not a complete example. I just kind of quickly crib this in here so you could get an idea of how this works. But we'd make a build and we use that devel equal yes so we get pip tools and then we can use pip tools now to compile uh inside the container which is important if you're using a Mac or Windows

and you each compiled um some tools some tools for example like numpy are gonna bring in different eggs and different um dependencies based on the platform. But if you compile inside the container, you're always going to get the same dependencies for whatever the target environment, whether you're using a Linux container or Windows container. This would be my same advice is to make sure that you run that compile and all the dependency bits inside of the container and not outside of the container. So let's run that so you can actually see how this works. So we'll do make build, which should already be built, uh, should be cached. Yep, that would actually work pretty quickly. Now if we do make compile. This will run the compile pip tools compile process for us, generate the hashes, which there they are.

And now if we go back over here to our base. txt, you can see here's our hashes. So that being said, each time in this project as I add new dependencies, I'll add the base dependency here, not all of its leaf dependencies, and I'll rely on Piptools compile to track all of the leaf dependencies that kind of come along with requests. I don't want to be tracking all these individually because when it comes time to upgrade them, I have to go one by one, whereas I know I need just a specific version of requests. I don't need a specific version of IDNA. I'll let requests dependencies to determine which one I get. Now we've have built this but we don't have it available in our image yet. The image we built doesn't have the that installed so we'll do make build again

And since our uh pip tools, our text, our base. txt has changed, it should have actually installed those packages for us, which it did quickly because it was cached. And now if we wanted to run our image, we can. Oh, so we'll run Uh having it roove the container after it runs, here's the image we're gonna call up, which is the one we just built, and we'll run our python main. py And there are code executed inside the container in a fully isolated environment, in a inside of a virtual environment that has been fully pinned and hashed. We can actually run uh just the Python interpreter. Uh we'll make sure we start up an interactive. console here

and import requests. This should work. Yep, it does. And so if we do requests. get HTTPS colon slash slash Bingo. We're now using requests inside the container and not on our system. So we we could even go so far as to not even have Python installed on our system. Now if we want to use requests in our IDE and have the IDE be aware of the fact that we've got requests, because if for example if I did this, uh requests. git It doesn't know about requests. And actually if I even tried to install it, uh I believe well it must be using the Python 3.

9 that's already got it here. That wouldn't work normally. If I change my interpreter, we're going to add an interpreter. We're going to use a Docker remote image interpreter. It picks up the fact we have a Docker file inside of our image. knows which uh tag even to use. So it gets the remote now we're going to be operating off of the remote interpreter. So when I actually click play to run our main That runs inside the Docker and not uh outside in the Python from our system. Which is I think is pretty darn handy and actually kind of demonstrates a lot of best practices here as far as like using Docker with multi-stage builds, keeping everything isolated. uh being able to control your dependencies uh

more closely and making sure that you have a a kind of secure um supply chain when it comes to the eggs you're downloading from remote repositories. So that is the really repeatable uh best practice way. And if what's nice with the Docker and Pip Tools option is A developer can now onboard have a clean laptop. They could install whatever IDE they want, VS Code, PyCharm uh clone the repository, run make build, run make compile, and start developing on that code without having to worry about getting all the th the world of things installed onto their own system. So that I think the ramp up time for new developers should be tremendously faster if you enter this world of Docker and PIP tools for managing your Python environments

That is what I have to say about Python bootstrapping. I am so excited again to be here. Again, if you want any questions Feel free to find me on email, um calvin at sixfeetup. com. I'm on Twitter at CalvinHP and pretty much every other social service under the sun. If you want to follow our blog where we actually talk about a lot more of these types of topics, you can find that at sixfeetup. com slash blog. And again, I'm excited I'll be around during the conference. Feel free to ask me questions. I would love to discuss and debate if there are other new techniques that people are using for this. Thanks a lot.