Choose Your Own Django Deployment Adventure

Speaker 1: Hey everyone, thank you all for coming. My name is Matt Mackay, I'm a developer evangelist with Twilio, and I am joined on stage by the amazing Kate, who I will let introduce herself.

Speaker 2: Hi, I'm Kate Heddleston. I'm a software engineer with a company called Run Scope, which this is our sweet t-shirt. Everything's going to be 200 okay.

Speaker 1: And this morning we're going to present to you Choose Your Own Django Deployment Adventure. And we were really excited that Brandon Rhodes kind of almost plugged us this morning. with some choose your own adventure slides because that is essentially what today is going to be. And we're going to ask you to do the exact opposite of what most speakers want you to do. We want you to pull out your phones. Everyone in the audience should get your phone out because we're going to need audience participation for this. So there are going to be many points, and we'll we'll give you the phone number in just a second, where you're going to text in your vote. for which path we're gonna go down as far as this choose your own adventure story.

Speaker 2: So yeah, so you'll be texting in. We'll put the number up on the screen. It's gonna be the same number throughout the talk, so that'll be easier We are doing a choose your own adventure story, and this is a bit of an experiment. We're also relying on technology, which is a very dangerous thing, as all of us in this room know. But basically you are going to be the protagonist in our story. You're going to be the hero or heroine. Of our young startup. So you're a software engineer, you're living in the hotbed of technology, Silicon Valley. You 're young, you're hip, you're fashionable, you have your tech hoodie, your backpack, or your tin bug two bag. Maybe you shop at Everlane even. And you have an idea. It's the single greatest idea since Pied Piper, the completely fictional compression algorithm from the show Silicon Valley, which is actually a very funny show.

Speaker 2: You're really excited about your new idea and the repercussions of your new web application could change the world. So, did you build Tinder for pets, the hot new app application casually connecting uh otherwise distant pets to each other, or did you build Melonhead? The application that determines the size of your head and which fruit is closest in size to your head. So what you're gonna do is you're gonna text either the word pets or the word melon to the number on the screen, and you'll start to see votes show up.

Speaker 1: So this is a reveal. js presentation and we've used WebSockets. So after every single text that comes in, you'll see that on the screen And in about 20, 15, 20 seconds, we'll cut it at that point, or if it looks like one is winning over the other, and we'll go down that path. So get your votes in now. It's 503. 946 -3662.

Speaker 2: Oh my goodness.

Speaker 1: Wow. Got a head-to-head race here.

Speaker 2: It's a little terrifying for us because we don't actually know what the audience is going to pick. Oh

Speaker 1: melonhead.

Speaker 2: Do we have seventy people in the audience? Someone just like it just keeps going. Uh you Not even a little bit. Not even a little bit.

Speaker 1: It's a race. Alright?

Speaker 2: Do you think we user tested our application? Come on. To

Speaker 1: get a me.

Speaker 2: It looks like

Speaker 1: uh uh we have to cut it! Oh no!

Speaker 2: We have a lot of people in the audience. I think I'm just gonna have to pick one. Um Pets, let's do pets.

Speaker 1: Tinder for pets.

Speaker 2: Tinder for pets is the winner. The clear winner of the 200 participants in that vote. Awesome. So Tinder for Pets. Alright. Tinder for Pets is a web application that provides services to pets to meet for pets to meet other pets. Before Tinder for Pets, Discovery was ad hoc and sporadic. Pets had to rely on the happenstance of meeting other pets at parks, going out for walks, or in the case of house cats, not at all. This is unacceptable, as you have discovered Now your app is changing the peer-to-peer landscape for pets in new and innovative ways. Pets can now say yes, lick, or no, paw stamp to another pet profile

Speaker 2: If both say yes, they can then connect for a play date, if you know what I mean. And it turns out soccer is very popular for pet playdates. So you've built your Tinder for Pets MVP, you've raised a million-dollar seed round, obviously, based on your awesome idea, and you're ready to deploy this to the public.

Speaker 1: So you've built your minimum viable product, as it says in the lean startup, which you just read. And now you need to get your Django application out into the wild. You need to get it into the hands the excited hands or actually paws of all of your users everywhere. So this is your initial deployment. So you do some Google searches, how do I deploy my Django application? You watch Kates. PyCon talk of full stack Python deployments, and this is roughly what you come up with. You have, okay, this is the structure of the WISGI app, but it's there's a lot of pieces here. So You take this basic diagram and you say, okay, one step at a time, the first thing that I need in order to put my application out there is I need some sort of server.

Speaker 1: You could buy some components off the shelf and then put it into your home and then serve up your application, but it doesn't seem like it would be rock solid for uh production applications. So you decide I'm just gonna get a virtual private server from one of the many companies that provides them out there. But then on top of that server you need an operating system. And you've read that while you can do your development on Mac or Windows You're going to have to use Linux in order to do this production deployment for your Python application. So on top of the server, the virtual private server, put Linux. And then you realize, well, okay, Linux itself doesn't handle web requests, so you have you need a web server. So you decide between Apache and Nginx. Now, the missing piece here is you've written your application in Django.

Speaker 1: A web server by itself, Apache Nginx, has a no-concept of what Python code is. And that's where Whiskey comes in, the Web Server Gateway Interface. There's two sides to this. One is the application or the framework side. This is actually in PEP. 333 or 3333 that is specified. And what this specifies is that there's an application framework side. That would be Django and your application in this case. And there is a server side, which is what is at the container that is going to be running your application. So the web server you configure to pass certain requests onto your WISGI server. Now you need a place to put your Python code. So you read about virtual env and that that is some way to separate and isolate your environment from other applications that you may deploy in the future

Speaker 1: and you put your application source code, that is what's going to execute the application source code. And then you also have your whiskey framework, Django in this case, and your application dependencies, any other thing that you may rely upon for your for your deployment. There's a few other pieces here, and there's actually one critical piece that we're missing. We have all this data about all these pets and we need to store it somewhere So we've decided we need a database. And for this case, we're going to pick Postgres. Postgres happens to be the one that most of the Django core committers are using to actually test their own code. So you say, all right, I'm just going to stick with Postgres as opposed to MySQL or something like that. So you put that as a part of your architecture. And then you have some JavaScript files you're using jQuery. You may want to move on to an Angular. js or a backbone front backbone front end at some point. So you have some JavaScript and CSS

Speaker 1: files you need to deploy. Those are served up directly by the web server. They're not going, they're not going through your WISGI server. You have those things, and then you have one other piece, which is There are some things that you have to handle outside of the HTT request-response cycle. So for example, if a pet hasn't logged in in over a month, you've decided you're just going to get rid of that pet's profile. They've probably found a pet that they really like And they're never going to come onto your application again. So every night you run a tac you run a function in task queue that cleans up all the old pet accounts So, this is the architecture that you've come up with, but these are the general concepts. So, let's take a look at the specific components So we're going to, for example, use Linode as is a as a server, a virtual private server provider. You install Ubuntu on top of that. Use Nginx

Speaker 1: as your specific web server, using Postgres as your database. Celery is a task is for your task queue, green unicorn is your whiskey engine, uh your whiskey container, and of course there are many options. You just decide this is what I'm going to go with for my initial deployment. And then you have Django as your web framework So that is our architecture. And now we're live. We're available to the world. And we've launched a TechCrunch Disrupt. It's amazing. We are getting so much traffic, so much press, so much publicity. But now, despite your successful launch We have one of two problems. Either we get so much traffic s through all that publicity that hackers are are that that that we are that our web server is starting to fall over and we need to handle that situation, or hackers

Speaker 1: are hitting our webs web application and breaking it down and stealing our data. Which one of these two is the situation that we find ourselves in? So the initial running looks wow head to head 33, 35, package starting to come up. And we'll cut it in about 10 seconds or so. Actually what we'll do is we'll do the first two 100 if we're gonna keep going. Otherwise we'll just cut it in a few seconds. Hackers, hackers Hackers it is. So hackers have broken in to our new web application and they have stolen our user data.

Speaker 1: We're not sure they may be Russian pets. They may actually be people that look like this. They clearly look like this and dress like they're from the 90s. Regardless of who broke into our system, we decide to hire a security consultant. And the security consultant says we've got four problems. We're using self-signed certificates. We've unencrypted our unencrypted cookies have been sniffed off our server or off of our user uh user sessions. There's been uh SQL injection into our database and we've been exp we've exposed our data through something called Heartbleed. And we didn't worry about all this stuff with our initial deployment. We were just trying to get it up and running so we could launch a tech wrunch disrupt So here's how we handle each one of these. The first one is a self-signed certificate. So if you have a self-signed certificate, your

Speaker 1: users are going to get an error like this. So if you do do your deployment and you for our in our case we're using Nginx and we deploy we s we sign our certificate ourselves, we're going to see something like this probably not the set you're looking for. The browser gives warnings. What we need to do is go out to a reputable set certificate authority like Digicert. And they will help us sign the certificate so that it is actually something that has been shown to be, okay, these this this site is actually who they say they are. So we have full encryption. So that solves the self-signed certificate problem. The second thing is User cookie sniffing. So here's what happens in a Django application. This is the scenario. Our homepage, we decide we don't have to worry about encryption on our homepage. But everything else has H is accessed through HTTP.

Speaker 1: So a user logs in and okay they're they're working through the application, they've got their session, but if they go back to the home page and that's unencrypted, then a tool like Firesheep, which is a plug-in for Firefox, can sniff those cookies. And if you're on an unencrypted wireless network or or just a wireless network in general, other other people say you're in Starbucks. Other a hacker could sniff your cookies and then impersonate you and your session. And so what you need to do here is have SSL everywhere. We've also had SQL injection. I really like this picture. It's just kind of weird. So what happens with SQL injection is If you are not using the Django RM, we don't have to worry about this with the Django RM, but if we have direct SQL queries

Speaker 1: and we are taking in user input and we're putting those right in and we're concatenating them with our SQL, then we can be exposed to SQL injection. And this is one of the top website vulnerabilities. The Django ORM protects us against this, but in our case, we were using direct SQL queries for certain cases. And in that case, someone could specify drop tables as a part of their user input, and that is actually executed as a part of the database. And then Heartbleed. Heartbleed was something that happened recently. What this is, is a vulnerability in older versions of SSL that allow hackers to read direct read memory from your server. So the way that we address these again, use proper cert authority, SSL all the things if we are going to be using SSL on for to encrypt our our

Speaker 1: our our sensitive data, sanitize all SQL input, so making sure that we escape that input if we're taking it in directly from users, and use the latest version of S OpenSSL with our Django server to ensure that we're not exposed to the heart bleed vulnerability. Alright, we've solved the hacker problems for now. For now, there's no such thing as a totally secure setup for our Django application. But we're running into another issue. Either users are reporting bugs and we don't even know that these bugs are happening, or VCs are starting to ask us about this thing called traction and we need to show them actual metrics and statistics around it What do you guys think is the situation we find ourselves in? So looks like bugs are winning for now.

Speaker 1: We'll go first to you cut it when you're when you're ready.

Speaker 2: It looks like Bugs is is winning and I d I really don't think we have over a hundred people in here.

Speaker 1: Alright. Bunches

Speaker 2: We might have some, so this section might be good for us. Um Alright, bug tracking and exception handling. So one of the first things you need visibility into is going to be exceptions in your application. So users are now hitting bugs and issues, but they're having to report them to you. So basically, pets using your application have to email in and be like, I can't do this with my profile and I have no idea why So uh with Django, there's the debug setting, and most of us know you don't want debug set to true in production, but you can set debug to true in development and test environments. This might help you figure out where exceptions are happening and what they look like. But you need to get the exception stack trace from production as well.

Speaker 2: So there's a couple of ways that you can do this. You can have Django generate an email every time that there is an exception. So you basically get an email anytime someone hits some sort of internal 500 error or even 404s. can s you can uh set the settings for that. You can also use freemium services like Bug Snag. They have free tiers where you can plug them in with your application and that will do a lot of the exception exception tracking and aggregation. Finally, you can use a service like GetCentry in Raven. So GetCentry was built by David Kramer, David Kramer, who has been active with Django and very active in Python. And the reason you want to use some of these more robust systems is if you have lots of traffic and people are hitting exceptions thousands of times, or thousands of people are hitting exceptions.

Speaker 2: You don't want to get 4. 7 or 4700 emails for a single exception. It's a lot of noise. And so a lot of these tools will aggregate these exceptions so that you can see What exceptions are happening the most, when are they happening, and you can see the stack trace. So now you have this stack trace. You're like, I see where this is broken. The next step is going to be reproducing this bug in your development or test environments. So the Django debug Django debug toolbar is great for this. You can inspect SQL queries, you can take a look at your settings, you can take a look at your static files and templates. If debug is set to true, you can see the stack trace. But basically you get to go be a bug sleuth and look around and try to figure out why this thing is breaking.

Speaker 2: Hopefully when you find the bug you feel like an idiot. This is something I always tell new developers, but there's two types of bugs that you run into. There's the bug you run into and you're like, I'm an idiot. I cannot believe that this I put this bug into production. The second type of bug is the type of bug where you're like, oh my gosh, this bug is super complicated. We're actually gonna have to rewrite our entire architecture because like this whole thing is just toppling as a result of this bug So basically, hopefully you find a bug where you're like, I misspelled this, it's just a key error, it's really simple to fix, I'm an idiot, and then you write a regression test, because let's be honest, you don't want anyone else to make this simple mistake that you made. And now you're fixing bugs like crazy. Users no longer have to email in every time they hit an exception.

Speaker 2: Whenever you deploy to production, you can see if there's a huge spike in the number of exceptions. Did this feature break somewhere? You can fix bugs hopefully before users ever have to complain to you about them. But now But now, dun dun dun, one of two things happened again. The first scenario, customers are still reporting issues, but there's no stack trace. There's no exception that's happening. So basically data is missing or data is inconsistent. And you realize you're gonna have to have to implement logging. The second situation is that customers are loving your application. They think it's awesome. All of these pets are like, oh my gosh, I love hooking up with other pets in the area

Speaker 2: We really want new features. I would love to be able to share hot pet profiles of other pets with my friends. And so now you're like we're gonna have to start tracking features.

Speaker 1: Looks like data is far and away the winner. I don't know. Data got like 40 votes before we heard what features flooded. Someone definitely wrote a script to text in. I think it'll stop it too much.

Speaker 2: Alright, data. What data? So basically data is missing or inconsistent. This is all of our favorite problem, right? Oh there's a problem. We don't know why. It's completely invisible and we can't figure out what's going on. So you realize you're gonna have to implement logging. Django has built-in logging. This is great. Import logging, this is kind of cut off, but logging. getlogger, and you can log in logging. error, you can actually just output log errors. There's five levels in Django. There's debug, info, warning, error, and critical. So depending on the type of log statement that you have, you can choose the level. So if I'm just debugging something on my dev environment or in test, I can set the level to debug. This allows me to just output

Speaker 2: the word here a hundred times like I do. Here one, here two. Oh yep, my code made it this far. Um info, warning, error and critical are more production level logging statements. Basically info is like this is information that we think We're going to need to be able to read through. Warning, error, and critical are these are things that are really issues. Perhaps you log whenever data is in an inconsistent state, or if you have a request object that is missing, a header that you would really expect it to have. In your Django settings file, you can actually uh set all of your logging settings. So you can set the handlers, you can set the level. Basically what you can say is in development and test I would actually only I would like it to to output all of the levels, debug all the way up.

Speaker 2: But in production I really only want to see info and higher So Django allows you to set these things so that you can see different log outputs in different environments. The really simple way to get started with logs is just to have it write the output to log files on each server. The second simple solution is to pay someone like Logly or Log Entries to do this for you. So Logly aggregates all of your logs, you pay them, they give you some really robust search tools on top of the logs that you have, a lot of data you can Look at cool graphs of what people are using and where they're going. And you can search things like session keys and search for keywords. If you want to get more advanced and you want to maintain the logs yourself, you can aggregate the logs yourself. But you're going to need to do things like log rotation, backup, and cleanup.

Speaker 2: So log files can get very large very quickly, especially if you're outputting a lot of information. And if you have a lot of production servers, so let's say we have, I don't know, ten front-end production web servers, you might even have a database server or two or three or replication or all of these different types of Servers, an asynchronous server. And these are all gonna have separate logs. Celery is gonna have separate logs, your Django application is gonna have separate logs. So you're gonna have to aggregate these yourself, either in one location. or again through using a third-party tool. You're also gonna have to rotate rotate logs. When they get really large, you're gonna have to zip them and rotate them out. You're gonna have to back them up if you're worried about keeping historical data And you're gonna have to clean them up when things get too large. So you can't keep them on these boxes if they keep getting too large and taking up memory.

Speaker 2: Alright, so logging fixes the data problem. You're now logging output, you can see what's happening. You're like, oh hey This is why data is missing, it's not actually getting saved, or we're getting these requests coming in from third-party APIs that aren't working very well. But now, again, we branch, and one of two things happens. Either deploys are taking a really long time, they're inefficient, You can't bug fix quickly. You only ship maybe once a week because nobody wants to fight fires Monday through Friday. Or Your team is scaling really quickly. You're hiring a lot of engineers, but you're having a hard time integrating them into your development process. This also includes management. deploys, um but development environments largely.

Speaker 2: So you realize that you're gonna have to focus on one of these two things, either automating your deployments or automating your dev team. So

Speaker 1: deployment got off to a quick start. We'll see if team catches up here in a few seconds.

Speaker 2: Alrighty, deployments it is. Deployment automation So, like most of the sections in our talk, any one of these could be expanded out to be a full-length talk. Deployment automation is a huge topic, and it's actually a really large space. There's a lot of things to automate when it comes to deploys, when it comes to infrastructure management in general. The first thing to automate is going to be your infrastructure configuration. So tools like Salt Stack, Ansible, Chef, and Puppet are all really good tools, very robust tools, well documented. For automating your web infrastructure configuration. And the reason that you want to do this is because you really don't want all of the domain knowledge for what's installed on each box in one person's head

Speaker 2: You also want to be able to automatically install the things you need for different types of servers. So if you had to go to every single front-end server and add a new installation every time you had to add a new installation, you would go nuts. It would also take a really long time. So these tools are great for doing that. SaltStack is Master Minion, Ansible is YAML files Chef is Ruby scripts, and Puppet is its own declarative language. But these are all basically for things like installing Postgres. installing Redis, installing Python, setting up user accounts and permissions. So you know each each employee at your company theoretically has a user on the different production boxes just in case And that's what you use these tools for.

Speaker 2: There are also some platform as a service companies out or solutions out there. Heroku is the most famous. Heroku and Elastic Beanstalk will deal with all of these things for you. So Heroku allows you to build your app locally, and then when you push to Heroku, it does all of the server configuration, it installs all of the application dependencies. It has a seamless deploy because it just uses get pushing for deploys. And it manages all the scaling for you as well. Elastic Beanstalk is Amazon's solution to this. It's much younger, so it's a it's a lot newer of a solution, but the idea is to be a platform as a service solution for people. Are expensive. Heroku can get expensive really fast. But if you're making a lot of money and you don't want to manage this yourself, they're great ideas.

Speaker 2: Infrastructure and DevOps engineers are some of the most expensive engineers and some of the most difficult to find So a lot of companies have opted for using tools like this to solve their infrastructure problems. Automating your deploys So deploy is a big one. I was actually talking to someone earlier about uh no downtime deploys and what that means, like several seconds versus zero seconds of downtime. But deploys can get really complicated, especially if you have one really large application and you're deploying changes to it. I don't want that. So one of the things that you can do is decouple the deploy logic from the infrastructure management logic. Infrastructure management involves what's installed on the box, what's installed on the server. Do I need Postgres?

Speaker 2: Do I need Redis? Do I need Python? Your application is actually separate. So because we have V at like virtual environments where we can install all of the application dependencies within that virtual environment, deploying Python code can be separate from updating your actual infrastructure configuration. While one might need to run before the other, and that's important to note, decoupling this logic and keeping it separate makes each task simpler. So rather than trying to do too many things at once and having too many dependencies that are very tightly integrated, what you have is you have separate steps for deploy that are each very simple and easy to maintain. And this allows you to deploy more quickly, to update things, basically to get your code out to production without things falling over and having huge fires.

Speaker 2: Automate your tests. So step number one, write tests. Step number two, you can use continuous integration tools. That's a thing. to automate the testing of your code. So every time you open a pull request, you can have test suites run on the code that's that's been opened. So you always know does this code pass tests? You can create test coverage tools Those are very popular to make sure that you have at least a certain percentage of test coverage for any code that has been pushed to production. And finally, one of the big things that you can do to make sure that your application is as simple to deploy as possible is to think about service-oriented architecture. And for those of you who are slightly new to service-oriented architecture, the idea is to separate different uh pieces of logic into services.

Speaker 2: So one Django application could be a service. Maybe you have a really robust Tool for emailing customers, like a notification center. And that can be separated from everything else. And you can deal with it through an API. You just Send emails through this API, it's your own internal service, but that logic is decoupled. So whenever I need to make changes to this one email notification service, I just deploy to that one service. Having an abstracted and decomposed system like that means that if one piece goes down, it doesn't necessarily topple your entire system, which is a good thing. Actually, one of the fundamentals of building computer systems. So thinking about building service-oriented architecture, separating out logic, having small pieces within your larger system is good.

Speaker 2: One of my favorite quotes is There is no such thing as a small change to a large system. So the idea behind this is to have more small systems, so you can have small changes to small systems. All right, you now have continuous deployments.

Speaker 1: But now we come to our most important decision point. We're doing really well. Company is scaling. That continues deployments. But now The VCs have asked us, they've said we're getting a lot of requests from Fortune 500 companies They want a white label version of Tinder for Pets. They want their own GE branded Tinder for pets. Capital One branded Tinder for Pets. Do you build that or do you clearly deny that as your votes have indicated already? Who said build? Do you stand up and fight against the VCs? Or do you just acquiesce and say, sure, we'll build whatever white label version of Tinder

Speaker 1: for Pets do you want? Clearly you've chosen wisely. You now have so much money coming in because you've decided to focus on your core audience. Instead of just dealing with dogs and cats. now have a have a separate product line for hamsters and gerbils. The money is pouring into your new application.

Speaker 2: Rodents there it's a really large customer base, rodents.

Speaker 1: And you are able to raise more and more money. Massive funds flow through your coffers. You are a great success, but in the back of your mind you wonder, is it all just a bubble? And that takes us to the end of our story today. This is a totally separate number. So obviously this talk here was pretty experimental, a bit different than some of the other talks where it's just a linear path through the the topics. Uh if you could send a text message to this number, it's five this is a sep as I said, this is a separate number from the other one And on a scale of 1 to 10, would you recommend this talk to a friend or a colleague? 10 being extremely likely, one being not likely at all, and just from one

Speaker 1: from 1 to 10. whether you'd recommend this. And what this is going to allow us to do is have a quantitative measure and you'll get back a text message response with a blog post that we've written with all sorts of resources on all these topics Some of which we did not even cover today. So you've only seen actually a very small portion of this talk. And we'd like to continue to iterate and approve on it. You'll also get back our Twitter handles. And if you could text uh uh tweet us with uh whatever improvements and suggestions that you have, it'd be super beneficial and we'd really appreciate the feedback. So again, 503-476 -3063 with a 1-10 score. Thank you all for your time today. Kate Hedleston, take a bow. Hi

Speaker 2: Cry

Speaker 1: Mamma Pai, Developer Evangelist Centurio. Thank you all very much for having us. This has been a really fun time to present this to you today. Thank you very much.