Django: a Data Shovel With a Future

Good morning everyone. Uh this is my first visit to Portland. I'm really happy to be uh uh getting to know already the city and some of you who live in it, uh, where I will be bringing the big PyCon conference in uh 2016 and 17. I'm going to, for a moment, think about software along two axes. I'm going to be very conventional and just use the horizontal axis for time. And I will graph vertically the idea of complexity, having more code, having more complicated code. In general, when we start writing code, our day, our month, our year, our project looks something like this. Over time, it becomes more complex.

Sometimes there are bumps in that trajectory, but that's generally the shape of things. And uh as I try to uh get up in the morning, I write a test or whatever, and I I start trying to implement a feature I write and I write and I write new code until I reach this magical state called working code. And when I was a younger programmer, I would stop there, declare victory, and move on. It turns out that this was a mistake. Once my code is finally working, it is probably better described as barely working. I mean, what have I done?

I have finally crested that the great border in the state space of of programs uh from all of the states my code could be in without having this feature I've just added. And one of the states in which it now possesses the feature, but unless it's the kind of feature I'm very familiar with, what are the chances that I will have crossed that magical state state state space border at the best and cleanest and most readable way to implement this feature. In my case, a very, very, very small chance, again, unless I'm kind of already familiar and have done the trick before. My guideline for re-implementing what are to me really new features

is that once new code works, then I'm probably about halfway done with it. Maybe I should now put it aside and reread it tomorrow to realize how difficult to understand this code I've just written is, but if I've spent an hour putting together what is to me a novel solution I generally find that an hour later in the day or tomorrow really helps clean it up and make it better code. So I tend to start, of course, at zero complexity because I have a blink. screen or or a procedure in front of me. I work and produce working code and that moment is a magnificent one because I can check in. I can uh plant a flag, I can plant a stake and say, I at least now have this feature working. But I generally find that the code itself needs more time, it needs more work, and that the complexity can then be ratcheted down through refactoring, through caution, through um

Cleanup, and that it's only then that I typically reach what I would call good code. Web framework history I think follows a similar pattern, both in the industry at large and also if you narrow your attention to Python itself. We started with the idea of of course in the very early days you just uh served the web off of your file system and threw scripts into that file system if you wanted to run them But once we got the idea of putting our software, not the file system, in charge, we started writing code. And we wrote more code and more code and magic to produce forms and magic to persist our objects and

magic to let URLs travel across our object uh persistence hierarchy. And when we had finally built that mountain of code all the way to the sky We gave it a name, and it was called Zoe. And it worked. I mean you and and and we had we had reached this point where we could now go out and accomplish things on the web. But just like when I'm sitting and and writing uh a new feature for a few hours. We didn't necessarily land on the simplest answer to the web first. And uh

once Ruby on Rails had inspired our community with this idea that the web could be radically simpler than we had made it. Right out of the gate, just a few months later, uh Robert Brewer, I believe, came out with cherry pie, which still has a loyal following today. Uh and uh Django very quickly followed that. It's interesting. There have been a lot of other attempts at web frameworks since then. But two of the ones left standing are those two first responses that our community made to Rails. I put Cherry Pie a little higher on the complexity scale because its pattern was that you build controller objects. and your URL kind of surfs down the attributes of the controller objects with its path components.

Django kind of went all like steampunk and just You compare your URLs with regular expressions. You treat them as strings. And that is a simplification over having to teach new programmers how to write objects. Of course, the community's experience didn't stop there, just as Django looked at Zope and said, I think the web is simpler than that. So of course the microframeworks then looked at ZOAP, shook their heads and said, I think the web is simpler than that, and produce flask and produce bottle and the other uh smaller, more boutique uh microframeworks that exist. Uh I should for the sake of record say that I have at times used a micro framework. I generally find that if I really need to add an ORM in a templating language and a cross-site scripting vulnerability package

that by the time I've kind of put all of that in there, I'm kind of back up at Django 's complexity level. It lets me reach that complexity level gradually and on demand from it all being uh something installed on my system uh or server to begin with. Uh but I actually generally find that micro frameworks are more valuable because they let me handpick the pieces I put together Not because your web app is going to be noticeably simpler when it's finished in a micro framework than if you had used Django. And finally, I should mention A few programmers did stagger alive out of the collapse of Django, of uh Zop. and all of its code. And they have actually

themselves looked at SOAP, thought about how they can solve those same very CMS-y, very content-centered problems more simply and uh actually another of under uh uh other small web frameworks have died as that uh almost everyone I know in that community has now gone to using pyramid, it's worth looking at if you want to see a whole different approach to what your URLs mean. So that is kind of the trajectory I think that we've uh taken. Django was a first very successful experiment in how the web could maybe be simple. What is then a web framework? What is this task that we can do in a complicated way or a simple way?

I think of a web framework this way. A web framework is a denormalization engine. It gets our data, hopefully, orthogonal and independent and unique in our back-end storage. And it mashes it and mixes it up into a single place, into a document that is delivered to a consumer, to a user. A web framework denormalizes data so that it lands in documents, which of course raises all of the normal problems of denormalization. As soon as my username doesn't just exist in the database, but on a web page that I have open in front of me, I now have a copy of my data that raises all of the normal problems of caching.

What is there that will keep that username in the corner of my screen equal to the value in the database if it starts changing? And so you have the full range of options all the way from, as long as the tabs open, it's just going to be out of date. All the way to using MeteorJS, where they promise automatic update of all the data on the page when it changes on the back end. the full spectrum of uh solutions to having a cache or having a copy uh of data uh arise uh and the full range of simple to complicated solutions to that are inherently present in what a web framework does, which is make copies of your data, usually denormalizing orthogonal data on the back end.

uh into some more complicated form. And I found, especially as mobile devices have gotten better, bandwidth has improved, storage has increased That a question that increasingly commands my attention is the question of how much data in a particular case should you deliver to the client? You can give a client, a web page, the absolute minimum, just the exact data and no more that the user has asked for. On the other end, you can just give the entire database to JavaScript and let the front end deal with it. If you've ever used the uh Sphinx documentation system to build a static site And notice that it has a search field

even though it's static. If you look behind that during the document building process, It builds an index of all the words in your document as a JSON blob and stores it to a file. That search field does not go talk to a search engine. The JavaScript reads in your entire full text search database as a JSON blob and does the search term action right there in the front end. So even on a normal Python project you might span in the solutions you're using this full spectrum between very carefully designed web pages that just give exactly what the user has asked to see and nothing more and other parts of your solution that hand

essentially a database to the front end and let it do its own searching. Imagine for a moment, because I think it's always fun to think of what we've accomplished and what the future might be like in this industry. What if hard drives kept getting bigger? They shall not do so because of things like physics. And it's not that you know the the way that we store uh data with magnetic uh polarities, I'm told, is starting to run into density problems at like, you know, the the atomic level and or way we can build and and purify platters and so forth. Uh while there are researchers looking at completely radical other ways to store data, which we know will be reached because

in a I believe it's stated in Star Trek the Next Generation that the drives store data on an atom-by-atom basis. Obviously it will take a long time, and we've been told that we can't expect the rate of improvement to keep going because we've essentially been writing a single solution for 30 years and it will really slow us up as we have to pivot to something denser than magnetic platter mediums. But just for fun, what if hard drives kept growing? In the last 30 years, from 1984 to 2014 Hard drives have gone from the biggest three and a half inch uh hard drive being 20 megabytes to Seagate a week or two ago announced an eight terabyte drive, thus beating out Western Digital, at least for a few weeks,

for those clients that need maximum storage. What of the next 30 years? What if they they were able to keep going? How big is the ratio between 20 megabytes and eight terabytes? Well if you do the math, I believe you'll find that if we did this all over again over the next thirty years About the time of my retirement, I'd be able to buy an off-the-shelf 3. 2 exabyte hard drive I could put Facebook on it five times. Google would need five of these.

Estimates vary, but some people think the NSA would need as many as twenty or thirty in order to operate. If that happened and if I lived to see the day, I'd be 70 years old. I mean, can you picture me as an old coot? waving my new three exabyte hard drive at the kids bicycling down the street and saying, I remember, I remember when you used to have a room full of computers to store this much data. And the children will laugh at me. Because they will be young and because I will be old. Uh for the record, because

I want to I want to look I want to look like a good futurist here if if this actually happens For the record, I don't think that a 3. 2 exabyte device would be a pure storage. I think we would probably go ahead and sacrifice some of that storage to putting an array of a few thousand small microprocessors on it so that instead of uh a single uh bus trying to read all of those exabytes out to do something to reduce the data I believe you'd probably have small onboard processors, a cluster, so that every few petabytes could have its own dedicated processor. uh to deal with it. But oh even if the rate slows up, we have a lot of storage potentially in our future. And even today, questions

begin to come up as I sit waiting for a page to load Why don't I have Stack Overflow on my hard drive? Would the daily diff really be that large that my um cable modem couldn't handle it in the middle of the night? Uh why isn't Wikipedia on my hard drive? Why isn't the IMDB, which actually is not that big, it's a few uh that they have it as text files and you can download it and and process the data yourself Why do I sit and wait for all of those banner ads to load when I just want to see that guy's name who is in that movie with that girl? In that future, we'll wonder, why not put the Library of Congress's full media holdings, video

and uh audio and uh books, on my hard drive. There will be questions like, I I went and looked at the IMDB database, added up the number of minutes of footage in all of the movies and television shows in history. you could have it all in high definition and by my calculation about 0. 008 exabytes. Maybe one of the selling points for the hard drives is they'll just come with an all movies ever partition on them. That you can blow away if you care enough, but that you'll probably never notice if you're uh not buying it for consumer purposes anyway. Um Maybe all you'll need is updates as new movies are made. I know Hollywood will continue making movies over the next 30 years.

Maybe it'll be 0. 02 exabytes by then. You might think I'm being silly, but we have already made taken this step and made this choice with our version control. You think that the latency of browsing between stack overflow pages doesn't add up to much. But do you remember what it felt like when you were never, again, waiting a few seconds for subversion to come up with an answer for you? Do you remember what it felt like? Actually, that's not what you remember. Using yet for a moment felt novel. It then felt quite normal. What you remember is going back to subversion for the first time. If the day came when you simply had Wikipedia, IMDB and Stack Overflow all on your hard drive, I believe that when you went back to your mobile device and browse the next day, you would have that

Let's call it the subversion feeling. That feeling that your whole life is once again spent waiting. So in one very important area, how we manage our projects, we've already made this flip. We've already made this pivot from the idea that an application should just give me what I need at the moment to How about you just give me your entire database? Yes, I'm not likely to look at the first hundred commits to Django. How about you give them to me anyway? Just give me the whole history and on the client I will deal with the question. of how much of it I look at. Incidentally it does occur to me that there are very nice privacy consequences

to me just having the Wikipedia on my hard drive and only my processor knowing what of it I'm looking at. A big win of Django as I have used it in projects. Is that Django is flexible? It is not opinionated about where my views land On this spectrum between share minimum and share everything. Now it's true that there are helpers Django has. class-based generic views that are opinionated about the relationship between a URL and what part of the database I'm trying to get to from it. But those are the exception in Django.

There's something you opt into when they make sense. Unlike some of its predecessors, Django, the framework itself, is not strongly opinionated about what a URL means. Begin a function definition, and I am invited, if I need to, to invent from the ground up what information will live there and what it will be. Let me tell a story about a Django project I did recently, where a very modest and small version of this pivot wound up radically simplifying the application. was a project for the New England Wildflower Society. They were named that in 1900 when they were founded in the the this uh

conservationist organization first came to be. But they have since then, uh though they're known by this name, uh they actually do all kinds of conservation study of native plants, study of the way that invasive species and garden species threaten New England's native biomes and all of the animals, other creatures that depend on them. And they have a big audience of students that use their resources, amateur botanists, and teachers who they wanted to serve better They went and found um uh Boston uh-based uh Python consultancy Jazz Carta. uh a group that started back in those uh heady days of Zope and Plone and those other big solutions

but has recently been using um Django on quite a number of their projects and having very great success with it. And Nate Ani uh told me that all of his uh developers were busy. He already had several of them working on this project, but they needed one more. And so I got pulled in and I got to help with this really, really fun project It tries to solve the following problem. Botany , in particular botanical identification is traditionally done with what are called dichotomous keys. Dichotomous is actually it describes a plant or something like that. that every time it branches, branches not several ways, but exactly two ways.

The dye in dichotomous tells you that you're always splitting into two choices over and over again before you reach all of the limbs or twigs at at uh the end of the process. And dichotomous keys are kind of a choose your own adventure series for botanists. They start up at a high level like plants. And the botanist designing the key asks, what are the two biggest groups in the world of plant categories? And they develop a question that, if answered correctly And only, if answered correctly, will let you look at the correct half of the plant kingdom among whose millions of species your plant lies.

And then the next question tries to split it into two big pieces. And then again and again. They actually did put their dichotomous key of New England native plants on their website. Here's one example. This is the top level for gymnosperms, all the things like pine trees and ginkgos that are fairly primitive in that the seeds grow right out visible in the open. All you've got to do when you hit the gymnosperms is start by determining whether on the plant you're examining seeds are born singly. partially concealed by a red fleshy aurl, and whether the abaxial surface of the flat leaves is bearing pale yellow longitudinal stromatal lines.

You should see middle schoolers' faces light up when they're presented with this kind of a question. This helps the expert. They look at that and they immediately picture, oh yeah, that group of plants, and they know which way to go when they reach this particular page of the choose your own adventure. This is not, however, something that tends to help students. And the New England Wildflower Society wanted to address this. They wanted to focus on something. that would allow and support a more user-directed search, search that does not uh descend step by step into the classifications biologists find useful, but the kind of search they called the project Gobotany

That would simply ask you to choose a filter that would let you look through a menu and say, well, alright, the thing I have in front of me has red flowers. And that would narrow the set of New England species to just those. Well you could then say, okay, it has smooth leaves, and suddenly have your result set narrow down more considerably. Uh I'm on a trip to Vermont and this is where I'm seeing the plant and that can narrow the range of species. They wanted the users to over on the left-hand side of the screen have a set of very common questions and then a get more questions button that lets you you even ask uh why I

give me I have a really good leaf in front of me. Give me more leaf questions. Give me more stem questions. Interestingly enough, I'm told that the early user tests proved that 0% of users ever notice the Get More Questions button. I don't know if they ever fixed that. But as you can see, when you select one of these filters, it gives you the multiple choice question. And when you select one of the options and hit apply The page uh adjusts uh in order to show only the species that match. In fact, if there are few enough species on the page to make this possible, it actually animates the throwing off the screen the ones that didn't fit

and the tightening up of your results set to the ones that do. I did that. It's a JavaScript. In our first iteration, we did something that seemed so obvious in retrospect and wound up being one of those suboptical uh moves. Uh it was one of in my case, it was one of my first times writing a web API, and we thought that to build a weight web API you build this kind of obvious one-to-one correspondence between the action the user's going to be perform uh performing and a URL that performs an analogous API call.

We thought that you could just study and make a list of the things That the user would be doing, and just build a URL query that would do each of those, and that that would constitute good design. Let us explore whether that is the case. If the person selects red flowers , smooth leaves, and a plant that's present in Vermont, well we thought, all right, we'll support a query URL. Where you simply concatenate one after another all of the questions that the user has answered so far about their plant and submit that whole question back to the server for evaluation to get the new list of plant IDs that should be present on the screen.

We noticed that doing a three-way query against our database, by which I mean it was a three-way join with the same table appearing three times. And then of course the student chooses a fourth filter and now you're doing a four-way join, five-way join, six-way join. I mean if it's a big group of plants, you can get up to ten -way joins, twelve-way joins We noticed that these searches get a little bit expensive. And so we started wondering: well, what if we uh put some caching headers in, uh, maybe have varnish or a CDN, uh content uh content something network in front of our app. Is it delivery? Content delivery network? Thank you.

Will that caching help reduce the server load uh by getting to uh over an hour, uh keep parroting back? particular search result of two students or three students or four students or the same person working again through trying to identify a plant happens to reach the same state in their search process? Answer No. And the reason is this, so for okay, a first answer that was easy to address is that the exact same search depending on the order that they choose these filters, can come out in fact in factorial different ways. That can be solved through a careful practice called canonicalization Where instead of allowing your application to have several different names for the same thing, you instead insist

that if the user creates a query, if two users create queries with the same meaning Necessarily they will produce the same text or the same query. A very uh trick with uh this kind of URL is to restrict your JavaScript so that you always order those filters alphabetically as you assemble the URL. Now whichever order they've clicked on these in You get exactly the same URL and have the possibility of pulling the cached version of that result. In our case, we probably only would have needed to do that had we gone down this road in the case of our front-end JavaScript, we were kind of our only client. The interesting thing is that if you're letting third parties build and deliver these URLs,

How can you make sure that they are hitting the normalized cached version of the URL and not one of the other infactorial ways it could be written? One possibility would have been just in case you ever go in this direction, uh you could redirect the other URLs so that um uh people who fail to properly canonicalize at least then get quickly uh moved over to the URL that you might have cached with a result. One of the interesting things about this transform, we noted, is that had we implemented it, it is purely textual and does not need the database. The fact that that query at the top should really be that query at the bottom.

actually has no bearing on whether this is uh an actual URL or a 404. This is a transform uh purely based on the fact that we disqualify the URL by looking at it This is the kind of logic that could be pushed out uh into a front-end uh caching and normalization layer. and not even hit the server that's busy answering canonicalized URLs with content And one of the interesting things is, you might think, well, then I'll have an app sitting there answering 301s all day. 301s, if you read the standards, can also be cached. Varnish will sit and spit back a 301 all day if you've let it cache it.

A CDN like Fastly will do the same. Uh caching non-200 results can be a big deal. And yet I find we often are busy setting all the right cache headers on our 200 results. And not even thinking about all of the others in the really, really great Reddit thread that the Onion did several years ago when they switched off of Drupal called The Onion Uses Django and Why It Matters to Us, where there's just a free-for-all of their answering all of these skeptical web developers uh uh and answering them about why this was a really, really good uh move that they made. Um one of the most interesting uh paragraphs to come out of that was their answer as to what the biggest performance difference they wound up making, which actually didn't have to do with Django specifically.

They said the biggest performance boost of all was caching 404s. Sending cache control headers to their content delivery network on their 404s, right? A view you very you might very rarely think to customize. They went into Django, customized their 404, gave it a custom 404 view. And they actually stuck a cache control header telling their content delivery network, um, no, no, don't come back to us every time for this URL. We give you permission to remember for a while that we don't have this particular URL. The onion having gone through so many different uh uh uh uh uh uh early versions of the website since the the late nineties

had a lot of URLs that had just been mistyped into popular web pages elsewhere. URLs that made no sense. URLs that the very first generations of web spiders had accidentally put together by misunderstanding how relative URLs work, and that were just still out there uh from a site that was so old they had a lot of old links that spiders were hitting every few minutes. That don't exist and will never exist and can't be deciphered, can't be redirected to real content anymore. And they found by giving their CDN permission to cache that their outgoing bandwidth was reduced by 66 %. Because think about it, once you have good caching on your 200s, um

your web server might not have much left to do Once the 200s, once all the onions web pages are out there in their CDN being delivered from a uh location geographically close to the customers, they found that most of what their server was then doing was 404s. And uh they uh uh drop their load average by 50% by letting non -pages be cached by their CDN. So, if we had wanted, we could have somewhat improved the state space even for third-party clients by letting these normalizations be cached. But even if we do canonicalization and cache the pages behind these URLs, how often will they be

revisited? And the answer turns out to be fairly rarely. What are the chances that two students will in their careers land at exactly this query? especially as you get to five and seven and nine different features of a plant having been entered. Imagine a group of plants that have a hundred different filters you can choose in use by students who maybe only apply five filters before they're uh they've narrowed the plants enough. Those are both underestimates but they already give the number 100 choose 5 for the number of canonicalized, properly canonicalized URLs those students could hit as they're wandering out into the state space of this chess game that they're playing with biological

classification. That's 75 million And that's only if they go five searches deep, students often go deeper. And in a state space of 75 million I mean, sure, there's a lot of oaks. It's very obvious to note their leaf shape. And so there are going to be some of these URLs that are hit more often than other. uh others, but you wind up with a very very big state space that your users are wandering through at the URL level level and a lot of different information to cache So a moment came when we conceptually kind of stepped back. I think it was actually during one of those freewheeling 40-minute Skype sessions that we always intended to be a stand-up

meeting. But somehow often became a little bit more. And one of the developers asked a question that caused us to think this way. This URL is asking our app, is asking Django, is ultimately asking the SQL database to do an in-way join against the the plant versus feature binary relation. At the bottom it looks something like this. You name the three same table three times. Make sure that you're using plant ID to build triplets of rows from those three tables, and uh then limit Each of those images of the table to only containing rows for one of the filters that you're looking at.

And it's the database that's doing this work for your user. It's looking at the table of all plant species and their associated features, this binary relation. It is using a where clause to grab, let's say, all the species that are trees rather than something else. And then it is doing the join, it is doing the intersection. To look for the plant IDs in common between those three, those four, those five different virtual computed tables in your select statement. And the developer suggested the following pivot. Instead of doing this, instead of doing what we had planned, Having the database,

the equal sign there, I mean doing a straight comparison on that database table to get some lists of matching plants. than doing an intersection between those lists of plants to get only the plants that fit all five of your filters. And having generated that answer, handing it back to the client. What if we do, in one sense, exactly the same thing, exactly the same computation, but what if we just draw that line somewhere else What if we instead have the server stop with having generated the list of plants that are trees or the list of plants with red flowers? give those to the client

and let JavaScript do the DRATID joining. Let the JavaScript take that step that actually produces this Very, very large state space, choose your own adventure state space that the user is exploring We found that we actually already had a JSON payload that for a particular feature uh this is this is when you get that little pop-up letting you choose the leaf shape this is what the JSON looked like behind it. When the person clicks on leaf shape, the JavaScript goes and hits the leaf edge URL. And previously it got back some text and maybe a little image with which to illustrate those options. We now

just went into that JSON and we went ahead and we gave away the database We just went ahead and for each of those options, we went ahead and just told the client, and if they select smooth leaves, Here's all the species in New England that that have that feature. If they select jagged leaves, don't come and ask us. We're telling you up front, here is the complete list of species that match that filter. A single line of code, if you use underscore JS, will get those lists of numbers and fairly efficiently mix them down. It even kind of works on mobile. to a single list of species. It's not

going through lists of integers and looking for similarity is actually something that computers are good at. and that we found you don't even notice the time it takes the JavaScript to do this quick operation. All of a sudden our search URL was gone. This disappeared from the project. And instead, if the user in any order went through the color of the flower, the shape of the leaf, the state that they had found it in, Vermont I mean. Not like other kinds of states. If they went through these in any order, exactly the same three URLs would be pulled once And they would contain exactly the same information, whether they were fetched very early or very late

in that particular student's search process. Suddenly the data we were delivering was independent of where the user was in their search and the front end could put together that list of 55 matching species by doing its own work to combine those possibilities it had been given. If we have in features, we have in URLs. End of story. No exponential space to have to cache. And actually we kind of noticed at the end that we didn't need Django. We could actually have generated it as a static site. Because except when the biologists update the database of New England plants, these little lists of matching plants never change during the day.

Very, very stable data that can have very strong cacheability properties. We found that giving our front-end client more data meant exponentially work less work for our servers All of which was an early lesson for me in thinking about this problem, thinking about this issue of data, Very often in a project I'm just racing to try to build obvious URLs that correspond to what the user wants to see and visit. And this was a first experience in stepping back and thinking about the relationship, the the pipeline that I'm building as a data pipeline.

uh to think of my web app as kind of a data shuffle, trying to get the chunks of data that make the most sense and throw them at the client fast enough that it can put in front of the user the results that the user expects. If I'm correct that very often the solution to building a really responsive, really efficient, really easily cashable app uh is to put data together in interesting ways. I think that as web developers we should have our eye on what is happening in the science and the data space. uh because uh after uh being uh a fairly obscure language in many parts of the world for uh many decades

Uh Python, uh now that it's what uh almost a quarter century old, is suddenly the big breakaway hit, or one of one or two big breakaway hits. in the area of data processing, while larger scientific disciplines are scientists are very conservative. They still use Fortran. This is very interesting. They're not comparing Python to Ruby or even C. Very often, Python is being presented by the younger grad students as a competitor to Fortran. And the professors are old. And so they shake their heads and they keep writing Fortran. And talking about how they used to have a machine room full of machines and and so forth.

But smaller and more agile sciences For example, astronomy, because it's a fairly small and close-knit community compared to science writ large, uh, has largely within the last uh two or three years moved over to Python. Recently I saw, I believe, Jake Vanderplas, who's doing a lot with the iPython notebook and with scientific reproducibility and statistics. I saw him give a talk where he showed a slide full of images of telescopes that are under construction and pointed out that every single one of them had an API that's in Python. Where if you get time, once that research telescope is built on the telescope itself, what you give them is a Python program that they can go ahead and test before you your precious few hours on the telescope comes up

That is a program that will point the telescope and take all the pictures you need in return for your whole research grant or whatever it is you you send to them. Astronomers, by the way, people who care about big data, they right now are building an array of telescopes in the southern hemisphere. that when complete, which will take a few years, will be generating an exabyte of images every day. They're having to wonder about things like how do you get enough power up to the observatory to reduce the data to the point where it can go back down the mountain every day? Because they want to, they're basically going to take a picture of the entire sky at high resolution every three days, essentially do a

digital uh a digital slow uh slow and sky survey every two or three days and they and that's too much data for them to look at. What they're gonna do is just give us a pipe where we can do whatever research we want on all of the blitz and motion that happens every night in the sky, it will be other researchers that have time to build the algorithms that look for near-Earth asteroids, for stars momentarily dimming as one of their planets plat uh uh crosses in front of their disk or whatever. Um Astronomers are going to be dealing are already dealing with big data, and it's scientists that are pioneering a lot of the work in scientific consultancies of figuring out how Python As kind of one of the simplest possible languages you can throw at a scientist, a grad student who's not a programmer, doesn't want to be a programmer,

they're busy with astronomy, and get them up and running and solving interesting problems. I think it uh it behooves us, and I've been doing some work recently to just on the side learn a bit about numpy arrays, adding up millions of numbers with a single line of Python code and no apparent loop Uh Pandas Data Frames essentially having a spreadsheet inside of an object in Python with columns and rows of data that you can query and uh group by. and sum with uh single method calls. You're able to do with uh these vector-oriented libraries queries like this. It might look a little crazy. We'll read from the inside out. Person is a data frame, like a database table, but in RAM or on secondary storage.

A list of numbers we're imagining in this case. Getting a single number in Python and saying greater than or equal to 21 yields which two values? True or false. So if you get a million ages, a column of a million ages, and compare it to 21, you get a million trues and falses. in a tight little vector of Booleans. And the rule is that you can get either a column like dot age or a whole table like person in pandas and index it by a list of trues and falses that's as tall as the table is. And it selects out the rows for which true is that Boolean

value. And so in a single line of code, you can get your spreadsheet, your data frame, and begin to reduce it down to just the rows that you're interested in using a very concise uh notation. um that has no explicit loops. They all happen for you behind the scenes. And the uh uh continuum uh analytics who uh spends their life helping sciences and businesses use Python for big data has just been writing Blaze, which given that syntax will not only use it as it naturally will against NumPy or Pandas, they can also translate it into SQL. They can also translate it into MongoDB.

They can also translate it into Apache Spark. or any of a growing list of backends. They are so often getting into situations where they need to run an algorithm they're familiar with But against a new or unfamiliar back end, or they have a um a customer where legacy data is on Oracle, but the new data they want to join it with is somewhere else, they are actually building not an ORM. um but a data mapping engine that lets you ask the b in the same way that an Orium lets you build an object from one database row They're letting you write single lines of code that ask about enormous amounts of data that might be spread across many back ends, or if you're using Apache Spark.

might be auto-sharded across an entire cluster of machines. Python is becoming an important tool for large data sets. And I think that it will behoove us, especially as shoveling data at the front end becomes more common and more possible, to watch how these data tools might fit into our projects. And Django is going to be a great web framework to be on top of, to be along for this ride with because it allows such flexible denormalization, providing common patterns if I happen to have an object per web page, but letting me grow go wild if I have something more interesting than that. A big deal, especially early on in its success, is it has no dependencies, which for people that still install stuff by hand, especially if all their organization allows them, is an old Red Hat Enterprise server, this is a big deal

URLs are simply text. Everyone else Django is competing with thought that your URLs should name objects in RAM or their attributes. Uh Django was the first one that was like, get the URLs out of my code. Let's say that the URLs live on the data side of the house. I was very suspicious of that at first because it felt so wild. I mean, what guarantees that the you are if you have slash person slash twenty-one What guarantees that a slash person page even exists if it didn't traverse an object on the way to person number 21? If you're from the world where the URLs visited a file system hierarchy, it felt like the intermediate parts of the URL had to land somewhere first before returning your data.

Django really achieved a simplification by saying, nope, URLs. text. Views can be simple procedures. It privileges relational databases. We should give Django so much thanks for the fact. that generation after generation now of Python web developer has learned a SQL backend uh as think of how many NoSQL backends have come and gone. over the time Django has been here. But people using Django are still on old, reliable, industrial strength relational databases by default, and it is a wonderful default. And Django was the really the first web framework, I think, in Python.

Given the fact that finding the simplest possible thing is hard. To be the first to market with some real simplicity. And I think that solution is going to last for quite some time. And I'm happy to be along for the ride. Thank you very much.