What would the django of data pipelines look like? with Lisa Dusseault

Speaker 1: I've been around for a while. The first 15 years of my career spanned a revolution in web development. The um using the Common Gateway interface to program a website, it gave you almost nothing. It's just like, oh, um, here's your URL, build a web page. Java servlets was al already like the f the first big improvement of that in in introducing an inversion of control where the the framework said, no, we'll call your code when it's time to do a web page. But also having having a framework that ran the other that that that was opinionated as it ran your code, saying make your code fit our framework and trust me, trust us, things will be better, and it was.

Speaker 1: Things continued this way with important framework features being added. Django included them all, did them really well, added a few innovations of its own. And um I was at the first DjangoCon in 2008, and by that time I felt like this had been the evolution in our tools. Like being a web developer in 2003 was having your ass hang out in the air, and being a w web developer in 15 years later. was so much safer, protected, supported. Um and and and there's so many more metaphors I could use. Another one I've got later on is the idea of working in a chaotic, messy workshop versus one that's organized with your models over here and your views over there. just changed. It made us all like 10 times more productive.

Speaker 1: Then in 2020, jump forward Um I was working on an HR startup uh and we were importing lots of customer HR data to our production database through a pipeline and then ultimately to our production database because we needed copies of people's salaries, job grades. uh managers, teams, job levels, and and much more information, their bonuses, their um uh their commissions to bring into our system and and be able to do analytics in our system And it was such a mess. We started building framework code just as we were going, trying to manage our own chaos as we went. And I can't stop thinking about this. Uh I left that company two years ago, but I couldn't stop thinking about

Speaker 1: what a mess it was and how it brought me back to this pre-framework. sense of of of of what a mess, where's the guidance, where are the utilities? Why do we have to why are there several different ways of checking if something's afloat throughout our code? And a framework is not just a library that you can do tactical things like Pandas is fantastic, but it's extremely tactical. You need to know what you want to do in a spot and tell it to do that. That's great, libraries are great. But a framework puts you into a structure of doing things a better way and from a foundation that you don't have to build as much. Uh and it can be quite opinionated, which I like.

Speaker 1: I like that a new member can join the team and if they've done Django before, they figure it out. And often there's a more declarative style too, like the way Django models declare model attributes rather than make them um methods. I also can't stop thinking now, even now that I'm no longer put on the data engineering hat, what the work conditions are like having your ass hanging out in the air So I've started working on this and with a great friend, Jeff Kaleski, we've been building a framework, not a library. This isn't what we built in the last startup. We started from scratch and we started from a developer experience perspective. We started from

Speaker 1: what should the code exactly look like? Specifically for somebody in a position of doing this kind of pipeline. There's some data, it could be more structured than this, it could be just a CSV. And repeatedly, like every week or every day, or when notified by WebSockets, you run the same pipeline on fresh data. uh and bring it into uh a data model that you that you know you need to end up with. Um And I want to apply these Django concepts to the developer in that position and see how far we can get to help developers in that position have better jobs.

Speaker 1: So this is the core of the vision, the develop building from the developer experience. What should be the developer experience of writing that list of steps? that is going to be applied every week or every day or every hour to slightly changing or greatly changing data. I would like the steps to be so declarative and so clear that they're this readable That was the core, the kernel of the of what turned from a library into a framework. And one of those steps. Any one of those steps should be this easy to define. If you have to write a custom step because you're doing something that is unique to the data you're working with. It should be you should be able to write a method, put it where you like in your code,

Speaker 1: and the only thing you have to do for the framework is to decorate the step and tell the framework this is a row step. You're gonna need to give this step data row by row. Great. The framework can do that. I like pandas for some things, but it can be very unreadable when the next developer comes. So I want I would wanted to encourage pandas steps to be wrapped in a step name give be given a purpose. When you have to name a method, you're also often forced as a developer to give it a purpose. So we're calculating the hour average here. We're not um the mean is the important part of this step, not the um group by or it it that's a that's a a means to the uh to the end. Haha. And encourage documentation and encourage testing of these

Speaker 1: specific things. So I mean even when you look at a unit test for something that is a few lines of pandas wrapped in a method name, now the unit tests are additional documentation as well as making sure you don't change what it does. uh by accident. Um and speaking of tests, steps should be this testable. You should be able to just Cons up a tiny little bit of data, not your whole data file. You shouldn't have to go through hoops just to get data to give to your test, to give it to give to your step to test it. It should be you should be able to give the step just what it needs because steps should be targeted like that. That makes a step like this much more reusable. I could reuse combine full name in any data source that had uh

Speaker 1: F N and LN and who cares if it has employee ID or not? We also, Jeff and I, built some um built-in steps for common problems. Um you can check if a field is unique when you get a data file. You can drop duplicates, you can filter rows you do want or don't want And for dealing with structured data when there's substructure in a field like location, having Latin long, you can flatten that if you need to. And that works for even things nested several day layers deep if you've got some JSON record data. But a lot of common tasks that you want to do are m uh

Speaker 1: type and value-oriented. They're not even transformations really so much as casting and renaming. A whole lot of work gets done in a messy data pipeline, renaming columns. And and making sure you didn't import the the wrong column because things got offset. So make sure the minimum value is zero or eighteen and the maximum value is a hundred or 2000, whatever, you know, applying those kinds of things. We thought that should be done declaratively. And again, it's a lot like um uh declaring a models attributes. Being able to say what the name of the column ought to be after the column transformations are done, which allows you to rename from a number of different names.

Speaker 1: If Carta calls it one thing and Fathom calls it another thing, you import both of those, but it's fundamentally the same field, so you can rename them to the same thing with the same declaration So now these things are magic. These things are one line of declarative code that often does several things in one line. Like I've got um each thing separated into a different into a different example, but you can um both have the date column with the date time format to cast it, but also give it a minimum or a maximum value to uh make sure you don't have The wrong date column by accident. And it should just be magic to fix column names and values with spaces around them or capitalization differences.

Speaker 1: Uh I didn't even have an example for that because the column um class just does it. Alright, I've dived into the details of steps and columns, but the framework wraps those in two more concepts, which is a phase and a pipeline. The pipeline handles opening files, pulling data out. It could also handle streams if somebody were to contribute that. and passes data to phases. The phases runs the column logic, runs the step logic, and changes the shape of the data for each step as it needs it. And there's some nice DB debugging out outcomes of that choice we made. But another important choice, and again, this was driven by writing the first code we wrote

Speaker 1: wasn't functional, wasn't working code, it was prototype code of what would this look like. if we had written this to be a great developer experience. And so we were driven by this vision of having a chunk of code that says, we're going to do the blood glucose calculation in this phase. So we can give it a name and we can tell it which columns it needs and we can give it its steps and at the end we've done a useful chunk of work and it's extremely readable. Now a phase doesn't have to be that small But it's often um it's often useful to at least break up some parts of your pipeline. It really helps narrow down the problem. And in our pipeline, in our complicated HR data pipeline, we we

Speaker 1: we built this is this is the pro real prototype of of the phase part of phaser, is that we broke our pipeline into several different steps. Including checkpoint files so that we could look at the data in between steps and see and debug problems very quickly And they we were trying to get the data towards a common internal format that wasn't even database ready, but it was getting closer to database ready. So the final transforms we tended to need to do before it was ready for the database would work for all of our sources. Uh and and we ended up having a lot of code that we could use in between those phases, could reuse it. It's modular. It's amazing. So that means a phase is a lot like a Django model

Speaker 1: because uh the list of columns is very like a field, or like a list of um fields in a Django model. Um the way it collects Um the way we wanted to be inspired also by the way it handles exceptions, which I'll get you. So this this list of things I wanted to be inspired by, we've only checked off one thing so far, but it was the like the big one, the core, the the logic, the the The the real I don't know the the core. I'd have no no other words for it. Um but there's a bunch of other things that I want to move through a little more quickly because getting inspiration from Django for these other things was also really useful.

Speaker 1: Uh run tools. I love these in Django. I made great use of them and started to build some run tools in phaser so that although you could trigger a pipeline from code that doesn't you could trigger it from code that is like a Django um view that triggers an uh a phaser pipeline through a button in the admin code. You can also run it with run tools, which is especially useful when you're developing and debugging. And we could use some more run tools if Phaser grows and gets adopted. We wanted to have exceptional exception handling. uh the magic that you get when you don't even have to handle a does not exist error because it just does the right thing, that kind of magic can re be reproduced in a pipeline.

Speaker 1: There are certain errors where If it's an error, you can just drop the row. That's the right thing to do. Like if it's not one of the employee statuses we're looking for, you can just drop the row. Oh, and one one tip, hot tip, um, hoarding is terrible if you work for data, work on data. You can always go back to older versions, earlier versions, and get what you dropped. Drop early, drop eagerly, drop that data. You have less to work with and less bugs later on if you drop your data early and eagerly. But there are also cases where you find an error that's so bad you want to drop, you want to stop the pipeline and like notify the developer. So being able to handle different things with good exception types was important. Utilities are so important to get consistent in this kind of project.

Speaker 1: We've got a contributor who uh said in her biotech um data team. Every every time she joins a team in in her domain, which is ser seriously this biostat statistics biostatistics and bio and um data gathering in preparation for research and um Drug development. She says every team I joined is 11 different ways of loading and saving CSVs. That's appalling. Having strong utilities that are sorted for suited for this purpose is a is a big benefit. But there's also it's also good to have ways to do things like row numbering. The phaser will throw row numbers onto the columns as it onto the rows as it's working with, and then will

Speaker 1: remove them at the end if you don't want them. But they're there in the checkpoints for debugging. and seeing the difference between um between the data you expected and the data you ended up with. Oh wait, there's we're missing rows between seven and thirteen. I wonder if something happened to drop those rows. Unified handling of null and nan values, the number of bugs I've seen where nans or nulls or nuns were handled differently, and that was the source of the bug. Oh, PTSD around that. The I did try Django Import Export when we were building all of this in my last startup, and I liked it a lot, and we continued to use it, but I ended up stealing some ideas for it and moving more and more code out of it. So Phaser ended up being the or the the phases that we built in that startup.

Speaker 1: were the first thing to run until it was ready for Django Import Export and then Django Import Export would bring it into the database. But I was so inspired by Django Import Export's really powerful diff tool that I outright copied it and had to write a CSV differ for that, which somebody might find useful, now you know about it, to do things like diff the column names Which isn't a native thing you can do. Unless you know you're diffing CSVs and have an idea for what could have changed Okay, so now we've been inspired by a lot more of the great parts of Django. The last two things are extensibility models and where your stuff goes. It's actually surprisingly extensible, although we've done very little work to l

Speaker 1: to explicitly be extensible, it's actually surprisingly extensible already because of being object-oriented. If you define your own columns, if you define your own phases, then you override the behavior you need to override. And the ability to just define new columns ought to handle a lot of um a lot of desire for extensibility. And the whole point is to define your own steps. Even th even though s even if ninety percent can be handled with one-liner built-ins, column declarations, and magic The remaining 10% is is important to your business or your your project. We have a few of the answers for where stuff goes, but only a few. We know where we we have conventions for where pipelines go and for where output goes, working

Speaker 1: working data, the checkpoints between phases, so that they can be diffed when you want to see what happened. Uh and we could use conventions for where tests go. So this is on my list of things to do. And that's gotten us most of the way through our list of of uh some of my favorite things in in Django that ought to feel the same when you're developing a uh data pipeline. We have a few other ideas. We have an issues list and not even everything we want to do is on the issues list And I'm sure if we had um more activity, people would have even more ideas about what it could do. But I know I'm running short on time. So um what I hope I've accomplished, we've accomplished here, I hope that you

Speaker 1: have been reminded how nice it is to start a new Django project or to come to a new Django project that's new to you. but you know where to find those models and views. I don't know if it's possible to convey how messy web development was in 2003, but try to imagine it. Because and enjoy that feeling of gratitude for a moment. Because although we as we're engineers, we know things can always be better. I did like to look back and think how much better things have become. If you have data engineers in in your company, I know a lot of them are super junior in their career. Data engineer is often treated in companies as a place where you land. And then as soon as you actually know some software architecture, whoop, you can be a back-end engineer now. And so the data engineering code base

Speaker 1: suffers from uh every single ex contributor is inexperienced and then they get moved on over time. So that's a big um flaw in some companies data pipelines as a way of as as a as a result of the way they're managing their engineers And I'd love to be involved in building better ideas, although I also would love to see um frameworks for data engineers. go into other languages and other frameworks. If you know people working on beautiful soup, I think it's great, but I think it could be a framework. uh and and and other languages like r um true r is powerful but surely we can do better than than r with uh some modern thinking there. So I I think there's a huge potential for this and I'm

Speaker 1: Uh welcome people's thoughts on that. Maybe we help this guy out a bit. Um The link to this presentation and Phaser and other things can be found via that. I'm easy to find online, Lisa Dissot. You only have to spell my name last name right. That's the uh intelligence test. Low pass filter. Hi pass filter. Any questions?

Speaker 2: Yeah. Ha!

Speaker 1: Of course I didn't test that. Well hang on to that and in five minutes I'll uh make it uh world visible.

Speaker 3: Cool. All right. Any any other questions for Lisa? Short pattern 50.

Speaker 2: Okay.

Speaker 1: I like your t-shirt, by the way.

Speaker 3: Oh, thank you. I was gonna ask uh who's gonna finally win the browser war, uh Mosaic or Netscape, but I was wondering if you had uh so I am uh I'm not a data engineer. I have a data team of brilliant data scientists I work with. And they've settled on a tool called Apache Airflow for doing some of these types of things. And I was wondering if you looked at any of the existing tools and what their shortcomings might have been that led you to develop something like Phaser?

Speaker 1: I looked at a lot of them and I looked at them repeatedly. I looked at them when at the start of our startup and and whether we should start out with a pat something like Airflow. And I looked at DBT and Dagster I looked at them halfway through, like two years have passed, three years have passed. I am now hiring data engineers. I should look at this stuff again. I looked at them again starting Phaser. And I think you do need those tools. Sometimes it's gonna be a yes and. I don't know what the answer would be if you wanted to take something like the structure and framework that uh I'm trying to give with Phaser, but then run it in an Airflow environment because you need the queuing management system that Airflow gives you to keep all of your many high volume data flows moving, which um

Speaker 1: that would be a whole thing you'd have to write that's not part of definitely not part of Phaser. Can those two be married? Maybe. But I think they solve different problems

Speaker 3: I also just wanted to say that Faser is an absolutely awesome name. Pew pew

Speaker 1: I always use Star Trek uh characters in my uh data uh for tests.

Speaker 2: Yeah, I I appreciated the Deep Space Nine references for sure. Um Yeah, any any we still got more time and if there are any other questions for Lisa.

Speaker 4: Thank you. Um I noticed in your dependency graph that you have um I think six people who are dependent on Phaser. Do you know if anyone's using Phaser in production or using it in any of their workflows right now?

Speaker 1: I I don't think so. But if they um but I should take a look at that. I don't didn't know about that tool Um

Speaker 4: it's in the community tab.

Speaker 1: Yeah.

Speaker 2: All right. Well uh let's give uh Lisa a hand and uh