Powering Energy Storage Beyond Excel with Calvin Hendryx-Parker

This video features Calvin Hendryx-Parker at DjangoCon US 2023 in Durham, North Carolina, USA.

Powering Energy Storage Beyond Excel with Calvin Hendryx-Parker
0:37:23
Published November 22, 2023
177 views

Companies with vast data and complex processes must streamline their operations to ensure precision, scalability, and sustainability. In this talk about transitioning from Excel to Django, attendees will discover Django’s superpowers that enabled a national installer of battery energy storage solutions to optimize energy storage configurations, enhance accuracy, ensure reliability and improve quality assurance. Attendees will learn best practices and gain valuable insights for streamlining data-intensive processes and shaping a greener future.

This talk was presented at: https://2023.djangocon.us/talks/powering-energy-storage-beyond-excel/

LINKS:
Follow Calvin Hendryx-Parker 👇
On Twitter: https://twitter.com/calvinhp

Follow DjangCon US 👇
https://fosstodon.org/@djangocon
https://twitter.com/djangocon

Follow DEFNA 👇
https://www.defna.org/

Video production by the presenter and DjangoCon US 2023 volunteers.

Summary

Calvin Hendryx-Parker explains how an energy-storage company moved from complex Excel workbooks to a Django application. The spreadsheets modeled battery degradation, power-delivery commitments, battery “tranches,” pricing, and bills of materials over 20–25-year projects, but became difficult to version, secure, test, and maintain across many customers. He recommends documenting formulas and their valid and invalid inputs, prototyping them in Jupyter with pandas, checking Python results against Excel while accounting for floating-point and precision differences, and then implementing tested business logic in Django with PostgreSQL. The proposed stack includes Django REST Framework, OpenAPI schema generation, JWT authentication, filtering, Next.js, React, Ant Design, and exports to CSV and PDF. He also stresses gradual change management: evaluate, prototype, review, build, and switch, while keeping the new application fast and usable without simply recreating Excel.

Key takeaways

  • Excel is easy to adopt, but versioning, permissions, security, testing, and multi-customer maintenance become serious problems at scale.
  • Document every spreadsheet formula, its source cell, dependencies, and valid and invalid examples before rewriting it.
  • Use Jupyter notebooks and pandas to prototype and validate business logic with stakeholders before building the full Django application.
  • Account for differences in Excel and Python numeric behavior, including precision limits, floating-point arithmetic, and the Decimal library.
  • Django, PostgreSQL, automated tests, and a responsive React-based interface provide a more maintainable foundation than a collection of spreadsheets.
  • Move users gradually through evaluation, prototyping, review, implementation, and migration, preserving useful exports and a fast tabular workflow during the transition.

Summarised automatically from the transcript.

Chapters

  1. 0:00 Energy Storage Modeling Introduction to utility-scale battery storage, degradation, and the sizing problem behind the project.
  2. 3:33 Excel-Based Battery Planning How the company used extensive Excel workbooks to model battery capacity, pricing, and deployment tranches.
  3. 8:11 The Limits of Excel Problems with spreadsheet scaling, version control, security, permissions, and testing.
  4. 11:19 Migration Strategy A measured process for moving users from Excel to a web application through evaluation, prototyping, review, building, and switching.
  5. 13:38 Jupyter Notebook Prototyping Using notebooks to document spreadsheet formulas, import source data, and validate the business logic before building the application.
  6. 17:34 Numerical Precision Differences between Excel and Python floating-point behavior, significant digits, rounding, and the Decimal library.
  7. 21:23 Django Application Architecture Moving validated formulas into Django with deployment tooling, models, data imports, and tested business logic.
  8. 22:55 Frontend Development The Next.js, React, Ant Design, OpenAPI, export, and API choices used to create a fast and familiar user experience.
  9. 26:04 Migration Challenges Keeping the Django application synchronized with changing spreadsheets while handling large tables, long forms, and user adoption.
  10. 28:27 Automated Formula Testing Adding tests to the Django implementation and using them to uncover data-model and JSON compatibility bugs.
  11. 32:28 Questions Audience discussion about spreadsheet change tracking, spreadsheet-like interfaces, and cloud spreadsheet integrations.

Transcript

6,818 words · auto-generated Show

Automatically transcribed, so expect mistakes in names and technical terms.

0:21

Speaker 1: Again, welcome everybody to DjangoCon. I'm excited to be here. My name is Calvin Hendrix Parker. I'm CTO and co-founder of Six Feet Up. We are a Python and AI for good consulting agency. And if you've seen us walking around today, we've been wearing our impactful uh campaign where we've actually been working on 10 impactful projects in 10 years. And the case study for this pro this specific Presentation will actually include some information, not you know all the names have been changed to protect the innocent, uh, about one of our impactful projects, which is around energy storage. And so if you're wanting to follow along, there is a GitHub repository associated to this talk as well If you're just wanting to get the link, it's underneath the six feet up

1:07

Speaker 1: GitHub organization. Just look for the DjangoCon US Powering Energy Storage URL or else that QR code will also take you to the presentation. And the link will be also at the end of the talk. So if you want to refer to it, it'll be there as well. But let's kind of start talking about energy storage project projects specifically. For this specific customer, uh think about your power plants. You know, they they're you know coal generation or hydroelectric or wind or solar. They produce energy, but they need to put energy someplace sometimes. Typically they're putting on the grid, and so you're going into your houses. One of the Big advancements we'll see coming, I think, in the near future is going to be longer term

1:56

Speaker 1: energy storage so that we can go and you know stockpile energy when it's nice and sunny. And then use it throughout the days. But right now we don't have those kind of capacities typically at our power generation points. So this specific customer Actually, it is built some unique and innovative ways to store short-term power storage. And so every project in this case is a multi-step process of figuring out What that sizing looks like. These calculations typically have to account for 20 to 25 year life cycle on the batteries that they may put in place at these power generation points. So these are not like battery gener battery systems you would put at your home. These are much bigger systems that are meant for kind of backing up the power grid itself.

2:44

Speaker 1: So it requires a lot of calculations to figure out how these batteries perform over time and understanding how they're gonna actually fulfill their commitments to the power generation companies that are buying these kinds of hardware. These kinds of things need to be able to predict degradation over time, you know, understanding that those kinds of curves, you know, and calculate what are called tranches. So tranches are like slices. It's a French word for slice. And so these are basically how they deploy more and more batteries uh over time. You if you wanted to guarantee a certain delivery power, you wouldn't want to uh If because batteries the way they degrade degrade degrade over time, you wouldn't want to put all that pa battery in at once because it'd be too expensive. So if you were saying over 25 years we guarantee a minimum level of

3:33

Speaker 1: power battery capacity, you'll want to know when you need to lay in you know the next tranches of batteries as you go. So that's kind of what this is all about, is like new hardware needs to be added, but you want to keep the system always operating above the minimum requirements. And so you don't want to add too much hardware at any one point because that's just too expensive. Plus the price per watt for battery storage As you know, a technology, everything gets cheaper, you know, every year, you know, in the future. So to do this, they built a slew of Excel Sheets as a startup, you want to go move quick. So you know basically that they built you know a whole array of Excel workbooks that b

4:18

Speaker 1: you know um like simulated all these various calculations for you know the kinds of uh the amount of battery that would capacity would be needed for any one specific project, the pricing for those kinds of things, the the bill of materials that would be required. And they had just inc incredible Excel workbooks put together to do these kinds of calculations all all over um across the various projects they're doing. But for using Excel in this case, I mean obviously it's it's easy to get started with Excel. It's you know most people if you've signed up for Office 365, it's included. Uh you've got some kind of a spreadsheet on your computer when you shipping when you pull them out of the box. Whether it'd be Excel or some other spreadsheet technology, but they're easy to get started with. It's super easy to bootstrap into, and especially if you're a small company of just a handful of people.

5:06

Speaker 1: it's easy to share. Like you put it on a shared drive and away you go. People are now calculating battery capacity storages and running their businesses on Excel because it was very easy to get started. The friction of the first movement is really, really easy to do. And then the you know you've got a rich ecosystem of tools built in, uh so rich sets of formulas, like fast calculations. Uh You can basically use references across multiple sheets, even multiple workbooks. You can include references to external data. And even now you can use Python in the macros. uh to build even more exotic behaviors into your excel spreadsheets. But then you end up with a problem. is you have tables and tables and tables and tables uh

5:53

Speaker 1: that get really really hard to manage. Uh but that's not the only problem. Uh I think if you look at the real kind of why not to use Excel um arguments though, is that Excel is great for a handful of projects, but as soon as you get into, say, thousands of customers, and maybe you're using an Excel spreadsheet per customer. You can see where the scale of managing the versioning of the files can really become a big roadblock to productivity. And another problem is what version are you working on? You could have 10 versions of this live across 100 customers and not in the have mismatches in the formulas and basically end up with uh potential disasters on your hands. And then versioning with file name patterns. Um, us as developers have been long accustomed to using source control and we love it.

6:40

Speaker 1: Uh, and using you know, conventions like tagging and and naming versioning is uh is all fun and good. But a lot of people who are using Excel don't have kind of that saviness with those tools. I mean you could check it in. But I think a lot of people just default into the you know my excel sheet. latest and you know or some date you know timestamp in there or v1, v2, v3. So you can imagine where that can also get very hairy and tough to manage. But uh another a bigger issue may even be if what you're putting into these Excel spreadsheets is kind of your secret sauce of how you operate your business. your one email forward away from or one in in you know errant share on your you know Dropbox or whatever file sharing service you're using away from sharing out your kind

7:26

Speaker 1: you know corporate secrets with the rest of the world So securing that business logic, you know, could be really an important aspect of why not to use Excel in this case. So another problem is that there's not a lot of access controls on Excel spreadsheets. So I've seen some pretty fancy ones. But there's a lot of hacks to get around, you know, protecting content on sheets. But everyone basically has the ability to change the content on any cell in any formula. There's no real permission checks going on here. And everyone can see everything. So there's and you know the last one is probably the most important one. No tests. If you're building these big extensive formulas, wouldn't it be nice to be able to pass a test case full of like edge case you know, type situations, you know, so you can understand the

8:11

Speaker 1: any regressions in your formulas or if new edge cases actually bring up uh logic that doesn't make any sense for that specific use case. Excel really doesn't give you an option for doing any kind of testing there. So what do you do instead? Well of course, we're at a Django conference. I'm gonna say Django. I shouldn't have to do a lot of hard selling here to tell people why Django is an amazing product to use, but Django does give you everything you need to get started fast. And including things like the Django admin. That can kind of be our You know, rough sketching of you know a easy way to get a UI on top of some models and some data to get things going. But we want to use Django because we actually care about our data and we want to get um you know we care about our data and our business logic first. Uh it's it's a very stable platform.

8:57

Speaker 1: We've you know been around for more than 10 years. You know, Python is obviously a highly has a very high security score. So when it comes to the platform itself, we feel like it's a reliable and secure platform to start building upon. And it's easily extendable with add-ons. So there's a great ecosystem of people out there building software for Django itself. And I'll show off some of the stack we used as part of this project. And then you get the ability to do things like scaling. You know, the problem of having thousands of customers can easily be scaled across hundreds of servers. So the ability to scale horizontally or use tools like Celery for long-running tasks. And then you get the right tests. So now you can actually take all that logic and and business understanding and wrap it in sets of test cases that can now be run on an automated

9:46

Speaker 1: process. So every time someone makes a change to those. They can be tested against a set of known good, uh understood test cases. So looking at the stack that we have uh put in place for this specific project. Obviously we're using the Django backend , but we also are layering into that Postgres. We do like Postgres because it's been robust, reliable, scalable. And it just has worked well for us over the I can't remember last time we used another one other than Postgres uh for this kind of relational uh type of a problem. Adding into that Django Rust framework, what we'll want to do is build a rich UI on top of this so that people still have that speed and flexibility or ease of use. of what they've become accustomed to with Excel, and kind of I'll make more note of that later, is

10:33

Speaker 1: you don't want to recruit recreate Excel in the web browser necessarily for people, but you do want to make sure you're building a fast and easy to use app. Um so see things like you know using HTMX. Uh in this case we'll be using React on the front end. So having Django Rust framework, which I think is you know the kind of default Rust framework that we can layer right on top of Django is great. Adding in DRF spectacular for open AI or Open API 3 schema generation. So those who are building the front end, this is really key to have that kind of documentation quickly available to them. Uh using Django Rest framework, you know, simple JWT, so having authentication with uh JSON Web Tokens as a nice plugin for Django. And then Django filter is a nice add-on for Django to filter down query sets. you know, based on

11:19

Speaker 1: the models fields. So making it very easy to write those kinds of queries. So how do you take this the next step now? Then we kind of got a stack of back-end software. And uh oh, I mixed missed the one part in there, which is the Next. js uh UI, so the React based UI. So we're going to talk through a little bit of that transition because everyone loves change management. That's a favorite topic of mine. Ask anyone who works at Six Feet Up how good I am at it. Uh Carol's laughing because I I uh sometimes I just pull pull the trigger on things maybe a little too quick. Uh but I think you should take this and keep this in mind as you're trying to move people from you know, I'll take Excel out of my cold dead hands uh type mode

12:04

Speaker 1: to you know some kind of a new web application, uh you're gonna want to make sure you do that in a in a very uh measured pace. Uh so the optimal stages we found, you know, it's gonna be this kind of evaluate prototype. review, build, and then switch. And let's kind of talk a little bit through each of those pieces specifically. In that evaluation stage, you're going to want to gather every formula you can find inside of those spreadsheets. This may seem obvious, but actually documenting them is probably your top priority. Understanding what people have been, you know, kind of encoding into the the the opaqueness that is a spreadsheet is uh kind of tricky. You know so every formula needs to have you know valid and invalid examples. So when we get time to go to the testing phase, we have you know, good data about what is good data in there and what is bad data in there, so we can actually write our test cases

12:53

Speaker 1: uh for what was implemented in Django. And then add information about the file versioning, cell locations, precedence and dependence. And then actually one of the key bits here is you know use like a Jupyter notebook. to maybe build out and prototype and test some of these things uh so you can quickly make changes and iterate with that with whoever your your client is, whether it's an internal client or an external client. I will show off that to you here. So You know, using uh Jupyter notebooks, we can actually actually let me show you the demo of that. Pull that up here. So in that repository that I linked to at the very beginning, I have a Jupyter notebook in there.

13:38

Speaker 1: Down inside of there is going to be, well, actually I'll show you the spreadsheet first because it may not be obvious what's going on. Here we are. So this is kind of a mocked up, again, like I said, change to protect the innocent uh type spreadsheet. But it's gonna have some things that are dealing with battery degradation and some of the sample formulas that are in here, for example. You can see here this starts summing, no, one of these starts summing the losses of incremental tranche DC output. These are also don't ask me questions about battery technology. This is not my area of expertise. Ask me questions about the Django Bart. That's fine.

14:25

Speaker 1: I don't know much about the battery pieces here. But I do know was that we were inside of here calculating like these output losses. Define the formulas. But this this also this spreadsheet is list is linked in that GitHub repository. So you can kind of start seeing what we did there. In the Python notebook, we would create uh doc strings with the cell like the headers and like the the column uh formulas We would just put those right in the doc strings. So we knew for this operation, this is where that came from. It may even be some additional things you could do here would be even put the location like workbook. cell location, like maybe starting cell location, so it's easy to find. Uh because you I saw there I had trouble trying to find where incremental tranche DC output was coming from.

15:13

Speaker 1: In this case, that's actually a static uh constant. So a lot of times in your notebooks you're gonna have tables of lookup data constants that are important for the operations you're doing. So I'll show in this demo like importing that kind of content into Django and then being able to use it for these kinds of test cases. So you can see here a couple of these where you know this is just a sum. That one's just a set of years. I think the rest that's the only one big example where we got some formulas is going to be this incremental tranche DC output. So documenting that in the Jupyter Notebook, being able to run through it, actually I'll I'll run through this notebook real quick. So we are Another nice thing about using a Jupyter notebook is you can quickly and easily import Excel files into

16:00

Speaker 1: uh pandas and do operations on them. So we were actually creating a data frame. And then converting that into data that's more usable by Python. And I'll talk a little more about that too. So here we're opening up that sample database or sample Excel spreadsheet. And bringing that in as our specifications. You can then see we looped over and are looking at just making sure the data makes sense So it's another nice thing with Jupyter Notebooks is you get that real-time feedback as opposed to like push some code, check it, see if it worked. Uh Jupyter Notebooks gives you that real-time uh aspect of it. I'll get down here to the very bottom. This one's bringing in uh specific constants that I talked about. The other one was bringing in data. This one's bringing in some of the constants We'll set those up as constants that we'll use later.

16:47

Speaker 1: So here's again that incremental trunch DC output. It's grabbing that sheet data from some specific areas of the notebook itself, or the spreadsheet itself. And then we bring in and uh I'll go down here at the very bottom. What we'll do is you know cut once you've got your formulas built up inside of Jupyter Notebooks. You can then compare with the Excel spreadsheet because you have the Excel spreadsheet data loaded up inside of here. And I'll show that right here. So at the very bottom here, we're looking at what Python did and what Excel did. So you've got Excel and Python for what values we expect, so kind of quick visual inspection. And you'll see already we're seeing a rounding error because of number of digits of precision. They're all the same except for that very first one.

17:34

Speaker 1: You can I and does it I who here is an Excel wizard? How many significant digits of precision does Excel give you? I had to look it up because I was like curious to know what this is all about. And it's 15, it turns out, which leads to interesting things. If the people who are doing work in your Excel spreadsheets are Dependent on some of these limitations of Excel and don't recognize that these exist, you'll obviously end up with odd or different results. Here's an example. I'm going to show a bad example of bad math. Well, bad math. It's doing math based on its current limitations of the platforms in each each platform, Excel and Python. This is the example of if you take one and you divide it by 9,000.

18:19

Speaker 1: Excel only gives you 15 digits of significance, which is the 15 ones you see there, and then the rest is filled in with zeros. If you take one and add it to that result, so this formula here is actually taking this number one and adding it to the result of the previous one, you'll see you get one and less Significant well less ones in the uh decimal part of that because it's again only 15 digits of significance. The one is now your first digit of significance there, to here, which is 15 digits. And then if you go to subtract, you're obviously going to get a different answer. This number and this number should be the same because all I've done is add one to one over 9,000 and subtract one. from one over nine thousand, but I have a different uh

19:05

Speaker 1: rounding there because of the number of significant digits. Uh that's that's always fun, right? Let's show you something with Python doing that. Actually we've got it right here. If you do this in Python, like 1. 1 plus 2. 2, why did that happen? Does anybody know? Because it's floats, right? I'm using floating point math in Python, which so 1. 1 plus 2 plus 2, you would have expected 3 plus 3. Instead you're getting three plus 3. 3 plus basically zero afterwards because of the way flo floating point math works in Python. This is not a limitation of Python. This is actually just how the

19:51

Speaker 1: floating point specification has been derived over time and then how the processors actually work behind the scenes to handle floating point numbers. What's what's the real answer? The real answer is you use the decimal library. So from And so if you do this instead, one point one plus 2. 2. You now have a 3. 3. And with a specific uh number of digits of significance here, if I wanted additional uh accuracy or significant digits in my math I can actually do that and you'll see that I get 3. 30 because there's three significant digits in that output.

20:38

Speaker 1: The trailing zeros are significant in uh in math. I I had to go back and learn a bunch of math to do all this kind of projects. I, you know, figured at my age I didn't have to go learn rudimentary math, but now all those classes my kids were talking about when they came home from school make sense. And like, cool, I actually understand what's going on here. So keeping that in mind, it's important to understand as you go through and do these various transitions from Excel into Python. You're gonna have to keep in mind those significant digits, floating point limitations, the fact that math inside of Excel can round in different ways than math inside of Python are gonna make things work differently for you. And that's why You've got this rounding here for that second entry. It's because of the floating point math. Actually, it's because of the number of digits of significance going

21:23

Speaker 1: on here because that math actually was done in with decimals, I believe. If I looked that up in some place here. No. You can look at the code and kind of see what's going on there. So that's one of the big recommendations there. Here we go. So the nice thing about doing this in Jupyter notebooks is you don't have to set up a full web stack to be able to start playing with it and actually get buy in with whoever your your client is, whether it's an internal party or not. You don't you basically can set this up quickly, recreate that business logic. Let them play with it because you can hand them uh uh outputs from the Jupyter notebooks as CSV or Excel files so they can actually validate that that's actually the ex what they got as what they uh what they expected and give them and they can give back early feedback

22:09

Speaker 1: So this is again that change management of giving them the ability to give early feedback often is really nice. So the next step after kind of going through the Jupyter Notebook piece is going to be You know, building an actual Django application. And now we're going to transition away from Jupyter. Now we've kind of validated that those business logic formulas are indeed what we wanted. And there's a lot of great ways to get started. There's you know the kind of the quick start for Django. Uh at Six Feet Up, we maintain our own cookie cutter template called Sixy Django. It's just an opinionated way that we look at deploying uh Django, which includes things like Terraform. Uh it's kind of maybe targeted more towards Amazon. uh looking at doing things like Kubernetes for local development and then

22:55

Speaker 1: basically hopefully gives you a fast development experience and an easy deployment. Well easy deployment. We'll see uh how easy it is. But it should be barely straightforward because we've templated it out that way. Now on the front end, this is where I mentioned some of the front-end technologies. We chose to use Next. js combined with Ant design. And I actually wasn't very familiar with Ant Design before we started this project. And actually not super familiar with Next. js. We've been doing mostly React or Angular type work. But Next. js, if you're not familiar with it, I'd recommend you take a look. It's basically an opinionated uh set of libraries around React, which are nice. Um kind of the most popular React framework that's out there. You'd be like, wait a minute, I thought React itself was a framework, but uh there's frameworks on top of frameworks.

23:41

Speaker 1: So there you go. But it does have support for client and server-side rendering. You get dynamic HTML streaming, file system routing, supports server and client-side data fetching. Which is kind of nice because you could have data sources living in the same cluster and actually have the server fetch them, render them server-side, and then give you back kind of the rendered results. So you can actually make the application more speedy and snappy doing some things on the server versus on the front end. And we can build you know internal API endpoints to securely connect to you know third-party APIs. So if you've got internal only APIs that are not available outside the firewall, uh it's good support for that kind of thing. Uh and Design is another is a basically a React component library.

24:27

Speaker 1: Really beautiful, easy to use uh library. I think one of the nice things about choosing Ant design is it actually has a charting uh extensions or uh add-on uh part up to it. So if you do ant design pro, which was a confusing naming to me because I thought Pro, okay, how much does it cost? Uh it's it's still open source. It's It's just the pro version includes more enterprisey dashboardy things for professionals who are building applications. It's not a paid version of the open thing. It's actually just a professional version of the already open thing. But it has really beautiful charting built in, so you're not bringing in yet another charting library. You can kind of stay inside the ecosystem of the ant design and have a common set of UI components. uh that we use on a regular basis, and so the whole team's already familiar with them.

25:15

Speaker 1: And it's easy customizing and theming. You mentioned open API. as being important here. If you've got a team working on the front end, it's really nice to be able to give them documentation for the API on the back end even before that is finished. And so having a schema generated from the open API docs, like with DRS spectacular, is nice because the front end people can get started before the back end is actually technically complete. So you can kind of mock up a lot of things. and actually work uh you know at building the front end faster instead of waiting for you know being kind of roadblocked by the back end only. Another thing we you know built in for them was things like the uh PDF and CSV exports. It's nice to be able to give you know stakeholders a report in a PDF format or if you're still transitioning from the Excel version of the project into the Django or web

26:04

Speaker 1: app version of the project. Some people may want to have those CSV exports so they can maintain parity and kind of run in parallel for a little while until they've fully switched over into the Django version of this. All right. What kind of challenges did we actually run into? Well, as you can imagine, as you you're working, people aren't just going to stop what they're doing on the Excel sheet. Uh and you can't obviously deliver full-blown version of this application in in a very short amount of time, depending on the scale of our scope of it. So people are constantly updating and optimizing their Excel spreadsheets. You know, every new version of that Excel spreadsheet needs to be evaluated for inclusion into the the refactor to the Django version of the app. You know, sometimes small changes in formulas can go without notice because they're so obscured behind the cells that they are

26:55

Speaker 1: coded or programmed into. So making sure you're kind of staying in sync with all the changes that the people, business folks using Excel are doing with your version of the app so that you can kind of start using it along the way is tricky because some people can change things out for an Aetheo. And then you know constant change of business requirements is it's definitely easier to do in Excel than it is in code. So there's a challenge you're gonna have to work through is how do you give business users that same flexibility they had before of Oh well, I just needed to change this one formula slightly. Well, that requires maybe a code change. Or if you wanted to build a framework inside of the Django for documenting those formulas or you know somehow uh putting the formulas as part of the data in the database. Something to consider, but every ch

27:40

Speaker 1: every one of those instances is going to be probably unique to your situation. I like having the code as Python code on the file system for things like testing, which is kind of nice. And from the UI and front end. the you know there's there's massive amounts of just tabular data um that you have to build a display. So thinking through the usability of that, the how fast it loads. uh giving them a fast experience. If you didn't see Chris's talk from yesterday about HTMX , that speed is really, really important to getting adoption. Because if they feel like this is A sub-par experience from the zippiness of Excel, uh they may not be as as excited to adopt this platform Another big challenge obviously is very long forms.

28:27

Speaker 1: If people are entering a lot of data into those Excel sheets to get things done, those forms can get really, really cumbersome and you don't want to be recreating a web version of Excel. One thing I did want to show Well, before I go here, was the another the one of the big benefits I mentioned was using um Python was having tests. So if I've got my models, so we've gone from the Jupyter Notebook version of the code, brought that in, and there's a couple different approaches you can have to put where to put those formulas and how they get tested. You could use FAT models where all the logic and the functions on the or methods on each of the models contain those functions or the formulas from the Excel spreadsheet. Or you could include them in like say utility helpers

29:13

Speaker 1: that are easier to test because maybe you don't need the full database available to you to actually run the unit tests in the example code on them on the project. This one has the formulas right inside the model. And so moving the doc strings over, keeping the formulas there so that there's clear documentation about where this came from. also now allows me to write tests. If you all are using, I think VS Code and PyCharm both support the Codium AI add-on. Which is kind of nice because you can see this little button here that says test this method. So you literally click it, and that's what I did earlier to produce this set of tests. So it can start building up test dummy data that you can now test inside of your Django application. So if you know what those good test cases or good use cases of data look like and the bad ones look like.

30:03

Speaker 1: you can start building those in here and and that codium plugin actually kind of nice because it'll recommend you know five six seven kind of space Use cases to get your bases covered. As I was playing with this earlier today, I recognized there was already a bug in the Django implementation of this code because we're importing the spreadsheet. Actually, I'll show that as well. We have a uh management command in that repository that will grab the uh sheet, the Excel spreadsheet. And do basically what we're doing in the Jupyter Notebook, which is pull that data in, populate some constants. This is where the error actually lives. And if I show you on the web browser.

30:53

Speaker 1: Refresh this page. Or no, I'll make it bigger so you guys can see it. We just had that bring in that constant data into a single model. And then we were using the JSON field for some of that like like larger data that we would want to use in some of these calculations. There's a problem here. It's like we were using integers as keys inside the Excel spreadsheet for a number of years. Uh JSON, according to the specification, does not support integers as keys. It's like gotta be a string. So it got magically transformed into a string. Through writing the test cases, actually I found that. So I came back here, and if we run the tests. Which is nice because the tests run fast, it imports the data.

31:41

Speaker 1: And you can see in here that uh, well, actually I started fixing that problem and got into the next problem, which was my test data didn't match the sample data. So now you need to start adjusting the sample data that Codium produced for me. But I actually have failing tests that can now be fixed, which is a great head start opposed to Excel, which has no tests at all. So just there's some example code that will be shared as part of the that repo as well. Oops. There we go. Now, if you have the luxury, uh you should use Django from the beginning if you can. Uh but you may not always, because a lot of these things are scrappy startups or a business unit that just wanted to get going. And so they're gonna use Excel to get started and you're gonna have to deal with it.

32:28

Speaker 1: Uh but transition to Django as early as possible because it's gonna give you that a lot of benefit uh for multi-user environments, having that sharing and scaling. It's just definitely the way to go here for using that. Now, if there's any questions, oh wait, I've got one more slide over here. Which is again that that link to uh there's a blog post on the 60 website that was about this project specifically. Goes into maybe a little more detail in some specific areas. And then that's the link again to the repository for this project. And you can always find me in all the various places. So if anybody's got questions, I think we have a few minutes left still for questions. And if we can use the microphone. Questions?

33:09

Speaker 2: Yeah, thank you. You talked about the challenge of having working in parallel while people were changing the Excel files and having to like update the Python code. So GitHub isn't able to compare Excel files. How did you capture what the changes were when they happened?

33:30

Speaker 1: That's a a really good question. I didn't work 100% on that project when they were doing this. So I just know that they kept they were responsible for the client was responsible for sending us updates to the Excel file with like a change log. And it may be common in actually in your org. that though there's a worksheet page at the very beginning that kind of notes the version history of that file. So as long as everyone is um like very consistent with how they track those changes and log those on maybe of a worksheet in the front that gives you like date, who did it, and maybe some description of the change, like commit log for an Excel sheet. That's gonna be your it's still very, very manual when it comes to tracking that change.

34:08

Speaker 2: That sounds really hairy. The other question uh that I have for you is if you're interested in energy at all, are you following any like podcasts or anything like that that you particularly like about energy policy, about batteries, about all this stuff?

34:24

Speaker 1: I I am not on any about that, but I I would love to participate in those. Oh question over here.

34:38

Speaker 3: So when you've uh you've got a bunch of data in it uh you know from Excel, you've got the people using a Django system now, um maybe they're comfortable in you know the Django admin, maybe they're not. But a lot of times when I've dealt with similar problems, I get feedback that they want to be able to see the data in a big table like Excel.

34:56

Speaker 1: Yeah.

34:56

Speaker 3: Are there any admin plugins or tools to quickly like add that UI element so they can see a bunch of tabular data to Visually inspect it.

35:04

Speaker 1: We have I'll have to get you what plugin. There was a React like plugin we liked for AG grid, I think is what they were using for the kind of an Excel-like editing experience. Again, keep away from it if you can, but if they're really insisting on having that experience, uh you obviously don't want to hand them the Django admin and have them just clicking through that. But is there are some nice React components for doing spreadsheet-like things that you can't even make editable?

35:32

Speaker 3: You said AG, what was the name of it?

35:34

Speaker 1: AG Grid. I believe it was the one we used um for another project where they had exactly that concern. They're like, we just want a big table to edit. Like, ugh. Okay. At least the formulas are still testable at that point. Like Any other questions?

35:52

Speaker 4: One thing you can do is to wire up uh Django to talk to Google Sheets API or Microsoft API and use their cloud storage as the data backing for your Django app instead of importing the data directly. Seems like that would solve some problems and open up up another class of problems?

36:14

Speaker 1: It did, especially on this case. Um there was a we couldn't even use the Excel online version of the app. They were strictly using the the the desktop Excel Excel application because of some limitations. Like you had to have some something they were doing in their formulas didn't allow us to do to even do that. But I think that it might be a good option. You're gonna deal with a lot of like synchronization and you know potential data consistency issues as you go back and forth between Django and and Google Sheets or even Microsoft Excel Online, I think. Awesome. We have any more questions? Well, thank you all very much. I'm again excited to be here. And if you see me in the hall, feel free to ask me questions.

36:59

Speaker 1: Happy to chat about this or the lightning talks or best or any other thing I've I've talked about. If you've got stories about going from Excel to Django and I'd love to share them as horror stories, I'd love to hear them. Um so come find me afterwards. Thanks again.

Questions this talk answers

How do battery tranches help plan long-term energy storage projects?

Tranches are staged additions of battery capacity. By modeling battery degradation over a 20–25 year lifecycle, operators can add capacity when needed to stay above a guaranteed minimum without buying all the hardware upfront.

Discussed at 2:44

Why is Excel a problem for scaling energy storage calculations?

Excel is easy to start with, but it becomes difficult to manage across thousands of customers and many file versions. It also provides weak access control, exposes business logic, and offers no built-in way to test complex formulas for regressions and edge cases.

Discussed at 5:53

What Django stack can replace an Excel-based business tool?

The example uses Django with PostgreSQL, Django REST Framework, JWT authentication, filtering, OpenAPI schema generation, and a Next.js/React frontend with Ant Design. The application can also use Celery for long-running work and provide CSV or PDF exports during the transition.

Discussed at 9:46

How should you migrate complex Excel formulas to a Django application?

First document every formula, including valid and invalid examples, cell locations, dependencies, and version information. Then prototype and validate the logic in a Jupyter notebook before implementing it in Django and wrapping it in automated tests.

Discussed at 12:04

How can Jupyter notebooks help replace Excel business logic?

A notebook can import Excel data with pandas, turn it into usable Python data, document the source formulas, and compare Python’s results with Excel’s outputs. This lets stakeholders review and validate the logic before a full web application is built.

Discussed at 16:00

Why do Excel and Python sometimes produce different numerical results?

The platforms can use different precision and floating-point behavior. Excel typically limits values to 15 significant digits, while Python’s floating-point arithmetic can introduce representation errors; Python’s `decimal` library can provide controlled precision when exact results matter.

Discussed at 17:34

How do you add automated tests to formulas moved from Excel into Django?

The formulas can live on Django models or in separately testable utility functions, with documentation carried over in docstrings. Known-good and known-bad examples become test data, which can expose issues such as JSON key conversions and mismatches in imported sample data.

Discussed at 29:13

How should teams track Excel changes while an Excel-to-Django migration is happening?

The client in this project sent updated spreadsheets with a manual change log. A worksheet at the front of the file can record the date, author, and description of each change, although the speaker notes that this remains a very manual process.

Discussed at 33:30

How can a Django app give users an Excel-like table for viewing and editing data?

The speaker recommends using a React spreadsheet-style component such as AG Grid when users insist on a large editable table. He cautions against recreating Excel unnecessarily, but notes that keeping the formulas in code still makes them testable.

Discussed at 35:04

Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.

More videos by Calvin Hendryx-Parker

More videos from DjangoCon US