Do the holes in Swiss cheese leak abstractions?

Speaker 1: Hi , I'm Tim. I work for Kraken Technologies. We're a sponsor. Please do talk to my colleagues at the booth out there. They have in fact arrived. You might know me from attendance at Django conferences. This is actually my seventh DjangoCon Europe, but the first time speaking. and Python conferences uh such as at Python PyCon UK and Cardiflies here where this picture was taken. Notice the matching shirt. Or in my previous career attending sysadmins conferences such as ten years ago in Washington, DC, wearing a panda costume. But these days I work with octopuses,

Speaker 1: specifically this guy. This is Constantine. I see somebody holding one up in the audience. The mascot of Octopus Energy. Octopus Energy is an energy retailer operating in eight countries at the moment and part of the Octopus Energy Group. The Octopus Energy Retail Operation runs on the Kraken software platform, which is also licensed to a number of other energy companies around the world. And if you'd like to know more about Kraken in general, I suggest watching Daniel Roy Greenfeld's keynote from DjangoCon Europe last year or Fred Jaeger's keynote from 2021, which goes into uh the Kraken architecture in a great deal more detail. But today we're going to talk about

Speaker 1: this, which is the secret source behind Kraken. It's the support site. We call our customer support people energy specialists, and they use the support site to do all of their customer support tasks. This page is an account page, the account of the customer Tom Sawyer, who doesn't exist. This is the test site from Octopus Energy New Zealand, I think. It's the entry point for energy specialists to do whatever they need for the customer. And behind this page are two of the core Django models in Kraken, accounts and users. And a lot else as well, but those two in particular.

Speaker 1: The support side is critical. So if something happens to it, such as a a server error or or page errors like this, or enough 500 errors that the monitoring shows severe disruptions, and the response time goes through the roof , then I am now a sad panda, and the energy specialists can't help any customers. Now that's what happened during an incident in August 2022 that I'll be talking about today. In the rest of this talk, I'll be explaining the two ideas from the slightly wacky title, Leaky Abstractions and the Swiss cheese model of outages or accidents. Then we look at the case study of this particular outage.

Speaker 1: And finally we'll think about how we might fill in the holes of leaking abstractions. So, what is an abstraction? An abstraction is a simplification of something much more complicated that is going on under the covers. And the value of an abstraction is in the details that it hides. But abstractions can be leaky. So this is Joel Spolsky, who you might know as the co-founder of Stack Overflow in 2008. Now it turns out that he Did a bunch of stuff before then, including in 2002, came up with the term of leaky abstractions. And he has a law, the law of leaky abstractions.

Speaker 1: All non-trivial abstractions to some degree are leaky. Now what does leaky mean here? It essentially means that the purpose of the abstraction, hiding the detail , the the abstraction fails. The details that it was hiding actually become apparent in some significant way. But if all abstractions are leaky like Joel claims, then why do we use them? We have so much detail that's relevant to our work as software developers that we can't pay attention to all of it all the time. So we relied on abstractions to hide some of the detail. What are some examples of leaky abstractions? Relying on Joel's examples again.

Speaker 1: TCP guarantees that your message will arrive. That's the abstraction? The reality? No, it doesn't actually. What happens if you have a broken network cable? No amount of network fanciness will deal with the fact that there's no no electrons getting through anymore. Another example is SQL abstracts away the procedural steps that are needed to query a database. And you could ignore that until query planners generate queries with pathological performance, which they can do from time to time. So that's one of the concepts from my talk title. Moving on to the next one.

Speaker 1: The Swiss cheese model of accident causation was devised in the late 1980s by James T. Reason, which is a great name, who was a psychology professor at the University of Manchester. It's used in aviation safety, COVID-19, layered security, and many other areas. In this model, as we've got illustrated here, The red arrows, and I think you can see at the top left, there are actually three of them, they represent threats. And each slice of the cheese represents an opportunity to mitigate the risk of a threat becoming a reality And the holes represent failures. So you can see that one threat, the one at the bottom there, has managed to find a straight line of holes through all of the slices, and so that threat has become a reality.

Speaker 1: Now, this model, as originally described, includes both active and latent failures. So active failures are things like unsafe actions that are taken by somebody And latent failures are conditions that exist that contribute to enabling the accident and include various human factors And the key point of this model is for an incident to occur, accident, outage, whatever thing you're talking about, the holes in the Swiss cheese need to line up just right. Or turning that on its head If any one of those holes got moved or covered up, you don't have an accident. But if you do get an accident, it means that all of the mitigations that you had in place have failed.

Speaker 1: So one interpretation of this Swiss cheese model applied to software development considers all the processes and practices that we engage in that help prevent the code that we write from creating outages. Also, it helps create ensure that the code does what we want it to do. An outage is obviously something we don't want it to do. The first four slices here, coding conventions and linting. code review, the test suite and test or staging environments would ideally prevent problematic code from being merged at all. But if the code does make it through this first set of checks, the last slice, which I've called health checks and monitoring, would hopefully identify a problem before it goes live in production.

Speaker 1: So it you know it would fail a final health check. or before the impact of the issue has grown too large. It's starting to turn up on monitoring, but you you catch it early. There's another interpretation of the Swiss cheese model , which uses the slices to represent a sequence of things that go wrong leading to an outage, and we'll see an example of that later. Now, before we look at the details of the outage, let's look at the relevant bits of the Kraken architecture And this is really only the relevant bits. There's so much so much else going on which we're not going to include. So this is the uh Kraken architecture as it's deployed.

Speaker 1: On the top there we've got AWS EC2 instances, and there are two types of instances we care about here, and we're showing just a single one of each type, whereas in reality we'd have a pool of them. The web instance, the one on the left, is configured as the support site server. So that's the server that was generating that page we saw earlier. And that serves web requests. The worker instance on the right runs cron jobs and also the salary service. The worker instance also applies database migrations when a new version of Kraken is deployed. And during that deployment, All instances currently running the previous version of the code are replaced with

Speaker 1: new instances running the new version. Both of these instance types have the Kraken code running on top of Django and the Django ORM. At the bottom, there are two Postgres database instances provided by AWS RDS. The primary on the left, which is used for writing and also reading, and the replica for reading only. Both the web and the worker instances route Database queries to either the primary or the replica with write requests going to the primary and read requests generally going to the replica. And we use a Django database router to handle this. All right, on to the fun.

Speaker 1: Here is what was supposed to happen. Kraken does the thing the the kind of thing systems built with Django do. It runs management commands. applies database migrations, services, web requests. So here we've got an example of this particular management command which was run by cron. It's supposed to query the database. And then for each of the rows it gets back, makes an external API call, and then terminate. That's a fairly normal thing to do. When we deploy a code with a migration, we expect that the migration will acquire any locks it needs on database tables, apply its migration, and then release the locks. as well as all the other code deployment stuff.

Speaker 1: And this is a Django app. We service web requests. We'll have Users, in this case the users are actually the support staff, the uh energy specialists, making requests of the web of the support site, and those requests will query the database and return responses All it all extremely straightforward. Spoiler alert, that's not what happened. Now here's what actually happened. I'm not going to build up the suspense or add any mystery. I'm just going to tell you each thing that didn't quite go right so we can then concentrate on why it didn't quite go right And we'll go into more detail of each of these points a bit later on. So Cron ran the Django management command, that's fine.

Speaker 1: It actually connected to the primary database, not the replica. And it opened a transaction. And then it read the rows from the user table. That's what we expected. And then it made some API, external API calls, and those were actually quite slow. And it got terminated by a deployment. Because when a deployment comes along, you've got a bit of time to finish what you're doing, but if you take too long, you're out. And it exited ungracefully. And in this instance, that meant it didn't close its TCP connection to the database. Which meant that the transaction it had opened earlier remained open and all the locks it had acquired were still held. So we've set up

Speaker 1: a bad situation. Part two. We deployed some code which had a migration. And it tried to acquire an exclusive lock on some database tables, including the user table. It couldn't get the lock because it was waiting for read locks to be released, so it joined the Postgres lock queue, which is essentially a queue of things they come in an order that are waiting to acquire the locks. and they will eventually acquire them in the order they arrived in the queue. And it waited Meanwhile, the rest of the world is carrying on with its business. Users mate support cyber reb requests, which were querying the database

Speaker 1: And they tried to read from the user table. Remember that support site page with the account information? Has users and accounts tables behind it? And they joined the lock cube behind the exclusive lock which the migration was trying to acquire. And they waited. And requests timed out, web requests signed out, and we had an outage. And sad panda. Right, that's a lot of information about what happened in this scenario. Let's show a picture. So here's our situation. In the middle, we've got the old worker instance. That's the one that's kind of faded out a bit, because that's been terminated But that's where the original connection originated. And the network link link to the database is still open

Speaker 1: and locks that were acquired are still held. And you can see there's a little uh a little if I can find my pointer uh a little lock there to indicate that. Now on the right hand side we're deploying version n plus one of our code and that's what's being created here on this new worker instance. And the first thing it tries to do is run the migration. It tries to acquire a lock and it it goes into the lock queue. And over here on the left, the previous versions of the I really can't see my pointer. Over here on the left the previous version of the the the web support site is still servicing requests And it tries to read and it also hits the lock you.

Speaker 1: All right, here's the other interpretation of the Swiss cheese model, where in this case each slice represents that something was necessary for this outage to happen. And if we can fix the relevant hole in any slice of this Swiss cheese, then the outage won't happen. Now there's a little bit of everything here. We've got database transactions and locking, we've got code performance, we've got the issue of the process termination and the networking configuration of the RDS instance. We'll go into these in a bit more detail, but let's connect this Swiss cheese model with leaky abstractions. This is the the point of the title of the talk. I've shown two Swiss cheese diagrams.

Speaker 1: They're both relevant, but we're going to concentrate on this bottom left one for now. I said that the Swiss cheese moff model was often used in in aviation. In aviation incidents, some kind of mechanical issue is often involved, such as a worn piece of wiring that short circuits, or a component with a physical floor that breaks. But in modern computing systems like cloud computing platforms, a redundancy and virtualization mean that physical hardware failures rarely cause disruptions. So what's left as a potential cause of outages? What leads us to write code that works correctly most of the time but occasionally doesn't? Now I'm claiming often it's leaky abstractions. So let's look in our outage

Speaker 1: where were the leaky abstractions. So the first slice of cheese in in that previous diagram was queries in database transactions. So we've had the transaction routing to the primary database. We had the select queries getting access share locks. That's the weakest form of lock in Postgres, and the locks remaining for the life of the transaction. So the dry run management command um, or rather the management command that we ran has a dry run functionality, and that's implemented by inheriting from a class, which I'll show you in a second. The abstraction here is that you can write your management command inheriting from this class and you get the dry run functionality

Speaker 1: and the implementation implementation detail is hidden. But that's actually a leaky abstraction. So here's the implementation. We're defining an execute method which opens an atomic transaction. That one there. And then we're going to get our results by calling the the the subclass, which is the normal execute action. method. And if we've set the flag, the dry run flag, what we're going to do then is roll back the transaction. So that's how we get the dry run functionality. And if we do roll back, we're also going to print that we've done our dry run and that we've rolled back our database changes.

Speaker 1: The oops, let's go back. Um the leakiness is that opening an atomic transaction does two things. First of all, it triggers the data database triggers the database router to select the primary database instance instead of the replica. Because the the idea is that we're opening a transaction. That's probably going because we're going to be doing some writing. Therefore we need to be on the primary. And it rather obviously opens a transaction. But as a result, any locks that get acquired, including just reading from the database, continue to be held until the transaction completes or is rolled back. Now this is leaky because um This forcing of the the the routing to the primary database happens even if the dry run

Speaker 1: option is not set. So remember this command was being triggered from a cron file. At some point somebody's done a pull request to add it to the cron file and they haven't got the dry run and uh flag included. Um it was probably there because they were testing it manually initially. Um But still we have this transaction beacon being opened. So there's our our first leaky abstraction The second one is that slow-cron drop. So it's consuming the results of the query and it's calling an external API. And we worked out based on the rate at which it was getting results that would probably take 20 minutes to complete. There actually wasn't a leaky abstraction here. It was probably just a old-fashioned mistake.

Speaker 1: It was The command was set so that you could add a flag which says, oh, only do 20 of these or 100 or something like that. Because we knew that we wanted to limit the number we did at any one time. It's actually a rate limit on the external API But we forgot to set that flag in the cron file. So yeah, bit of a mistake. The interesting thing is this Cronjob had been running for about a year before this incident happened. But what had happened is over the period of a year the the size of the query the results of the queries getting back was was gradually increasing. So as it got bigger and bigger, the the risk that it was going to take too long was increasing as well.

Speaker 1: The third of our slices was this ungraceful cron job termination. So we had deployment, which terminated the cron job. and resources weren't cleaned up. Now the leaky abstraction here is is what happens with process termination. And you might think that a process will release the open resources when it terminates. And by resources we mean things like memory, CPU threads, and network connections. But how the process is terminated makes a difference. In theory, the more gentle term signal should enable the process to clean up resources properly, including closing any network connections. Whereas the kill signal, which can't be caught, results in the process being killed immediately without any cleanup happening.

Speaker 1: Cleaning up resources is fraught with complications. In the case of Python, all objects will be destroyed, which in theory means that their dunder del method gets called. And in the case of a network connection object, that would close the connection. But here's the documentation about the Dundadel method. And you can't read it because there's a lot of it. But in there there's a there's a note in grey, and then we've got a C Python implementation detail, and then there's this big red warning block of text So there's a lot of complexity in here. We don't we didn't quite get to the bottom of exactly what happened, but we knew there was a very strong risk that that this this process termination was was not happening cleanly the way we thought it was supposed to. Moving on.

Speaker 1: TCP parameters on RDS. So you can tune how the TCP works, and RDS has a way of setting those parameters. Half open TCP connections, that's a connection where the other end, one half has gone away, they need to be detected at the RDS end. And as long as the TCP connection is open, any transactions persist. So it has default TCP keep alive parameters. That's where it will send a packet every so often just to see if the other end is alive. But with those default parameters, the connection will remain open for two hours. I think in the long dark history of TCP, two hours might have made some sense, but when you're working inside an AWS data center, two hours is completely unnecessary.

Speaker 1: So here's our leaky abstraction. A TCP connection is either open, established, or closed. A useful abstraction about a TCP connection, open or closed. As a developer, When you're writing code that opens a TCP network connection, maybe using the requests library in Python, it's reasonable to check if opening the connection failed. But if it succeeds, you're good to start sending and receiving data on it. But that's our leaky abstraction. This is the TCP state diagram from W. Richard Stevens' classic classic book, TCPIP Illustrated. This is the open state And and this here is the closed state. Um I don't know what that state is, or

Speaker 1: all of those states. And and the interesting thing is This diagram doesn't even have the half-open half open state on it. So that's the state we ended up in, right? Half open where when the process terminated According to RFC 793, a TCP connection is referred to as half open when the host at one end of the TCP connection has crashed or has otherwise removed the socket without notifying the other end. So that's our leaky abstraction as far as a TCP network connection goes. Last, uh this is the fifth of our five slices here. Um database migration. So it makes the the details of this particular migration was making a foreign key to user ta

Speaker 1: the user table non-nullable and adding a constraint. And for that it needed to acquire the exclusive lock. And the exclusive lock was on the source table and also the destination, the user table. And that waited in the lock queue, and as we've seen, the read queries were waiting in the lock queue behind it. And we had our severe disruptions. So database migrations. In Kraken we have guidance on how to write database migrations to avoid various well-known problems, including the one that occurs when a database has a different schema compared to what the currently running Django code thinks it is. Now that can occur during deployments and you saw in the diagram a few pages back, that we had both the new code running and the old code.

Speaker 1: And this uh blog c blog post which is linked at the bottom there, and it's probably too small to read. has describes how you can deal with this particular situation. And our guidance in Kraken includes the right way of dealing with this We also require that, or we avoid migrations that require a table lock to be held for a long period of time. So for example, we use add index concurrently rather than just addindex. And the add index concurrently doesn't require a table lock for for extended period of time. So our leaky abstraction here is that database migrations can be deployed smoothly during normal operations. And this is the one that kind of

Speaker 1: m scares me the most. Um because Any migration that requires an access exclusive lock can potentially cause an outage. And many forms of of alter table uh require an access exclusive lock. For instance, add column. And you know, we think we we add columns to database tables. Not all the time, but relatively frequently, and without thinking too much about it. And depending what else is going on, that actually could cause an outage. Obviously I've given a particular example with a where a cron job is involved, but we might have other sources of long-running queries that, depending on how long they run for, could potentially cause an outage.

Speaker 1: Uh my mind was blown with when I finally reached this realization because we'd been operating on the under the assumption that, oh yeah, sure, all these kind of migrations, uh database migrations are are okay to to to deploy without any special um any special care. All right, let's let's let's finish up the end of the story about the outage. What was the post-outage remediation we did? First of all, a spoiler, the outage actually recurred two days later, when we had another migration, and before we'd actually understood all this stuff, what had happened or had a chance to make any of the changes that I'm about to describe. First of all, we removed the dry run functionality from that particular management command.

Speaker 1: That meant that it no longer queried the primary, but used the replica instead, which is more or less what we would have assumed it would be doing anyway. We also added the command line option we'd originally forgotten to limit limit the number of rows it processed in any particular invocation. And we reconfigured RDS, uh, the TCP keep alive parameters, so that the half-open connections would be closed after at most two minutes, um, which is much more reasonable. Maybe it should even be something as slow as 10 or 20 seconds. I don't know. But we went with two minutes. Some things that we didn't actually reach a resolution on. In the real world, you might have come up with any number of actions you want to do.

Speaker 1: Some you get to, some you don't. We're still not totally sure why the process wasn't terminated with a term signal. It it 's possible it might have been, but due to the complexity of what I showed with the the uh the the Dell method in um in Python that it just didn't work. Uh but yeah, we're not totally sure what happened there We also thought about the idea of having a default lock acquisition timeout on migrations. So that that would mean that if the migration doesn't apply after a short period of time, you know, maybe five or ten seconds, that it's just going to fail. The problem is that's going to cause the entire deployment to fail, and that's really annoying. So maybe there's an argument to be made that we should have this and that we should be better prepared to deal with deployments failing, uh, but we haven't had that conversation at work yet.

Speaker 1: Alright, back to our cheese and the holes that we want to fill in. What can we do about leaky abstractions? Now here are some ideas. Some are more practical, others less so. The first one is we need to acknowledge that abstractions are leaky. We can't just pretend they're fine. 99% is great, but 99% is not 100%. We don't want to rely on something that only works correctly 99% of the time. We might be able to patch some of the leaks that are unavoidable. That aren't unavoidable, sorry. That would be impossible otherwise. But Spolsky's law says that there are always going to be some

Speaker 1: that we won't be able to patch. We can write conventions, our coding conventions, to avoid scenarios where a leak, a leaky abstraction can occur. So for example, don't iterate over query sets within a transaction. And I think that's going to be something useful to do, but I think it's also going to challenge a few people in terms of, oh, but I want to, you know. They'll have to change the way they think about how we use transactions. How could this fail mindset when writing and reviewing code? And if you were at Wilhelm's talk before lunch, one of the examples about uh or quotes about testing was

Speaker 1: it makes you think about worst-case scenarios. I think we should think more about worst-case scenarios when we're reviewing code. No guarantee it's going to help you find these, but at least your mind will be open to look for them. I don't know if anyone's seen the list of fallacies developers believe about X. There's some for names, there's dates, there's times, there's addresses, there's all sorts of things. I'm wondering if we need lists of fallacies developers believe about database migrations, uh deployment. There's a bit in our documentation which says uh migrations like this probably should be deployed outside work hours. Now that was five five fine five years ago when we only operated in the UK, but we operate around the world now, and there's no such thing as outside work hours for everyone.

Speaker 1: And database locking. I personally didn't understand how the locking cues worked before I started looking into this, and now I know a lot more. Now a couple of those bullet points from the previous page actually line up with the other of the Swiss cheese, the software development Swiss cheese model. So here we've got Talking about conventions, well, we've already got a place where we've got coding conventions. That's a kind of a control that we can use to avoid issues. And Code review is already one of the well any of the other controls. And you know, we try and hope that at least one of these controls will prevent an issue from hap from happening. But um Adding an extra, you know, we've learned lessons from this outage.

Speaker 1: Let's add those into our conventions and talk about them when we teach people how to do code review. All right, all that's left is very short confusions, uh conclusions. There was a lot of confusion trying to understand what happened with this, so It was a Freudian slip. Know what leaky abstractions you rely on. Write conventions as much as you can to avoid those leaks. And finally, look for failure modes in code review And I've got this slide saying, do we have time for questions? But I think we might be right to time. Okay, we might have time for one question if we if we do have one question.

Speaker 2: Thank you very much, Sam.

Speaker 1: Where where's where's Russell? He's always got a question.

Speaker 2: We've got time for one question.

Speaker 1: Adam.

Speaker 3: Thank you very much, Tim. That was a great talk. Some of it sounds unfortunately familiar. I was wondering if there's anything you think Django or packages could be doing to assist you with these migrations at scale.

Speaker 1: I there is something I'm thinking about in terms of of Django and there's a uh there's a blog post written by um somebody at a company, I I'm afraid I I don't remember the name, I've got a note of somewhere, but they have talked about a thing called what they've called asynchronous migrations. And the idea being that an operation that's similar to a to a standard Django migration and that could be implemented like that, but we want to be able to run it at an arbitrary time So rather than in our case when we deploy we run our migrations, what we want to do is we send it out there saying, hey, there's this migration. When you get round to it at a convenient time, please run it. I personally think that would probably be really useful, and I'm hoping to spend some time at the sprints seeing if we can pull out this bit of code from this company's open source product.

Speaker 1: and see if we can pull that out and make a package that would potentially be useful. I don't think it belongs in Django Core , but it could be a useful add-on. But apart from that, I'm not really sure if there's anything that really belongs. is obviously an improvement that could be made in Django. But I'd be open to suggestions. Thanks.

Speaker 2: Thank you so much, Tim.