Oh, I Found a Security Issue
Published October 26, 2026
This video features Andrew Northall at DjangoCon Europe 2026 in Athens, Greece.
TALK: Digitising Historical Caving Data with Python and Django by Andrew Northall
Endless quantities of old but critically important data is trapped in paper journals and reports across the world. Is it possible to extract this data and make it available to the masses, even when the data is highly specialised and of poor quality? That is what I did for reports of caving incidents and accidents dating back 50 years. The National Speleological Society has published American Caving Accidents since 1967, documenting thousands of incidents, and despite being freely available as PDFs, these are unindexed, poorly scanned, and basically impossible to use as a learning tool. In this talk, Andrew Northall shows us how he built a system that programmatically processed these documents into a structured, publicly searchable database.
More information: https://2026.djangocon.eu/speakers/GJUXR3/
Follow DjangoCon Europe on social media:
*X - https://x.com/DjangoConEurope/
*LinkedIn - https://www.linkedin.com/company/djangocon-europe/
*Mastodon - https://fosstodon.org/@djangoconeurope
#django #djangocon #djangoconeurope #djangoconeurope2026
Automatically transcribed, so expect mistakes in names and technical terms.
Speaker 1: So the next speaker is Andy. You've seen him uh with a pink t-shirt most of the times these days, but now he's wearing a white t-shirt as he's going to talk about uh caving data and how he's He's using Django to
Speaker 2: uh something like that.
Speaker 1: Something like that.
Speaker 2: Yeah, yeah, something like that. Thank you very much.
Speaker 1: He's a Python and Django developer. He's also like to tweak his Linux uh
Speaker 2: I like wasting my time by hosting things myself for more money than it would cost to actually
Speaker 1: I think I'll I'll get out of your way. I think we have more important things to say. But yeah, if you if you n want more things about Athens ask us but now I think it's more important to To learn about caving data, thanks a lot Andy for all your work as volunteer and as speaker in this conference.
Speaker 2: Thank you very much. Um I think that was a bit mean. I was completely ready about uh I don't know ten minutes ago but I just really enjoyed you talking And I think it's about time you're still on the stage. So thank you very much. Thank you for coming for my talk. It's great to see that there's so many people here. And I'd like to thank you again for not scheduling a workshop next to my talk. which is why there's so many people here. Um so I most of you have probably heard me talk about a load of nonsense last night during the lightning talks just literally whatever could come to my mind. I'm very good at talking. Problem is uh staying on topic, so we'll we'll see how we go. But I know this is a technical conference and I have to do a technical talk. So I'm going to do that and I'm going to start with some things that I like.
Speaker 2: So some things that I like, I like caving. It's this is probably my primary hobby and uh I go caving a lot. Uh you can see me here caving and uh you can see I'm actually leaving the cave and how happy I look to be leaving a cave. um which is great demonstration of how much I like it. Uh I also like coding, because I know most people here probably don't like caving. Um this is a picture of me coding on the floor of a nightclub. It's probably the best uh indication of How much I like coding. I don't know why they let me in with a laptop and I don't know why they let me sit there, but they did. And it's I'm very grateful. It's a great photo for this uh for this uh presentation. Uh in case there's anyone in the room who doesn't like coding. Or caving, also
Speaker 2: like holidays to Athens. I think that covers everyone in the room now. Hopefully we've all got something in common. But let's go back to caving. I'm sure most people know what coding is and uh I probably don't need to explain that too much. But uh as for caving, um what is it? I thought I'd try and explain, so I went on Wikipedia, and uh apparently caving is the recreational pastime of exploring wild cave systems Which is probably the least informative sentence I've ever read in my entire life. So I thought that I'd try and uh portray to everyone here who maybe hasn't been caving, which is probably everyone, uh what it's like to go caving and how it feels and why I'm so personally enthusiastic about it. But I didn't want to
Speaker 2: give it to you in my words. I wanted to give it you in in in in the in the words of people who do it all the time or perhaps the words of people who've who've uh considered it deeply. So I went and I found some quotes on the internet And this is the first one I found. Nobody could convince me to do this even with a gun to my head. That's not really what I wanted. So I found this one. Okay, uh I wasn't doing very well at this point, so I thought I'll probably have to uh try and explain it myself. But what these quotes show is that caving has an image problem. So people think it's really dangerous. I can't really understand why people would think going in a cave would be dangerous. I've never uh personally um b had any injuries or died from it as far as I know.
Speaker 2: But you can kind of understand why people might think it's dangerous because it's a kind of niche hobby, it's kind of closed off. Um there's not a lot of people who who talk about it. And uh has anyone heard of the Thai Cave Rescue? Yeah? Well all these uh kids were stuck in a cave in Thailand. Yeah, everyone's heard of that. No one's heard about my great caving trips. Um so there's a lot of media attention on the on the very bad things. Um but absolutely uh no confidence or publication of of what we're doing right. So most people think caving is like this. This is my partner. And you know some caving is like this, a very small call uh very small hole 500 meters long and you do this for about an hour.
Speaker 2: 500 meters doesn't feel very far when you say it out loud, but when it's like that it takes uh a very long time. Um you don't have to do that. Some cavings like this Which is lovely. This is in uh Tennessee in America. And uh as you can see, the people are getting smaller because the kids are big. So about two million people visit caves every year. Most of them are on guided trips and most of them in really easy caves. But there's some people I'd say about 50,000 people is the is the best estimate I can find from the numbers that I've got who uh are really, really into caving. I'm one of them. And uh these people who are really, really into caving find that they start going caving all the time. It becomes their identity.
Speaker 2: Uh they start doing unpaid work for caving. They start doing caving projects. I don't know if there's any open source maintainers in the room who might relate to doing unpaid work all the time for their hobby, but I think it's a kind of similar thing. This is the good caving. This is over in Spain. And uh this is a cave called Puerto Coventosa uh in Cantabria. It's a lovely place, it's absolutely gigantic, uh as long as it's not full of water like on the on the uh right hand side there. It's uh it's pretty fun. Um when the boat punches and starts to sink when you're halfway across the lake it's not so fun but there we go. And again another nice photo of uh cave in Tennessee.
Speaker 2: This is why I love caving. It's great. I mean it's it's it's the only the only hobby that you can you can do and you can go somewhere where no human has ever been before. You can discover things. You can be the first person to step foot in a place. And for me this is really exciting. I hope that I can encourage more people to do it. And that's why I come to How Safe is Caving? Well, it's very safe. um as far as I'm concerned. Like I say, I've I've never had any injuries. Uh uh I I don't know anyone else who has. Um I can't really prove it. You just take my word for it, I suppose. I think it's very safe. How could we prove it? How can we show that caving is safe?
Speaker 2: Cavers achieve high standards without formal organization. Again, maybe like uh some communities you might be a member of. I mean Django is extremely high standards. Uh it's respected, it's stable. And uh I mean there's some form like organization now with the DSF, but uh mostly people just contribute and mostly people look after it. And if people didn't just do the things that they're doing without anyone forcing them to do them, it wouldn't work. It's the same with caving, but with Django we have lots of visibility. It's all online, it's all on track, it's all on GitHub. Um caving we can't really do that. Or can we? Because we have these organizations here, all of these organizations are safety related and uh they all They all have uh
Speaker 2: really frankly horrific incidents. I mean uh aviation crashes make the media and uh you know NASA, etc. Uh Uh these organizations have been responsible for things that have caused lots of deaths, and yet people generally have a positive image of them. Why is that? Well it's because they are transparent and they publish what goes wrong and they publish what they're doing about it and they publish uh how they're gonna learn from it and how they're gonna go forward. Um could we do the same thing with gaving? Yeah we could because we have this organization called the Let me try. National Speleological Society. And this is a American organization, and they're the biggest caving-related organization in the world.
Speaker 2: They're a nonprofit. And uh they have thankfully been recording records and uh reports of caving accidents and caving incidents for near enough uh 60 years now and uh some of the incident reports go back up to a hundred years So we can just use these, right? We can just make a database, it'll be really easy. Unfortunately, this is the uh this is the sort of thing that we're dealing with. So it's Not exactly structured. When they made this in 1971, this was top quality data. I mean this was really, really good. Um they never thought about some hypothetical futuristic person who might be using hypothetical futuristic LLMs to to process this and and try and make it into something that's usable.
Speaker 2: They thought this is as good as it will get, we're doing a really good job. But Stuff like this is is is difficult. I don't know how many people are familiar with OCR technology. It struggles when the letters aren't there. And it it struggles when the letters look like another letter and it struggles when basically everything in this photo struggles. Because they didn't foresee this futuristic future technology, they started adding all different formatting into the incident reports. So this is the year after the one I just showed you and they've added a load of dots between every incident and uh they did this to make it easier to read and then the next year they kept adding dots but they added more dots.
Speaker 2: And then the year after they removed the dots and started adding a line. Didn't really work for me. They also started adding all sorts of little bits of information in between all the reports. So they thought, oh we'll keep the reader interested. Um we'll just add a story. There's nothing to do with caving, right in the middle of the reports. And then we'll we'll do that the next year as well, but we'll put it at the front and then the next year we'll put it at the back. And then actually we'll just do a uh half the incident, uh half the journal is about nothing to do with caving at all. They didn't really take into account uh Postgres standards for dates. Um I don't know if anyone's ever tried to put fall 1985 into a Postgres datetime field. It wouldn't accept it for me. If anyone knows how to do that,
Speaker 2: let me know. So how do we deal with all this? Um well I tried to deal with it for probably three years ago I started this project and uh it was not So fun before LLMs. Um we had uh about ten volunteers and uh I I wrote some Well, I called it to them an advanced pipeline that processed the incidents. So what it actually meant was I was kind of doing a little bit of processing with OCR. and uh then manually editing them and and splitting them and figuring out how many dots in each incident and doing a find and replace and all this and uh everyone else was very impressed but it was a huge amount of manual work We made a web UI with Django
Speaker 2: and we uploaded all of this text. We got volunteers to go through it and they'd format it and they'd pass it. And uh they'd correct spelling mistakes and so on. And I think after about three or four months of work we had about 200 instance done, um which uh was less than 10% and as we went on, as time went on, uh no disrespect to the volunteers, they didn't want to do the most boring task ever created in the world. and they got slower and slower. Uh after about three or four months, uh projection for finishing the project was five years. So actually finished it in less than five years with the upcoming pipeline, so that was good What do we do now? So we have a new pipeline. LLM's heavily involved.
Speaker 2: We also have a library called Dockling. In some way I feel bad for the volunteers because I actually just threw away all the work that they did it with one asterisk. Uh but With this pipeline, with LLMs, with document extraction, we managed to extract almost 3,000 incidents In about 12 hours once I'd written the code. Once it ran, it took about 12 hours and the whole thing was finished. Which again was very, very bittersweet. So step one, extraction, and extraction is getting the text, the raw text, out of documents. Dockling is a library that came out of IBM research. It came out
Speaker 2: literally just after I finished doing most of this myself. Um which is fine. I mean they did a better job. I'm not gonna say that mine was better. In fact it was uh I threw it all away as soon as I heard of the library, but they You know, uh it would be nice if they'd released it a little bit sooner, but I'm very grateful for it. Dockling is fantastic. It's a document extraction tool. So everything I showed in the slides earlier, we've got this uh poor OCR, we've got uh multi-column layouts, uh we've got uh anything that's difficult, we've got little figures under text like this Dockling can handle this. So you can see the yellow box on the screen there. And under experiments, it's actually going around and joining to the next column. I had to do this myself. I had to figure out all sorts of X and Y
Speaker 2: things Um I didn't realize that coordinates were involved in OCR. I left school when I was fifteen, so this was really hard for me. I didn't know I have a clue what I was doing as I wonder if it worked. And uh without LLMs, by the way. I did it myself. And uh it follows it round to the next column, it ignores the figure at the top. Which again is very difficult for me. And you get nice kind of formatted text. So you can you can run a uh Docklinger stretching on something and you'll get a bunch of text that looks kind of like this. The two columns is just for your benefit. It will be, you know, basically one gigantic column of text. And uh this was across, I don't know, 50 incident reports with 3,000, 50 journals, sorry, with 3,000 incident reports in, so it was quite a lot of text.
Speaker 2: I used to manually split this again, thankfully. Don't have to do that anymore. So what what do you do when you've got a gigantic wall of text the type that no man has ever seen? You use an LLM. And actually a very cheap LLM is very good at this sort of task. So you pass it some amount of text, 100, 200, 300 words, some amount of text. And uh you say, tell me if this is a complete incident. LLM goes, yes or no. And if they go no, you give it a rip more text and a rip more text until it goes yes. And then you split it off. And then you say, well, how much uh extra text did we have? And it'll say, well, you had this many extra words, and then you start the whole process again.
Speaker 2: And you'll you'll end up with a bunch of text files, 3,000 text files. with one incident each. Unfortunately they still look like this. Um so they still look like this with one incident each in. So how do we deal with this? We're gonna have to format it. I I have very high standards when it comes to grammar. Literally no one else cares, but I mean I do, so I spent a lot of time on this. We need to format and structure it, and we need to correct OCR errors Standardize the grammar, capital letters, full stops on the end of sentences, but also very critically, we need to make sure that we don't actually change any of the content. We don't change any of the meaning. We don't hallucinate anything or the LLM doesn't hallucinate anything into the into the uh document. When you're dealing with this sort of data, it's very, very important.
Speaker 2: And that everything is readable and usable and can actually be consumed for the purposes of research and learning. I'll tell you exactly how I did that in a moment But I will just say first, very important when you're doing anything like this with LLMs, is to make the LLM check its own work. Because LLMs are great, they'll produce some work and then you pass the same work back to the same LLM and the LLM says, This is terrible. Uh uh I would never ever use this. I don't know who created this, but they need a lot of uh training or you know it's a kind of junior developer level and you pass it back and it makes it better and then you pass it back it says the same thing again. This is terrible. Eventually you get to the point where you have some good
Speaker 2: standard of work. How do you do this iteratively 3,000 times and normalize stuff and put it in a database? Well you make a pluggable architecture. of LLM-based formatters, it sounds very important, doesn't it? LLM-based formatters and checkers that allows easy future revisions or expansions. So it's actually very simple. I made a Python class, uh abstract base class, and uh it has a name. The name's important because this is a a unique name. And uh a unique name means that we can run an operation on an incident and then store the state of the operation in the database. So I have one incident, incident ID one, and I have ten operations. I want to run them all on incident
Speaker 2: one. And then when I add more incidents, I don't want to run them all on incident one again. I want to run them once, obviously, to save some money. And that unique name for each operation allows us to do that We have should run. So we don't want to run all operations on all incidents. Maybe we if it's a US-based incident, we might have a operation which Checks that the US state is formatted correctly and it adheres to you know uh is a US state. Um and We have a try and accept loop so we can run this. It's async uh so it can run concurrently and we have a try and accept loop so we can figure out if it failed. If it does fail, we log it. If it doesn't, we will save uh the state to the database.
Speaker 2: This is how we save the state. It's very simple. We have a Django model, data operation log. Foreign key to an incident, operation name, version. I don't know why I didn't just make a New name with a one on the end, but there we go, it's the same thing. And uh and the status. So if it fails, we can we can store that and we can see what exactly needed. to be uh corrected. And then we can actually get that LLM to iterate. So if it fails, we can set the status to failed. We can Put the error message in the log, and then the next time it goes round and tries to complete this operation, we can actually refer back to previous operations and see maybe we needed to do something particular
Speaker 2: And with LLMs we can do that. We can actually adapt as we go through. So there was all sorts of edge cases in this data, and there's all sorts of problems that came up once or twice throughout the entire data set Because we iteratively go through, we store the state of what we tried to do, and we store the error, the LLM can adapt to this. It can actually self-heal and produce a very good result. How do we ensure that we have good data still at the end? Well, actually we just use exactly what I've just shown you. We use the same data operations to check the work. and they can run in a specific order. So we use the same data operations. When we have the should
Speaker 2: run, we can check if a previous operation has run. And then if it has run, we can then make another operation to check it. This works really well. So this is the iterative quality loop that I talked about. We keep going back and we keep checking, particularly if it's important data, we can check multiple times in multiple ways from multiple angles, either with LLMs or without LLMs Whether we're happy with the results. So for a US state, for example, we don't need to use an LLM. I mean some people would. I'm sure you've seen this repo on GitHub where people do math with LLMs. You can, why not? But once we've done all this, we finally need to put it into Postgres
Speaker 2: How do we put it into Postgres? Well, it's not actually that simple. Locations were a big problem. And uh some of the data had a location like uh uh a region. It had a town, a region, a street name, you know, whatever, whatever, uh very precise locations and some other locations just had Arizona or Nevada. It wasn't very helpful to be honest, but that's what I had. I couldn't go back and ask them it was in 1921 or something like this. So we have a tree structure and this is really really useful because we can store any precision level of location. without having to represent this strictly in Postgres, which frankly would be impossible.
Speaker 2: Imagine all these foreign keys and empty empty empty links and so on. So uh Django tree bit tr tree bid is a great library that allows you to do this. You can it has a a table that says this is the precision level and this is the name or identity of what of what you're talking about. So you have a region with a name of region name and then you can insert above that a state You can search below that a town, and all this happens kind of automatically and invisibly to you using Django TreeBid. We had a lot of duplicates in the data as well. And uh these duplicates were almost exact duplicates, but not quite. And
Speaker 2: you couldn't really check with an LLM if these incidents were duplicates because you'd have to compare every incident to every other incident. But what you can do is create embeddings and then create a similarity vector with these embeddings. And figure out, okay, if this uh incident is more than 80% similar to another incident, then we can SDLM check these incidents and uh and and determine if they are a duplicate. Just to kind of note, I never actually trusted the LLM to throw something away on its own, so I actually then manually reviewed these. Strange or weird dates like fall 1985. I handled these with a custom Django model field, it's actually uh quite simple. Um You store it as a string and then at the end you just have one character which represents the precision.
Speaker 2: So instead of 2026, 06, 01, uh 2026. 06, 01, M. And the M just meant that this date is to a month precision and the O1 can be ignored. It works really well, actually. I actually found some libraries that do this already. Apparently I'm the only person in the entire world that wanted to store full as a date though. So all these libraries allowed for month or for year precision, but not fall or spring, or kind of like in between fall or spring, which is actually a real one that I'd handle. So this is the uh tree beard model. It's super simple. Kind of kind of geographic area, name of the geographic area, and the code, um which is
Speaker 2: uh kind of internal internal reference. And you can see at the top we have valid parents. So a state can have a country as a parent, a region can have a country or a state as a parent. So a region doesn't need to have a state, we can just have a country. This was really important to me, so I could actually uh ensure that we have the highest precision of data, but without going crazy or using ridiculous levels of uh Django modeling. Just code a front end, that's it. I mean just just do that. Um it it kind of used to be a joke, but it's not a joke anymore because Claude now will just code at a front end And uh Claude did a really good job of just coding a front end. And uh I'm happy with the results.
Speaker 2: We we we documented 2,739 incidents. which was 2,739 more than were documented before. The National Speleological Society, which I mentioned earlier, is funding this and they're actually trying to expand this to a worldwide instant database much like aviation and uh I mean they also want me to code that for them which uh may or may not have time to do but um All these incidents now accessible, people can learn about what's happened, people can learn from it, and we've got nice presentable reports summaries about the AI summary though I didn't actually implement this
Speaker 2: Uh when I asked Claude to code a front end, it implemented an AI generated summary in in in the front end. Um so I'm not really sure how to remove that. I've tried to ask Claude to remove it, he hasn't done it yet. Results. As I said, two thousand seven hundred and thirty-nine incidents published. I'm really proud of this. I left school when I was 15. I've mentioned that about three times. I'm actually not very clever. But the data set has already been used in research. on how to prevent and reduce incidents. There's a uh cave scientist, uh microbiologist called Hazel Barton, and uh I think she's widely considered to be the leading cave scientist in the world.
Speaker 2: She's um presented documentaries for IMAX, uh BBC, National Geographic, and uh has a team of about ten PhD students doing whatever PhD students do. I don't know. I have no idea. Genuinely if anyone knows, please tell me. Didn't go to university. Drink coffee. Okay, thanks, Colton. That's all you did. Yeah. And you're still doing it now. Anyway, she she's doing real research using this data, which I feel really good about. I really do. And I hope that she's able to find some insights and and trends that can actually help us reduce uh incidence. She's writing on paper. Writing a paper. Um she said she might put me as a co-author on the paper. I mean uh I'd I'd be quite amazed if this happened.
Speaker 2: Uh I don't even know how to write a paper. Again, Carlton, maybe you can tell me later. A surgeon in the US is using the dataset uh to investigate the frequency and typa injuries. Again, um it it's It feels really good for me personally, but I think more importantly it shows that there's a lot of data out there that Couldn't previously be accessed or easily used just a few years ago. And now we have the technology and we have the means to process this data and actually make a positive impact using it I did this with frankly like kind of minimal work after the LLM stage. And
Speaker 2: I wonder if anyone can think of any data that they might know of, maybe some hobby that they have. Anything that could be made available to people and could be recovered in some sense and made useful to society once again. Thank you to everyone who helped with this project. Uh I did put a little asterisk about the volunteers earlier. I said their vo their work maybe had gone to waste. It didn't quite go to waste because uh I was able to use all their manually created incidents as a verification set for the LLM-based work and uh we found that actually it was almost 100% accurate, strangely. Um not not quite but almost
Speaker 2: Uh because we had so many checks. So thank you to the volunteers. The work wasn't completely a waste, even though you did spend hundreds of hours on it and I threw it all away. There is some saving grace there. I don't I don't know if they appreciate it, but maybe they do. And uh that's it. Thank you very much.
Speaker 1: I'm really afraid to go into caves. Sorry about that. But it was really good.
Speaker 2: The talk was a failure.
Speaker 1: No, it's a struct failure. I mean uh I'm afraid of heights and you know caves. Okay so that's it. Uh
Speaker 2: fine.
Speaker 1: Maybe you can do one or two questions. I have one here.
Speaker 2: Hold the button down. Sorry, I'm uh I've no pink t-shirt now. I shouldn't say anything.
Speaker 3: Both speaker and tech support. First of all, thank you for the talk. Strange Hobby says a lot about a person. So
Speaker 2: I didn't hit just say that again, sorry.
Speaker 3: I said uh strange hobbies says a lot about a person.
Speaker 2: Oh thanks.
Speaker 3: I'm interested in the the the What you mentioned the step about iterating with the LLM to split the incidents. So I'm curious um were you limited by the context uh size because With a million tokens, you should be able to just stuff all 3,000 documents and have it split it in one go. Any reason you didn't do that?
Speaker 2: That's a great question. It just doesn't work. I'm sure most people here have have tried to get cursor or maybe I don't know uh claw code to pick something out of a code base even when you actually pass indirectly the context. It just misses things. I mean the the the attention of the LLM, if if this is the correct word, the attention of the LLM cannot focus with such a a a large amount of data. I mean this is a simplification and humanization of of of of the thing, but it it it's looking at so much data and it will not accurately replicate it. I I
Speaker 2: Uh I don't know how to demonstrate this, but this is what happened happened to us. That's that's basically it.
Speaker 3: Yeah, I had a similar experience. This is why I was curious. Thank you.
Speaker 2: Yeah, thank you. Uh
Speaker 1: maybe you can do one more question. Okay, let's do this one.
Speaker 4: Hello. One question. It will be it will be one. It will be one let me finish. But the real question is. You used the Trib Bird, which is the first time I see you, and very glad of it, and thank you. to spatial data as it is or or is it only used as an indexing like use a phone um a phone c an old phone catalog
Speaker 2: so you said spatial data Yeah. So you g you want to know perhaps if you could model I don't know. But let's talk about it in a moment. I I would think that with spatial data you wouldn't actually need a tree structure, but maybe you know why you would. And I don't. So let's talk about it after. Thank you.
Speaker 1: Thank you very much. I I think this is it. You will find him with a pink t-shirt after this talk. So uh feel free to grab him if you have other other questions. Thanks again uh for the talk very nice one maybe I'll try Kev maybe maybe I'll try Kev
April 15 - 17, 2026
Athens, Greece
Note: We understand that names change, people change, and bodies change. We respect each individual's journey and privacy. If you have any concerns about a video or need us to remove content, please don't hesitate to contact us. We will handle your request with care and promptly address any issues.
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026
Published October 26, 2026