Hidden Dangers Of AI In Developer Workflows: Navigating Security Risks with Human Insight

I this is my first Django Con. I don't know what it's about the scheduling, but there's something in the air because I also am a former Drupal person who also used to work at a company that competed with platform SH, but that was ancient history because I work in security now. I highly recommend getting those slides. Shout out to Marietta who is gave a talk earlier on QR code making because I made that with Python about QR code today. So very awesome. Alright, I'm gonna talk fast because I got a lot of slides and a lot of ideas to cover. But This is the point of security. And this doesn't look like the point of security because there's no CVE involved. There's no Screen flashing on fire. No, this is the point. I work in the world of security

and we must keep this in mind. And I every by time I talk to developer or uh security people in the world, I remind them, Why we're doing this is because human beings aren't their jobs. Human beings need to be able to free have the freedom to go do other things in their life. So that's what I tell security people. I just gave this exact same talk at Blue TeamCon over the week uh weekend. Uh Full of security people. So I modified it slightly for this audience. Hopefully it goes well. Because security is a giant dumpster fire. And we fight this every day. And it's getting worse and it we don't know what to do. So if we lose track of the fact we're doing this for humans, we'll lose everything. But this is a talk about AI, not just security. I

love AI. I use it for all sorts of crazy stuff. I'll talk about that later. And there's some good use cases, some bad use cases. But there's a danger to this, and it's the same danger that we had from the thinking machines in the Dune universe. Who who here is a Dune fan? Yay. It's a lot of hands. That's actually more hands than any has ever gone up when I uh asked that question. But the fourth book in the series, God Emperor of Dune, uh God Emperor in the Trial of Siona in the middle of the book, she asked, why did we have to destroy the thinking machines? What was wrong with the thinking machines? And this is what he says, like, what do such machines really do? They increase the number of things you can do without thinking. Things we do without thinking, there's the real danger. I'm Dwayne. I live here in Chicago. I help people figure stuff out. That's my entire mission in life, and I'm very happy and proud to be here as part of DjangoCon talking to you in my hometown.

Hit me up on social media. Happy to talk to you about all that stuff. Also, I work for a company called Git Guardian, and we are a Django shop for sure. Our entire dashboard system and how you would ever touch us or work with us other than the API is all Django-based. And we are hiring for a engineer in Paris, needs to be in Paris, needs to speak French. French part they'll Get around, but uh it has to be in Paris and office. Uh that QR code goes to that. Again, thank you to Marietta for telling me about how many QR codes. Up front, I'm gonna make a giant line in the sand oversimplification and say there are two kinds of AI in the world that I work with or care about, predictive AI and generative AI. Predictive AI is not part of this talk. Predictive AI is guessing.

Which thing is next, and saying that's probably the right thing. And if you're sorting tickets, there's a bunch of use cases where predictive AI just makes sense. Gen AI is the one that can introduce vulnerabilities into your code. Because that's what just happened way back in August 26, 2025. Remember way back then? That's uh seems like forever ago. Um well for me it does. I've been basically doing conferences since. Uh this happened. Anybody get hit by the NX uh breach a couple weeks ago? No? Okay, okay. There's this company called NX, and they are an AI platform that helps you optimize your builds and scales your CI. Remember that for later. So what happened was they had a supply chain attack where someone inserted some malicious code into their one of their libraries that basically did a prompt injection.

when you ran that bit of code and the prompt was actually this and it dumps all of your local keys and then puts it into a public file. Double 64 base 64 encoded to make it a little bit harder to spot, but not that hard. Uh also did it in public, which is really weird, but I'm not here to talk about ins and outs of that. What we do know from this is my company did a investigation and we found 230 uh 2,349 distinct secrets. When I say secrets, I'm talking credential. of some sort, AI um git GitHub authorization token, uh OAuth token or something like that. Uh half of them are valid. The full report is if you get the slides, all those, all the links to all this are in the in the slides, uh are in the speaker notes. Um

how this happened was someone was able to hijack a GitHub action CI workflow Wait a minute. Doesn't this company optimize? Yeah. We're in a world where AI is literally eating itself now. This is the future of security and this is terrible. But that's actually not what I'm here to talk about. How I think we got to that state of AI eating itself and what we can do about it. Um boils down to not just these three things, but these are the big three in the room that I want to talk about. Development is hard. We just gotta admit this up front. It's hard, but it takes humans talking to other humans about what we're actually trying to do in order to develop stuff that's useful. I mean, that's one of the beautiful parts about Django. It's like how you all talk as a community and how you're actually trying to solve problems together as people.

But we all know that development doesn't work in the way that sprints work, it's generally like, hey, we're doing this. Go figure it out, developer. And that's what the client's paying us for or the customer is expecting us to do. And back in the good old days before AI, like way back in 2021, we would go Google it. And where would we inevitably end up? Anybody? Stack Overflow. Yay. I love you people. And what's the best part about Stack Overflow? Is the comments section. That's right. Because it's a giant flame war. It's a giant argument between people that say, no, I'm right, no, I'm right, and eventually someone wants to be technically right, which is the best kind of right. It's the best kind of correct. And how are you technically correct? You point to the docs. Eventually you point to the docs, and then you figure out how to do it.

Meanwhile, this entire cognitive process, even though you weren't there the two years ago when those two people in the our comment section were arguing. You're engaged. You're engaged with the human process asynchronously. And you're forming opinions. Like I think they're right. No, I think they're right. Okay, the doc say that okay, that other guy's a jerk. Yep, they're right. And you're engaged and you're learning. So you go back to the team and say, this is what I built and here's what I learned. And you can defend that PR. So when they say Okay, good overall work, but this part needs to change because the way our internal systems work. Don't worry about the individual use case up here. I'm just picking on Jenkins, because why not Jenkins? Um we're gonna call a vault instead of an EMV file, because why not? Like Don't worry about that part.

The part is that you should worry about is another human being says, look, I think what you did was great, but we're gonna do it better. And together, you make an awesome product. That's an ideal situation, I know, but that's my experience of working on teams. With AI, we have a lot of how does this do this? Uh give me something, and it just does it. And it's probably correct. It works on your machine. It's probably secure, right? Like AI wouldn't tell you to do something stupid, right? Then you take it back to the team. Well before you do that, probably, I forgot to put this in here. I have a whole other talk on probably versus determinism.

Uh Genai is just guessing. So is predictive AI, but Gen AI is just guessing the next token that's mathematically closest it thinks to give you what it thinks you want. But it's not thinking, it's just trying to fulfill the prompt. But just like any probabilistic system, the more you roll the dice, the more random that's gonna be. And that leads to other problems that I'm gonna talk about today, but The bigger problem here is, well, I cover your face. I didn't do that right. But other than that, uh the bigger problem is you come back with the working code and you have no idea what it works. I'm going to show you something real quick. Sorry for the screen slide and sorry for the horrible outline of this. But here's a Django site. And while Marietta's talk was going on earlier, or right after it, I said, hey, how do you how would you do that in Python?

Uh how would you do that in Django? So I have a fully functional Django site that's a QR code generator. That's uh make the background red, just to show you this is a live site. Yeah. I have no idea how that works whatsoever. I am not a Django developer. I now have an app I can go tear apart and understand how it works. There's no security aspect to it, it's just running on my machine. I'm never gonna launch that, but I literally have no idea how that works. I don't have any idea how those things function, but I have a functional thing at the end. And then you put it through the PR process and it's Sometimes you written the test already that will catch the security flaw. Sometimes you put the right tool in place to say, yes, we will we have caught that. But sometimes you don't. The problem with testing frameworks is they only test what you are testing for.

But it leads to this problem. Uh I don't know if you agree with these numbers or not. Up level has gotten a lot of flack for the way they did this methodology. I don't think they're wrong, even if you don't disagree with the specific percentages. The underlying fact is we have never seen more PRs coming through, pull requests coming through the system than we have right now. We've also never seen more bugs coming through the system than we have right now. We've never seen more security issues coming through the system as we have right now. So think way back to it when I started, that company that optimizes the PR process is saying we'll let the AI check the AI's work. That's terrifying to me. Did anybody notice this line? I know it's hard to read and there's a big room. But anybody noticed this line earlier? I don't even know if you can read this line. But it's telling you very specifically to hard code your credential.

Don't do that. Do not hard code credentials. Do not put plain text credentials anywhere. If you're just doing a local dev to experiment, sure. But if that thing leaves your machine. Please, please, please use a vault, at least environment variables that are safe in some way. Because we're in this giant dumpster fire of I don't know if I mentioned the giant dumpster fire of security, um, but we're in it. And I work for a company that One of the bits of research we do, and you can read this report for free, you can go through Tor, we don't really care. We just want you to read it. Don't we're not asking for any email or nothing. Um We look at every new commit that hits GitHub public. You can too if you want API. github. com slash events. That's the feed. It's public by the nature. So you can look at

these and we scan every commit and everything that becomes public that was private becomes public or that was already on GitHub, those events. out of the feed and we say hey is there a secret in here and if there is we email the committer right then and there and like hey you did this it's automated we look at a 1. 3 billion commits last year Anybody want to take a wild guess how many secret and just yell it out. Secret how many secrets did we find added to GitHub public? And if you're looking at the slides, you can't guess. But how many how how many secrets do you think added added added to GitHub public in just 2024? What? Half a million? One volume. Ten percent. Twenty million. Seven hundred and fifty billion Yeah, I will just uh I don't have enough time to play the giant guessing game. It's 23. 77 million. Four point one

four point six percent of all repos on GitHub that were updated last year. Contained a hard-coded credential. You can get the report to see all the methodologies on that. That's a 25% increase from the previous year. Every year we've done this report since 2020 has been a giant jump up. The first report had just over 2. 1 million secrets. Part of it is our methodology. Part of it is this. It's four percent uh four point six percent for the general populace. When we could tell specifically for Copilot, because Copilot leaves certain fingerprints in code that we can identify, once we see that, yeah, it's 6% of those repos

contain a secret. AI is making the situation worse. Even if it's just a little bit worse, it's still making it worse. I'm not gonna go into the full details of the report, but we know this problem is eight times worse. Not the not this problem, this problem is eight times worse in internal repos. Because you have the false sense of security, like no one's gonna see this code. This is an R repo, in our private instance. Please, if you ever see a hard-coded credential, just don't. Don't uh go talk to your security team and say, Hey, I need a way to not use a hard-coded credential. And I guarantee you they will have that conversation Hey, I I need to authenticate to the system, but my current way would make me hard code a credential. They will thank you. They'll freak out a little bit that you're talking to them instead of them having to come and yell at you. Trust me, security people are great people.

They're just weirdos. They really want you to be safe, but a lot of them don't know how to do anything but yell, you're wrong. But that's just how they live their lives. So if you go to them and say, I want a better way, they are very receptive to that. Very receptive to that. Mainly because nothing's on fire when you ask that question. If you prevent the fire with them, they're your best friend. Now, the good news is there are tools you can use right now for free. I have a personal favorite on this list, but there's a bunch of things on this list. Um you can even write pre-commit hooks. You can build yourself guardrails. So you can say, hey, I'm gonna make this commit. Uh-oh, I accidentally hard-coded your credential. I shouldn't commit this and it will just stop the commit. I'm not gonna go into that full, that's a whole other talk. But the all the docs are out there. I'm happy to talk in the hallway after this.

So hallucinations. I'll just let you read this. I know it's a little small, but Abraham Lincoln's my favorite quote from Abraham Lincoln, artificial intelligence hallucination is akin to the shadow of the loom, that loom's large. lacks substance. It mimics the semblance of reality yet feels the essence of truth. Old honest Abe. Uh that's fun and all, but What if you ask ChatGPT, like I got stuck using Orient DB, which is actually a pretty good database? Uh how how do I use that with uh Django? Give give me PyPy packages. Well the second thing on the list here is something called orient dash or py orient or m. Turns out that doesn't exist. Uh I did this yesterday. Hallucination problem

has been a known problem for two years now, three years now. Uh Mackenzie Jackson, I used to work with him. Uh he made a whole video and study about this. He works for a company called Aquito now that does awesome work on research into open source. Yeah, it's actually getting worse, not better. Um I did this last week with an NPM package, and it was the fifth thing on the list, not the second, but it was still top five a week ago and yesterday for two different systems. Welcome to a little hell that some of us in the industry are calling typo or hallucination squatting. Who knows what a hallucin or a typosquat is? Typosquat is where you accidentally mistype the name of a dependency and someone is squatting on that domain and ready to give you a package that does exactly what you think it does, plus a little bit more.

And that little bit more we'll you know call a CNC server or uh we'll call uh we'll send your data somewhere. Well, hallucination squatting is the exact same thing, except how it gets those typos. It just keeps asking for packages and keeps squatting on those hallucinations. Here's the terrifying part about this. Nobody knows how bad this is. Nobody. There is no way in the system right now to tell if the thing you're pulling down has is a hallucination squad or not, unless Unless you go line by line through every bit of code that you pull down, please do that. Diff is an awesome tool. If it's a new version you're getting. . Pull it down, run diff. Just see what's different. It's gonna take five seconds.

Maybe you'll see a URL you don't recognize. Don't install it. There's also commercial opportunities out there, uh people that really, really good solutions. I'll give a shout out to Shangar, they're a partner of ours. Don't pull from the internet, pull from them. That kind of solves this, because then they are at fault. There's the throat to choke if something goes really bad Uh, uh also, uh don't hard code your secrets and stop putting them on PyPi. Uh this is from uh two years uh year and a half ago, but uh I used to work with Tom. Uh this is before he worked for me uh wor worked with me actually. But uh yeah, he found almost four thousand working or uh seven hundred and sixty-eight working credentials across all the repos that he scanned on PyPy or all the packages he scanned on PyPy. He's actually currently working with PyPy itself to like how do we eliminate that?

I have not heard an update on that recently, but hopefully this is getting better, but stop hard coding your secrets, people. All right. Last one. I actually changed this recently because a year ago when I first came up with this talk, my biggest fear was my data was gonna leak, my code was gonna leak, it was gonna get out there, and that's gonna contain a secret. Now we live in a world of Agenic AI, and I think this is like bumped up to that's still a problem. Don't get me wrong. Your data leaking is still a problem. But now we're giving AI agency, literally giving it the ability to do stuff. Uh if you ask AWS, uh, this is how they explain agen agentic AI. And I don't think they're wrong, but I think this part is uh hopeful, and hope is not a strategy.

They're hoping you're paying attention. Are you really watching everything that the AI is doing, like when you test uh cursor or um whatever other system you clawed, desktop or whatever, to go do a thing like, hey, make this happen? Are you watching everything it does? I used Cursor to make that Django site you saw, and it didn't PowerShell. I do not use PowerShell and this is a Mac. I have no idea why it chose to do it that way. The only reason I know that is because I paid attention. I actually saw it. I also know there was no serious danger from doing that. But agents are clumped in with this field called non-human identity. I spend a lot of time thinking about this field, uh this this this part of our industry, the non-humans.

Humans, it's not a s human identity is far from solved, but humans have fingerprints. We have pass keys, we have multi-factor authentication, we have so many ways to prove you or you. You have DNA. Like there is a way we can ultimately prove you or you. Uh I don't like the term non-human identity because so many things are non-human. This floor is non-human, that ceiling's non-human. Uh the cup of coffee you're drinking is non-human. But what I actually care about is the workload, the running piece of software that's executing for a reason. What we gave this reason was to go do things in my name with this authority. So many bad use cases for that. And I don't have enough time to go through everything, but I will point to the OWASP POP 10.

This is the top 10 list you should be tackling, and I think the most important you can get off of this list is the overprivileged one Uh number five on the list. Uh is does everybody know what OWASP is? Sorry, I'm not in the security context anymore. OWASP with the Open Web Application Security Project. It now is the open worldwide application security project. I forgot they renamed it, but it's still OWASP. It's 23 year 22 years old now, uh older older in Django. Um and it's a project to keep us safe if we're running web apps. Uh over 300 projects, tons of tools. The top 10 you probably have heard of. They have a top 10 for a lot of specific sub things. This is one of them. Make sure you're not over-privileging these things. But that's hard, especially when you're saying, act as me. And if you're acting as you, you could throw these flags of YOLO, trust all, dangerously skip.

That's the problem. We're letting them act as us. You know you could do that. The AI has no idea that's a terrible idea. It just knows here's a prompt. I have the ability to do as a human. Human would do it. I'm going to do it. Hooray! I think those problems though, oh yeah, there by the way, there's no tooling that can fix this right now. There just isn't. We're working on it, Git Guardian. I'm happy to talk to you about possibilities on that. There just isn't an off-the-shelf thing I can point to like the others. Larger issue, point thinking versus system thinking. When you're working on something AI, it is very, very, very easy to get sucked into why doesn't this work? Why is this doing this weird? What is going on with this thing? And lose track of what you're actually trying to get done.

If you don't know about editing Edward Deming, we would not have DevOps without Deming. We would not have Kanban without deming. We would not have so many things like system thinking without a deming. There's a deming award in Japan, but we for some reason don't have one here and almost no Americans know who the hell I'm talking about. He gave us so many ways to think about this is one of my favorite books about him. John Willis uh from the DevOps world wrote it. Kind of a big praise for Deming Guy. The slides are available afterward. You can get this for him later. But basically One of the things he describes is if you have a system, find the bottlenecks, and that's what you should work on next, fixing those bottlenecks. However, if we go down this AI route, we can get to a point where

Instead of stepping way back from the system, by the way, all of the animations and like ever almost everything in this deck is public domain. I pulled it from Wikimedia. None of this is AI created. Nothing in this deck other than the Django side I showed you. This already exists. This costs us way less less electricity and it's free. And there's no rights issues. This is public domain. Anyway, instead of stepping way back from the problem saying, what are we actually doing here? Oh my god, what if we just took down that middle section of wall? Like that would help a lot, right? No, we get told, hey, this is going real slow. Make it go faster. Uh move code faster. Do it faster. And we start trying to optimize how wide this door is or how many people are flowing from the back without stepping back because the AI is like focused on that problem. That's the last thing we touched.

And the context window said the feed was the last thing we needed to touch. And of course no one's saying do it safely, right? No one's saying, hey, the thing you did, make sure there's no security implications from that. Or we'll fire you. They say do it faster. Do it by this date, or there's somebody waiting to take your job, and AI is going to make this may help us pay them less. That's the reality I've experienced out there. It's a reality people have told me. And that gets us back to here. Instead of step stepping back and saying, what's the security context of us doing this? What's the larger picture I'm trying to get done and how can we do it safely at scale together? I don't want to leave you just like here's a bunch of problems, good luck. I do have solutions here overall. This is one of my favorite paintings, uh

hoping to prison to despair. I do not know where the original is. I've never seen it. Just came across this one day in the archives, uh internet archive, and fell in love. We can solve this together, but it takes us as people creating a virtuous cycle. We need to raise awareness with everybody you can raise awareness with to start building the right processes. Like how do we get out of this mess? To put the right tools in place. There are tools that can help with some of this. And then train people on how to use those tools so they can document, hey, well, these are the best ways we got results from those tools. And you can automate those tools. And you can start Training people and raising awareness and start a virtuous cycle of how do we improve this over time. Because ultimately this is about communicating with other humans.

My favorite definition of DevOps comes from Kelly Albright, who's a um Drupal developer, or old Drupal developer up in the Boston area. And he said, DevOps is just about shortening the communication food feedback loop. That's it. It's nothing about tools. It's nothing about deployments. It's shortening the feedback loop. And I've always carried that with me. There is this magical thing that happens before you deploy code called the PR. This is where AI has no freaking business. other than to maybe run tests automatically and tell you what the results are or say, hey, there's a major problem and block it. This is the time to have a conversation because it should be Wow, we can't do that because that's a giant inner antipattern that you almost introduced thanks to Claude

Also, what the hell are you doing? The actual deliverable is supposed to address this set of concerns, and our business is concerned with this business logic in a much wider sense. And remind you that. There's also great projects out there. One last shout out to OWASP. Go read these. They're free. They really want you to read them. I love the OWASP top 10 for uh non-human identities. Really good stuff. So we can stay safe at night and alarm bells don't go off and you can be at home with your family playing an impossible game of Monopoly. No kid has that many hotels. No kid's still in a game that has that many hotels. That's beside the point. This is a family doing something they love. And that's what I am in security for. Hopefully, hopefully, hopefully, we can get everybody on board to create that virtuous cycle.

So we can all go home and do things we love instead of worrying about who's breaking into our stuff all the time. I'm Dwayne. I live here. I have a podcast. Please check that out. Happy to give you all the information. And again, we are hiring, but that's this is the right slide. We're hiring. Uh I'll put it in the jobs channel in the Discord or yeah, the Discord. No, we're in Slack. In Slack, and uh there is the slides URL and um QR code, thank you very much.