How to design and implement extensible software with plugins with Simon Willison

Speaker 1: Good afternoon Django Con. This talk is about designing and implementing extensible software with plugins. When I pitched this, it wasn't really a Django talk. It was one of those sort of like a slight angle from Django. It is now a Django talk. I have Django plug-in stuff to show you that I will be live coding shortly. And I realized that this talks really it's almost about the plug-in sort of living the plug-in life um lifestyle. Like I have over three hundred projects on PyPI at the moment, which I'm mostly maintaining and fixing bugs in. And that's because I've got so heavily into plugins that most of the work I do now is building plugins for other packages that I've built. So I'm going to show you what that looks like. We're going to try and sell you on plugins as something that is a really powerful sort of architectural concept that we should be using for all sorts of things that we're not using

Speaker 1: so far. And so this is how I got into plugins. This is my project dataset. I've been working on this for about six years now. It's an open source tool written in Python. for exploring, analyzing and publishing data. So this came initially out of work I was doing with newspapers where you've got data behind stories that you want to expose online. And since then it's grown into a tool where if you're a person who thinks data is interesting, which I think everyone here is, this is hopefully a really powerful addition to your toolbox for that kind of thing. And dataset, I quite early on I realized that I wanted it to have a plugin system basically inspired by WordPress. Perfectly decent sort of core CMS and blogging engine with 10,000 plugins that mean it can solve any publishing problem in the world. So any problem that you have around publish

Speaker 1: sort of content management, WordPress plus a stack of like Plugins of varying quality and such like will probably get you to a decent solution. And I want to build that for data. I want I want to be in a wo live in a world where any problem you have that involves taking data and exploring it and analyzing it, there is dataset plus a plugin that will get that do that for you. And so dataset has a plugin directory and we've got 158 plugins listed at the moment. I wrote almost all of them, but I didn't write all of them. Like Success criteria for plugins is very much when other people start writing them on your behalf. And really, I've begun to think of plugins as the absolute perfect form of open source contribution. Because If somebody files an issue track I issue on one of my issue trackers, I have to go look at it.

Speaker 1: If they submit code, I have to review that code. If somebody builds a plugin for my system They can go all the way from idea to shipping their thing to launching it to other people installing their work without me being involved at all. I can literally wake up one morning and my software grew a new feature and that's fantastic And so I've, as somebody with far too many projects on the go, this is a way of working that I really, really enjoy. I'll show you a couple of examples of plugins. So this was actually the first plugin I ever built. It's called a dataset cluster map. And the idea was there are lots of databases out there where you've got latitude and longitude columns. Can we see this here? Yeah, there they are. And what this plugin does is it notices those columns and it draws a map. So it's very, very simple. It's a little bit of JavaScript that says, hey, there's a column called longitude and a column called latitude.

Speaker 1: I'm going to stick a map in there and I'm going to put all of those points on there. And I can actually load 35,000 points onto the map and it works because browsers are fast these days. But this is really nice because just taking this database, this is every power plant in the world. The moment you put it on a map, you start noticing patterns. Like there are two little power plants down here. This is McMurdo Station in Antarctica, which has one oil generator, and it's also got a windmill down there. Which is kind of cool. And the sort of thing that you won't notice if you're just looking, browsing around through a table of data, but the moment you start doing these visualizations, things like that jump straight out at you. The other thing I love about plugins is that they let me basically go off to things that are probably bad ideas.

Speaker 1: And this is actually sort of uh it's incredibly liberating. Like if you've ever worked on a project, there are lots of ideas that come up where you're like, well It'll be kind of fun, but it's not a good enough idea to be worth besmirching my beautiful code base with this thing, especially if it turns out to be a mistake later on or I have to maintain it for the rest of the lifetime of the project. Plugins break that um break that barrier entirely. This here is an example of what I think of as spite -driven development. Right. I um I was working on a project with somebody who was really into GraphQL. They were like GraphQL is the future of everything, like rest is over, this is this is this is this. And I didn't think they were right, but I didn't feel like I could argue with any sort of form of authority because I'd never done anything with GraphQL. So I sat down at the weekend and I knocked up a plugin

Speaker 1: for dataset that added a GraphQL API for your SQLite database, but for all of your data. And um unfortunately the joke was kind of on me because I ended up quite liking GraphQL. It's it's actually um the thing about GraphQL, it's got one of the best development tools I've ever seen, this little um interface here, where you can run these GraphQL queries Tweak them, you get like um you get auto-complete of the available fields, all of that kind of stuff. It's really nice. And the single biggest downside of GraphQL is it's very easy to construct one of these that triggers a thousand database queries. It turns out SQLite eats a thousand database queries for lunch. The whole sort of big benefit of SQLite is because you don't have a network server that you're talking to, a thousand database queries is like a thousand C function calls. It's nothing.

Speaker 1: So Weirdly, in in sort of spitefully trying to prove that GraphQL was was bad, I ended up sort of convincing myself the other way. But I would never have done this in dataset core. Like there was no way I was going to take this experimental technology that I thought was a bad idea and bake it into my open source project just to explore it over the course of a weekend. Whereas this this gave me the freedom to explore that. And I've abused that freedom many, many times since then So that's my main plugin project, is dataset. The other significant project that I've implemented plugins for is this tool that I've been building called LLM. LLM is a command line tool for interacting with large language models, things like OpenAI 's GPT-4 and Claude and all of those kinds of things. And when I first released this,

Speaker 1: all it did was talk to OpenAI. So you could inst you could pip install LLM and then you could say LLM and then a prompt and it would run that and feed you back the result. But after a few months, um all of the other models started getting really good. So I added a plugin system to this to let you add additional models using plugins. And I can demonstrate that right now. So I've got say hi in French. That's ooh, I've got to install it first. Pip install LLM. I gave myself a clear, clean virtual environment to make sure I couldn't cheat. So there you go. I say LLM say hi in French, and it says hi in French is salou. But if I type LLM models, I can see that so far All that I've got available to me are these open AI models, right? GPT-3 and 4 and 01 preview and all of those things.

Speaker 1: So I'm going to install a plugin. There's a plugin that I wrote called LLM. GPT4AL, this is an open source library with a slightly confusing name that gives you easy access to models that run on your own computer. So if I install that. And then run LM models again, it'll show me that now I've got access to all of this new stuff. And let's uh increase the size of the window there. And these are models that get downloaded. Some of these I've already downloaded, so I didn't have to sit in the talk waiting for a four-gigabyte download. But now that I've got that, I can say things like LLM-M Mistral, which is one of those models that I can run locally, a poem about a Python programming parrot. Let's see what this does.

Speaker 1: And so this is actually running a model on my laptop, which was look at this. I mean it's a dreadful poem. They're always dreadful poems. If you need help with your Python program, just call it this parrot. It won't let you down. It will solve your problems without any fuss or commotion. Commotion rhymes with pro yeah. Terrible poetry. But my laptop wrote a poem. That's cool. Um And this is and that the the the fact that now all it so I've got plugins that add local models, I've got plugins that add support for API-driven models. Like this tool now supports, I think last time I counted, it was over 200 different models that you can talk to, and that's all driven through this plugin system. Quite a few of those weren't written by me at all as well. So I do get that dream of waking up in the morning and my tool can talk to some new thing I'd never heard of before.

Speaker 1: So that's pretty cool. And this here's the plugin directory. These are all of the plugins I have available for LLM at the moment. I think it's 25, maybe 30 now? Um oh actually I will show you one more um because this one's quite fun. So I built a plugin, so LLM, you can have plugins that add new models to it. You can also have plugins that add new commands. So if you run LLM-help, you can see here are all of the things it can do If I do llm install llm hyphen cmd, this installs a plugin which adds a new command. There's now a command called cmd. Which will generate and execute commands in your shell. If you think this sounds like a horrifying idea, it kind of is, but um, it means I can do things like say LLM CMD U

Speaker 1: Spotlight to find PDF files mentioning Django. And that then outputs the command it thinks I should run. Notably, it doesn't just run it because then who knows what will happen to my computer. That looks kind of good. So There we go. And that actually that worked. That was quite a good demo. And I use this thing all the time. You know, it's like, I can't, I actually use FFmpeg as a piece of software now because it has the most obtuse set of command line options you've ever seen. But the language models know how to do it, so so I'm now now using it on quite a regular basis. So that's sort of fun. Um what I'm going and One th a point I wanted to really emphasize is that Python is an astonishingly good language for building plugins in. Like the very the dynamic nature of Python makes it perfect for things that load additional optional code and all of that kind of stuff.

Speaker 1: The introspection support is really good. And there's also this feature baked into Python packaging called entry points, which it turns out is is basically a plugin system that's baked into the core packaging specs that we all that we all use. There's even a section at the bottom of this. I think it mentions plugins on this page. Yeah, here we go. Entry points for plugins. So you can actually roll a very effective plugin system. just using these entry points. And the key there is that when you install new packages, their functionality magically becomes available to the rest of your program. But we don't have to use entry points directly because we can go a step further. There is this absolutely phenomenal library called Pluggy, which was released by the it was built by the PyTest team.

Speaker 1: PyTest is a giant stack of plugins with additional plugins elsewhere. They built this first and then they extracted it out of PyTest and made it available independently. And this is what I've been using for all of my tools. It's Really well documented. It's quite small. It's like a couple of thousands of lines of code for the whole thing. And it's beautifully well designed, really stable. It is a fantastic framework to start building these different things on top of So, I'm going to do some live coding and show you what this plug-in lifestyle looks like. I'm going to ship some software on stage. Today, about two and a half hours ago, Google Gemini announced a bunch of new models. This happens about once a week at the moment. You can read through all this, but the crucial thing is that they are called

Speaker 1: um Gemini 1. 5 Pro 001 and 002, these things here. Uh yes, I will oh I'll well I'll There's no honestly their webpage isn't that interesting. But what I'm going to do is I'm going to add those to my existing plugins. So I've got this plugin called LLM Gemini, which currently provides access to three Gemini models. I'm gonna pop that open here and open up. Oh no, you can't have the latest update. So all this plugin does is it implements a plugin hook called register models. And when you're building extensible software

Speaker 1: Really, this is what you're doing. You're defining these hooks which can be called by the used by these plugins to add additional functionality. In this case, I've got register models, which registers three instances of my Gemini Pro class that I wrote And I'm gonna add four more. Let's do this. There we go. Oops. And I messed that up, didn't I? Gemini Pro, LM model. So that should do it. That should add those four different models right there. If I run oh if I run LLM models again, there they are. I'm gonna test them by pasting in a quick set of four commands.

Speaker 1: Let's do I'm gonna tell it to I'm gonna say say hi to all four of those. Hi, how can I help today? Hi, hi, how can I help today? And hi. That's interesting. So 001 Has a longer answer than 002. They actually said in the documentation that they've been trying to cut down on the default um the default wordiness of these models. But that's it. We're done. We've just added a new feature So I'm gonna say I'll run black on that first. I'm gonna commit it and and two models Going to push that. I'm going to bump the version number in my Py project file. Let's knock it up to

Speaker 1: 0. 185. And now if we go to GitHub for that package, here we go. My changes have just landed. My unit tests, which are a little bit thin for this project, are running away. And I can now deploy this plugin directly from GitHub to PyPI using the GitHub PyPI automation that was released, I think, about a year and a half ago. The way that works is you create a new release. I'll call it LOP1A5, give it that, new Gemini 001 and 002 models, full stop. And I'm going to mark it to pre-release. And that should be it. So what this is going to do, it's going to run a GitHub action that I've got configured

Speaker 1: for That release. Uh where are we? This one here. Runs all the tests, packages that deploys it to PyPI, and in about 30 seconds time Anyone in the world who attempts to install my LLM Gemini plugin will get that new version with those new models in. I do this a lot. Like often in often over the course of a week I will ship five, ten, maybe e even fifteen of these different releases. Because the projects are all so small, they're very lightweight, that I can update them independently of that core tool. Imagine if I had to ship a new version of LLM itself Anytime any of these companies released a model, that would that would not really be a sustainable way of working. But there we go. That's all worked.

Speaker 1: So this is what I mean by the plug-in lifestyle, being able to iterate very, very quickly on just tiny little aspects of an application. Safe in the knowledge that the worst thing that could have happened right here, if I'd broken this plugin, it would have just broken that one tiny little fraction. Everything else keeps on working. Thing of that these things as well. In fact, let's see if that did work. So if I do LLM models in my global install here, I get back, this will give me a lot because I've got lots of plugins installed. Oh, hold on, which Oh that's annoying. I've already I I already installed a Test version of that plugin. But imagine that that hadn't given me the model and upgrade I'd I'd have upgraded.

Speaker 1: It would have all been all been all worked absolutely fine. So that's sort of this is the the core idea with plugins then is we're building for a world where we have this huge flexibility. We spend a lot of time thinking about the hooks that we should provide, like which parts of our application should be extensible. But it gives us this enormous flexibility in terms of building new features, collaborating with other people, and having other people ship stuff to production on behalf of our applications without us even having to review pull requests, which I really like. But I promise you some Django related stuff. When I sat down to write this talk, one of the questions I asked myself is what would Django look like if it had a plugin system? Like what what kind of things would be useful for plugins in Django

Speaker 1: And then I popped open a text editor and started just exploring and poking around with it. And um now as of this morning I have a plugin system for Django that I'm ready to demonstrate. So I guess this is the the launch announcement for my weird new Django plugin system. So I built this. This system, it's called DJP. Because Django Plugins was already taken on the Python package index, and I like three-letter acronyms, my packages, so DJP stands for Django plugins. And what this lets you do is it lets you build reusable applications for Django that people can install just by pip installing the package with no additional setup. I'm sure anyone who's installed like an application for Django will have done the thing where you install it, and then you open up settings.

Speaker 1: py and you add something to the installed apps list, you add something to the middleware list, you pop into your URL configuration, you add something in there. There's like Three or four steps needed to get code working. This eliminates that. That's the problem that this is solving. And so what I'll do is I will demonstrate this in action. Let's do Let's do that now. Okay, so I'm going to pip install Django and then I'm going to Python-m Django start project. I'll start the project called demo. Oh, did that I'm going to remove my previous demo and start a project called demo. Here we go. So now I've got myself a brand new Django app and I can do manage. py migrate. Manage. py. You know, I'll create a super

Speaker 1: user just so I don't have to later. There we go. And so now if I do manage. py run server, here I have a Django app with our lovely little welcome screen there. But I want to start installing plugins into this application. And I'm going to start with a very simple demo plugin that I built yesterday. And all this one does is it adds a custom HTTP header to every page on your Django Django application that outputs a Django composition with the name of a piece of music composed by Django Django Reinhardt. So it's a lovely sort of nod to our to our heritage as as Django developers. And so the way you install that is first you need to

Speaker 1: configure DJP, you need to configure the plugin systems. There is a one-off um set of steps that you have to go through to get all of this stuff working. What's happening there? So you go into settings. py and is that legible Yes, great. So we go in settings. py and I type import djp and then the bottom of this file I say djp. settings globals. This is A horrifying hack. What this lets this means that DJP just gets to futz with the settings any way that it likes. But we're done, it'll work. So I do that. And now if I do pip install Django plugin Django header

Speaker 1: and run my server If all went well, when I refresh this page and view headers in Firefox, go on. Oh. There is a Django composition header. That worked. So we installed a plugin. It did do something moderately unspeakable, but it did it did it did pull things off. And this works for applic full full-blown applications as well. There's one extra thing we have to do here. We have to go into urls. py and we have to do import DJP. And then we say URL patterns equals that plus djp. url patterns. So we're essentially letting the plugin

Speaker 1: system just chuck its own stuff into that list of things. And I've got this other plugin I wrote called Simon called Django Plugin Blog, which is a full-blown blogging system. It's actually based on a a little um like TIL tutorial I wrote a wrote wrote a little while ago. So if I do this There we go. And then if I do manage. py migrate, it created the migrations. And now if I do manage. py run server, and go back here. What this should have done is added a blog at slash blog There it is. It's got an RSS feed. It hasn't got any content, but it has got an RSS feed. If I go to slash admin and log in as

Speaker 1: I think that's right, it registers itself with the admin, so I can add an entry saying first post. Hooray. This is a blog. I can add myself as the author. And we are done. There we go. It's a blog. Look. Woo! Um and it's got a year archive page and it's got the the RSS feed like works and all of those kinds of things. But crucially This I feel like this is the dream, right? We've talked about in the Django community, we've talked about reusable applications for such a long time. And The ecosystem has never it's it it it feels like I can't really just pip install a blog into my Django project so uh up until today. And now I can. And it's got like custom templates work and all of that kind of stuff.

Speaker 1: So This feels like something interesting. There's something very interesting about being able to effectively like have a requirements. txt file with a list of features that you want. And when you install that, all of those features spring into existence. And I know that works because that's how my dataset project has been working for a few years now. I've been building this system called Dataset Cloud, which is the hosted software as a service version of Dataset. So paying customers get their own private instance that they can sign into and invite their friends and upload their files and so forth. And if you go to the debug URL that shows you the list of plugins, you can see that all this product is, is just a giant, I think there's about 58 plugins now. A whole collection of plugins adding all sorts of different features.

Speaker 1: I can customize this for individual customers. So if a customer needs a feature that nobody else needs, I can drop in a custom plugin just for them. This feels it 's really exciting. It's a really fun way of building software. And it's a way of addressing, to a certain extent, there's this challenge I've had with software I've worked on in the past. Well, you always have to think of your features in terms of a matrix. So every feature that you add has a chance of interacting with other features. And so as the software gets larger, you find that okay, we've got 10 features, but that means that there are 10 ways they might interact with other features as well. And all of that is stuff that you have to think about. That is entirely true of plugins as well, like plugins have that effect But because of the nature of them, it sort of encourages you to design your hooks such that you minimize the conflict between the different things.

Speaker 1: The entire art of designing systems around this architecture is about saying, okay, what are things that plugins could do that really are standalone that won't interfere with other things? Like with Dataset, one of the plug-in hooks that I have is something called database actions. So I can Plugins can basically add themselves to this menu and say, I can do a thing to a database. I can query it with AI assistant, I can do export, I can um edit the database -based metadata. Each of these um menu options comes from a plugin. And they all sort of fit together in that list, which is mostly good. A problem I'm facing at the moment is that the way pluggy works, I don't get to control the order of these things. So I've kind of got this growing list of things that you can do to a database in a semi-random order, which isn't great from a usability perspective.

Speaker 1: So I'm thinking a little bit at the moment about how I can regain control over this and have a bit more editorial control in the way these things are presented. But it all works. It's a really powerful way of of of building and constructing software. So I'm going to be a tiny bit more ambitious and I'm going to build and ship a plugin live on stage. Partly to mainly to reinforce this plug plugin lifestyle idea again and to show you what developing one of these things actually looks like. And the plugin I'm going to build is one that adds Gzip to a Django project. This is a complete cheat because Django in its core library has a really good gzip middleware, which and all you have to do is add that middleware to your middleware list and it'll work. But I'm going to make it so you don't even have to do that, right?

Speaker 1: You can just pip install Django plugin gzip and it'll do the right thing. So let's create ourselves a GitHub repo. Um gzip. Plugin for Django. And I always like to have an issue for these things just so that I've got something to reference. So initial design adds the Gzip. Middleware using TJP. There we go. I'll make that an enhancement because it's nice to have labels on things. So let's create this plugin. The good news is that I'm a power user of cookie cutter templates these days. So anything like plug- and anything that I'm going to be on a repeated basis. I build a template for. This is my new Django plugin

Speaker 1: cookie cutter template, which if you run this, it will get you up and running with um It'll get you started with all of the things that you need for a plugin that has tests and continuous integration and all of those kinds of things. I'm actually going to uninstall the Django blog. Django plugin blog because otherwise it'll interfere with what I'm doing here. So I think I need Cookie Cutter. Okay, I can run um oh wrong line. I'm going to run cookie cutter gh colon simw slash Django plugin. And this will fetch that in and ask me a few questions. So I'm going to call this Django plugin gzip.

Speaker 1: I'm going to give it my GitHub username and author that goes into the metadata. And that's it. So now if I do code on Django plugin Gzip, here we have a nice little Oh, copilot, this hit my rate limit again. Actually I'll show you what that looks like. So we've basically got a license apply project. We've got a bunch of test configuration, the module itself that will act as the plugin , a little bit of um GitHub configuration as well. And I should just be able to run, oh no, I need to do pip installed that. And now if I run PyTest, I ship with a default test that passes All that test is doing is making sure that when you run

Speaker 1: when you that that you can hit the index page and get back a 200 request. I really like this kind of test because it means that even if you write no other tests, if you do something, if you screw something up so you get a 500 error, this will protect you. So we've got a test. I'm going to let's do test-driven development for this entire thing. Although in this case I will cheat and just paste in the test I wrote earlier. So uh where's my where's my plugin gone? Here we go. So I'm gonna write, I'm gonna drop in this test. So what this test does is It hits the homepage and checks that there was no content encoding header in that response. And then it hits the home page again with the accept encoding equals gzip and checks that that did come back and say that the content is encoded as gzip.

Speaker 1: So, oh no it passed. Oh no. It failed. Excellent. There we go. We have a failing test So this is the magic of these plugins. When you inter when you implement a plugin for Django, there are four hooks that you can use to mess around with what's going on. There's a hook called installed apps. That you can use to return a string list of strings to be added to the installed apps section. There's that list of URL patterns if you want to hook up URLs for your thing. There's this magic settings one, which can mess around with settings in any way you like. And then the one we care about here is this middleware hook at the end. The Django middleware, I happen to know, is called Django. middleware. gzip. gzip middleware. So I can stick that in there and that

Speaker 1: will probably do the trick. Let's see if it passes the tests. It didn't. I wonder if that's because I need to do this, DJP. before. So because Django Middleware, the order does matter. I've set it up so that when you're using this plugin hook, you can wrap things in a before or after class, which will cause them then to be jumped to the top or the bottom of the middleware list. That didn't work. Why not? Um definitely worked when I tried it out this morning. That's I don't have to put it after do I? Oh

Speaker 1: no, it's as a list. Huh. Is that the correct init. py? Yeah, it should be. DJP URL patterns. Um Django utils, Django response. Just running test Django plugin gzip. Um I thought I did. Maybe I didn't. Is that well, I may have to oh it's before, isn't it? I may have to give up on my grand ambition of shipping a plugin live on stage.

Speaker 1: I'll figure this out and uh I'll I'll ship this to GitHub once I get it working. But that's the core idea. So the core idea with this system is if you've got middleware that needs configuring, you can wrap it in the middleware plugin. If you've got things like that blog demo I showed you earlier that need to register their own um their own uh URL patterns, they can do that And it turns out almost all of the other functionality in Django that you might want to have a plugin do is covered just by these four hooks. If you want to add additional management commands, those are automatically picked up from the installed applications. The same thing for things like template tags and context processors. There's a whole load of stuff in Django which Really, if you can impact these two these three things, if you can impact the list of installed apps, the URL patterns, and then maybe futz around with the settings as well, all of this stuff just starts to work.

Speaker 1: So I've built a few plugins to I one of my one of the things I've learned in building plugin systems is never ever ship a plugin hook without accompanying it with a plugin that uses it. The few times that I've broken that rule, I've come to regret. You have to have at least one plugin that actively demonstrates the thing that you're doing. So I built three plugins for um I've built three plugins for this Django system so far There's the blog one, the headers one, and the other one I built is called Django Plugin Database URL. And all this does is it takes the DJ database URL package and it configures it for you. So once you've installed this plugin, anything that's any settings you have in a database URL environment variable like you might get on.

Speaker 1: on something like Heroku will be picked up automatically. This one I built basically to illustrate what to demonstrate how this settings uh that how the settings uh plugin hook works I promise you I'd talk a little bit about designing plugin hooks, because that really is one of the least sort of well-understood and documented parts of all of this. And I'm still, I'm sort of four years into my plug-in hooks journey, and I'm still figuring that out myself. But I can show you the most complicated set of plug-in hooks I've accumulated. This is um My dataset project has been evolving its plugins for I think four or five years now. And so most of these came out pretty well. Most of these are are working quite effectively.

Speaker 1: Oh hang on, that's stable. I need to go into latest to get the most recent ones. So I've started grouping plugins into different families as well. I showed you the plugin earlier that adds things to that menu. I call those action hooks and I've got those for tables and views, queries, rows, database, and the homepage. So that's the same format of plugin for five different places. I've got these template slots plugins that I'm experimenting with where a plugin can say, I have some HTML and JavaScript that should go at the top of this page. So anytime like things like the um the mapping plugin I showed you earlier, which needs to inject additional code into a page within the interface, I've been experimenting with um having these sort of named areas that plugins can extend

Speaker 1: I've also been there there's a pattern of plugins where one of the things you can do with them is use them to register additional things. When I showed you the LLM plugin earlier, I think that was called Register Models, and that's a hook which all it does is returns a list of additional classes or objects or functions or things which the application can then use elsewhere. And so that's really one of the two ways these plug-in hooks can work. They can be you can have them return a bunch of things that get used later, but you can also and you can also have them just do something. You might have a plug-in hook that says um I've got one for event tracking that says a interesting thing has just happened, like a table has just been created, and all of the plugins that implement that hook will get called in a row and they'll be able to then act on that information

Speaker 1: So I will be I'll be writing this up in quite a bit of detail later on. Because I'm still trying to figure out what are the patterns that work most effectively myself. One thing I'll say for Plugie that's incredibly useful is Pluggy is designed to be forwards compatible in terms of adding new features to your plugin hooks. And so the best way to describe that is to look at Let's say we look at the let's do the uh the that that database actions hook from earlier. So right now, this plugin hook is defined as taking four um four parameters. It could take the data set, which is my sort of magic object that does all of the work. The actor is the current user.

Speaker 1: The database is the name of the database that they're acting on, and then there's an HTTP request object as well. And so you can build plugins right now that use all four of those, but you can also, when you define a plugin , a plugin hook, you can have it take just a subset of those options. It can take none of those options, or in this case we've got one that just takes the dataset actor and database but doesn't take the request. And this is super useful because it means that If I add mute new options to this plugin hook in the future, if I upgrade it and say, Oh, I'm gonna pass in a list of tables as well All of the existing plugins will continue to work. Those new options that we add to the plugins are effectively invisible to the things that don't care about them, which gives us a really powerful upgrade path. we can continue expanding out these hooks and adding new options to them in

Speaker 1: a way that doesn't cause problems for the the plugins that people have written already. There's one other trick that I'm using quite extensively in dataset because dataset is a very heavily, very heavily async. Almost everything in dataset is asynchronous functions And so I came up with this pattern for writing plugins where the plugin hooks themselves still currently have to be regular functions. They can't be asynchronous. But I find myself wanting to be able to do await things inside of my plugin implementations. And so with dataset, a plugin is allowed to do a thing. Or it's allowed to define an asynchronous inner function and then return that function to the to the framework itself. I call this the await me maybe pattern, because I like the joke.

Speaker 1: I have a function called await me maybe that you can call on this. But this works really well. This means that plugins can effectively return a asyn an async function that should be awaited on at the right moment and it all fits together perfectly. Pluggy are working on asynchronous support in plug-y core at the moment. So I haven't kept up to date with that. So this kind of hack shouldn't be necessary at some point. But it's been working really well for with for me for the last couple of years. If you are interested in this stuff that I've just shown you, I'm going to be hanging around for the first half of the sprints on Thursday. If any wants to like sprint on the idea of Django plugins and explore that further. I'd be delighted to talk to people about that. And I will be posting very detailed notes and write-ups of all of this later on.

Speaker 1: And I think I have five minutes left for questions I have six minutes for questions.

Speaker 2: Really, really good. Uh do we have any questions in the audience? All right

Speaker 3: Hi, thank you for your talk. So this might be a naive question, but uh I'm sold on plugins. But why this was originally in Django? Oh was this ever considered? Was this ever discussed?

Speaker 1: Yeah, I mean Django has signaled And signals were a very early version of effectively the same idea, the idea that things could subscribe to signals and act on them. There have been there was a the the Django hyphen plugins package on PyPI is an older idea that was based more around model level pack um plugins. So that's been explored. I think what's new here is basically this is all about plug-y. Like the the the fact that And it's that and the the the fact that the Python packaging system has matured to the point that metadata and packages for plugins works really well. Like it's it's a proven pattern now. So you know, twenty years ago none of this stuff existed and that there was nothing we could have worked with. I feel like today Pluggy is now super mature and stable. The packaging infrastructure is amazing. I feel like we're in a really good spot for for

Speaker 1: taking on these kinds of ideas.

Speaker 3: Makes perfect sense. Thank you.

Speaker 2: Who else do we have Right. We'll go here and then I'll move back.

Speaker 4: Thank you, Simon. Uh given your horrific abuse of globals, uh do you have a Uh obviously a high-level idea of like how would we do that without doing something that horrific, right? Like is there like a better approach to that?

Speaker 1: The globals trick isn't necessary and actually a earlier version, I only added the globals trick last night Um prior to that, if we if we roll back in the in the um here we go, release naught point This one here. Right, if I go back to this commit and look at the readme, this should show what it looked like beforehand. So yeah, what it looked like beforehand was this. So you say installed apps equals list of apps plus Djp. installed apps, middleware equals this one, djp. middleware has to wrap the middleware so it can add things at the beginning at the end. And then that thing at the bottom. And the thing at the bottom was there because I wanted to have plugins that could mess with templaters and all of that kind of stuff. And I looked at that and I thought, you know what?

Speaker 1: If and I actually I messed up twice in forgetting to do one of these steps myself. So on that basis, I'm like, you know what, if I'm hacking globals, I'm hacking globals, and I I f I I I rigged it up to do that. But if the globals thing turns out to be a horrible idea, this is this is sat right there. It's something we could do instead

Speaker 2: We st we still got a couple more minutes. We got another question here.

Speaker 5: So I I think this could be really productive and a lot of fun actually just to be able to add new functionality to a Django app, but are there concerns at all with uh it being a level of abstraction that would make you know be too magical and And let people not know what's going on, or are there ways to mitigate that at all?

Speaker 1: That is a great question. And this has come up for me in all of the other projects that I've worked on with plugins. There is Like you need tooling. So actually one of the features I've got in DJP is I think if I go up here I can do manage. py help. And it adds a command called show plugins. So if you say managed. py show plugins, it shows a list of the currently installed plugins. And I want to do that there also like Pluggy has lots of tracing options that you can turn on as well. So I feel like, yeah, that's an absolute considerate of. You can build debugging tools that make that make those problems a lot less worrying. So yeah, it's it's it's about tooling at that point, I think.

Speaker 2: All right, we got time for one more. Any other questions? All right. Oh hoo. All right, here you go

Speaker 6: I was just curious, is there um are there hooks for removal for uninstalling a plugin to get rid of the state left behind installed?

Speaker 1: That is a great question. So the question um are there hooks for removal? So By default, Pluggy, actually Pluggy, the the Setup Tools integration of Plugie is an option itself. You have to call a special function to load all of the plugins that are registered with the metadata You can do additional things on top of that. One of my projects, I've got an environment variable I can set where I can opt into specific plugins. And I'm still trying to figure out what the pat what the best patterns for that are. I've certainly found that with datasets. Sometimes you want to you've got dataset with 50 plugins installed and you want to just turn one of them off. So it can be done, it's additional code that you have to write. And that's something which would be good to bake into a system to make it to give you that additional level of control over what's running and what isn't.

Speaker 2: All right, well, thank you, Simon. And if you have other questions, you can meet Simon out in the hall and speak with him. Let's give a round of applause for our speaker.

Speaker 1: Thank you.