Building a little YouTube on Wagtail

Speaker 1: Okay, we're at time, so we'll get going. I hope you're all having a great time. Recordings are available right after the talks if you wanna catch up. But for now, I'll hand over to Tom. Tom uh co-founded Touchbox, the agency that started Wagtail, and Tom started Wagtail 11 years ago now. And uh Tom is gonna talk today about uh building a little YouTube inside Wagtail which sounds really quirky and interesting. Over to you.

Speaker 2: Thank you, Thibaut. Hi everyone. Yeah, it's it's a bit um it's a bit aggrandizing, really saying a little to YouTube because it's sort of the it's uh it's a tiny thing compared to that, but uh I thought that would at least catch the attention of the uh of the people reviewing the talks. Um and it is the so just a little background, as Steve said, I uh have been a I was uh one of the co-founders at Torchbox starting and uh in 2000 and uh I've been really involved in Wagtail right from Wagtail's beginning and uh I think But yeah, that involvement with Wagel was really the sort of highlight of my time at Torchbox for the for the last ten years. But I'm I'm not at Torchbox anymore. I'm still sort of around uh on the edges a bit, but um I'm now doing bits and pieces, including

Speaker 2: Building writing code which is uh which is fun and something that I hadn't done much um in in the kind of years up to now And one of the bits of code that I wrote is uh a project for a friend, uh friend called Chloe, who is a Pilates instructor. And I'm going to start by just giving a quick demo of the site and then I'm going to talk about how I built it and uh and why I feel like Wagtail turned out to be a surprisingly good fit for this So I guess the nearest analogy to this site is something like um Netflix or or Amazon, the sort of the TV app. Actually , For those of you in the UK or who or who are familiar with it who aren't in the UK, iPlayer, BBC iPlayer is probably the closest link because as well as on-demand content, so

Speaker 2: all like the the previous The videos that have been encoded in the past is also live, so not quite like live channels, but there's there's live content as well for people who want to turn up to the live classes. So you're seeing now the view in fact everything can see everyone can see this is public on the internet, but uh only people who are subscribers can actually watch the videos. And so you'll see to start off with a list of videos with um in chronological order that I can I can search. One nice feature I like is that uh uh I've taken a legal snapshot link of uh image from each video. I think about 45 seconds in I experimented with some different ones and then as you roll over it's generated this an animated image which starts in the same second so it's kind of you get this nice feeling

Speaker 2: as you roll over that you can see Chloe moving around doing her exercises And so as you can see from the left hand side you can filter by various options. There's also this ability to go to featured videos, which is like curated. slices of of the content. And this is again similar to what you'd be familiar with if you've if you've ever used Netflix, which is or Amazon Prime or iPlay or something like that, which I'm sure most of you have. uh where you have these kind of slices of curated content. And uh so an example here, short workouts, if you if you have less than 20 minutes to do your workout. And for each of these you can click view all, and in most cases they just they're basically like a front of a view for a predefined search, so this one with it with with all with durations under 20 minutes.

Speaker 2: You can also favorite videos, so I can click the heart on this one and they become available from my favorites. So just a quick way of going back to them. As I said before, as well as the on-demand videos, there's the ability to watch live. So there are no live sessions right now. But here are two which are scheduled for if I click this one for 111 hours time Um and uh as this ticks down to zero seconds then it s it uh flips in immediately into the the actual live second session. And then once the session has completed, it gets converted behind the scenes into an on-demand video, which you can watch here. There are programs which are kind of curated lists of videos. So uh Um here's a session for uh for September

Speaker 2: and uh each one has a a day and you can have rest days in between. All this is managed through Wagtail. And there's a bit more, but I'll I'll stop going through it now because that's that's kind of the heart of it. And when I started, I I was pretty clear at the beginning of this about what the features would be, how this site was going to work, but I wasn't I wasn't that clear on on the technology. And in fact, it wasn't really obvious to me at first that this was going to be a good fit for a wagtail. So I thought maybe I would build this. I'd probably use Django. But can plain jungle with models and maybe using DRF for an API and uh and possibly uh uh a next front end. But as I started thinking about how to model the videos themselves and uh

Speaker 2: and the the the workflow of Chloe, who's really the main, the really the only editor of this site. more and more it felt like White Tail would be a good fit. And in the end, um I think it's worked out really well. There are some edges to it which don't feel like a the most natural fit, but I think uh it's ended up being surprisingly smooth. So I'm gonna talk a bit about I suppose that the the main challenge for uh building a site like this is how you handle video. And uh a tool shox we had, you know, many of our clients would uh and still do want uh wanna handle video and although it's possible there's a there's a webtel plugin for for managing uh yourself generally the recommendation is to use um something like YouTube or Vimeo or some hosted service and then to embed them in your site.

Speaker 2: So you're not having to deal with the the the kind of potentially multi-gigabyte files and the and the the different ways of streaming and protecting all those files. The tool I used is a platform called MUX , MUX. com And I should say that although I'm going to say lots of nice things about MUX, this is not really, this is definitely not an advert for MUX, and there are there are quite loads of alternatives. So there's one I've used for another project called BitmoveIn, which is kind of very high-end. You can do all of this in AWS and probably a Cloudflare have something similar and so on. Mux though uh really it kind of hit the sweet spot for me in the way that it's um it focused so clearly on on developers and and uh kind of A strap line for Mux

Speaker 2: could be something like Stripe for video. So Stripe, you know, well known for having the best developer documentation, a really good developer experience. And I think Mux probably probably tries to be the same. Th I guess the the key feature that MUX provides is something called HLS, HTTP Live Streaming, which is pretty old technology uh came up Apple came up with it in 2009 and really the idea of of HLS is that you um uh so you up you start with a high quality video so maybe a 4K video that you've captured on your high quality camera. and uh and you and you you s you s send that to some service which converts it into lots of different slices of the same content. But so you and those slices

Speaker 2: are made up of chunks, but also each chunk. um has what's called an ad uh adaptive bitrate. So it means that as well as the the high quality 4K one, you'll have a 1080p one, a 720p one, maybe even a like a 480 pixel version at increasingly lower bandwidth. And that means that people who are on small devices that can't see 4K or who are on low bandwidth um see the the lower one first. And it means you get this smooth experience. and um and it minimizes buffering while maintaining the the the the optimum quality. And FFmpeg can definitely can create HLS files for you. In fact uh almost certainly mux and amazon and all the else are just using ffmpeg behind the scenes and uh uh yeah you I mean they're all

Speaker 2: sort of just a bit like FFmpeg. in the cloud, but um there's there's quite a lot more detail to it than that that that you get from tools like mux. So this is within the MUX admin and uh and you can see it's a brutally developer focus. So like I can't even see uh see the the thumbnail of the image here. They don't have titles, they just have you know these long UUIDs. The reason I can't see the the image is because this is a protected video, so that means you need a token to watch it, which is how how you handle um uh non-public content. When I look at the asset details, which are things like thumbnails and subtitles and so on, I just really see the H

Speaker 2: the HPI responses to all of it. Um but that's so that feels it's it's it's not very friendly, but uh it's uh it it's kind of telling me that this is uh very an API driven service. And it's so I guess another way of thinking about things like Mox is it's not um Mox is not like a replacement for Vimeo, it's like a tool you'd use to build a replacement for Vimeo. Which incidentally might be a useful toss because I think Vimeo's just been bought by some nasty VCs and it's turning into something else. But I I could be wrong about that. Marks also seems pretty pretty reasonable. So here's the billing page which is showing that in the last month I think we're storing four and a half thousand minutes of video, streamed 15,000 minutes and that currently comes to $9. 99.

Speaker 2: And so it's pretty, you know, you can run a what 's a pretty significant video-based service on the back of tools like Muxet for a relatively little cost. One of the challenges for uh for handling video is what to do with all these with these massive files. And um and and because of the sort of traditionally synchronous nature of apps like like Wagtail, it can be quite hard. So y you don't really want to upload a two gigabyte file into Wagtail because it'll use up a thread while it while it's a Python thread while it's doing it. And you have to think about storage. Maybe it's probably you'd use something like S3, but then you've got to push it out there as well. The way that this works, and this is common with other large file uploads, but the way I'm handling it now,

Speaker 2: I'll show you briefly. So Uh on the left hand side here we see upload video and when I click this this is making an API call to Mux which returns with a URL and the URL is a URL that I can post large contents to So just in the in the in between clicking that button and loading this page, MUX has replied with the URL. And I can now choose a file. And I'm going to choose one from This is my cricket team. This is a guy who hit three really juicy sixes in a row if you uh not in England or maybe one of the uh test playing countries you might not know what I'm talking about, but uh this is uh uh sport played. uh in my local village and um so this is a relatively small file 10 megs

Speaker 2: but typically this would be a thousand megs but what's the the the upload here is not to wagtail that's the important part I think it's it's to in this case marks but a third party storage service and uh I'm getting updates through my browser but I'm not tying up the the Python back end while I'm doing it. So if I go back now to before I click on the link here, if I go back to MUX and we see already it's uploaded to see there's a new asset, we can identify this one pretty easily because it's so short, it's only eight minutes and eight seconds instead of 35 So that's now been uh outloaded to MUX and then MUX will ping my my Wagtail site. So uh on every activity mux will generate a webhook and I'm Wagtail in this case is listening to for web webhooks and when it hears that there's a new

Speaker 2: uh uh item assets in much terminology available then uh it will sync and update it here so now I click here and I'm seeing the latest ones I get the thumbnail generated automatically, a title which has been generated basically from the quality and the time it's been updated. And if I click on this now, now we'll see that this is taken to my Wagtail page. And this is where I started really feeling the benefits of Wagtail, that um, although I could have modeled this in Django already I'm getting, for example, immediate real-time preview. So I can click and watch this now. And even from this preview I could be playing it on my big screen and I could see the captions which aren't any in this case and and so on.

Speaker 2: But I'm also getting workflow, the ability to uh to stay save different records, to uh to attach taxonomies which are managed through snippets, um I get uh immediately a uh a slug, which um is gonna be better than this once I rename it, I can uh make it free or not. I inherit from Wagetail just make my life so much easier as a developer and also importantly make them so much easier for in this case Chloe the person who's managing this content who's not especially technical and has never used Wagtail before. Um so yeah yeah that's that that video upload part was something that I I took me kind of a little while to work out my head how how I was gonna make that happen.

Speaker 2: It does mean that you have two records of everything now. So there's the the record on on mux and the record in my um videos, in my worktail pages. Those are held together by keeping a record uh a reference to the MUX asset ID here and whether or not it's public. And that means that I can do things like um so so when I'm serving up the page in Wagtail and it's looking up the asset ID and the playback ID and then serving it directly from MUX. It also means I can use that to track things like so now for example I don't really that that video uh is not going to be relevant for Chloe, so I'm gonna delete it. Um And uh

Speaker 2: I also want to remove it from MUX. So I uh I have built in this little MUX asset manager which is checking the MUX API. and seeing if there are any videos I've got locally which are not being used. And here we can see that while all of these ones are in use, that one I just created has been orphaned. I've just got to resize my screen to see that. Um so that was one of the challenges about um uh uh of keeping these two records in different places, but uh in the end, relatively straightforward to solve and uh and I feel like Yeah, the advantages of not having to handle all these assets really outweigh the the minor disadvantages of of needing to track these items in two different places.

Speaker 2: I'm also going to show live videos because these this works in a quite a different way. So with a live video, this means this is I think it may be easier to explain it by by showing it. So if I was going to create a live video now, it could either be like right now or I can schedule it sometime in the future. So let's make one called Wagtail Space. And in this case, I'm gonna do it for immediately the future. So it's in the UK here, it's quarter to five, so I'm saying it's a 5 pm. And the action of saving that. uh will generate uh what's called a um uh uh RTMP URL and extreme key So RTMP is a is a it's like a a a transport, a bit like HTTP, but for

Speaker 2: for video usually. And um it's uh it's it's become quite standard. The tool that probably is best known for for streaming content like this is called OBS Open Broadcaster Software and it's gained a lot of popularity in the last few years because it's heavily used by streamers. um like uh game streamers or and so on. And it's um but it can be used for any sort of video inputs like your screen or your camera. uh your phone and you can have like fancy bits of hardware or software that combine all these and then you send them off to some stream and that stream could be like YouTube for YouTube Live. Or it could be MUX, which can also handle live content. So OBS is the software that you use, and software that in this case Chloe uses, and she just needs to plug in this stream key

Speaker 2: and then once she starts doing that then it will appear live on her on the live site and I'm gonna delete it quickly because I don't want any of her users to uh accidentally come across a an empty Wagtail Space live program when they want want to watch Pilates classes. That so this was uh live live videos are uh uh uh first off I thought maybe that it could be like a uh a sort of uh a different type of of video page. And then I realized that it's not really a good fit because live videos are more like They're more for sort of holding state than for uh managing content. But what happens after a live video has happened is that they then get converted. So you can see here this converted point.

Speaker 2: In the background, Marks will turn it turning it from a live stream into the HLS format I was talking before. I get a webhook about that, and then I from from the converted page, from the converted stream, it will I can't remember where where it shows me here. It will create a Wagtail page from that. So in fact, increasingly the on-demand content here are is conversions from the live. uh from previous live sessions. Okay, I'm gonna move on from from showing you the back end and talk a little bit about uh the API. So the next thing was my um my decision to use an API. So although in uh in the beginning, as I said early on, I decided

Speaker 2: not to um not to make a uh a JavaScript front end. And in fact I'm I've been happy with that decision. So the the front end is pretty straightforward. Django templates. There's a little bit of some little bit of JavaScript for things like liking and And some modals, but really not very much. And I feel like because I've really tried to optimize for speed, and these pages are pretty lightweight that um y I think the the advantage we get from the you know the extra immediacy and perhaps of a of a JavaScript app just weren't worth it. And I'm I'm yeah I've definitely been happy with that decision and the fact that I get URLs and history and all this sort of thing that you get for free and also just the the the speed of working with with

Speaker 2: with standard Django templates has definitely made that decision feel right for me and for my sets of skills. But I wanted to have an API too because I was interested in some other types of client for this content. And I guess normally in the past I would have used uh Django Rest framework, uh, or in indeed Wagtail's built-in API. But um Because not all of the content here is handled as Wagtail pages, then it felt like the the Wagtail built-in API wasn't going to be quite right. And I was also keen to work with Django Ninja, and one of the reasons was because of the great new page on the Wagtail documentation that uh I think maybe Tebow wrote, um, which really makes it straightforward for understanding how this all fits together. And on the whole, it feels like Django Ninja is uh

Speaker 2: uh you know it's it's sort of easier to to reason with at least at least for me than with with DRF. One of the things I really like about Django Ninja is that you get a built-in spec for your API. So this is if I go to slash API slash V1 slash docs, then I'm getting the the full spec for the API. So as you can see here, it's mainly about retrieving details or listings for programs, analytics, live videos and so on. And also the OpenAPI. JSON file, which doesn't look very fun here, but is but uh is great for computers to read. So um I have found that I've been able to build a mobile app based on just based on that spec and using AI tooling.

Speaker 2: And I'm gonna try I haven't done this before, but I'm just gonna try sharing now my phone so I'm gonna go to advanced iPhone by cable and I hope that at least in a second I don't actually see any notification of this. Oh yeah, I can see it on my screen. I'm hoping that to you are seeing my phone. And one I can click on now, this white circle unstarred one is uh is a mobile app. So this is a native iPhone app. I built it using React Native on top of the Expo platform. It's not terribly fancy, but it does all the kind of things that you would expect, like browsing videos, being able to Of course, watch them, the live video features,

Speaker 2: the ability to handle programs , nice sort of back and forwards iterations, favorites and so on So no really extra features from what's in the uh in the website, but uh a nice UI and probably a good platform, I'm hoping a good platform for uh um for future development, particularly perhaps around no, a a nice feature I think would be offline videos. So um if you're going holiday, you don't know if you're gonna have good in good Wi-Fi where you're going and you wanna do your your Pilates workouts, it'd be nice to to to select some videos for download and uh that's the kind of thing that's hard, pretty I would say impossible to do on a browser site on a but um uh

Speaker 2: but certainly possible with a mobile app I was next going to talk about deployment and I am going to just talk very briefly, but I see that I'm sort of running out of time, so I'm probably going to skip on. quite quickly to the final part of my section. I use fly. io for deploying. I really like fly. io. Um uh it's it's Docker-based um works really portably. There's great visibility over um everything that's happening and particularly this is what uh perhaps what I'm what I like most about it and what's been most used to me is the built-in Grafana for for log management. And uh just a little aside, when I launched it um I was really happy with the performance of the site So really fast and then just gradually over the next two or three weeks it was getting slower and slower and I actually been spent quite a lot of time thinking about

Speaker 2: okay I've obviously underestimated this I need to add in caching I need Redis I need two machines at the front to to handle the load And then two, I mean I should have done this right at the beginning, but right near the end I started looking at the logs and I was being hammered, decided to be hammered by AI crawlers. So uh Claude, open open AI , uh the Googlebot. And there are it's up, I think at one point it was 200 requests per second. And they were all like just misbehaving, looking at the different search facets, cycling around. And that's why though my the site was crazy. And so uh I wish I'd r realized this sooner because uh it would have helped me do what I did quickly which was just uh block all the bots uh and immediately my

Speaker 2: s the the site went back to normal. Here are the logs and it's really interesting seeing things like the webhook uh payloads, um, which are what I was talking about for the way that I it gets communicated with from MUX. I know I'm running out of time now, so for the last section of this, I'm going to talk about the AI tooling. So there isn't really any AI in this app. Uh but I I used AI a lot to or particularly large language models to help me build it. And I guess the sort of the standout point of it is that I didn't write a line of code for this app. Of course I read a lot of code and I made a lot of decisions about code.

Speaker 2: But all of it was written through prompts and reviews. And that's a really feels like a s you know a such a dramatic change. And um So st strange to be kind of saying that, you know, when two years ago that would have seemed crazy and even now it seems sort of impossible. Um It certainly accelerated the way I work. I uh there are some standard ways that probably lots of you are familiar with using tools like Cursor and There's another editor called Kiro that I like in Clawed Code. But a couple of the less common ones that I want to talk about briefly are um So this is an approach which I haven't seen much elsewhere, which really takes advantage of the massive context windows of of Gemini

Speaker 2: 2. 5 Pro in particular. I use just files, so just is like a kind of modern alternative to make. And I've got a justgendocs command which uses this tool, filter prompt. It's one of Simon Willison's little command line tools And uh it ignores a lot of files like migrations and logs and so on, and just and it but takes everything else and converts it all into a big to a single text file, which is in a sort of uh XML format. And then that that file is currently 1. 3 megs big on my and and I upload that to Google to AI Studio, which is weirdly still free. And because of its massive context window, it means that AI Studio can see the entire code base. And I don't write AI I don't ask AI Studio to write code for me, but I do ask it in this scenario

Speaker 2: like that. to give me uh sort of top level advice. So like a sort of bird's eye view taking into account the whole code base. So here's an example of when I wanted to start working on the mobile app. past the whole code base. I talked about what I want in the app. And here's an example of the output about what's possible and uh and you know a a well reasoned um starting point that would help me then split that up into features and break it off into other into other uh to other LLMs for for other tasks. And finally, I'm going to show another technique I've used, which again is not, isn't, you know, there's nothing kind of new or special about this, but uh it was interesting to me because I find that when I'm working on stuff like this, I'm kind of my head is buzzing with ideas about it quite a lot of the time and not always when I'm sitting at my computer working.

Speaker 2: And uh I really like the ability to trigger um to trigger pull requests to AI from other places. And in fact, this one, I don't know, this is too much information, but this one I think I'm pretty sure I did this one in the bath. So uh I I realized there was there wasn't a way for uh non-members to easily sign up. So using the GitHub app, the mobile app. I type this message in and put it in the link it should be, and I signed it to Copilot. And Copilot comes up with a uh with a work in progress pull request. And nicely it even prints out a screenshot. Uh and didn't get the styling right. Uh didn't manage to collect static, I guess. But um came with it and it's this was a minimal change that I was happy to to merge. But um I thought that was uh yeah, that's

Speaker 2: So this this we were still of course learning about the different ways to make the most of these tools, but um I guess yeah the sort of headline point with this one is that I was able to build the whole thing with without writing any any code. So I guess and the last thing I'd say before I wrap up, because I know I'm over time and I can see Thibaut smiling at me and probably have run out of time for questions, is that um I think Wagtail is a good fit for handling sites with a lot of video and uh or other large formats. And um uh I uh um Yes, that's it. I think it's good for video. I'm happy with the choices I made. Thank you, Wagtail.

Speaker 1: Oh, thank you, Tom. This is excellent. We're a bit over time, but we do have like time for rapid fire questions. Ralphast, when you said subscribers earlier, is there a paywall in here or is it just user signups?

Speaker 2: There's no Pay well it's um yeah I mean anyone can go to that site and look at it but uh currently you yeah you need to be a subscriber to uh to watch the actual videos. There's a one-month free trial, I think.

Speaker 1: Great excellent. More rapid fire Ralph also asks the media player. Do you use a specific one for the media player?

Speaker 2: I used. Uh I think I use them some MUX them have their own player, but um they actually they the a lot of the people who work at MUX now w were on this project, VideoJS, which is one of the standards um browser player so it's it's very closely related to that.

Speaker 1: Okay. 30 seconds more questions. Lots of good feedback in the chat. Oli asks for clarification. uh writing the whole thing without writing any code. That's just a native app, right? How much is the Wagtail sites?

Speaker 2: The whole thing. The whole thing. All the Wagtail site. All the front end, all the back end, all the the mobile app.

Speaker 1: Okay. Um if you want to say more about that, I'm thinking people will find it very interesting, but I also uh reflected on the hits you were getting from AI crawlers. Do you want to say any more about how much

Speaker 2: Yeah, I mean that was really shocking to me and it made me think, you know, what are how other people around must must be dealing with this problem. And it was only, you know, it took me a few days to work out that I should be looking at the logs, but Most people who are having this problem are not going to be looking at the logs and seeing this, you know, the the actual the cost that's been generated because all these companies are, you know being funded by the VCs and they're all desperate to suck up as much of the internet as possible. And I think they're doing it in in sort of ill disciplined ways. And of course, you know, the other bots like Google bot and so on have learnt how to behave generally over the years. Um but there this is Wild West at the moment and I think uh yeah it's it's it's uh it was it was shocking to me and something probably that people need to be aware of.

Speaker 1: Excellent. We're right at time of stop there. Thank you again, Tom. I think people love the book.

Speaker 2: Thanks everyone. Plague T Bow. Bye bye