Faster Thumbnails for a Faster Web

Okay. Good morning and welcome to Faster Thumbnails for a Faster Web. So, uh, this is my first time giving a conference talk. Uh please forgive me if I'm a little bit nervous. A quick little bit about me. So I'm Alex. I'm a co-founder and director at the Developer Society. I've been working with Django for just over 16 years. However, these days my role at work is mostly focused on supporting our developers, the society, and a little bit of DevOps. But I still love Django and Wagtail. And as you probably haven't seen us at conferences, this is a hello from the Developer Society, or Dev. We're a not-for-profit cooperative. That works for charities, nonprofits, and other organizations that are focused on a positive impact on society.

And one of the fun challenges in working with charities and nonprofits is that they usually have limited budgets. There isn't an infinite amount of money to host them. But as someone obsessed with DevOps and infrastructure as code, I enjoy trying to optimize setups and get the most out of them. Which brings me to this talk and what I hope you get out of it. While some of this might be a little bit basic, I'll be showing you some of the problems we've had on some of our larger sites when it comes to images and thumbnailing them I want to talk to you about best practice for images, why it's important, and what we can do with Wagtail, and a few things to look out for. I'll be showing you the one solution we've gone with to speed up our Django MicTail sites, and it might help you too.

As a small disclaimer, I'm definitely not an expert on this. All I wanted to do was speed up our sites up a little bit. But found a few interesting things along the way. So, the past 12 months have been busy for us. There's quite a few sites we've launched for organizations that we're very proud to work with. However, after launch, when you finally get the stats on the traffic you're generating and the bill that follows it, that's a bit of an ouch moment. Here's a graph of the subset of our sites that are hosted on AWS. You can see where things were mostly stable and consistent, and then we launch a few new sites. This is where it starts to hurt a bit with increased traffic being served. Some of you might question, why don't we just switch our CDN to a cheaper alternative?

Or commit to a certain amount of data transfer to reduce the costs. But when you realize that 90% of the data transfer cost for one of your new sites comes from images, you don't want to commit to that. You want to fix it. We were thumbnailing on some sites, but quite often it was just a single large image that was optimized for desktop only. And there were a few other silly mistakes. Firstly, I'm going to give a quick rundown of different image formats, how well they perform, and the current state of support in 2024. And to help with this, I'm going to use one of my favorite selfie photos from a few years ago. Say hello to my alpaca friend Peru. I highly recommend alpaca walking to everyone. It's definitely a fun experience and even more fun when you get that perfect selfie with an alpaca.

So let's start off with JPEG. All browsers support it. I'm taking my original two-megabyte photo and resizing it to 800 pixel width in Magtail with the default quality settings. We've reduced this photo to 166 kilobytes. Not bad. Next up is WebP Looking at browser support, for 2024 the question should be: can I switch to WebP as a default over JPEG? All the major browsers with active security support can use it. Maybe if you're confident that you don't have legacy users. Although the picture HTML tag means that you can avoid picking one over another, but it would be a nice future if we're not generating so many thumbnails. Anyway, Wagtail 2. 6 added support for this back in November 2019, which uses 80% quality these days for thumbnails by default.

And it's reduced to 112 kilobytes. So smaller than JPEG. And finally, AVIV, the newest format. Edge only added support for this in January of this year, and there are still a few Safari versions under active security support that can't use AVIV just yet In good news though, can I use did add AVIF to a list of baseline features for 2024? Wagtail 5. 1 added support for AVIF back in August 2023, which also uses 80% quality by default. That reduces it down to 177 kilobytes. But hang on a minute. That's larger than the 166 kilobyte JPEG. And this is where I hit my first what? Those numbers surprised me.

The file sizes seemed a little bit off, especially if WebP and AVIF are supposed to be more efficient. This is where the default quality settings in Wagtail are probably a little bit too high. The challenge is trying to figure out the best quality settings for all of those different formats. And that if they give roughly the same results. There's a fantastic post on industrial empathy, along with code on GitHub that can re-encode in picture into multiple formats, sizes, and quality settings. I've tested this myself on the Alpaca selfie, and for images comparable to an 80% quality JPEG, it suggested these numbers. I definitely recommend using your own images to play around with it though and see what white

must white what might work best for you. My results showed that WebP needed to be similar to JPEG quality and AVIF could be 15 to 20% lower than JPEG quality. But with smaller thumbnails you could get away with slightly lower numbers So, after applying these quality suggestions, we have sizes that seem a little bit better. But yet again, AVIF is slightly larger than WebP. Something else is at play here. The numbers just didn't attach up with the other tools I was using. It took a little bit of investigation, but eventually I found the culprit. This is a photo taken with a camera and contains exif metadata to store all sorts of extra detail.

Currently, Wagtail is keeping exif data when creating image renditions. But it's not just a little bit of extra detail. My thumbnail contains a seven kilobyte thumbnail. When exif data has taken twenty-two percent of the total file size, it's something that needs fixing. So, we ended up creating a small package to remove the exif data. As WAGTL's image processing is part of the Willow package, we can register an optimizer that keeps the colour profile data but strips out excessive EXIF data. This is a bit extreme. You might want to keep some exiv data. For image copyright and attribution maybe. But for us, I was quite happy to remove it. And after adding that to the project

We finally found the numbers we're expecting. JPEG is the largest, WebP offers a small improvement, and AVIF offers a significant improvement. With right quality settings and making sure no additional data is sneaking in, we're starting to make savings. Now we've been through the formats, let's take a quick look at responsive images. Modern HTML allows you to provide the browser with a list of image dimensions and formats, and then it'll pick the best image it can support and the most appropriate for the size of the device. Most mobile devices are pretty small for width, so this makes it easier to send smaller thumbnails. With our most problematic site, over 58 % of our traffic was from mobile devices

So reducing the image size even further makes quite a difference. For us, this site is a little bit of an outlier for how dominant mobile traffic is. But if you look at your own stats, it still might be significant and might surprise you. Taking that alpaca selfie and reducing it down to mobile device width drops the file size to just a quarter of the original thumbnail. That's a massive reduction in cost just by serving smaller images. And WagTel is actually pretty good at generating multiple renditions in one go, if you use the right template tags. It'll fetch the source image once and use multi-threading to simultaneously create the images as quickly as possible. But

if you add desktop and mobile variants for one image in three formats, that's six thumbnails that need to be generated before the page can be served to your user. If there's multiple images on a page that need fresh renditions, that can add up. So, what can we do about it? You can potentially solve this with Wagtail alone. Wagtail has a view that can be used to dynamically serve thumbnail renditions and not block the rendering of the page. This requires a quick bit of setup. Firstly, adding serve view to your URL config. After that, you'll need to swap your image template tag to image URL and using a slightly different syntax. Pretty easy to get going, and just like standard thumbnails, it'll only generate the thumbnail rendition once, and any following request will use that save

file. One slight issue with this, it's a Django view serving a media file, which is something you usually want to avoid. You can easily mitigate this by having a caching proxy or CDN in front of Django. But if solving the problem is as easy as switching to the dynamic image surview, why am I after a better solution? Wagtail's image handling is fantastic, but sadly it doesn't support generating thumbnails for Django images. Only supports Wagtail images. The site where we've got the biggest problem is mostly user-generated content. And whilst we could put those images into a Wagtail collection, that would be a little bit hacky. My dream would be that we could use Wagtail thumbnails for other image fields, but sadly that's an old feature request.

I'd really love it, but I can definitely understand why it hasn't been implemented. Also, as Matt will probably be mentioning later, it's impossible to keep everyone happy. And when it comes to thumbnails on sites with thousands of images, the amount of files starts adding up. This is a screenshot of one of our non-Wagtail sites. We're into hundreds of gigabytes of thumbnails. And this is one where we haven't even got around to adding WebP or A of image support I'd love a solution that works with our pure Django sites and not just Wagtail. We could just use another thumbnail library, but we still have the problem of it generating multiple thumbnails in one request. Or the other common solution, they'll generate all the other thumbnails when saving that original image.

After searching and investigating for alternatives, one caught my attention. Image proxy. It's not a thumbnail and library. Instead, it's a separate server that fetches the image from a source, takes the thumbnail options directly from the URL path, and returns an image. It's free and open source with a but. Basic thumbnail operations are available in the open source version, and there's a pro version you can pay for. Chances are the majority of what you want to achieve can be done with the open source version. For me, this seemed like a dream solution. It's fast, written in Go and uses libvips underneath, so it can stream images instead of processing the entire image in one Go. Looking at those numbers and seeing the A-VIS results, yes, it really can go that fast, but there's a catch.

You can set the speed or CPU effort that is used to encode an image I found that there was less than a one kilobyte difference in the Alpaca selfie between using the fastest default setting and something more similar to what Pillow would use. Anyway, it's just a proxy, so it doesn't save thumbnails. For some people, this might be a little bit of a con, but for us, we're not storing hundreds of gigabytes of files. Just put it behind a CDN with a long cache duration and the amount of hits to the back end are minimal. So Our first step in using image proxy was to find a good Python or Django package. And whilst there are a few small packages, most of them were a little bit basic.

Instead of using them, we decided to create our own Python package to generate image proxy URLs. Why? I wanted something Pythonic, but more importantly, I wanted something to use as a good example for Python's static typing, with high-quality doc strings to make the package super easy to use without having to refer to image proxy documentation all the time. It also feels a little bit like using Django query sets, which makes it fun and intuitive to use. With the right setup, it's pretty easy to use with pure Django sites Just pass in the file name to an image, start adding processing methods, and you can get the URL to the image you want. But as this is a Wagtail conference, you might be thinking about Wagtail images

They're still Django images, but they allow the editor to set a focal point. We can easily add an image proxy gravity into the thumbnail, which is taken from the focal point in Wagtail Any additional thumbnail options after this will take this into account. So, after all this, how much a difference did it make for us? These are the stats from just one of the pages on first load, so only a small sample and not a full set of results. A huge reduction in size for the user-generated content, although that's just mostly following best practice. Pages remain fast to load and generating thumbnails has minimal impact on Django or Wagtail. Anyway, as I've said earlier, I'm not an expert on this, but hopefully you've learned a few things.

I've had fun showing you image proxy and libvips, which is definitely an extreme solution, but it helped us solve a problem. Maybe some insights from all of this can be used to improve Wagtail and the packages it depends on Or maybe creating third party packages for those of us that need it. And on that note, thank you for listening.