Caching everywhere

Hi, good morning. I'm Yuri. I'm a principal engineer in Brazil. I work at Boozer, but I'm left in the company going to the Nether Netherlands right after here. Uh what I want to s to talk about here, I want to talk uh about a lot of things related to caching And I know it's really complicated to understand at at the start because you don't need that to start a Django project. You probably can handle your project for years before you really need that

but when you really have scale you can't do uh you can't work without caching and lots of the things I will talk here. Boozer is a startup growing really fast in Brazil and I had to use all these things that I will talk about here probably I learned most of them there because I need to Starting with what's caching. Caching is a component that stores data so that future requests you don't need to Process that again and you can serve that data faster. And Python has a lot of

tools to help you with that Uh starting with the batteries included in Python library. We have funct tools, it has a uh least recently used cache, so you can And it it will start to cache that and the next time you call the function with the same arguments, the same parameters you Python will just use the result he it already processed process it Uh it has two ways to do that. One is R L R U cache, and the other one is the cache decorator.

Both are really similar. The cache decorator don't have the limited size, so be careful with memory space. When you are doing cache, you have a lot of trade-offs. You if you're are doing too man too much cash you you use too much memory it will be expensive but in the other side if this data is used too much or the processing of this data is too slow probably it's better to pay for more memory than for for more CPU It's never a easy answer. You have to know your your data, your processing, your limits, and stay with that.

Here an example being executed of this function. When I call the fact factory function, the first time I call that with 2, so it will pro I I added a print statement just to show the cause. It will process for two, for one, for zero and return the factor two, it's two The next time I call that with four, it will call for four and for three and will not call the other ones recursively because it already knows the effect R2 so it will return twenty four result without reprocessing that. It's a small example but if you are doing that for a big number

it's a really important optimization. If I call too many things recursively in that case I can even uh crash the crash python so still in python library we have cached property It's almost the same thing that cache decorator do, but it does for properties. So you have a class with a property every time you call these properties it will be processed again. But if you have uh a value that you want to process one time, probably because it's slow to process that, you can use

cached decorator, cached proper to decorator, and it will avoid the more calls. Uh for the same thing. If you in this example, if I have a big sequence of numbers, the calculating the deviation is It's low, so you probably don't want to do that more than one time. It will not change with the same data All these examples are examples of memoization. Memoization is functional caching based on input. And I reserves for this presentation, I really not sure about the difference

about what's caching and what's memorization because with the right abstraction everything is a function but uh people say this is different and I I don't know. Looks like a legacy term and we handle that different, but it's almost the same thing. I'm talking about this term because Uh we have other libraries that do almost the same thing that the standard library do, and they always talk about memoize memoization because That's how it's called for functions. Uh like Py memoirs, jung memoise, jung cache memoise, and other variations memoization and other things.

I don't know which one is the best for you. You can look at many of them and choose. I use mostly Django Memoise because I I'm using if Django. It in integrates with Django Cache. I will talk about that later. But most of them add useful features. like time to leave so I I set a cache now and after ten minutes this cache is useless so I I don't have to track that, the library will handle that to me. I can use custom caching algorithms, so not only least you recently used I can I don't know first in, first

out and other things. You can construct custom cache keys so it's not only based on the parameters But I can transform these parameters or other things to do to create better cache keys and handle that cache better. And the the Python libraries, Python built-in functions don't support unhashable objects, and these libraries can support because they can transform these cache keys. And for junk Django libraries, Django memois, Django cache memois, they have um they easily uh bind

with Django cache framework that I will talk about and it's really easy to work with Django and make that work. Here an example it's using Django MemoIs. I think probably works with fodder. Just have to change it in part. I'm doing the same thing that I did that that I did uh before, but now using the memoise function it has a timeout so after 60 hundred seconds, ten minutes, uh this cache will expire. I can do that because I don't want to store a lot of not used

cache and use my memory better. But it does the same thing while I'm doing that on memory I can do that distributed. I can use uh Redis or a main cachete as backend so more instances of my application can reuse the same cache. It's really useful when you have slow functions and you can not reprocess that one time for each instance and save on local cache. Jump cache framework, I say I talked about three three times before. Uh cache it's a cache abstraction with many backends.

It has built-in in-memory, database, file file system. Man cache Redis. You can plug any other cache backend you want. You just have to implement it or you have already have some third-party implementation. Uh but for most cases you can just use one of them. If you're uh and here I will start to talk about some examples I had and problems I found when I started to using cache framework because I didn't know how to do that and I did all the things wrong in different ways

so I can talk about that now one of them even with caching if you don't care about how many calls you are doing to the cache it's slow because Jumbo cache framework for example when you cache something it will serialize and you deserialize using Pico or JSON if you want And for if it's a big payload, it it restore your serialize the serialize, but you in some cases you don't need that uh because it's uh static data for example but you you have uh some overhead there because of that. If you are using main cached

Redis or any other uh ek uh distributed or remote cache you have another problem that's network so uh call that could be fast like a factorial now I have a network latent latents to my cash back end. It's uh small problem for one or two calls but if you are if you don't care about that you will start to have any plus one issues not only with your database but with your cache two Caches are faster normally, but uh you can avoid that. I talked about memoization and one of my

The most interesting and useful libraries I use is Junkcashback. It does the same thing, memoization, but When I ask for a value that it it knows but it's expired for some time, it will serve this data, this stale data and will Uh create a task to refresh this caching background. So I don't have to wait this processing. It's really useful for my use cases because uh in my website I need a lot of performance and even this

small things that I can process later help me a lot to scale and that that way we can do that we can serve users faster and And now talking about stale data or data in general uh distributed data any database or websites your data is never up to date If you when you query your database you have some milliseconds and then you send it to your user you have I don't know More hundred milliseconds, maybe a

second, and that uh probably is is still usable data, but you have to know your product or limits how much I can serve uh uh older data and old data But uh it's always a trade-off. I prefer to have a better response time than uh refresh data because in my example I It's uh like in e-commerce between the user adding something to the cart and paying for that Probably possibly this dot this product is outdated

and I I can't sell that anymore because I don't have stock or change the price or anything else But I even if I don't use stale data, it's stale for the user in some cases. So I have to trade off between performance and data quality. I don't know the name. It's scalability, reliability, and other things that you always have to think about. It's always the it depends, but uh you have to know your data too. Know when you can use this state of data, how to use that. I have uh my best example. uh for this e-commerce part we serve stale

data we we have a function that serves stale data and this we get these results and updates only pricing and stock to have a more up-to-date data because even I don't know one minute too much for that but for the list of products it's okay if Uh something changed in one minute, but I can it's worse to show products out of stock or With a wrong price because it changed. Here I have an example of the my mainly website I worked for a long time. It's in Portuguese, so don't try to read that.

But what matters it is pricing and stock that's the right part there And all the other things are cached and I can serve mostly without processing anything, but the pricing I have to update all the time, like I said There are caching validation. There are only two hard things in computer science: cache caching validation and naming things. I talk only about caching validation. Uh it's very difficult to do it right. Uh in most cases, caching validation doesn't matter that much. You you are okay with stale

data. But uh when it matters it's probably better to have a small time to leave for your data than to create magic uh work workflows to invalidate or delete this data from different places. I I will still talk I will talk about the other parts of where we can cache things so when you have this cache distributed it's a lot difficult than only in python or Django Django has a nice decorator to cache your page, so you can just call cache page decorator and it will start in

The cache config the default cache configured you can define in which cache you want to store And not only that, it will add control cache control reader readers so you you can Tell to the browser that you can cache that for I don't know fifteen fifteen minutes and that's okay because I'm caching it here too. uh if you call me again I will just return the same thing. And now we are talking about caching pages, it's probably more data than just a function call. So

be extra careful about about caching space. It should be really monitored and If all caches you add, uh you have to think about that. If it's okay to use space, memory space with that In some cases it's better to just reprocess that if it's not used too much. Django has not only the cache page, it has the cache control too. Where you can add custom configuration to your cache control properties. I will talk about them later Uh but uh you do that to configure

configure your cache control with different properties and You can customize that a lot I talk about. I talked about cache page, it's the common use case, the documented use case, the more documented use case, but you can use Django Cache for anything. You can just access the cache and set something there after some time you get that back and reuse the way you want. You don't need to just use the functions jung would give it to you

or they memoise libraries. So in this example I call in cache get And here you have a implementation detail that I have a lot of problems. That is when you access caches. They all all the caches you you call that as a a key, so caches default for example. That's default cache Django uses. So It's not thread safe, so you have to call that only when you want to use.

If you define that globally, like the first cache I'm defining there. It you have sometimes you you have a problem because the the cache connection dropped or Two threads are are using the same cache and it will crash your backend library. I had these issues when I changed to uh library in Mancash library in C and I discovered that a lot of people had the same problem. Here I changing from the presentation to show Even Django had the same problem. It's a pull request

for Django that I did when I discovered that. It instances cache here and that is a thread pro tr tradium safe problem So using a property I can access that only when I need to use so it's safer this way And I just yeah after that I checked a lot of libraries and many of them did that wrong So uh it's uh in just uh interesting detail that when you try to really Try new things and different things you hit some limits that you don't expect.

This bug had 10 years when I solved that and it was there working for most people. Talking about direct access, you can use the not so public APIs of Django to to reuse the connection management management measurement because it works really great it has pooling and you can add lots of configuration charging and other things to Redis and main cache and you can access that the way I described it there

It's not even the same thing because it's not public, but you are just accessing the the client that Django creates for Django and you can reuse that. It's better because you you are not creating new connections, you are reusing what's already open uh it's a good use of resources but be careful because it's not expected I think maybe someday it changes So I talked about Django a lot in Python. It's just the application layer And there you have different different application caches, like I explained, but you have other layers.

You have your web server that just sends requests to Django. and you can have a shared cache there. You also can have a CDN, a content delivery network, that's has other shared caches so you can store shared caches in two places. Maybe you don't need the CDN so you can just use the web server. CDN are Explain what why you need that and when. And in your browser you have uh private cache, only the user of this browser knows what uh use this data so it can store uh

more things when you configure that uh a request to be a private cache only the browser will cache you should do that when I don't know you have some uh User ID information, session IDs and other things. At this level, cache is really useful because network is slow and network latency is in all these steps I showed. and between the my computer and the my server I have some milliseconds and I can't get rid of that

and for it Step I add in there I have more milliseconds and it's slow. So when the browser cache the I don't know the JavaScript library you are loading, it's it reduces a lot the loading times for your application, for example. Here a small table with latencies between cities. It's mainly because of light velocity, you can't go faster than that. So From Edimburgo to Sao Paulo where I live, it's 200 milliseconds. In Europe it's faster, but it's still a a thing.

And you have to always think about that. One funny thing that we have in Brazil, Brazil is one of the most expensive Amazon regions. So most people just put their website in New York, Ohio, and other places in United States and this gives I don't know a hundred fifty milliseconds latency and it's really slower than if you put your server near your users and you and it's it's per se it for conversion and for user experience these second these milliseconds matter a lot.

I can measure the difference when my server is far from their user. It's really good experiment to do, but not easy. In a website you don't have only one request, so it's not only that latency number. Your connection probably is not good as good as This servers did that worm it we're measuring that so you have other layers of latents And for many requests it's even worse. If you have microservices in your application side, you are adding other

network connections and requests and it's it will always be a bit slower. You can of course you can do that be good enough, but it's lower And it I like to always think about networks because they are really critical to the results I expect. Caching is part of the internet. Without caching we probably couldn't have what we have today. It will be uh would be a lot of data transfer that we don't need. Browsers cache a lot, even you don't know that, but they are caching JavaScript, CSS.

uh lots of HTMLs, API calls if they are config the application are sending the right HTP headers to define The browser behavior or the client behavior, not necessarily a browser. Other parts of this stack Use the same headers to improve uh performance uh on the this chain Here I have an example of a request. It's a request to a JavaScript file and some headers that are related with cache control It has a max age if

my last request has is older than this maxage it will try that the request again. It has other headers, e tag less modified often can be used to improve caching and browser know how to do that. Uh below I have specific cloud front headers, just you know it's a CD C DN And I don't know it's not really useful in most pri m most case, but uh For example, uh the last last one, I don't know if it's good to

read that. Uh it it's saying hit from cloud front. It cut it it means Uh cloud front server this request request without calling my server to provide the file because it it was already in the CDN. Cache control is the most important header to know. It has a lot of properties, options. I don't want to explain off then because they are really exp is specific and Pro yeah I I need two more hours to explain all of that. Um max

age, no is no cache, no store, no transform, must validate uh any others the last two I really like and I will talk about stay oh while revalidate stay all if error they work like John Cashback I told In the presentation. It will serve I stale data while revalidate this data. So if the data the request expired instead of Calling the server and giving the answer, it will give it uh stale answer And call the server to update his cache. Stay away if you error, it's the same thing.

If the request to the server fails, it will uh serve the stale data while it's not possible to update this cache. So if my application fails, maybe you don't You don't even know that because it it will serve some data that that's okay for me for some time. Uh browsers are not too Smart, they just use HTTP headers and you can trust that most of the time. You have some really edge cases to learn in the future, but you can just trust them.

Uh it and it's it it's uh local memory, local disk probably, and disk memory, so It before making the request it checks if the cache is already there. If it's there, it doesn't make the the request because it's not necessary. It checks if the the cache is okay, it's still valid , I can use this cache and other things. All these headers I talked about uh uh are still are valid here. Other clients um I don't know we we have Everything talks

HTTP and it's up to them to implement caching and other strategies. Request package don't care about caching. Request cache can help you with that I thought um con CDNs serve from distributed edge re to reduce lattice so it's distributed around the world to be near your user cache requests based on the same request headers or you can have crystal rules. I'm without time here so I have a problem. You can compare that their performance, it's really difficult. For example, CloudFront and Cloudflare

are really similar, but when I'm using AWS with CloudFront, that's a AWS product It's really better than Cloudflare with Cloudflare because it's I don't know they have uh better network between them I CDNs have cache control headers and it's not a thing yet, but we we will have a CDN cache control header uh specified to handle like cache control but only for cdns. For now we have specific headers to do that like the S maxage that define a Timeout on MaxAge

only for CDNs and sewer gate servers. And you can use this type CDNs we will use this stay UI we validate, stay over with error and other headers to do that better And at the web server part I have nginx and nginx is the most user web server in the web. And in many cases you don't need the CDN. You can just put your caching Nginx, it it can do that with all the most of the these things. It can't be distributed like a good CDN.

And you just have to configure that. It's difficult to understand this configuration, but if you know Nginx, it's a one-line configuration. or some lines configuration to do that better. You can that for errors and updating. And that's it Thanks.