🐍 ❤️ 🦀

Hi everybody to my and welcome to my emoji talk. Okay now I'm including it's not about emojis. Um it's not everybody as fluent as in in emoji speak as uh Adam is, if you go look up the the link. The talk for everybody else and for myself is uh Python loves Rust. Um it's about my excursion into using a different programming language called Rust. and then figuring out or trying to figure out how I could use that with Python. So I'll give a very, very, very brief introduction into Rust. The learning curve for Rust is Kind like a staircase and we get to maybe step one or two of sheer endlessness. Um so you can will be able to write a bit of Rust at the end, but there's a lot more that you could do with

practice and learning over time. And also stuff that I have no freaking clue about what how it works or how to use it. So who am I? I'm Marcus Holtelman. I'm a staff engineer and team lead at Clate. We built a piece of machine about the size of a 3E printer for analyzing chemical liquids. and then do quantitative and qualitative analysis on that. So you can take a liquid, take a sample, do a measurement, and then we can tell you there's such amount of whatever concentration in there. Or we can tell you these two liquids are identical. I've also been a long-term Django Pond attendee and speaker at previous events, mostly about migrations and Django security, so I figured maybe let's take a different topic this year.

When we decide to pick Python, we as a language for our project, most of us are aware of its pros and cons. Python is easy to learn, it's It's on t to to write it's approachable for beginners for newcomers. And let's think about all the scientific web system administration tools and libraries that are out there that we can just use like that and install from PIP and uh with PIP from PyPI and have at our hands. But then Python can be slow at times. There's a whole bunch of problems with that you can run into with typing. Like Python doesn't Python have has types, but it's not strictly typed, strongly typed. Um but you will eventually run into type errors. So type annotations can help with that if you run myPy

there But then next thing you write type annotations and you have more t uh code for type annotations than actually code in the function. So that becomes then a bit of a burden, or can be. And so yeah, the type on type errors still exist, and we will never mini be able to completely mitigate them. And well, at times when our programs need to deal with concurrency, well, we can split up the work into individual chunks, we can throw more hardware at the problem, or Mm sometimes that's also not possible, either because it's too expensive or just not feasible. And if you think about the energy and hardware prices these days, then maybe something else than just more hardware might be a better solution.

That's then the time when people start using either PyPy or Python and maybe even go back to the roots of Python or CPython, the Python interpreter we use or probably use every day. Um we're gonna write uh C extensions. We write code that is written in C just to make it fast and then use it from Python. This is what a piece of C code integrating the Python application binary interface or ABI could look like. It's straight from the Python documentation on writing C extensions. But I'm not gonna go into detail what it's doing. It looks terrifying from my perspective. If you know C, well you can figure your out your way around it, sure, but it's also still C.

And all this code does here, it provides a Python module called spam with a function in the system. That you can pass a string and then the thing calls this and executes the string. So this is essentially the same as import RS and then OS. system Obviously, this approach with that approach you can integrate any kind of C library or anything, any kind of C code and write anything in C. You need to deal with the di the the the ta types of the underlying memory and maybe map between Python types and C types. But that's pretty much it But then again it's C and for decades we know that writing C and or C<unk> for that matter is something we probably shouldn't do but we still do because there's n

or wasn't really a better alternative. Writing C codes is hard, writing C code that is safe is pretty much impossible. So an alternative, and you might have guessed that from the talk title, we're gonna use Rust. Rust was designed by Graydon Hoare in 2006 while he was working at Mozilla. In 2009, Mozilla um was started sponsoring the project and became uh and and then started using Rust for their browser engine rendering engine called Servo. In 2014, Rust's first stable release was uh published And well nowadays Rust is known for its speed and its memory safety. And also its novel approach on how to deal with memory.

When you think about C or similar languages, each time you want to use some memory, you need to go and allocate the memory in on your on your computer. And every time you stop using that piece of memory, you need to explicitly go and say, I do not use this memory anymore. This can be tedious and error prone and we've seen that every other day with CVEs being reported in some kind of C libraries where Somebody forgot to either free something or free something twice or didn't allocate memor the memory pr um at inappropriately. So it's hard. Python uses reference counting. Every object has a counter. Every time you have a new reference to this object, the counter goes up. Every time the reference

is being removed, the counter goes down, if the counter goes to zero, well the Python garbage collector can just wipe the memory and free it. In Java, there's the um garbage generally the just the garbage collector. Every now and then the garbage collector looks through all the memory that the program allocates, figures, okay, this stuff is not used anymore, free it, and you and it's done. These three approaches were pretty much the approaches you had out there, and pretty much the only approaches any other language that were out there used. Rust takes another approach called ownership. Each object is owned by exactly one thing. One other object. And the owner can do everything with that object.

Every object, however, can uh either only ask for a reference to another object or be passed on to the other object. Think about your car. You can either own the car and you s it's in front of your door or you're sitting in there. Or you borrow it to your friend for them to t use the car. But it's not like you have your car and your friend has your same has the identical the same car. It's just not feasible This would be a weird situation anyway. Um so the other thing is that you can then go and say, Okay, I have references to this car, which well, with cars it's kind of Bit weird to think about that. But you have op references to an object, to a to a thing in

a piece of memory, and you can pass around these references. And because they all refer to this one object in memory, they are still tied and bound to the owner of the object. If the owner goes and says, well, I don't need this object anymore, I wipe it from memory, all those references will die and will be wiped as well. So the key difference with Rust and Python's and Rusty and Java's memory management here is that Rust memory management happens at compile time. So C being uh Python being an interpreted language, we don't really in that sense have compile time. Um , interpreting time or interpreting step. Um but in C

well you can do anything um with your memory, whatever you want. In Java and in Python Allocating memory and deallocating memory will be something that the interpreter with the Python CPython or the Java executable will do at runtime, which is also the one of the reasons why at times those programs tend to be slow because the garbage collectors, the the tools for memory management go and say, hey, I need a minute to actually deal with memory In Rust, I said, this happens at compile time. So at compile time, the compiler will tell you, wait, you cannot do that because you do not own this piece of object, this object here And it won't even let you compile the whole code.

And that can be a very, very, very steep learning and curve in when when getting into Rust. And you need to get over that at some point. Otherwise writing Rust is just not possible. But once you've grocked this whole knowledge and this this this concept It doesn't necessarily make the whole thing easier, but you can figure out ways of how to deal with uh certain problems. Alright, so let's get started with writing some C Rust Hang on. This looks like C and Python and this is Rust. And it's kind of this combination of Python with this Fn and then This this print which looks like it

was taken from Python. And then on the other hand you have this main function which is the typical entry point in a C program. And the curly braces in Yeah, this is feels like a kind of a combination of C and Python. Fn says it's a function. That's the counterpart to def in uh in Python. Main is the function name. similar to the CWORD and then print lm is the something called a macro, um where the compiler does some magic and turns this into a print this string into standard output. Similarly to classes that we have in Python, Rust has something called structs. And they are say similar to the stuff in C.

Classes or structs can encapsulate attributes, which you then which hold references or hold data, information, memory allocated essentially. And then you can pass this whole struct or an instance of the struct around. And what you can also notice at first glance, Rustly uh Rust is strongly typed. Strictly typed actually. For example, that line with pubbook rec and then the angle brackets book tells Rust that the attribute on the struct is a vector, which is like a list. And can can only ever contain book objects or instances of a book And when we then go further and we instantiate a struct, we need to pass

all its attributes and arguments in uh while during the creation. And in the last line you see this VEC exclamation mark. That's again a macro and essentially tells Rust to create a vector with those elements. So it's like the list again. the list syntax in Python. What we can also see here, Rust is more than one has more than one type of s uh string. Much like Python, but in the Rust world, these strings are not Unicode versus byte string. These types of strings differ in where and how they are allocated in your memory. The simple double quote something, double quote string, for example, is compiled into the binary, into the executable or library that you create.

It's a fixed size and immutable string of UTF-8 characters. The string, capital S type, however, is dynamic in that you can mutate it. It's kind of like a vector of characters. You can insert some, you can remove them, all that. The string class, for example, here, on the other hand, is something that is not allocated or written into the executable, it's something that's allocated on the heap. The str type for on the other hand would be something if you if you create that manually during the runtime would be something on the stack. Both come with pros and cons. Bytes are completely different story and they are I'm not gonna even touch them here.

So byte strings from Python are different topic in Rust. In the print Ln exclamation mark line you can see some kind of format string. We have that in Python these um as well. Well we have three forms in Python these days The curly brace colon question mark curly brace syntax uses the object debug thingy or function or representation if you want um to to output that and that's kind of what Python's wrapper dundal wrapper does kind of takes the object and turns it in something Readable or representable. And you can let Rust automatically create those refer

this wrapper, so to say, by adding this um this debug uh derived debug statements uh um above those structs. And in the end, when we compile the whole code and run the executable, you can get that output that you can see at the bottom. But where structs can only hold data, you can implement functions on them. It's a bit different than defining methods in classes in Python, but kind of Again similar, like this a piece of memory and there's pointers all over the place and your computer figures out how to resolve pointers. Um in the example we want to implement a two-dimensional point That has a single method

distance. And that method takes another point and then calculates the Euclidean distance between those two points So think of a like two uh two-dimensional plane, pick two points, and then what's the straight line between between those two? So we start with a struct again. This time we use sixty-four-bit floats. We then start with an implementation section for this type. First we define a constructor. The common name for that is typically new, but you can actually name it whatever you want. There's no fixed naming required. And it takes well two um sixty-fit 64-bit floats and returns an instance of this

truct with all the attributes created and set. This is kind of a combination of Python's new and dunder new and dunder init methods. New creates the instance and in dunder init initialize it with some values. And if you look carefully, you can spot this self-identifier here, which was let's be honest clearly taken from Python. Second, we implement a distance method. The method takes one other point and then returns a float. Well, this is the just the math doing the logic behind that. But actually it doesn't take a point, it takes a reference. that you can see from this ampersand. Um if you if you do

or have written C before, you probably are familiar with that. This is such that the point when passed into the function isn't owned by the function. Remember how I said earlier an object is only ever owned by some by one other thing And if you think about using this, if you don't pass in the point with a as a reference, passing in the point to the function, you wouldn't be the the outer function would lose its ownership. The distance function would own this instance of point. So the print at the end wouldn't be able to access point two. This is something this this is the whole ownership topic in Rust. This is something you

eventually when you want to write Rust you need to rock. You need to be able to to Yeah, understand. So why am I telling you all of this? Well, because we are now going to look into how we can combine all this knowledge and turn that into something we can use from Python. There are some wonderful people behind the Pythonium Triacide project. They have published several Python and Rust libraries and tools to integrate Rust into Python and vice versa. There's the PyO3 package, which provides bindings for the Python interpreter and Python types to Rust. So it's a library you use in Rust. And there's Maturine, which is a tool that helps you turn this Rust code or this Rust package into a Python wheel to install with pip.

What we'll do now is we take this point example that we just had and look at it step by step and turn that into a Python usable library. And we do that side by side. We start with some utility imports from the PyO3 package. And then we extend the point struct we had before with some more macros or compiler instructions, so to say. For example, we tell it that this struct is supposed to be a Python class. You see this Py class at the very top there. Um we also tell Rust that the x and y coordinates should be readable from Python. Since we don't set or define a set

macro instruction there, X and Y won't be able to be set from Python. Similarly if you don't define get but only set, you would be able to set the value but never get its like content the specific attribute in Python. As for the method impl um implementations, In our case, only two more macros actually are needed. All the other rest happens behind the surface of this or by using Pyro3. Firstly, we need to tell Pyro3 that we have this impl section, implementation section, and that it contains Python methods. And then we define an explicit constructor with this Pound um new um identifier there.

Otherwise we wouldn't be able to create the instance of a point from within Python, only from within um Rust. Well and apart from this, this intersection actually remains unchanged Last thing we need to tell Rust or Python in that way that we create a Python module. And that we wanna add something to that module. Like we want to add the class, the the point class, and we want to add um And we want to add the Python the point class to this Python module. So we can actually go and import that class from Python

Then we can use Meturine, as I mentioned earlier, to build a package and install it into our virtual environment. And then we are back in our well-known land of using Python and the Python REPL and can import this point class from this library. We can instantiate two points You can pass one to the distance method and can print out the result. So you remember how I said earlier that we didn't define the set indicator on those attributes. If you try to set that, Rust or Python is straight out going to prevent you from doing that And because this is not a property in the Python terminology

where you have some underscore something attribute in your class, there's actually no wave Well not really anyway and um to set this attribute from Python. There is no underscore y or underscore x that you could overwrite and then the property picks it up. It's just not there. There's a library prevention the the the the Rust code essentially prevents you from doing that. So what we can what can we do with this and this knowledge? Well, everything that Rust is good as and that Python is terrible at. For example, anything that's computational expensive. Or passing arbitr arbitrary bytes into some form of object.

Or following some kind of schema validation and all that. Like everything where Python goes in is is tends to be slow or tends to be a lot of work for Python. If we can do that in Rust, for example, and Rust is good at this task. Well, we can write that in Rust somehow and make it somehow available to Python. Let's look at some kind of time series data In case you've worked with sensor data or machine data before, you may know that fairly often, not always, fairly often, the format this sensor data is sent between services is some kind of JSON. But it's not actually in total valid

JSON, because all the individual records are um condensed into a single line and then multiple objects are re um separated just by new line characters. So this is also why this often is called new line delimited JSON. So every every row itself is valid JSON, but not the overall blob. If you have streaming data, for example, that can be very convenient because you just read line by line as as data comes in and deal with it. In this example, um each point contains a timestamp at which the data was collected, for example, the name of the metric, such as temperature or speed or viscosity or pressure, or you name it. And then as well does the corresponding value of that sensor.

And then because you have a pl probably plenty of sensors in your place, in your factory, in your machine, or whatnot You probably want to identify these sensors more precisely and that's where you then have this mapping of labels or tags or keys or what they're called And this is, for example, then the place where you put in the sensor ID or the sensor name or something that you can use to distinguish those. So what we now have is something called a set newline delimited JSON. So and what we're gonna do next is we take this JSON and try to parse that with Rust And then provide it somehow to Python. First, again, we start with a struct.

This defines our data type. This defines how our data looks like. And we're going to use the Rust library library called cert that implements a fairly generic way of serializing and deserializing. That's also where the name comes from. Way of loading on and parsing this data that we had just had. So we have a timestamp field, we have a metric name, we have a value and a map map of the dictionary. And As I said, Rust is strictly typed, so the map can only have strings as keys and strings as values. Then we're gonna have a function that takes care of parsing a string and turn that

into a this this kind of struct that we just had. As I mentioned earlier, in newline delimited JSON, lines are separated by, as the name suggests, new line characters. So we split this whole string or whole body at the newline character. We can then filter out all lines that have less than two characters. So an object in JSON always contains of at least the open and closing curly braces. If you don't if you have less, then it's definitely not valid JSON. Um or not not a valid JSON object anyway. So everything that has last characters we can throw out and ignore for the beginning. And then we actually go and parse

each line and we're gonna do that in another function. And then once we call that and get the return value of that line, we can collect all of that and return a vector. So a list of those objects. As previous mentioned, to parse a line we are going to use the cert library. More precisely we're going to use cert JSON, which is a JSON parsing and serializing and deserializing. Implementation for Rust using cert. And if the parsing succeeds, Rust will automatically return an object of this type sensor data And there's a lot of stuff happening in this line of c in these six lines of code where some of the magic that makes Rust

occasionally very hard to comprehend is happening At the end of this function definition, we have this PyResult sensor metric. This is a type definition of the return value of this function. And then this death is a kind of like a deserializer class. And then between this um third path to error, deserialized deaths match Blob there , the Rust compiler can automatically derive that the value of this object, this OBG, should be a sensor metric Type. And then because it's a sensor metric type, it can look up this these instructions that we had earlier

in with the or see that it's deserializable that the instruction that we had earlier and it can automatically do its well still for for me it's still magic um to turn that JSON into a correctly parsed object correctly passed instance of the struct. This match in syntax is kind of the same similar to the what we have in Python since um in Python since three point ten I think. And yeah, similar to what we had earlier, we need to add the class to the method uh to the module. We're gonna add this function to the module. And then sorry, and then we can go and

load this newline delimited JSON file, parse the content and then get objects and have the central metric data. So you could for example now go and use this function in uh parser class in DRF for example. If you have a DRF project, you could go use this as a way of parsing data in DRF. But of course, that is not all. There are some actual Python and Rust packages out there that are usable. For example, there's um

the org JSON library, which provides a far superior JSON parsing and serialization than Python standard library does. It's also in specific situations significantly faster. Um there's Pydentic. Um who of you uh whoever of you has or may have used Pydentic before They are looking or they are rewriting its core, parentic core, in Rust because it's so or can be so much faster The cryptography package uses Rust to parse X five hundred nine certificates. Um And there's Rilpy, which is Ril is the Rust com uh counterpart to Python's pill

or pillow. So it's a Rust image library. There's Python bindings for that. So heck maybe the Django image field at some point could use that library. Who knows? Which brings me to the last slide Let's see what the future brings. I mean there's a lot of potential in Rust, there's a lot of speed that we could gain using Rust, but then we still want to write Python because it's easier and more approachable, as I said in the beginning. But I I could imagine that at some point, like cryptography does, there could be a few lines of Rust in par in in Django I don't know, maybe the URL resolver, maybe the request parsing

in D DRF. I don't know, who knows Like this is something that the community that all of you and everybody else who is not here, as we learned earlier in this morning by in in Katie 's talk. That's something that the community needs to decide. Do we want that in the first place? Should we do this? Should we not do this? And then abandon this idea for a while? And where do we want that? How much do we want to use Rust? And the problem I see right now specifically is that both PyO3 and Niturian, while kind of stable They still have a fairly quickly changing API and and duplications going through there. And this is just not really in line with how Django 's stability and

release policy works right now So maybe we could go and appro um use something, use Rust in in some features right now in a provisional um with a provisional API. But yeah, I said that's up for all of us to decide as a community Thank you.