Taming rich text content in Django

Speaker 1: Okay, uh then again, yeah. So it's Paris Casitiaris. Yeah. Wonderful. Um telling us about taming rich text context content in Django. Wonderful. That's what we're here for.

Speaker 2: Okay, thank you. All right. So before I get started, I need to say how excited I I get every time I have this mic. because it uh reminds me, do you know the Eurovision contest? Yeah, you know where it's uh it's that vibe, so I like it. So thank you very much. So Let's talk about prose. Um it's a strange word. Does anyone know the word prose here? No one? Okay, so uh when we say prose, what we mean is written or spoken language in its ordinary form without metrical structure. Meaning no poems, no songs, no code.

Speaker 2: So what I'm doing right now is spoken prose. An example of written prose is this are we okay? Is this article here? It's a text from a blog post we did when we announced what I'm talking about today. Whoops, and that was it. Thank you very much for attending. Okay, let's should I have to give it the one? Um yeah, I have I have an extra yeah. Yeah, let's go.

Speaker 3: It's like a football, so you get extra time added.

Speaker 2: So welcome to the second part of the presentation. And this is all right. So when we talk about web applications, prose is just multi-line Text but plain text thank you I can use this plain text is a bit bland And not all text is cut from the same cloth. We have plain text, we have code, we have rich text, and we want to see how all these are different in our case. So let's spice up the original example. As we can see here, we have some bold lines here. It's the same text as it was before, but it looks a bit different. We have some words that are bold, we have some inline code. So

Speaker 2: this is what we call rich text. Rich text is just text which is enhanced with some formats and styling to help to help it stand out. So we can have strong letters or bold lists, tables, images, subscripts, whatever, all that stuff that make it look more interesting while it's it has the same content Let's see what makes rich text special in our case. So, contrary to plain text it is stored and coded and rendered differently. What that means is that in plain text we have it's it's just a string. We

Speaker 2: just save the file and then we just load it and display it and it's the same thing as we stored it. With rich text It's different. We have to store it in a way that we can understand how to display it to the user to achieve its goal and more often than not rich text is long form. This means that we use rich text to write books, articles. Whatever bigger than an email title, for example. And because we have because of the first bullet, because we have to encode it and render it differently. There is no

Speaker 2: interoperability guaranteed. What that means is that there are many formats that we can save and encode a rich text rich text content, and there are many formats where we can Display it. So it's different if we if we display rich text in a web application or if we render it natively in an iOS application or in a 3D canvas and all that stuff. When I talk about encoding, I mean nothing more special than this. When we save a file as a word file, it's some sort of encoding for rich text. The same goes for Markdown and the same goes for HTML. Some people might know and point out that Markdown is a superset of HTML, but most of us don't know

Speaker 2: use it this way so I have it as a different place here. And let's start getting a bit more specific. Let's talk about rich text in web application. A straightforward way to handle rich text is to store it as HTML, which is the native way to render it in a web application. Cool. We should be cautious though when we handle rich text in our web web applications because essentially HTML is an application itself and And strange things can happen. And we have to be extra cautious when this content comes from our users, which are an untrusted source. Not because they're bad, but because they can be. So let's see a few caveats

Speaker 2: in rich text. We will um focus in three areas: security, performance, and workflow. It's already getting weird. We're talking about bold uh italics images. What's the whole security thing here? I know. Let's find out. So when we open up the potential for rich text, we open up the potential for XSS attacks, HTML hijacking, and JavaScript global namespace polluting. So in case you didn't you don't know what is an XSS attack? It's a cross-site scripting attack. Which means that HTML got injected in our applic

Speaker 2: in our web application without us intending it to be so. And it runs on a different server and it can grab data from our users or do even weir weirder things. HTML hijacking means that it can hijack how the page is being displayed to the end user. And JavaScript global namespace polluting, meaning the window, is something that I was three days ago years old when I learned about it. So I will I will spend just a minute to tell you about this because I found it really interesting. How many of you have written HTML in your life? Okay, so did you know that if you add an ID attribute in an HTML element, it gets populated

Speaker 2: in the global namespace in the window. So if you add ID equals Django Copenhagen in a div, this this div object will be available at Django Copenhagen as a variable. Did you know about it? So you learned it up today. And okay, so this can really interfere with how we do things. So this is some innocent rich text in HTML format. We have hello Copenhagen in strong. Thank you for this opportunity. So this can be a cross-site scripting HTML which has injected a script tag below. Here we have some HTML hijacking

Speaker 2: where we just end the web page early and we stop it from rendering more content. And this is what I was talking before: is that I can add jQuery as the ID and if you're um let's say unfortunate enough to use jQuery you can be even more unfortunate so you cannot use it anymore in your page after this you know after this strong element you know uh renders so a few security tips here is that when you handle rich text in your application you should always sanitize it before storing and when I mean sanitize I mean you know remove any HTML tags that you Do not explicitly need and remove

Speaker 2: any HTML attributes like ID that you do not explicitly need. Next is that you should also consider sanitizing before rendering in some cases, because this might be you might not be a hundred percent sure how it was stored in the database. And in some other cases when you have to render something really dynamic, you have to render it in its own browser context, meaning in its own, let's say, frame. When does that happen? For example, in GitHub, when you render an iPython, sorry, a Jupyter notebook. uh document it is being rendered in a different frame so because it's super dynamic you do not want this to interfere with other stuff of the application.

Speaker 2: So now that we talked about security, let's go and talk about performance. How much weirder can this get? We are talking about bold and italics. What does performance mean? So we should pay close attention to performance when we deal with rich text content because rich text content takes up more space than its plaintext counterpart. Which actually makes sense. It's rich in data because if I want to display something as bold text, I have to store this information. As we said before, its content is often long form. So it's different expecting title for a message and the whole content for the message. And

Speaker 2: more often than not, when we need to list the the our our content. More often than not at the least part we don't need you know the whole the whole content of each part. We might only need an ID or stuff like that. And we have to keep this in mind. So let's go and say a few performance tips about rich text. We strip it of redundant HTML and attributes besides being a security guard this also helps us save some space which can be really really important and we see an example later. Next, we can store this content in a separate database table to avoid fragmentation in our database because if we have

Speaker 2: rows where every particular column is just a few bytes and then we have a whole blog post or a document there, this causes fragmentation in our database. And it can get weird. And we can we need to fetch we should fetch this content only when we need it. Let's take an example. These are some release notes from GitHub. This is rich text, as you can see, it has lists, it has uh links. It has um inline code below where it says full change log you might not be able to see it but it is inline code so it it is a bit of text. If you take only the plain text part of this and you encode it in UDF-8, it's

Speaker 2: about one kilobyte and it looks like this. Good. If we take the rich text part of this the HTML content of it, it's ten times bigger. It's eleven thousand bytes, eleven kilobytes, which As you can see, it's the HTML here. It has quite a lot of tags that don't make sense for us to store them if we want to just display this thing. And you know, it's just 11 kilobytes. It's it's not a lot, but if this gets bigger and bigger and bigger because we have articles, documents and all that stuff, you don't wanna mess with it. Now let's go and imagine another example. Let's say we have an application like X, you know, the Twitter thing after it was renamed. And uh we can imagine we have posts up to 500 bytes

Speaker 2: of text, which is this is about 280 characters, and multiple users can post and users can reply to posts with posts. So this is getting a bit distributed and it's important. So this is the two that I made when I was super happy to learn that I was gonna do this exact presentation here. And this tweet, um, if you take the HTML inside the inside the rectangle that I have is about 500 bytes, a bit less. And we will see some performance tests if we imagine this application with that with these tweets and we will imagine two different tables. One we'll have the tweets with the the content of the tweet in line in the database table, and then we will have another another table, let's say two

Speaker 2: tables, one that will have an ID, a title, and a link to another table with the content of each tweet. And we will see how two different queries behave. It's select star, I mean bring everything from the table, and select only the ID and the title. And because a picture is a thousand words and I need a chart to make my presentation legitimate here , let's move on and see that. So The orange one, which is the left one, is when we have the content inline. So I will move here and show you that here we have the query execution time in milliseconds Here we have the query row limit, how many rows we bring with the query, and we see how that thing scales.

Speaker 2: So if we want to list the tweets and we have a table that includes The content when we have 1,000 tweets, it's okay. When we have 10,000 tweets, you know, we start and seeing the difference. But when we go to a hundred thousand tweets , We can see that listing all tweets here takes multiple times more um time to get fetched from the database compared to just Bringing everything in a table where I don't store the content. So I wanna be fair and we will see how this compares in case we didn't bring everything. I mean we didn't do select star, but we did select

Speaker 2: ID and title. So even though I might have inline content, I will only bring what I need. And here things are much much better, but there's still difference. And here, to be even more fair, I had to scale this up to one million to see this difference being quite visible. And here Uh you cannot see it, but the green one, which is the right one, which is the table that has the external ideas of for Ainkey is just a bit more but we can see that as it scales still even though I I bring everything I need here Have certainly more time spending to bring the information from the database, which might not seem a bit here, but it's more than a hundred

Speaker 2: milliseconds. And when you have an application that has high traffic, a hundred milliseconds, many of you know that they are crucial. And this is only in the database. in the database query time and it's even it's the best I can get because here I only bring the ID and the title. So we talked about performance. Let's talk about workflow. We need a user-friendly experience so I can author rich text content and render rich text content. I need to know when something is safe and when it's not. And I need transparent management. I mean I need it to be super easy for me to do all my stuff without caring about security and performance.

Speaker 2: So after saying all that things, it's about time we talk about something Django related. So uh in my company we created this library, it's called the Django Pros where we did all that stuff, we encapsulated all this functionality in the in the library that we could use in our applications and in our clients' applications. It's easy to integrate and use, it's secure and robust. What comes in the box is field and model for storing rich text content either in line or in a different model if you wish. We have a widget for its text editing, which for now is the TRICS editor. And we have URLs and views for attachments, because you can drag and drop images, and it integrates with the admin site.

Speaker 2: How do you integrate Django Pros? You install the pack ads, you add it in the installed apps, and then you can use Either's a document model, so you can have everything in a different table and get the amazing performance that we showed you before, or you can use the rich text field to have it inline. And if you want attachments, you just add the URLs in your URLs file and then you get everything you know working for free. So it looks like that. You can see. Here you just add pros to your settings file Here you can have a model named article where you can have an excerpt with a rich text inline content because it's not gonna be big and you ha you can have a different one to one field for

Speaker 2: any key for a document to store big data in a different table to get all the performance benefits for free, a few easy views. Here is how you how You render this, you render the content of the model, and you have to mark it as safe explicitly because we need to know that this is nothing to joke around. This is HTML. This can have This can end my jQuery in my website. It's very serious. And if I want attachments, you know, I just put the URL there and it just works. So um wrapping up in a bit I wanna say um how Django Pros compares

Speaker 2: versus you name it. So if we have Django pros versus Django your WYSIWYG editor, I'm plugging plugging this for free here, is that Django Pros does not care Okay, it cares because it has the editor built in, but we don't care what this editor is, but we wanna focus on the workflow. But if you care about the editor, you can use these tools. Why Django Pros versus content management libraries? It's because we do not want to provide a CMS, but we want to make it easy to integrate rich text into your application. So, um the future we have some ideas about making the editor front

Speaker 2: ends pluggable so you can use whatever suits your needs best. We're thinking about supporting multiple output formats using attachments with pre-signs you are pre-signed URLs because right now you have to send the attachment to Django as you found out but we would like to make it straight from the browser to the storage and it would be easy it would be nice to be able to embed also content that's you know more platform specific let's say tweets or social media cards and all that stuff so wrapping up you can find about jungle pros in github And PyPy it's Django Das Pros. I will share the slides with you. You don't have to note it

Speaker 2: here. You can find it later. Who am I? My name is Paris and I'm co-founder at a company called Logic. You can find me in social media, in uh Twitter, GitHub, LinkedIn. I use a different username everywhere to make it really hard for you to find me I'm one of these guys. And my company is called Logic. We provide professional services for web development with Django. We publish our open source code. in our GitHub organization and you can learn more about us uh at withlogic. co or email at hey at withlogic. co. That's it. Thank you very much

Speaker 3: Thank you so much. So we've got time for questions.

Speaker 4: Yeah, hi. My name is Janice Wilhelm. I'm the chair of the W3C Web Editing Working Group. So we are the ones that like standardized in the browsers that they can work with with rich text editing mostly. A lot of this was Uh new to me, things I hadn't thought about. Um one thing that comes to mind is um over the last few years we have discussed quite a lot about which kind of rich editors actually work. Like talking about the the front end, right, in JavaScript. And it turns out that uh the ones that just try to manipulate HTML or let exec command or so do it don't really work because there's there there's just not enough information in there. And it's

Speaker 4: like exec command and so is is is not really possible to uh configure in any way Um and if you get to things like collaborative editing, it's it's the it's just not there. So the ones that really work are the ones that have a model of their own. So they store the the contents of the document not in HTML but in their own format which you they then can serialize to, for example, HTML, or which they in turn I mean you can also have it as a JSON or something And um the the three editors you mentioned, they all do it. Um the CK editor, Pros Mirror and tricks. Uh and so it should also be possible, or you I mean it is possible, to instead of storing the HTML, to store this format they have internally

Speaker 4: and send that to the back end. And install that. And then you don't have the problem that they can like it magically add JavaScript or like IDs to things you don't like and so on because you have the deserialization uh uh uh method, you simply don't have anything about that. You only uh allow the things that you whitelist, you only allow those kinds of uh attributes and and uh elements. Yeah.

Speaker 2: So um is it open? I think so. Okay, so thank you very much for the for the comment. We have thought about it and I can say that our initial um actually thank you very much for the work you've done with the W3C because we've tried to doing rich text in the past and it's It it was real real hard. So uh we initially intended to use Pros Mirror so we can do this thing, but Pros Mirror didn't have anything that worked out of the box because when we started this library you needed to compose the whole thing yourself but tricks works worked quite easily but I'm not sure if you're able to export them model of tricks. But I I'll I'll check it out for sure.

Speaker 4: Yeah send me an email afterwards. I'll I will I'll show you.

Speaker 2: Thank you. I will

Speaker 3: Next question.

Speaker 2: Are you

Speaker 3: are you using this in a In a CMS?

Speaker 2: Um actually we're we're using it in in a few client applications. So in uh in our company we have we have clients that we build applications for. So we use it in in in their applications and we use it in some websites of our own. So we have we have a website called remotework. cafe where we can find cafes where you can coffee shops where you can sit. and work from in different cities in the world. So the content that we put there is authored from from this library. Yeah. So we do use it.

Speaker 3: Cool. Uh any questions from the inside?

Speaker 2: Thank you very much.