How to distribute Django projects with Ubuntu Snap

Speaker 1: Without further ado, Johannes joining us from Lund, from another country, but not so far away this time. Um and uh

Speaker 2: eighty kilometers, I think.

Speaker 1: Yeah, eighty kilometers. And uh Johannes is uh talking about uh um Ubuntu snaps but in uh relation to that uh Fidus writer I would just say I've now tried it. I think it's amazing, really uh interesting way to write academic papers. So Check it out too. And a big hand for Johannes.

Speaker 2: Thank you. Yes. Okay. Yeah, so I will today talk about distributing apps using Ubuntu Snaps and specifically Django apps and the experience that we have had at with one particular app, which is what Benjamin just mentioned, Fidus Rider. For those of you who haven't heard of Fidestrider before, it's an open source app. And uh it's kind of like Microsoft Office 365 Online or Google Docs or any of these others, except that uh the like You can see in here you don't have uh font sizes you can change. You can just say that this is headline level two and so on. Uh but it's meant to be used for non-technic by non -technical users.

Speaker 2: It has a templating system, you can make books with it. It's quite extensible uh extendable, so there's a number of plugins for it and so on. And it's within this space quite stable. It's nine and a half years old. And uh th throughout those nine and a half years we've not always had financing. So we like needed a way to keep this stable because what mo mostly happens to these small startup projects is that somebody gets a bunch of money for a while, uh you know, they have an app, it runs a year. App is gone, but the text have a much longer lifespan. So people will not put their text into an app that's gone or potentially gone within two years. So we we we really try to make this like as as long-lived as possible and as sustainable as possible

Speaker 2: over even when we don't have financing at at some stages. And one of the really big selling points for this is it's you can self-host it. Okay. So on top of all these other things, you people can run it on their own server and they really don't need to have any communication with us Um and so for the first, yeah, I don't even know how many years, maybe five, six years, uh, the way we would have it is we would have one server that we would uh maintain ourselves and then there would be a growing number of universities and private uh uh individuals and professors and so on who would start running their own instances And we would try to give them support for this, right? And sometimes they would have money for it, sometimes they would not.

Speaker 2: Sometimes it would be, you know, a private individual in a in a in a country far away from here and they would be they would have their own research on their own server and and like we couldn't They wouldn't want to give us access, but also, you know, they they couldn't maintain it easily. Um, so we have that problem. Um so we so Yeah, a few years ago we looked at like, okay, so like what are the options out there? Like how can we how can we improve this and make this more subs uh uh sustainable? Um And what seems to still be the case, you know, and please teach me about something new that I don't know about, but is that uh like companies that run some kind of

Speaker 2: Django app have some specific solution of how to deploy this, right? You have m one to two developers who can who can do this within the organization. There's like specific steps that they have to go through. When a new developer comes in, and this is like a well-trained developer, it takes them up to multiple days to get the the app that the organization is running up and running on their on their laptop. There is Docker, which you know we're supposed to make things super easy because you can just like you know one command and it's just running and everything's just working. Uh unfortunately it's that's often not quite as easy and uh I have been in situations where I had to help like uh especially designers and people who are like technical

Speaker 2: but not exactly back end technical and and like try to help them get things set up and it would just take forever and and reason really wasn't uh really wasn't feasible for for other one outside of this developer community. And then we looked at like, okay, what are the specific requirements for us? Because we have this self-hosting aspect. Um so we needed it installed not just on one main server and some developer laptops, but we needed it on like hundreds of of servers. and uh or potentially thousands sometime in the future. Uh and most of these we do not control. Of course, the one that we provide we control, but anybody else can run it and we do not control those instances

Speaker 2: Also, as you know the app grew, we started having things like a spell check, uh a grammar checker built in, where we use language tool, another open source project, but it's a Java project. And somehow we also need to get that set up. So if if you need to give people instructions who are like, you know, uh this is the first thing that if the first time they set up a server and they need to set up both a Django app. and a Java app, it's it's uh it's not exactly easy. Um so yeah our our users are like professors, PhD students and and others who are like Have some basic understanding of a computer, but but not really enough to to to uh deploy normal apps to to to servers.

Speaker 2: Um So yeah, and then addition to that, because we we started we we first it like was just one app and then we found out like oh actually there's like a specific community that makes books with this. And so we made some plugins for them. And then there's others who just use this to like export to LaTeX. And others they just want the Jets export and so on. And so we we we started making various uh sub-modules like plugins for it And not everybody wanted to have everything enabled at the same time. And so somehow we needed to get like all this uh we needed to be able to people enable and disable plugins and and specific configurations while still being uh being uh easy enough for for yeah normal computer users.

Speaker 2: Um so we looked at what's out there, and this was at least what what what what we found at the time. There was uh flat pack, which is uh like uh a relatively new and and quite popular uh system for distributing packages. Um the problem with it is it's desktop only. So I was being a server-based app, this didn't really this didn't really work. Then there was app image. Um this is desktop focused. I believe it's theoretically possible to run server-based things, but it's not really made for it. Um and there's no built-in auto updating. So what what what we really wanted is to like be able to like push a button and say like all the

Speaker 2: all the ones that have this point release. you know, please update to the next point release without us having to enter their their server. Um and so that was really only Ubuntu snaps. As far as I'm aware, that that does all this. Um now of course it's being criticized for this company Canonical controlling the app store. But it does include auto updates. It's server-friendly, meaning you can make these server-based apps, although as far as I'm aware, there's hardly anybody doing it. An advantage is that it's because it's both desktop and server. Uh for a while we were like some were asking, can you make a desktop app first?

Speaker 2: Because I really just you know, want to write on my own laptop and then like click the export to LaTeX or uh export to JATS button and then get that uh out. And this way we could just say, well, just install the server version on your own laptop and you're good to go. Like that that works just as well. Or almost the same. You can run it in your own browser and there you have it. Um yeah, so that's that's how we ended up with Ubuntu Snaps. Um now, yeah, so you we have the the desktop and and server in the same app. And there are also some advantages for uh like how the update works. So if what it does is it tries to update And if the update it tries to

Speaker 2: so it takes the data that is associated with an app, right, it tries to use it with the new version of the app. If that fails, then it rolls back the update. So you're back to the old version. So you should never end up in a situation where there's a broken update has somehow destroyed the system for a user. case is that the uh the user ends up with an older version that is not being updated and then you can go in manually and help them and so on but in most cases that that really shouldn't uh help. The fact that they have this app store may means that it's just it's really just like a one-click in the search in the Ubuntu App Store for Fidus Writer, click install and you have it running. Of course, this basic install is not, you don't still don't have a domain name, you don't have you know

Speaker 2: an HTTPS certificate, but those are fairly easy steps to add uh on top of it later on. Um And also it has a like different different channels. So it has like you can have the edge channel. So that's like the newest version that's maybe not as stable. And then you have the stable channel where this is always like a version that is like rock solid and and should never break because it has been in the edge channel first for a while and and We've done the the extra testing with with those users who are willing to do it. Um yeah, they um now when I uh Just because some are not familiar with how this how this works in snaps

Speaker 2: is the by default, snaps update automatically. Right. So you just have the system running and it's it like it pulls down the next version immediately. However, you can manually like disable this. So for some reason, for security reasons, so if some want to like monitor exactly what is installing, they can disable it. And then if they uh so so one question we received is yeah, well what happens if you if you disable it And then you enable it again two years later. Right? Does it just then like jump from version two to version 416 and like you know clearly the data won't be able to be Read by the latest version. And there is actually they have something built in for this, which is called epochs, where it's it it knows like which version can read which version of the data.

Speaker 2: And you mark which version of the data the current data is. So it will then update for say you have one version that can read data of version 5 and write data of version 6 And then another one that can read version six but and write the data of version seven and so on. And so it will update those versions that are needed to get from wherever the user is currently to where what the latest uh release is. out there. So that's all really, really smart. And then the last part is that that uh like you know we don't have the resources to bundle this for a bunch of different architectures. So the way it works is we we just put everything on our repository, click a button on their website, and then the final package is being built for all these various architectures.

Speaker 2: And so people can run it on different servers, even though uh Uh yeah, we just have the the AMD sixty four, which is uh probably the most uh common nowadays on laptops Um so what did we need to do? Well in the uh fibers writer part we needed to do some code changes The first thing is that we put we we packaged everything into PyPy packages. So that we made like PyPy packages for each of the plugins and of the main app. Um and we modified the like the you know how the when you have a so the plugins are Django apps. Right? But you know how normally you have to like both add the apps to the installed apps list in the settings file

Speaker 2: and additionally add uh the URLs. to be linked from the from the main URLs file and potentially other places. So we we basically uh uh yeah made that uh uh Simplified that so that just by adding any application to the installed apps list, it would automatically draw uh drag in the URLs. So that way, the people who have to maintain this, all they have to do is add the name of a plugin in the installed apps list, and vroom, that's just working for them. Then additionally, we created a repository, the Fiduswriter Snap repository, which is based on the Nextcloud. uh snap repository, which is open for everyone to see.

Speaker 2: And I think that's a PHP app. So we we we learned a lot of what they did and how they did things, but obviously it wasn't quite the same because it's not a it's not a Python app. Um but we did things like the for example they have a MySQL server packaged into it and we we copied a lot of of those things. Um but some extra things we bundled in there is we bundled the nano editor. And uh then so we created a like an extra command, which is this fightuswriter. configure, which you can execute, which basically opens the Nano editor on your configuration file. And we made that a little more uh yeah child safe by adding like after you edit the configuration file, if you made some mistake, that means that the file is no longer

Speaker 2: valid Python or that you can't run like manage help uh manage py help with this file it will fail So that way, like the user should not be able to accidentally destroy their configuration file and thereby destroy their server. Yes, if just quickly to see what that looks like. Um if I try to execute it here Right, so this is like a normal configuration file, and what I what uh like a non-technical user may do is write something like you know uh some setting.

Speaker 2: Like that. No? Forgot to put the the quotation mark or put like a different type of quotation mark, something like this. Now they try to save it. So exit and now tells me now this is no longer valid. Do I want to fix it or not? If yes, I get thrown back into it. And uh then they try to fix it and eventually they'll they'll send us an email saying like your program is wrong, uh, you know, and then we we try to help them with this, but at least they're not breaking their server accidentally. And so far this has taken care of like almost all the stupid mistakes that people can do by misconfiguring the configuration file.

Speaker 2: Yes. Um and then the last thing is we needed to add like a type of migration specifically for those things that had to that have to change from one version to another that are not in the database. And I'm I can show you how what that looks like. So this is an example of this. So this is like a poor man's migration uh system. No, but what this does is It's uh that what we did in this case was we had a plugin that used to be called GitHub Export. But we added GitLab X uh GitLab support for it. And so it should no longer be called GitHub export, just Git repo export. So we needed in all the configuration files when they switched to the new version

Speaker 2: it needed to say git repo export rather than github export. And so we just Wrote this very basic uh you know Python script which is being run on anybody doing the upgrade from one version to the next one, replacing anywhere where it says GitHub export to git repo export Now you may say this isn't really safe because maybe somebody in their configuration file, you know, their their child's name was uh GitHub and and so somehow they put you know this word in there and like well what not. Yeah that's true but remember like it's it does the rollback no so if somehow this breaks the whole system We are still being rolled back to the to the version before this. And so uh we we don't end up in a situation where where where we have a broken configuration or broken uh server.

Speaker 2: Now what is it uh what does the uh distribution look like now? Well yeah as you can see it's like kind of going up. There's some spikes. This is whenever I I believe this is whenever Ubuntu features us in their in their in the app store, then like lots of people click on it. So then goes up and goes down a bit after that. But still it's like at 433, that's more installations than what you know our small team of four people who are not financed constantly would be able to help with and and and fix and and install and so on. Um you can see it's like many different places, fifty in Germany. In Germany there's like actually several universities using it officially and who also paid for it.

Speaker 2: Denmark zero. Okay. We even put a Danish name on it, so I'm not sure what that is about. Um Yeah, so and and here just an example. So this is the Technical University of Munich, okay. They have an installation and uh Yeah, where 's here you are the because of the we were just talking about the the 12 factor app this morning, right? So I mean this is kind of what we're doing is kind of separation of the app and the data Right? Because Ubuntu does this. But not quite, because in this case, what they needed is like a specialized plugin for just their server.

Speaker 2: And they needed one so that their students can log in with their student login system and also a mechanism so that all students would be deleted after a certain period of time to be compliant with GDPR. And so this was like very specific, something that they had programmed. It's like the like the the the calls we had to make was not anything standardized. So we had to program it exactly for them. And so this plugin lives in the user data of this of this uh uh Ubuntu Snap. And and you you know eventually this could break. But at that stage we see like we get an alert like, oh, they didn't update. Why didn't it update? And then we can go in and fix it for them. But so far for the last

Speaker 2: Yeah, I think two major versions and a number of point releases, it hasn't that we haven't had really any problems with it. And it's just updating by itself and and just working. And yeah, so our conclusion with this so far, yeah, it's this is it's it's we working really well, but it's not It's not perfect. So it it does work to get like the app to just be installable with a single click and so on. It's really easy for us to maintain a lot of instances everywhere and and Okay. And also to let uh people that don't want to let us onto their server, to let them you know help us in in in in fixing some bugs and so on.

Speaker 2: Um but There's still a number of issues with it. So the first is this is not very common. And I don't really know why, because it seems like other than Nextcloud and Fiduswriter, there should be so many other apps. That people for some reason want to run on their own servers. But there doesn't there doesn't seem to be like a common mechanism for for letting people install those kinds of apps. And so because it is so uncommon, like our help is looking at what uh Nextcloud did in their snap, right? I tried to ask somebody in the Ubuntu Snap forum, but there isn't really a lot of there are not many answers there like before we asked them. And in Stack Overflow, there's almost nothing.

Speaker 2: And so there's also a number of bugs in here. And you can you can s if you look at the code, you can see out of the file stride snap code, which is on GitHub and which is also open source, you can see there are several times where we like work around a major bug that they have in their system that hasn't been fixed for multiple years and probably because there's like hardly any demand for getting this fixed. So that's an issue. Then also

Speaker 1: And yes.

Speaker 3: If you are if you're running a FIDO standard on server, what exactly is The advantage of using snaps over being a native package.

Speaker 4: So uh yeah the question is if you're running fi Fideless Rider a standalone, what what is the Um advantage of the snap.

Speaker 3: Yeah, yeah, exactly.

Speaker 2: Yeah, so the it has this the the the data rollback that I'm at least as far as I'm aware they we don't have in the same way that It's DAP isn't made for that the same way. Also the i idea at least with it is, and I mean it is in in theory it's working that you can install this not just on Ubuntu, not just on Debian, but also on Red Hat and and many of the others that are not depth based. It's just that so Ubuntu has like made this like, oh this is the common uh tool you can install on everything. But then and so they have created these tools for even Red Hat distributions to install it there. But uh the the Red Hat company is not very happy about it. So the the Red Hat people are are kind of uh

Speaker 2: against it because they like flat pack, the the other one. Um

Speaker 1: I have a similar question. I know that you can uh answer the same question about each and every alternative, but uh I I was interested in in why Docker could be uh disadvantage for the people that you find using the snap is better.

Speaker 2: Because it's uh uh it's harder to install Yeah, that's not like Ubuntu like Docker, my experience with Docker installing it on Ubuntu is first of all you can't use the Docker version that is that comes with Ubuntu. You have to like install some separate list of packages from uh from uh yeah get them somewhere from the internet and you need to have docker compose for something which I never quite figured out why that's uh why that's like a separate thing and and I mean you end up spending uh like I've I've on on some of my last jobs I had to do to use Docker and it ended up being like at least half a day or so. uh or day or two getting a a new uh system set up with it and you have to at least these really long instructions.

Speaker 2: It's never just one line. So

Speaker 1: saving your users from this confusion.

Speaker 2: Yeah, and then I can handle it. I'm a developer, but like I wouldn't I wouldn't uh I wouldn't ask somebody I wouldn't ask somebody else to do it. Whereas this, I mean I could anybody who has I I put my my parents only know Ubuntu. They never had a Windows laptop, no so they I just put Ubuntu when they first you've got their computer on it, they've never seen anything else. And I could ask them to just like, you know, install Fidus Rider, they know how to how to install packages and then they can open their browser and see it. I I think that's easy enough for them.

Speaker 1: Yeah. You have very lucky parents. Um yep, uh more questions. In that case we can uh get a little bit closer to our schedule But thank you so so much for the talk.