How to turn your Website into an App (and why maybe you shouldn't!) with Russell Keith Magee

Speaker 1: My name is Russell Keith McGee. I am a principal engineer on the open source group at Anaconda. They pay me to attend conferences like this one to talk to people. like you so I'd like to thank them for that support. They also support the Python open source ecosystem by providing direct financial support for organizations like NumFocus. and by employing developers to work full-time on open source projects, projects like Jupyter and Number and the open source project that I founded, Beware. For those who haven't come across it before, BeWare is a collection of open source tools and libraries for creating native user interfaces in Python for desktop, but also for iOS, for Android, and single-page web apps. Android supports Beware to ensure that Python remains a viable programming language for the platforms that people are increasingly using as part of their day-to-day computing life, phones, tablets, and so on.

Speaker 1: But we are all here today because we are web developers. Personally, I am not doing as much web development as I once did, but I am a long-term fixture in the Django community When I started working in Django back in 2006, the iPhone didn't exist. Apps were real apps and websites were real websites. However, in the intervening 16 years, things have changed. Phone apps are now an unavoidable part of our online lives One of the reasons that I started the Beware project was that I was running a startup that was a Django website. But there were tasks that my users wanted to perform that needed an app. And at the time, as a Python developer or even as a web developer, there wasn't really any good options for building that app. It's taken me almost eight years, but there are now some options, and that's what I'm here to talk about today.

Speaker 1: So, you have a website. You need an app. What do you do? Well first off, let's actually interrogate that premise. Do you need an app? In many cases the answer is no, you don't. You know the adage that this meeting should have been an email? This app should have been a mailing list. As software professionals, we owe it to our clients to interrogate why they need an app. If I look at my phone, I have a proliferation of apps that don't need to exist. And because they don't need to exist, they end up being developed on the cheap, which results in a bad user experience. Web apps crammed into a mobile phone form factor just so some executive can say we have an app. This is on us as developers to push back against clients that ask for things, to interrogate why clients want apps. And if their use case doesn't actually add up, convince them that their development time is better spent elsewhere.

Speaker 1: Sometimes a website is the right answer to the problem. Especially if you're able to take the effort that you would have spent trying to build an app and direct that into proving improving the user experience on small mobile screens. Because websites do have distinct advantages over apps. Features like deep linking are in the blood of how websites work. You can deep link into an app, but it's extra effort and it isn't necessarily immediately discoverable. Websites are single deployments where deployment has an immediate and universal effect. If you've got a situation that is time critical, you can push an update and everyone has that update pretty much immediately. That's almost impossible to do with an app. And apps, especially mobile apps, are distributed through the walled garden gardens of app stores. Your ability to play in that garden is 100% at the whim and mercy of the owners of those gardens.

Speaker 1: If an app store rejects your app or doesn't approve it in a timely fashion, you are pretty much out of luck. And then there's the mandatory profit sharing that app stores enforce. There are arguments for not building an app. You should only be building one if your problem will actually benefit from the specific strengths that websites and web apps lack. So, when is an app the right solution? Well, native apps allow you to work with local data. If your data files are multiple gigabytes in size. Uploading to a web app probably isn't a viable option. If your data is sensitive for any reason, you need to be very careful about where you upload that data. Local data storage and manipulation may be preferable or even required in some cases. Uh sorry, in those cases. Remember, your attack surface is only as big as the data you actually hold.

Speaker 1: You can't leak data that never left the user's device in the first place. Local data also matters if your users are going to be working offline. When I got on a plane to fly here, I was offline for 35 hours. That is time I cannot access a web app or cloud data. When I'm at home, if I get in a car and drive for two hours, I am in Outback Australia. I cannot guarantee that I have a reliable sell signal. And if I can't get a sell signal, I can't use a web app. but I can still use native apps on my phone and laptop. Native apps also allow you as a developer to deeply integrate the user experience with the device. Integration with task managers, notifications, task bars, application lifecycles, and more. These integrations are difficult or impossible to do in a web context because the entire web experience is delivered through the lens of a browser tab.

Speaker 1: Native apps have almost unlimited access to native device hardware as well, things like cameras and peripheral ports and GPS readers. Web APIs for these exist and they are definitely getting better with time, but they're also very heavily sandboxed by the browser. Native apps have much lower level access to these services. And when I say device hardware, that also includes the CPU and the GPU You can do some very impressive things in the browser with WebAssembly and related technology. But if you need to do some computational heavy lifting, a native app is going to give you much better results because you have direct access to a multi-core CPU and GPU acceleration. Now, some of these properties can be true of some web apps, but when compared with a vanilla Django website, these are areas where having a native app will be an immediate advantage.

Speaker 1: There is one other very important consideration though. You need to have the skill set to build one. If your development team doesn't have the skills to build a native app, then best case your progress will be slowed while they learn the skills they need. So ideally, you want to reuse the skills your development team already has. So, we're all Python web developers. We're here at a Python web framework conference. So how do we develop websites? Oh, and we know how to develop a website in Django. Are those skills transferable to app development? Well, broadly speaking, yes. If you are developing a modern Django web app with a you know React or View front end, you are probably already doing a lot of what you need to do to support an app. In order to support a single-page web app, you have already made the major separation that needs to happen. You've separated the back

Speaker 1: end that provides an API from a front end that consumes that API to present a user interface. Once you have done that separation, there is very little architectural difference between building your front end in, say, React or building it with a native or cross-platform GUI toolkit. However, that's not the end of the story. There are two facets of this kind of API separation you may have considered already, but take on a whole new level of importance when you start looking at native apps, especially native apps that are going to run offline. The first is data storage. If you are only expecting online operation, nothing really changes from the way you deal with a single-page web app. The UA UI requests data from the server as it's needed, it's always online, it's always accessible. However, if you are planning for your app to work offline, you are going to need a local cache of at least part of the database available on the client's device.

Speaker 1: Your service database is no longer a single reliable source of truth. It is now possible for multiple copies of the data to exist and be out of sync and not be able to immediately notify each other of that discrepancy. So, how do you manage this? This is essentially the CAP theorem that plagues NoSQL databases on steroids. The good news is that it's a solvable problem. There are solutions to these problems and we can borrow ideas from the implementation of multi-primary and NoSQL databases. The bad news is that the right solution is highly problem-dependent. One solution, clear domains of authority. Define as part of your data contract that either the client or the server is the source of authority for a given type of data. For example, only the client app can write event update records.

Speaker 1: Fine, the specific rules don't matter as long as it's always clear who currently has authority to create or update those records. You can also write uh use write permission for a data class or the write permission for the data class to change over time. The sort of rule set can uh that sort of rule set can use states on an object or you can have an independent token object. If you're going down the tokenization path, it's essentially transactional behavior. You request a lock for some context, you are granted that lock, and you release the lock when you're done with it. Another approach from databases. If you've ever run a database in a multi-primary mode, you might be familiar with a write-ahead log or a wall. Any candidate changes written to a staging log. But those changes don't actually take effect until all interested parties have ref uh have uh uh logged in and declare that they're synced up and agree on the current state.

Speaker 1: What happens if there's inconsistencies? Well, then you either need to roll back or resolve the conflict. You either reject all the ambiguous changes or you define merge rules to always use the change with the most recent time step. Uh timestamp or you put ambiguous changes into some sort of purgatory state to be manually resolved by the user. Which of these solutions will work for you It depends. It's very application dependent. You need to think about what your data are, where your source of truth lies, and how you can use that knowledge to prevent and resolve conflicts when they occur. Fun little historical side note, I mentioned earlier that I had a startup that needed a mobile app, and you might think my experience here comes from that, and you'd be right. Partially. My startup was eight years ago. I was also doing this in 2006. Sixteen years ago, two years before the release of the iPhone.

Speaker 1: A technician was an Australian Army officer training radar operators while holding an iPac Pocket PC running Windows CE that was running a Django local Django server and SQL Lite on the device. Technicians would record training results while inside a secure facility that didn't have Wi-Fi, that'd come out of the room and upload all the results to a central Postgres database for analysis. So, anyway, back to the topic at hand. I said there were two things you need to keep in mind when building an API that will service an app. Data storage is the first. The second is versioning. Versioning becomes critically important and you need to consider both forwards and backwards compatibility. If you've got a website and you deploy an update, you can be pretty certain that within a couple of minutes every user has a new version of the site. There may be caches to flush, but It's a manageable problem on a relatively short time scale.

Speaker 1: That's not true of an app. Users won't always install the update. And even if they do, any change to a file format, any change to a cache format, any change to an API potentially leads to a state where old code needs to read the new format and new form new code needs to use the old format. The only advice I can give here is to consider versioning early and often. And you may already be doing this. Good API design usually includes some kind of version marker. But again, it's one of those things that goes from being sort of a good idea and best practice in the context of a website that is constantly updating to something that is essentially essential in the world of building an app where where the app may not be updating. Okay, so let's say we've done all that and we want to turn our website into an app. What do you do? There are a spectrum of spectrum of approaches you can take. Which one you use will depend upon your specific requirements, skill sets, and resources.

Speaker 1: One option is actually to stay completely web native. Over the years, web technology stack has added a lot of features that lets a website be more app-like. This can be as simple as just using adaptive layout, that won't make your website present as an app, but it will make the website a lot easier to use in a small foam factor There are a number of options for making your website work well offline. Web storage APIs provide a simple key value store. Indexed DB provides storage for larger structured objects. App Cache Manifests are a legacy API to force your website to pre-cache certain APIs so that or sorry, certain URLs so they are available even when you don't have network connectivity. That has been deprecated but still works. The more modern approach that browsers suggest you use is called a service worker. They provide a JavaScript API for injecting logic into the browser

Speaker 1: between the sort of the request of the resource and the server that provides them. And you can use that as an entry point to cache resources that the app needs, synchronize those resources with the server, and so on. If you provide a web app manifest as well, not to be confused with an app cache manifest. Your app can then service as a portable web app or PWA. PWAs are very much being promoted by Google, but then even Google doesn't use them consistently. Gmail, Google Sheets, obvious candidates for being PWAs don't present as PWAs. And support can be a bit spotty between different browsers. Now, I have skimmed over a lot of details here, partially because the end user story is a little bit of a mess, but also because this approach doesn't have a good Python story. Modern offline apps really hinge on the service worker approach.

Speaker 1: Writing a service worker is a long-form talk unto itself. You effectively end up writing a client-side reproduction of non-trivial amounts of the logic on your server. And it all needs to be in JavaScript because it's client-side web. As web developers, that's at least on the table, as Python and Django developers, it's not a great option. So is there a better approach that doesn't require us to redevelop our server code and still get an app? Well yes, you can build a native wrapper that consists of a lightweight web server and an app window that contains a browser that points at that web server. There are two well-known tools that take this approach. The first is Electron. Electron lets you build a desktop app, but not a mobile app. Electron doesn't support iPhone or Android. It is a JavaScript tool and it's most at home running a node web server, but you can coax it into starting a Python web server like Django's

Speaker 1: The code that runs in your app's web server won't be exactly the same as the code in your server's web server. You only need to serve content on the app that is related to the client-user interface. You'll have some additional sort of context to manage the exchange of data between your server side and the client side, all the data management stuff that I mentioned earlier. But you can share resources like data models, validation logic, and so on. If you need mobile support, there is PhoneGap, it's called Cordova, and it's open source guys. It's essentially the same idea as Electron, but supports iOS and Android rather than a desktop app. The downside is that it doesn't work with Python at all. Your app code needs to be JavaScript, so you're back in the same boat as a service worker, writing a bunch of JavaScript server code to support the deployment of your app. Can't we just do this in Python? Well sure, and because I enjoy a good pun, I'm going to call that approach positron, because positrons are electrons, but a lot more positive because they're in Python.

Speaker 1: You build a native app in Python that starts a web server. The native app's GUI consists of a single widget, a web view, that is pointed to the in-app web server. And you can do it 100% in Python. The only part that isn't Python is any client-side JavaScript that you use to render your website as an app So how do you do it? Well, this is where BWAR comes in. You create a virtual environment. You install briefcase, BWAR's app packaging tool, and you use it to bootstrap a new application. This will ask you a bunch of questions about your application. Name, description, authors' names, and so on. Fill out those details, select Toga as the GUI framework. Toga is BWare's cross-platform GUI GUI toolkit. We're going to use that to build our app. With those details, BWare will generate a full stub project including configuration files and an app. py.

Speaker 1: What do we put in that app. py? Well, app. py declares an app class. And when that app class starts, we're going to tell it to set a single set up a single toga widget, a web view. We create a Python threading event which we'll use to track when the web server exists and is ready to serve. Then we create a thread that will run a web server method on our app class and start that thread. And we tell the app that when it exits, it should run a cleanup method. Then we wait for the server exists event. When it's triggered, we get the actual web server host and port and set the URL of the web view to that address Lastly, we set up the main window of the application. We set the content of that window to be the HTML WebView widget that we created earlier, and we show the app window to the user. All that's left is to define the server threads. Now, at this point, I'm going to wave my hands a little bit because the code

Speaker 1: that's needed to start a server isn't especially complex, but it's just complex enough that it doesn't fit well on slides. And it's at least a little bit application dependent. If you want to see some working examples, the Toga repository contains two examples, one of a site serving completely static files, and the second serving a Django site. In both cases, the missing pieces basically look like this. There is a web server method that starts a HTTP server instance of some kind on localhost port zero. Port 0 instructs the operating system to select a port rather than hard coding a port 8000 or something like that. You have no way of knowing what ports are in use on your user's operating system, so you let the operating system decide. There's also some nebulous setup the server code, but once it's ready, you trigger the server exists event, tell the server to go into a serve forever loop.

Speaker 1: Remember, this code is running on a background thread in your application. So we have to signal the main thread to say, oh, it's okay to proceed. You can actually show the user the app now. Lastly, there's cleanup. When the app exits, tell the server to shut down cleanly, so you don't end up with straight uh straight ports lingering. That app is uh sorry, uh that's and that's it. That's all you've got to do. You've just wrapped up a web server as an application with a single HTML Web View pointed at that server. The app is completely self-contained in the HTML and JavaScript of the website that you've got. If you want your native app menus to trigger behavior in the app, you can inject arbitrary web JavaScript into the web view. It's also possible to have Python code be triggered in response to a JavaScript. event. But there's no simple API for that at present. That is something that we actually want to we want to we do want to add. So how do you run the app? Well you want a quick development test, you run briefcase dev, and you'll get a native window that looks something like

Speaker 1: That for a static site, or that if it's a Django version. There's a web browser there, it has no address bar, no tabs, just the operating systems window frame and a web view. And if you look, you'll have an icon in your in your taskbar, you'll have a menu up in your uh up in your toolbar So, you can then turn to building out the website that you want to serve as an app using all your usual Django tools and CSS styling and JavaScript, whatever you want to do. Once you've finished iterating on that design, you can then package that app for distribution. Briefcase Create builds the stub of a standalone app uh app for whatever platform you're running on. Briefcase build will do in some compilation if it's needed. Briefcase run will let you do a test run to see that application actually running. Briefcase package produces a distributable product. On a MacOS, that's a signed and notarized DMG. On Windows, it creates an MSI installer that ships the app into the user start menu.

Speaker 1: On Linux, it'll create either an app image or a flat pack depending upon the options you use. And because it's beware, it can also be deployed as an iOS or Android project. Just add iOS or Android to your briefcase calls, and you have a mobile app. Now. Now, very small asterisk here on iOS. You currently need to add a four-line workaround for an annoying async bug that we're trying to get to the bottom of. We are working on it. But bugs and workarounds notwithstanding, you can have your website served as a native application using Python tools as a desktop or mobile app from a single Python code base. Alright, so that's wrapping a website as an app. As I said in the discussion about Electron and Phone Gap, it's unlikely to be the same code that you're running server-side as your server-side app.

Speaker 1: It is a slightly different web server that only needs to serve client-facing web pages and it manages the exchange of data between the server's database and your app's database. If you only want to run sort of as a local, there's no sort of server component, you don't even need that. You just need to have the the code that displays the local user interface. But you can use the same web development skills, your same Django skills for developing that server. And you then bring in your designer who knows the JavaScript and the HTML to make to make the f make it look the way it wants to look. But I'd like to suggest that this is a bad idea for app development. Operating systems provide buttons and menus. They have a consistent platform-dependent style and all truly native apps on a platform use the same buttons and menus. This is a form of user interface affordance. As a user on MacOS, I know how MacOS menus work.

Speaker 1: I know where I need to click or whether I need to click and hold. I know that when a button is blue, it means that the return key is bound to that button and the button has focus. This means that user interfaces are easier to use and easier to discover. Web design does not exploit this. Every website has a different look and feel for a button because they've all been coded from scratch using HTML and CSS primitives Some of the most frustrating experiences I have had as a user on a website have begun being because a website has a pull-down menu with some weird customized behavior that just does not behave the way that I'm used to having a menu behave. There are web design themes that will make your web page look like a native Mac OS or iOS or Android app. They're never quite right And so you end up with a package that is very clearly an app on the outside, but nothing about how the user interacts with the app is completely consistent with all the other apps on the user's operating system.

Speaker 1: It's the uncanny valley but for user interface design. Apps that almost feel native, but something isn't quite right. And that friction isn't just cosmetic. It's something that impedes users getting on with using your app. Learning a new set of user interface language that exists solely in your app is a cognitive load you are putting on your user. And worst of all, it actually took effort to make this user experience worse. You had to build all the primitives and develop this user interface language. And I bet there's a lot you forgot about. For example, who here can guarantee that every website they have ever built has full WCAG accessibility compliance? It absolutely can be done. But you have to make sure that you remember to do it when you're building your user interface from scratch. If you build a native app using native widgets, you get accessibility pretty much by default

Speaker 1: because the operating system provides it. Now, I know it is not as simple as that. In a world where native app development requires a completely different skill set, learning multiple new programming languages or maintaining multiple code bases, it makes sense to settle on a single web technology stack and to maximize the utility of that stack. But it doesn't have to be that way. We are all Python developers and with tools like BeWare you can pivot from having fully native apps written in Python. Don't just wrap a Django website in the skin of a native app. Build a native app. Toga is a cross-platform Python native GUI toolkit. I have just shown you how simple it is to embed a web browser as the sole widget in a native app. Buttons and text inputs and labels are just as straightforward, more so because you don't have to worry about the web server threading in the background or the styling of user interface elements.

Speaker 1: So, don't rule out the idea of actually building a native app using your app your development team's existing Python skills. And if that is a bit too much to swallow in one sitting, there is a continuum between those two approaches. You can build a GUI that uses both the embedded WebView and native elements in a hybrid approach. Toga. webView is just a toga widget. It can live in a toga app like any other widget. So you could start with a purely web user interface and then over time migrate some of the parts of the user interface to be completely native. lessening the amount that actually needs to be inside that web view. And over time eventually migrate to a completely native app. Over that whole time you don't have to leave the warm embrace of Python. One last approach that's worth considering. Flip the problem around on its head.

Speaker 1: Start with the app and use tools that produce our website from the app. Toga is a cross-platform GUI toolkit that supports MacOS, Windows, Linux, Android, iOS, and the web. And you can use Briefcase to deploy your Toga app as a website. Now this is still very early days, but as an example, let's say you build a native application that does Fahrenheit to Celsius conversion. Type a Fahrenheit value in the top input, press the button, Celsius displayed down the bottom. That's tutorial one of the Toga repository. About 50 lines of very verbosely documented code delivered here as a completely native Mac OS app. It could just as easily be Windows, Linux, iOS, or Android. And here's the exact same code, zero lines of code changed, served as a website. The only difference is that you run briefcase run web instead of briefcase run

Speaker 1: macos. And to be clear, the Fahrenheit to Celsius conversion here that happens when you press the button is being done in the browser in Python. It's using PyScript and PyDide to provide a client-side WASM Python interpreter. The entire GUI is rendered as Python code modifying the DOM to put the GUI there. Essentially the same thing as React is doing, but it's doing it in Python. Briefcase and Patoga provide the thin layer on top to provide the GUI, uh the GUI API and a way to easily deploy and test that code in the browser. If you know up front that you are going to need a website and an app with similar functionality, why not start with the app and have the website fall out as a side effect rather than try to cram the website into an app? The bottom line is though, if you do need an app and you are a Python developer, you have options. If you're a web developer, those options aren't far that far removed from

Speaker 1: what you're comfortable with right now. And if Beware and Toga aren't to your liking, there are other Python GUI frameworks you can use. QT, Kivi, WX, Windows, cross-platform, hybrid, app to web support will vary, but they exist. Check local guides. In terms of beware though, there's still a lot of work that needs to be done. I've flagged a couple of bugs and some missing features. The support of Anaconda means it's developing much faster than it ever has before. I am here for the sprints. If anything I have spoken to you about today is of interest, come have a chat. If you'd like to get involved, I've got plenty of things you can work on, no matter if your level of experience, and everyone who contributes gets a challenge coin. Thank you all very much. On the nose

Speaker 2: Thank you so much. We have time for a question.

Speaker 1: We do?

Speaker 3: Hello Russell, thank you, brilliant. Um I couldn't you talk about um hybrid apps. I wonder do is Can you embed a TOGAR app inside a say an iOS application built with Xcode, for instance?

Speaker 1: So the application that that actually ends up running is an app Xcode application. So if you needed to bring in native Objective-C or Swift code to do something, you could absolutely do that. What briefcase is actually doing is templating an Xcode project that has all the stub to get a Python interpreter started and get the thing running. If you need to modify that, you can go in and edit the project as much as you need, add extra widgets in pulling libraries from wherever they need to go.

Speaker 2: We had time for a question. It's so cool. Thank you once again. Give them a clap again. Do the thing. Thank you.

Speaker 1: Thank you very much.