Django for life (sciences)

Um thank you very much. Thank you for uh being awake so early on the third day. So my name is Nicola and this is called Jungle for Life Science. Very briefly who I am Uh I am a software developer from Belgium, I'm working on my own, can call it a freelance consultant, whatever And I like to think of my way of working of being more like an artisan than a shine worker, if that makes any sense. And my my niche uh for the last 10 or 15 years has been uh software for biodiversity research, and that's what I want to talk about. today. So um what we will do, uh I will present a few concrete examples of what I do because I guess

that um software for biodiversity research is a bit abstract Uh I'll tell you why I think Django is very good at it. Uh some tips and lessons learned along the way. Uh I try to keep those tips uh useful for a broader audience, even if I do something a bit uh a bit on the side and finally some possible improvements uh that can still happen in the framework and the community. Um what do I do? Software for biodiversity research. Um basically biodiversity is the the variety, the diversity of all forms of life on the planet. Uh and I don't want to bring bad news in the morning, but as you probably know, uh

there's a big biodiversity crisis right now, which is kinda similar to the climate change uh issues. It's also human-caused and basically the species go extinct at an alarming rate. According to the latest estimates, in the next hour we will lose between one and three species probably. And most of them are probably not known and not described from humanity and it will just be lost forever Sorry about that. So if you want to protect nature biodiversity, you need to understand it, to understand it. Uh you need some people to study it scientifically and to study it you need good data and that's really where uh we can help as software developers

We can help sharing data, exchanging data, transforming data, visualizing data. That's really where we can as a community help scientists do their job and do their job better. Yeah, that's what I do. I will show two concrete examples now and then uh more during the the talk Um first one uh it's a tool we developed to visualize the bird movement, the bird migrations, uh using the data coming from weather radars. because basically if you want to uh study bird migrations uh you have a few options. Uh the first one is just to use your eyes and look at the sky and look for birds It doesn't scale really well at the at the scale of a continent, for example.

It doesn't work when it's at night and many birds migrate at night. uh it doesn't work when it's cloudy so yeah it's not great and that's probably why until a century ago uh people in the northern hemisphere had basically no idea of where all the birds go during winter. And there were crazy theories like about that, like the transform to other animals, don't kind of things. And finally a century ago Uh people spotted a stork, this stork over Germany, it was still alive, but he has an arrow in the neck and the arrow really looked like the arrow they can see in uh in Central Africa and that finally gave the idea. But for most of the humanity we have no idea of what the bird did. Another option d using the ice

is using uh specific bird radars to do that But it's specific equipment is very expensive and it's very difficult to convince authorities to install bird radars everywhere if you were a bird scientist So some smart people say okay but what we can do is that we already have in most inhabited uh places on earth a very good um array of uh of weather radars because we need radar radars to follow precipitations, thunderstorms and all things that are really needed for human activity. So the radars are already there. And if we smartly filter out the data we can probably distinguish uh the Nico on the radar that is coming from a bird than the Nico coming from an airplane or other phenomena

uh or clouds basically. So Other people much smarter than me developed uh algorithms for that and where we could help was basically to do all the plumbing and to put all the data from a a wide array of networks together and make them communicate in a common format and uh put all that in a central repository and build build a web application to show nice graphs about that and and you have tools like this one where you can there's a small uh map of Europe on the left you can select a radar anywhere you can select the time and have an idea of the the number of birds that are flowing around uh at the time. Um just a quick question for the audience. Uh if you see on this graph on the left

There is a big peak. You can see January 1st around midnight. Any idea of what it is? Yeah, exactly. Fireworks, people partying and the fireworks met make the birds really scary and they all fly. And you can basically see that on every radar uh every every year Um another project, this is about tracking invasive alien species because invasive alien species do great harm to ecosystems and like every kind of invasion. The earlier you you can uh work on it, the the better it is. It's much more difficult to eradicate a species if it's everywhere than if it just arrived in a specific location. So you have people whose work is to is to basically do that and manage spaces on the field.

And this tool allows them basically to be alerted as soon as possible. So data is downloaded every night uh in the web app. uh from a data aggregator called JBIF where all the biodiversity, open biodiversity data is basically. And the users can basically create alerts and say I am interested in this and this species in this specific location. And I want daily email for that and they receive the emails and uh so they can go to the web app, get the exact location and uh and go work on that. So Those are two different examples of things we can we can do for biodiversity research and I will show you more uh with different uh examples I have worked in um different companies, different organizations and now

on my own, but there are a few things about those projects that is quite different than than um probably most uh most uh common development works uh and that has an impact about how we work so I will briefly cite them Um basically it's often small projects, which in a way is easy, but you're almost alone for that. I mean you work in a team, but the other people are scientists and you are the computer person and everything related to software development, to data, to databases, to to front-end development, to back-end development. It's just you or maybe a few people but uh basically you work on small things but all alone. Um

you have to work in close cooperation with uh with scientists and it's um it's it's interesting because uh You're working with smart people that don't really have um programmer mindsets, but at the same time everybody code in a way because those people need to code for their PhD to do data visualization analysis or everybody that saw There's code, everybody codes, but um they're not really developers and that that's interesting to try to to fit a niche and to be to be helpful uh alongside those people. Um something different is generally about the code at least. The environment is not uh competitive, people are not fighting having competitors to sell a product or something So they are really uh open to

work in the open, work in a collaborative way, have everything public in the default, put things on GitHub. That that's basically part of the culture also because science should be in theory at least repeatable. So if you do an experiment and you need software to do this experiment, other sh people should be able to do that. And to do that they need access to the code of course. So uh that that's a mindset that's more and more ingrained and um we have the luxury to be quite open and that's something we can use at our advantage, I will Talk about it later. The funding of those projects is a bit more problematic because basically funding of science is still very much targeted at publication People work on a project, they receive money, they work for two years on it, and then they write uh

they write a paper and there's no more money and oh it's fine. The paper is there, there is a result and no more money. Of your course if you are building websites and web applications and data pipelines and it's useful and it's used and things should stay online but hey the funding is finished because it's been three years and there's a paper was published. That doesn't work well. I don't have a magic solution for that, but we we have something to that's something we still have to do and I have a concrete example for that. It's uh It's an online database of Afrotropical MOS species. It's basically a website showing a very specific database. It's been developed like 15 years ago. There's no more money flowing in And then you still receive emails very often from

small group of researchers all around the world saying that the work is super important and they cannot do without your tool and that it's super important to keep that alive and that it's incredible and blah blah blah blah blah blah. And uh yeah but there's no money and it's using a very old version of Ruby on Rails and so How can we how can we keep that online? And that's a moral dilemma I've seen a lot. And a lot of things are working just because people want it to stay alive and it's not um very professional but that's sometimes the best thing we can do um if we if we really want to help in the in the context the science is is funded uh right now Um why do I think Django is awesome for those projects?

Uh I like to think of myself more of a generalist and someone who like use the best tool for the job, but the fact is that I always come back to Django and to Python because Basically, I think it's often the best tool for the job. Um I'll list a few reasons uh now. The first one is probably obvious to many of you, the quality of the documentation and the community It's very complete, there's a great attention to detail. I still think the official tutorial is in many cases the best entry point to the framework. I remember when I start using Django just doing the tutorial and at the end of it it was wow I already feel confident enough to start a real project with that and I think that's something we take for granted but um

It's not like that for every tool and every community and every framework, uh unfortunately. Um that's really something important and in my own work I when I'm writing documentation I always try to thing to be as good as the the Django documentation. It's a bit bit like the unattainable target in the terms of documentation for me. That really helps. Uh the other one is quite obvious too. I think it's the ability to very quickly prototype and especially since uh when you're working in my position you also have to collect requirements from users and from customers. And in my experience, having a quick prototype and having people playing on it allows really to get great feedback about what

should work and what should be changed and those kind of things. Uh you can do long meetings, you can do visual mock-ups of the screen, but if you can quickly build a prototype and send it to people, you get much greater feedback. And uh I think that's very important and something uh Django allows me to do Uh and this is especially true if you use the the whole the whole framework, use the template, the form, the admin application. Uh you have everything there. If you are starting to use Django only as a back end, for example, it's more difficult to do a quick and dirty prototype. So that's something I do all the time. And after the the challenge is to reuse or rewrite this prototype as a proper solid product but um that's a very useful tool too.

And at the same time I say I want a framework that allows me to develop very quickly. But at the same time I don't want a framework that changes every night. And that's also something I saw in other communities and with other tools. You have a version one of the framework and you use it, you learn it, it works well, and then one day they announce version two and oh it changed everything. You have two Just rewrite everything. Um that might work for companies that have a product that in continuously maintained But for me with tons of small web apps sometimes very old that should stay alive for a long time that would just make my life crazy and so the the approach that Django has I mean I mean having a very good solid basis and

very good core and then it still evolves but it's optional uh optional stuff it's new features it doesn't reven the wheel every every week And when it changes things, you have great documentation about it, about how to manage the migration, you have long-term release. a c certain kind of slowness uh and um it's really really uh important and that really something that uh sold me uh I used to use uh Ruby on Rails um ten years ago and uh yeah I had a very different experience and can really see that the those things the The speed of release and the quality of the documentations are really the things that made me use Django all the time, and I I don't really regret

that Another reason is the admin site. I know that the documentation said don't use the admin site for your users, don't send people Uh to the admin site, of course that that's true. The admin site is not done to build uh I don't know an e-commerce website and so anonymous visitors to buy things on the admin, that would be that would be crazy for a variety of reasons. But for what I do the admin is often good good enough and uh and it really it really helps to develop things quickly and I have another example here. I built this catalogue of the butterfly of Belgium basically a few few years ago. You have all species of uh butterflies that appear in Belgium there you can

yeah it's an online database nothing nothing too fancy you can explore from a variety of um Of sources, you have pictures, you have details about species, their distribution, uh pictures about that. Well, you you see the thing. But before we build that How these uh people um w were managing their data? Well it's quite simple. There were maybe five data editors, everybody has an access database on this laptop Everybody was updating the data daily and on Sunday they just meet over coffee and people start copy-pasting data from one database to the other. And then after after a few years, someone say, okay, I stopped working for six months, I take my version of the database, I copy-paste everything in Word, and I publish a book.

And those people have crazy useful data for their niche, but that's all they have as tools and that's what what they did and that was their way of working. Uh of course when you can quickly build them a website like that and you do a small Django admin and with that they can uh they can all update the same database in real time, have everything published have some data validation. and automatically publish the data to to common um data exchange places like the Global Biology Information Facility or download plant information from Wikidata to make the webpage more rich or things like that. Uh that's super helpful for them and these those people are were blown away but what's what's possible actually and and for me it was only like an admin

pi file of four hundred lines of code so it's really really worth it and uh The admin was super useful for that, and yeah, that's basically a few screenshots from the admin, but you will learn nothing there. Uh another reason why Django is great is Python explicitness. As I said, I um Always come back to old projects after a long time. I cannot deal with magic uh with that. I'm not my brain is not big enough to remember everything and how it works So I really prefer uh ten lines of explicit simple boring code than to something too smart that that I cannot uh maintain basically and that's really something I pay attention to and this philosophy is quite quite present in the Python uh world I think.

That's um that's very good. The rest of the Zen of Python and the that way of of thinking is also useful, but I wanted to especially emphasize the explicitness. Another reason is the integration with PostGIS for geographic work and with GeoDjango. because most of the data I deal with is as a geographic uh component and I would argue that most important data for the challenges of the century of a geographic component. I mean if you're thinking about climate change, biodiversity, migration inequalities. You cannot do much with the data if you don't know where exactly it happens. So to me, uh having the ability to store location in a database and deal with that is a super useful and I use that all the time and we have GeoDjungo

which is great for that. And this is a small example of that with just a few lines of uh SQL code and Python glue. I'm able to do things like that, like having a few thousand points, aggregating them into hexagons at some zoom level, assigning a color. and sending back the the data directly to the browser in a binary format. Well I have everything at hand to do that uh quite quickly and that's uh that's super useful. Another similar reasons, the data science ecosystems. If you need to transform data, visualize data, analyze data, you have basically all you need in uh In Python libraries, so that's um that's great.

Um some tips and some lessons learned uh along the way. Um I have one I think the most important thing if you are working on your own, you will always have to take decisions about if you should take a new approach, a new tool, a new technique. And it's very tempting and many things are released every day and uh and basically there's a very fine line that you you have to You have to find your place on this line because if you just become too conservative because you don't have time and you you just become the the old man yelling at clothes and In in div software development, if you do that for 10 or 20 years, you don't use version control, uh, you don't write tests, and

I think it's not not acceptable. in the twenty-first century. But uh on the other side if you if you are too excited by everything that happens you you end up uh storing your very basic data in a blockchain and change a JavaScript framework every two weeks and basically you just cannot deliver or you can deliver something but you cannot maintain it and if you're working mostly alone Um it's your problem later, not the problem of the next guy. You are the next guy basically. So that's kinda like my obsession to Not become too hard and not get too excited about the new stuff and uh I think it's an important um think and I don't really know how to do that except try a lot of things, experiment, learn a few

experiences and at the same time try to always Ask you what is useful for me, not what do I want to do, what is exciting, what is nice, but what is really useful and what is the cost of using something new. Something very similar about the dependencies for the project. I like to think of the dependencies of like a backpack At the beginning of the journey it's very tempting to put a lot of things in your backpack because it's nice, I might need it, and and blah blah blah. But later on you have to carry that. And if the journey is long, it can become very very difficult So I think it's um it's important to be quite conservative about that. Only use dependencies that are popular if possible, because there's a higher chance they will be maintained. uh

don't use them for for fut futile stuff and things that are not really necessary like um like cosmetic stuff and Sometimes it's fine to just copy-paste a function from Stack Overflow instead of adding a huge package to use one function in it. Well, if you just need that function and you think it will be like that Probably more maintainable to just copy a single function to none to not to install a package and its dependencies and uh and all that So I think it's important to always consider the long term cost and not uh just add stuff that is that is cool or looks cool An example of that is should we use single page applications with like a Django backend on a front end written in React or something like that?

Well, um yeah, I don't know. It's it's very useful. Of course, Facebook didn't even react for for fun. It solved the problem for them, but uh Most of us are not working on the same kind of scale and the same kind of problems than than them and uh I've seen that too often. I think people just use React because everybody uses React And sometimes it doesn't make sense and it's difficult, you have to maintain webpap, uh Webpack, you have to maintain different web applications. make things work together. It's in my way it should be carefully considered and we have other alternatives now like HTMX. Uh never had the chance to use it but I'm sure I will love it So I would like or even if you are using Vue.

js, you can include it in a very lightweight way, just a simple tag and use it for a small part of your web page and continue using uh Django templates and forms for for all the rest. And I think for smaller projects uh that's an approach that that sometimes work quite well. Um another example NoSQL could be tempting to use NoSQL databases. But if you are already using Postgres, which I encourage people to do by default, uh there are already a lot of things there. You have already uh NoSQL like fields. Uh able to store JSON and query that for example, you can store GIS data, you can do full text search uh directly in Postgres. So If you can use it to the max, you can avoid adding other dependencies to your project

and I think it's great when you when you can. About the infrastructure, I also I think I understand the benefits of the cloud computing. But for the personal development standpoint, if you're thinking on the very long term, I think in a way it's more more interesting to and invest time in getting better at Unix Unix basics and uh That's that's some knowledge that can transfer easily from one vendor to another. And um yeah, I think it's sometimes more useful uh personally to spend one thousand hours getting good at Unix than getting good at a specific AWS product. Um I understand why the companies prefer to have an army of AWS

experts, but I don't think it's always good for for me and for my project. Some things I put mostly as a reminder for myself if you're working too much alone and if you're an introvert it can be dangerously comfortable to become isolated Um it's it's important to remember we are social animals, we need interactions. So do something about it. Speak at conference even if it makes you uncomfortable, especially if it makes you comfortable. Uh or find find other things that work for you. And what what I do personally uh since as I said the the developments I do are not We don't really have competitors and we are not afraid of competitors because we are working in a more collaborative way.

I take that to the my advantage and sometimes I call people I know from other work other other places and say hey can I just come here to your place today? I work from there. I pay the sandwich for lunch and we talk and we exchange a bit on what we work and since there's not much secret involved, it's generally accepted and it's super nice and I call that being a work squatter. I have to yeah it feels a bit weird but in in my case it works super well and people are really enthusiastic about that. And uh it's good for my mental health, it's good for fostering new ideas and and it's good for the business because you you have exchange and that gives new project and those kind of things. Another tip, uh

another way to take advantage of the fact that uh sometimes you work in a non-competitive environment is that people are not um are not afraid of uh having their code stolen by competitors so you can't put everything online from day one very often and uh That can be that can be quite good. I have this tendency to put everything on GitHub from day one in a project when I can. It has a few benefits. I can host there the documentation. I can log in in issues, not only the result but the rationale behind some design decisions, which is very helpful three years later when you come back to the project. We can use GitHub Actions

as a CI environment, those kind of things. So in a way it it it helps if there is no real um development infrastructure in the company you're working for. You can do that in public and use free tools like GitHub. And another benefit of putting everything in the public is that basically it forced me to do things a bit better because yeah my code is readable from day one so that pushed me a bit more to write better documentation and better tests and and better code and I I use that um thing to my advantage to to force me to do things a bit uh a bit better. Another tip is to write boring explicit tower code. I talk about that generally I don't

really have time or have a big enough brain for magic. So I like explicit stuff and uh for that reasons for example I don't use much the jungle rest framework or the class-based view because I I prefer something a bit more obvious and to have the code in front of my eyes where I see everything what's happening Maybe just me, but I think when you're switching between a lot of tools and a lot of projects all the time, it's it's super helpful to work like that. Um what to test when there is not enough time to test. And you still think tests are important. I think I will skip that because I'm running out of time, and especially we had a fantastic talk yesterday about mostly this topic.

uh how to spend less time writing tests and still do good stuff. So I I prefer to send you to this wonderful talk than to Uh spend too much time there. And finally some things that I think could be possibly improved uh in the future. Uh as I said I use a lot of Geo Django, it's wonderful, but you can feel that it's a bit a niche tool and it receives less development effort than other parts of the framework. And also it's not easy because um for some uh geographic operations it becomes really complex and so using the ORM is not always the best way and you are tempted to write row SQL more than for other kind of developments

Yeah, I use the admin size a lot and having it more configurable is always is always good. There are things frequently for that, but uh could have more. And yeah, a last one Sometimes I'd like to have an easier way to share a complete Django project and allow less technical people to deploy it, to configure it. uh like having dynamic settings, con a configuration with R to set those settings, a deep an easy deployment thing. That's something you can you can build yourself with a combination of Docker and package to store settings in the database and things like that. But Yeah, I would like to have something uh easier and more integrated to do that. But that there's not a lot. I'm really, really happy with Django actually. Um thank you.