Lionel Jouvet - AI That Plays Nice: Building a Privacy-First Grant Review Tool

Speaker 1: I think we should give uh Lionel a big round of applause. This will be the last full talk of the day before uh we will have a lot of uh lightning talks after. So super exciting uh remainder of the Today. Yeah, let's give Leona Hill a big round of applause. Okay.

Speaker 2: Then thank you very much for having me and having us. I'm going to speak to you some today about how we can play nice with um LLMs. When we are working into a Django environment, I'm going to uh first um remember you two years ago I did kind of uh It is about what we were working on, uh Django Lama. You know that if you want to use an LLM locally, OLAMA has been for a long time a good way to run your local LLMs. and then now I'm going to show you what it is. And first I want to give you some environment about what we are doing. Then first I have to speak about our team. Maeva was the first one to set up our team and we are

Speaker 2: all working with Django in different projects. Uh first Louisa, she's a fire woman. When there is a something that goes wrong, she's the first one uh in line to uh get the fur fire down. uh you know that guy probably. The one that is shouting at the LLM there uh is Julien and Julien is a full stack that have been producing a Django uh web app that is used to um by our university and many other universities in Denmark. to match scientists with funding. And we have Alex with another full stack here that I've been developing on a Django framework. uh using React as well, um uh a web app for um um the mission uh ocean

Speaker 2: for the EU and uh doulé that as well have been developing with Django another web app that is used by many uh young universities in uh EU Django has been the uh bread and butter for all of us. And we are a bit special for a group of developers. We are working in the administration of a university And uh then uh as you can hear from my accent, uh I come from the University of Southern Denmark, Southern, uh that's where my accent comes from. Um yeah. The uh group in which we work is um uh people that are research supporters, meaning When you are a scientist you spend an unholy

Speaker 2: time uh trying to get money to do more research um to be able to help the scientist We have teams that are research supporters that are help helping them to be able to fill up the grounds to be able to get money. And us behind we are there to be able to help our research supporters too, and our scientists, to make tools that are dedicated to that. And it is what I'm going to show you today with this problematic. Then the problematic for um recherche supporter, as you can see here, it's uh we have to deal with a lot of different types of fundings, um national and european. In here to give you an idea it is all the deadlines. through the year uh of uh European funding meaning that uh scientists come to a research supporter

Speaker 2: 20 at a time uh three weeks before the dead end uh the not dead end deadline it can become a dead end and um asking them for support and help to read through their application to be able to improve it and often it is last minute. Then our colleagues um I don't know how to say that exactly but They have to deal with so many things. They have to be able to review the whole d read the whole document. They have to look at the money behind. They have to Um it is so many things to do. I wish I had a drawing to be able to show uh yes, more or less is that. It's they have to do a lot uh in a very narrow time. And the application they get, this can be from someone that just started kind of a draft for an application to someone that have been already spending a lot of time on it

Speaker 2: Meaning that for the research reporter it is very important to help the guys that already uh have something that is concrete, uh finished and for the one that just started the draft to be able to give uh quick feedback so they can move on. And it is what I'm targeting with that. Yes. One other thing that is uh probably of interest to you into the web apps I've mentioned earlier, my colleagues have been developing the uh or um deploying these web apps on uh Asia uh then we um in a lot of for a lot of you you are using platforms that are um available At SDU we have um the eScience Center that have this supercomputer with servers that we can use.

Speaker 2: Um and we have um uh uh tiles we can access that uh are running uh uh uh ubuntu in the back and where we can deploy our own stuff which is really interesting for us because we gain sovereignty We don't depend of Microsoft, we don't depend of uh Amazon, we don't depend of anyone, then it is all self-contained, kind of a sandbox. And that's um start to be interesting for us when a scientist is applying for ground. A lot of these um applications contain some um intellectual property, some kind of advanced work. And trying not to lick it as uh you would um if you um use um LLMs that no are not based uh in the EU, that are not GDPR compliant.

Speaker 2: And um I don't know if you heard about it but um even people from um how what's their name uh the guys that give the uh surveillance uh tools to a lot of nations are complaining about the way the uh this L LM provider um are dealing with their data. Then even if them are um Uh complaining about it, imagine that what it is for us. We all remember using uh Gmail at the beginning when they started to release um free access to these um emails We never thought that the data that we were putting in it could be used in such an extensive way. LLM is a really good way to capture even more of that. Then on our own uh small dimension at uh SDU we wanted to develop a set of tools that where

Speaker 2: the use of LLM is less um um dangerous uh when it comes to uh intellectual property And what we vouch to maintain , it is an automatisation. We don't want to take away the job of our colleagues. People that have been working in research support have développed an expertise over many years. They know exactly the right feeling to add to a prop uh to an application. And our goal in here it is to give them the help for things that are not developed enough yet, so they can just put their brain power to the things that really will make a difference. I'm telling you a lot of words but I think it's I will be giving you an overview of what this app is. Uh to be able to um

Speaker 2: um explain what we do. I don't have my classes. There. Then as you can see in here we are working so you will see what we have the dashboard from this e-science center. You can see that we have access to compute, we have access to storage and we now for the past 10 days, 15 days, we have access to uh inference. As you can see, we can run different types of modules and in here you can see Postgres, which is the database I'm using in the background, and Django, which is running the website Uh one other thing I like very much with uh Django

Speaker 2: it's the fact that you have this static where you can put all your CSS um If you have JavaScript that is uh code that is um in different pages, you can all store that on the side. Um one place Everything is clean and it is really exciting for that. The color scheme, then as you can see, I'm not a graphic designer. Yeah, but it runs. What is interesting in Twitch then um this tool is aimed for uh a research supporter but other scientists as well to be able to analyze a proposal and give some feedback on it. To be able to do that, then as the obviously to be able to read through a proposal we use LLMs

Speaker 2: This plateforme can run either locally on a laptop in Docker, using Olama and running the LLM locally, then totally outside of any connection with Internet. with strong limitations that I'm going to speak about later. Or use now either ten days ago we were using the API of Mistral, GDPR compliant um servers Uh and now we do the inference uh dire directly into the UCloud uh into our own secure environment. Um And um to be able to do an analysis you need to be able to define what the ground type will be. Then if you have a EU application or a DFF, what is uh the uh another um

Speaker 2: Why now the LM for uh come in different places into this tool? This is one part that will do the analysis I'm going to show you how it works. You click on start analysis. You are going to choose one type of grant, in that case we a European programme. You can select some proposal. I'm going to select it in here. You give him a name, Toto Vous pouvez choisir si vous voulez le retourner localement ou online. Enline is à l'autre de votre computer. Then you the um you are going to parse the document.

Speaker 2: It can be um um a PDF I hate PDF. It can be a word document, it can be a markdown, and so on. And then automatically the tool is going to cut this document into the different sections that are interesting Why do we want to cut this document? It is um as you know, uh LLM have um a strong limitation and it's often it's a context window. You can feed them with a lot of information but very quickly they are going to forget stuff at the beginning, in the middle, at the end, to be able to limit and mitigate this problem of loss of memory We break the document in small in different parts that allow us to use different agents that we

Speaker 2: develop ourselves. where you can give a different persona a different question. For example, this is what you see me doing just now. The NPAC section I'm going to look at my NPAC specialist with the question and finally the last part the implementation. I'm going to take my implementation specialist. And all the process is then set up in that way. Now what is interesting with Django est que this or what is this difficult more like with Django. LLM can be very slow to do the treatment And if you take a normal naive way as I did at the beginning, I had a lot of problems of breakup because there was the disconnection

Speaker 2: with my API And what is important too, it's uh then you can see the analysis is running and you can start from standing in front of your computer, not doing anything, which is really annoying. But when you come back to your dashboard, you are looking at previous applications. What you can see on the top of the document is that you can still access the analysis um and see that it is still running. Why I'm telling you all that it is I'm going now to explain how that was set up in Django to be able to do it. If I can get my presentation back. And um at the beginning I was very uh naive in the way I implemented it.

Speaker 2: Um I'm uh using um engine X and Gunicorn uh into managing the different people. Many people can connect to the app. Then when you go into the app, then you are going first to meet your engine X that is going to give you load And then after you are going to ask for um um uh an analysis to be an analysis and an API call to the uh LLM to be done And that will be sent to your Unicorn and to the chef at the back which is uh Python that is dealing with it.

Speaker 2: What is interesting it is at the beginning I had problem and I was trying to find a week on the same um uh zytophilic uh line for the people that do like beer. It is that it's a bit like when you go uh into a pub and uh then you have been giving your load and you ask the chef, uh now have we like my LLM uh to analyze it. And uh you just leave the your beer on the on the stand and you say uh we need for one minute I'm just going to the jet I'm coming back. The problem with the API can take much longer And the first thing that will happen, it's uh your engine X that is going to tell you that's more than one minute, I'm pissed off. And uh the idea is yeah, uh there's something wrong with going

Speaker 2: with it. And then after you have gunicons that come back And uh the process was already uh problématique. And then when you come back, you want to obtain your uh analysis and what you get is plenty d'erreurs. Lately your load, uh your work has been corrupted your database is not happy as are and um a way to deal with it to be able to uh work in between these different two different um life span dans le système a forme de pouvoir utiliser le database to dealer with this workflow And um it had been the ground why I did that I wanted to have a simple system. I wanted to use Django

Speaker 2: as a um as only Django as possible, try not to add other layers, other libraries to put in. And this is uh Nginx and Unicorn format using the database to be able to manage the work at the back was uh very uh practical. The other thing that for I think for a lot of you when you want to develop um Django environment you are ready to um or you plan to have a huge amount of users In the infrastructure we have at SDU at the university, I'm likely to have twenty -fifty people at the same time that work on it, never more than that. And with this frame of work, I am kind of comfortable with this uh technical choice. Then um when I go and I load an

Speaker 2: um uh I load the PDF for example, then I will wait until we get the return. Um the parsing can be done either uh locally or We use sometimes the Mistral API to be able to have a much higher quality uh optical character recognition recognition, uh OCR. As you know, passing a PDF is really challenging. When you do it locally on your CPU, usually it's almost impossible to get the structure of it out. the titles for example. You can have tables kind of okay autofit. Usually the Mistral API for the OCR is much more um stable

Speaker 2: When it comes to the call to the LLM, then we can use uh our um Postgres uh queue that allows us to Get the job done and we wait until we get in return. We don't keep our worker busy. That allows a much more stable flow And finally, um when we have our um uh um uh process running too using this server send uh event we can actually do other things while that is running in the background you um that allows the user to have um more interaction with the tool, uh we start to be able to be dependent uh on the uh

Speaker 2: result to come back. Um In the development of this web app, um it had been often when you develop um such a system, you have In French we say cahier descharges, everything is written down what the type of uh workflow you need to use, what uh the type of architecture. The development of this web app has been um organic through time then it had been a long feedback uh in between the uh scientists the research supporter and me and uh to see what needed to be added to it, uh how we can improve and what is interesting as well. It is the type of evolution that were in the background I told you at the beginning that two years ago when we started to work on this project, the only API

Speaker 2: that were available in four LLMs were ChatGPT, uh American system Uh Mistral came in a bit uh later and no data could leave this app because at the time I had no trust at all into OpenAI um um um user rights uh and into cloud or into any other American platform. The only path I had was uh OLAMA uh to be able to run it locally where I could control uh the different lm I was using. And I don't know if any of you use uh locally run LLMs. Any A few, okay. Um you use them to code as a helper. Helper, yes.

Speaker 2: And um then uh because one of the limits of these models is if you give them a full uh load of uh code, uh they forget ever uh everything but they don't have a memory for it. And um what we saw in the past two years in evolution for this type of local LLM, we saw um local LM that had much better capacities, um context windows that started to be of the order of Thirty thousand, fifty thousand um uh tokens uh reliably And um one of the ones that started to got my attention was the one developed by um IBM, the granite. I don't know if any of you used the advantage. No.

Speaker 2: Good. Uh then the advantage of this uh platform it is the LEDM that they developed actually was for a production then it was much more stable than other LLMs that we can find, not as powerful but working pretty well. But now coming back to uh why now we use uh APIs that are much more powerful, first was Mistral Mistral had a good model a year and a half ago but never moved ahead and we have been stuck with the same model now for a long time and the capacity are very reduced. Two weeks ago the um e-science center surprised us by g giving us free inference uh for employees. Then when you are an employee at SDU you have uh you are given enough inference

Speaker 2: um quota to be able to vibe code the whole year to give an order of magnitude. But also to be able to run models such as this one to be able to ingest a lot and to be able to uh the model that is behind is GLM uh 5. 3 that you may have heard about that was released a month ago. And that gives us a very good large context window, one million tokens, and in addition a much higher quality output. Um then yeah I'm speaking a lot about what it meant to us. Um and the The constraint we have at uh at the university, it is to protect as much as we can the um

Speaker 2: intellectual property. But then if these levels are really high from a health project, for example, where there is personal data involved, we would go to health Then uh we'll go to a local um then we are sure about uh the quality uh the um of um security Um the problem is yeah you need a BPR uh hardware. Um it is very slow and a very limited context window Otherwise we use um the UCloud LLMs and GLM five point three at the moment but is going to um be uh updated with time, uh or Mistral uh API if they improve their model Um where

Speaker 2: we have up to s up to I like your system. Uh then the speed are brilliant. We have almost six hundred, seven hundred token uh tokens per second, uh going quickly, uh good massive uh context window. In the case of Mistral actually the problem is it costs something for us to run. And what is interesting for you, what is the most expensive is not the inference, it is the optical character recognition. three compared or in the use of it. Um then we come back to the thing about the OSC Um then the second part of the project that is um difficult it is that these documents are uh as soon you give um a template to a scientist

Speaker 2: is going to mix it uh mess it up. Then you have a nicely organized well it's a big title, the you put your text here, the b um You it comes back, it is broken. And um usually uh a lot of people in administration administration like to deal with PDF Each time you try to parse PDF, you finish with a huge amount of imperfection Meaning that you have to be very clever when you do uh your uh section uh sectioning of your different um uh parts. Uh regex is really uh even a very clever regex structure will miss uh then we came up with um a more hybrid uh way to do it

Speaker 2: And um the it had been the um akilli uh hill uh of this whole structure which is kind of the most um annoying part because it should be the most benign way or the more obvious part uh of a project cutting a text in sections and uh yeah there's something of uh magic in tweet Uh you take your your text , you do your chunking and uh so there is a different reason for that. Um when you Uh when you are a scientist, you never believe one person. You uh you need you need to peer review uh your your work. Then if you submit your proposal to one person uh it's one person.

Speaker 2: When five percent to tell you the same thing, then you start to believe them. And it is a bit what we did into this tool as well. It's We configured a lot of different agents, personalities that will be specialized for a particular area of the feedback. Then for uh once we have our chunking that is done, then we are able to uh mitigate this problem of uh context window, the size of the part that will be treated. as well as uh getting some quality. But because of the fragility of the system , we still always need uh control of a human behind to make sure that it has been cut properly at the right place. Um what it means then without an image.

Speaker 2: Um then it is that in this tool uh if you found that there is a problem into um Uh the slicing you can always go and re-slice into um like manually to be able to do it. Um and I haven't found a way around yet. Very frustrating Um and LLM again, then how do we integrate the LLM into the construction of this tool? You don't want to spend hours to dev to define a new grand type. Then I developed this grand type architect. And ici vous pouvez gagner un nom. Si vous cliquez go , vous faites une mistral search And it will try to retrieve as much information about the criteria for the ground and um oh the lingua of grants

Speaker 2: uh in the EU must, should, could, uh doesn't mean the same thing for everyone. And it is really important to know all these context to be able to build personas that are meaningful. You can also um add your own um um um criteria and it will be done automatically but there is a lot of automatisation that have been done using LLM to be able to build the persona for example and um When you want to build your persona, um that's not that one. Um uh The agent that is going to review the proposal they need to have kind of a character each one of them. There is a base um prompt that you need because you need to be able to input all the context about that grant in particular

Speaker 2: You need to be able to define what are the um uh landscape of what this ground is about. But then after you want it to be more um criticize for if it is for mathematics, for biology or for sociology, history. And then in here we develop another tool. A lot of our colleagues are not at all prompt engineers. They just want the tool to work. I want that the text will be coded for each one of the feedback and so on. And then we build as well an LM at the background and other region that helped to build this prompts Then we see that um within the framework of Django we were able to integrate this LLM to help uh assist um in to um developing this tool um

Speaker 2: and we need to be careful because Yes, you know, when you ask L LM to do something, you rarely get uh the right thing out. Then there's a very strong uh push behind to be able to make that the output of this LLM is correct when it comes to these architects either for making a new ground type or persona um to get sure that within the prompt that we do we have a very strong um um constraint on the format of the output and then after a lot of um safety and validation to make it that it works. Can it go wrong Yes. Um and uh but uh not only uh LLM do mistakes, um

Speaker 2: when it comes to text Some other do. That's not an LLM. It could be. Lazy large man or something? Then uh nothing bad can happen. Only good happen. Then the horror story I had it is that one I need to share. You know LM are always complacent. They always to tell you it's good, it's good. Then I ask the LM to be a bit more rough That's what it gets out. Imagine you are a scientist, you spend uh a week uh working on this document and uh I don't know if you can read from here, but uh buzzboards. Um I don't know. Uh it's read like a wish list. Uh what is there in note? Um

Speaker 2: unadvise, burn this draft Otherwise enjoy your rejection later. And that in the category uh parting shot. It makes me think of the scene in Terminator. Sarah Gondor. It's very important that a um user before giving it back to the scientist to read through it, it can be a bit rough sometimes. To finish, as I told you, the text uh the text tags that have been used into this um uh project it is uh Django , as much Django as as we can. Postgres that is saving uh um

Speaker 2: that makes the day when it comes to interact with an um unreliable connection to um an LM API and for all the work that we do locally, a Docker, so we can uh move from uh one platform to another without too much trouble. Knowing that Only OLAMA is needed. Once you have the app running on Docker, the only thing, the only component you need outside is OLAMA that will be running your local LLM.

Speaker 1: Yeah, right on time. Um thank you. And also sometimes Time for questions, like uh let's let's there is a question

Speaker 3: Uh this one is from me. Uh thanks for your talk. Um how specific is uh the tool to grant reviews like uh what they call or if it be applicable to other use cases? Or is it because it sounded a lot of it was just like stuff you have to do in a lot of other cases. as well.

Speaker 2: Tell me if I ask um pro uh if I reply properly. Then uh um there's a lot of work to be done ahead to be able to get a proposal. Uh is that uh what what you could no, okay then good I did well to rephrase.

Speaker 3: No, I meant more for like uh other use cases of document ingestion. Like where it's uh not at all grant related like just like like the the whole thing of parsing parsing the different documents and and and and and and putting it in section and and putting context and just having a system. That works with the documents.

Speaker 2: Yes, okay. Um then uh it is true that some of the uh components of this uh web app uh can be used otherwise. We are thinking of developing uh local tools to be able to do this PDF parsing and to be able formatting. The goal of this app is to target the research support service, but a lot of this component can be reused uh outside. It can be used as well, not only for research funding, but it can be used by startups or if you want to build a different ground. But yeah, passing document is interesting.

Speaker 1: Will it be uh open source? So uh everyone can

Speaker 2: I hope so

Speaker 1: goes with the reviewing.

Speaker 2: We are speaking for a long time with management about to be to have that disp uh available. We are extremely relevant on uh open source then uh on the sp uh Postgres on Docker on um uh Django Python um um uh ZAI um or Mistral besides they're not really open source but uh yeah then um we speak a lot about it uh we may find ways where we could do that but it is At the moment the universities see it as a competitive advantage and uh then it is why it is um yeah

Speaker 4: One of the interesting uh and challenging things when working with LLMs is the uh guardrails and the classic classifiers.

Speaker 2: Yes.

Speaker 4: Uh did you manage to go through? I know the answer would be uh quite verbose, but just just if you can some sum up. Uh how did you manage to work with classifiers and uh guard

Speaker 2: Then uh to answer how to manage the um the output of the LLM and the setting up guide rails and the type of um uh AI system we work with than LLM but other type of classifier. In the guide race uh we are lucky in a way that we um it's unlikely that a scientist put some um um weird prompt in their own document they don't gain anything from it. The people that can access the web app need to be the um a member of SDU. Then we are protected in a way against misuse of it Um there is a feedback as well from the user by the research reporter before handing it back out. into the tests that have been done by the

Speaker 2: it is in testing phase at the moment and uh we haven't seen any um outside of the um edge case i showed you there's no much uh problem now for other types of um algorithms Uh you probably uh are thinking of uh Jeff that um that is a kind of um did a buzz uh this past two weeks. where you can uh use some of these LLMs that are uh twisted to be able to you be used as a uh classifier Uh we use other type of classification uh algorithm like BERT, uh if you heard about it. Then the then In our team we do a lot of NLP, meaning that uh we try to use the most simple solution for a task.

Speaker 2: Then when it comes to um an algorithm that allows to do a classification of uh of text There is often lighter algorithms that can run mainly on CPU, for example. Access to GPU can be challenging. But we are playing with it too a lot.

Speaker 1: All right. uh that came in here. Alright. But we did yeah um so we'll have um we'll have some uh another short break for coffee uh at some point. But we're going to switch over to lightning talks immediately. Uh let's give Lionel a big applause.

Speaker 2: Thank you so much.