Mohammad Abu Khashabeh - Pre-commit hooks that catch Django problems early

Speaker 1: As uh your body need to to sleep and the mind is negotiating. So I'll be uh quick as much as I can. Uh I would like to talk about the pre-comets. This is what the topic would discuss in the first place. We're gonna walk through everything related to the pre-commits, actions, hooks, best practices and common common bitfalls, ending up with a quick demo

Speaker 1: Okay. To be begin with, this is uh a break brief, uh a quick brief about myself. I'm engineering lead and uh Django contributor Um I spent the last eight years working with uh Django and Python ecosystem I'm jumping into front end whenever needed. Working with DevOps, which is uh this is the part I would like to contribute. uh more uh I'm really enthusiastic about the DevOps and when the Dev and Ops is being met. Okay Many of us have introduced issues while working on the div

Speaker 1: uh for one of the features on Django. maybe with missing migrations, leftover PDBs or breakpoints for the newest version of Python. Also fail test. and much more things that actually failed on the local environment but somehow it got deployed for the pre-prod environment and caused some issues. Okay, uh this is what we're gonna talk. Uh the pre-comets. I think the name is uh self-explanatory, right? Because it turns and runs before the comet exists.

Speaker 1: There is always when working with Django uh there would be a migration file generated Let's take this example in the first place. Migration file being generated and overlooked eventually when deploying the feature. This is one of the examples. and go on for all of things we have already mentioned. This is how we can actually get rid of and use pre-commits as our guardrails uh to move forward with the with the safe deployment I would say. It comes up with variety of hooks that uh actually checking the migrations

Speaker 1: uh cross-check the debug statement to make sure there is no breakpoints that cause uh the whole process uh to freeze. Uh uh we have Python uh check as well that uh you know um make sure uh the readiness and um uh you know uh the whole process is working as it should Uh yeah and one of the things that I think the other talks have addressed uh which is um the security uh um uh the security keys or the security variables within the project Uh uh this is what I wanna talk to later

Speaker 1: so you you may hear it twice The local pre-commets protect who installed them, but the CI protects the whole repo. This is what we gonna take for later Uh where things to point actually there is two main files. The first one is the pre-commits file, which is uh uh uh all the pre-commits about. It has all the configurations which called dot prefixed by dot pre-commit YAML file by the way which is in uh configurable So uh it's a YAML file and the second thing is um uh

Speaker 1: pre-commits within the CI which what we will explain uh later a bit Where actually the pre-commits are running. Okay, uh there is something need to bear in mind uh that the pre-commits not seeing all of our disks. It's only look at the files that would be staged Because you know there is a bit confusing when working with a pre-commits that uh uh the pre-commits should come across all the files which is uh genuinely not right so it's have the visibility on

Speaker 1: the files that have been staged in the first place uh yeah Uh this is one of the uh common bitfolds working with pre-commits which is you know the Django migrations, files, uh Python files being generated by Django itself. uh need to make sure the pre-commits, whatever the type is, not to go through the migration because this is system uh systemically systemically generated by Django so we need to exclude which um pre-commits gave us uh the ability to do that but there is uh issue that um

Speaker 1: uh many Django developers came across uh exclude that uh uh at the top of the file but there is something need to take care of not to start you know the exclude with leading uh slash because this means the migration only exists in the root project so by doing so which what Pointed as does uh needs to be uh treated as uh like pointed uh to make sure that pre-comets excluding all the migrations folder whenever it uh exists okay uh I think the order of the pre-commits

Speaker 1: is uh actually the product it's not the test so uh pre -com it's actually uh would say uh do the uh first cleanliness across the project then it goes to the format um you know it it it's sequential so it goes through all uh the hooks being installed but this is the right direction that any uh pre -commits should uh come with uh ending uh up with any migrations that this is uh actually recommended to keep the system checks uh for example the Django migration at the end

Speaker 1: Okay, not uh every hook is uh the same or equal. It makes all sense uh actually pre-commits offers variety of hooks that serves our use cases uh starting with formatting uh checks uh um Making sure there is no uh security keys is being compromised or hard-coded, uh uh check private keys and uh much more but try to recap uh the most uh ones uh in this slide okay This is one of the things that I think uh relatively connected to the other sessions, which

Speaker 1: this is I think one of the things that we should focus on, given the AI uh era we uh we we're uh actually uh living in. making sure there there is no security, there is no public keys, there is no PIM files are being uh committed for the repo which is this is how the pre-commits gonna help uh to overcome Okay, believe it or not, uh I quoted this from GitHub uh statistics. There was uh 39 million of security key being breached or compromised in the twenty

Speaker 1: twenty four uh twenty-four. So this is one of the things that uh let us uh mo uh f uh focus more maybe on the security and uh um uh I would say the guardrails for the um uh securities and checks that comes before even the commit exists Okay, this is also one of the um common bitfalls um try to uh addressing There is uh without which is one of the command on the pre-commits uh file uh that says without pass files to false. Actually

Speaker 1: without doing so this is will cause an issue or actually we wouldn't get the best of pre-commits because what actually uh does here is um what actually the pre-commits will come up with checking only the file that has been changed not all the uh working tree so this is one of the things that I think we should uh um yeah take into consideration while working with pre-comets Also, there is uh one thing that's obviously uh being missed in all the projects that I used to work on, which is pointing that uh

Speaker 1: uh the system checks is being triggered whenever the application has been ready. Uh this is the uh you know the boiler plate for Django if we didn't uh do this most likely that the pre-comets or the chicks will pass uh silently without any issues Okay, how bad commits still passes? I think this is one of the uh Interesting things. The first thing that the pre-comm pre-commits is not being installed at all. The second one is There is a gate to overlook the pre-commits, which is no verify.

Speaker 1: I quoted this uh Jeff. I think this is the official meme uh for Windows antivirus checks. Okay, so the other thing is the debug statements This is what we uh were discussing. Anyone who's working on the Django project uh could easily uh just overlook all of these comets. uh with no verify. That's the point the uh that we said uh in the beginning of the um session that the CI protects the repo actually and pre-commets protects uh who installed them. If you can

Speaker 1: check if anyone who uh overlook uh or passes the pre-commits locally The CI in GitHub Actions, which is the file pointed out, uh will take care of the rest. If you missed anything, all the pre-commits or the checks will be uh caught here in the uh GitHub Actions So that's why uh it's the CI is complementary to to the pre-commits uh moving forward Um we are all uh I think this is uh related because the repo is not the server. Uh all of us have heard that it worked

Speaker 1: to uh uh It get to work on my machine. That's why the repo is not actually the machine. That's why we need to maintain the pre-commits across the uh repo itself and the GitHub actions or the CICD in general Okay, this is the third time actually. There is some recommendation not to use pre-commits. Pointed over here actually when there is a database action or database request that needs to be done So it's actually recommended not to be uh

Speaker 1: involved within the pre-commits, either on CI, which is what obviously will cause the CI or the pre-commits to work successfully locally and failed on the cloud. There is also many things. When you have many hooks to run , I think this justifies uh how people could uh just simply overlook or trying to pass the pre-commits because you know when we working on the big chunks of changes and trying to commit them uh you will end up with many errors and validations that may come may come up um uh along the way uh

Speaker 1: either failed test either you know formatting issues and formatting conflict uh and so on so forth. So um this means we um not to recommend to pass the pre-commits bus but also uh reducing the chunks or the size of changes that are being done. Okay, here is a quick demo. Okay, so this is the pre-commits uh file we pointed out uh actually we uh s uh we specify we uh where

Speaker 1: or when exactly we would like to run uh here in the pre-commits uh this is the execute that like i said we could address on any files we would like to Okay, zooming in. Thank you. Okay, here is the list of hooks that we could um use. Um I tried to uh you know uh gather uh all that we could do. Um You know there is variety of uh of hooks that we can uh use like trailing white spaces, um check Jason, YAML, etc. This uh I uh

Speaker 1: I I truly believe that uh the God process comes with good uh governance because when the pre-comits is being um you know unified locally on the CI this means all the developers they are working on the same repo will same the will follow the same structure and you know the same rules and the same tests uh Okay, um there is one of the coolest um uh hooks is the Pi upgrade. If you are working on let's say Python 3. uh or Python 2 for example, uh the pre-comets will automatically replace or change

Speaker 1: um the code that have been written uh into the earliest version of Python We have iSort, we have uh Flick8. This is you know uh for formatting um uh stuff in the first place. Uh one question, who knows uh the default uh hook for the pre-commits. The default hook for the pre-commits The default is the formatter. This is what pre-commits come up with from the early beginning. This was you know add-ons along the way to adapt the needs across the projects.

Speaker 1: One of the greatest things that I would like to highlight each time actually the security again, which is bandit, one of the uh commonly used uh hooks or libraries that come across the files and identify any um I would say hard-coded uh securities, uh any uh breached or compromised uh variables or known uh things for the projects. uh so it would be caught in uh the pre -commits uh at that time. Uh and here uh for sure the Django system check. This is make sure the Django and its readiness to move forward with the deployment because

Speaker 1: if uh again any migration file any you know reference before assignments and uh dummy um uh errors that happening on the dev environment will be caught sooner before the commits are being exists okay Here is um uh quickest tip. Um This is how to get ready with the pre-commits. First thing you need to make sure you have installed the pre-commits. Because the commands should be exist locally. This is already done from my end, if you can

Speaker 1: see. I think I need to zoom again. Okay, um after uh afterwards we need to make sure that the pre comets are actually being installed. Not the pre-commit command, but all the hooks within. It comes with variety of uh you know checks, uh versioning and rules. So whenever pre-commit is being installed, if the precommits dot precomits, which is the origin file for the pre-comits , will be created, if not exist. So if you can see here dot get hook

Speaker 1: if you can quick look at I think there is. Oh, T. Okay. This is actually uh what uh pre-commits will do in the background to make sure Git is being uh forced to look at the pre-commits before jumping into any action, which is the get get push. Okay, uh there is uh a quick um Think we could do is

Speaker 1: running pre-commits. This is uh assuming this is the first time we uh use the pre-commits locally So we are running pre-commits on against all files. This will come up with uh changes that nobody gonna review on GitHub. If you can see, this is my uh changes that uh came up with. If you can see uh there is uh detected uh private key which is hard coded in the code that's why this is was for a purpose Also there is a debug statement. Debug statement which is pointing to

Speaker 1: uh you know breakpoints for the earliest version of Python or PDB for uh whom used to uh to uh to do uh earlier. Okay, there is uh also you know uh This is actually how GitHub or the CICD is are the second layer of security for the commits that the developers may come up with. just to make sure uh actually um just pointing that I I made a quick repo just for the sake of uh you know

Speaker 1: the uh the session that has all the files to run the pre-commits. Here is a quick actions how uh uh the pre-commits actually uh running and how github can refuse or allow these commits to be passed If I rerun all the jobs, you can see that pre-commits if I used to do non-verify locally, uh GitHub w will never let the C D uh to process with the rest of the uh deployment

Speaker 1: Yeah, this is the repo uh I was pointing out. Huh? Just thank you. I think this is uh dedicated if anyone uh make up uh may uh have any questions so we can address

Speaker 2: Wonderful. How many here run pre

Speaker 1: I think the most

Speaker 2: hey yeah, uh questions. I was wondering uh when you do run it in uh in CI I know that pre-commit has a has a local cache so it's uh fast. Uh locally it doesn't have to install everything. again and again. It was also fast in the example here. Does it also cache uh across

Speaker 1: Uh actually uh it depends on the GitHub runners or the processes that you are run in the first place. Uh I think there is also option to cache uh pre-comets on the CI uh just for the sake of speed, but It's limited to the I would say um uh installing Python and the other dependencies. Uh no not more.

Speaker 2: You you you mentioned that uh pre-commit runs on the files that you have changed. That's the default behavior. Um and and then I was wondering if if you have uh uh check migrations uh because you've changed your models uh in in some way. But actually you forgot to uh create the migration. Is is that a thing it can uh can detect so you can have a command also that kind of runs general

Speaker 1: Actually there is two things that will code this uh overlocking. The first one is the Python check that will point out that there is a migration file not being created. I think will show up in the terminal. The second thing is using the hook uh make migrations that which is dedicated in the first place to check any migration are uh are being overlooked

Speaker 2: Question

Speaker 3: Okay, so a bit uh in the previous talk we uh had some experience from Ruby on Rails. Uh and I have I'm not personally familiar but I have seen on uh uh article title level that they have something like a local CI which is basically they run all the CI tests on the local machine with motivation that local machines oftentimes are much more powerful powerful than the CI machines. And then they do the sign-off where the they it's not cryptographic signature but they kind of tell that GitHub that this has been passed. So what do you think about running uh test K test suite from the pre

Speaker 3: -commit.

Speaker 1: Pre-comets. Uh actually it depends of how uh uh uh how much the number of tests tests that are being uh you know run in the first place. uh uh for uh the test suites again uh this is means when involve ma many checks within the CI or the pre-commits, this will take much more time. So it depends how we gonna adapt this. This actually from practical experience with the developers, they will end up with no verify locally. So the same thing will goes for CI. So my answer would be this depends on the nature of the project and how many tests

Speaker 1: will be run or triggered on that change. I hope uh I took your accent

Speaker 2: Is there a way if something is caught in in CI to um let's say I put push a a secret to uh uh because I forgot to pre-commit install and then suddenly I got my SSH key on the git repository repository. Um should it be doing more than just failing? Can it do more than that?

Speaker 1: Uh you mean alerting?

Speaker 2: Yeah.

Speaker 1: Uh actually uh I don't think so, to be honest, but when the CI itself is being, you know, cancelled or rejected or fail failed for some reason, this is uh you know a red flag because something goes wrong actually you will get uh you know notification from github you your action is being uh yeah failed So yes. Um I I uh I would like to point out uh for uh and highlighting the security topic because uh I'm pretty sure pre-comets alone won't solve the issue. You know uh that um simply we c couldn't just uh put that the uh secrets within uh Python files, right? It could be in YAML

Speaker 1: or whatever it is. So this is take us um uh to the point of you know uh security managers and uh yeah other topics not related. What I'm trying to say that pre-commit is something uh uh very healthy to to move forward with but you know that this should come up with other complementaries. like like I said maybe security uh checks on the CI um security objects like uh kube city cube kube city um config for the security credentials Uh I think this is already was uh pointed out in the earlier uh session. Yeah.

Speaker 2: It's an old tool pre-commit and there's some new ones around too. Uh that uh have do you have experience in in uh I think preck? Is that

Speaker 1: preck?

Speaker 2: Yeah We can we can talk about it after.

Speaker 1: Okay, can take it.

Speaker 2: Thank you so so much for the talk, Mohammed.

Speaker 1: Thank you.

Speaker 2: Um It's exactly the time that uh we are going to have uh bit of cake and coffee and that's

Speaker 1: interesting.

Speaker 2: Yeah. It will be Uh I s I assume since it's not up here that is a reason that it would probably be downstairs. Can we s can we send out a a scout? Yeah. So so yeah, there's a break now. Um and uh the next speaker is uh Lionel. Um and uh there's a small change in the program because of that talk that went out uh then you are on a little bit earlier so we're back here at uh four o'clock not sixteen forty five as it says in the program but sixteen oh oh uh where Lionel will present present his talk and then um we'll probably continue with lightning talks after that. Yeah. See you then