End-to-end testing Django applications using Pytest with Playwright

Speaker 1: And next up is Jacken Reef with end-to-end testing Django applications using PyTests with Playwright.

Speaker 2: Hi. In this talk, I will present a solution on how to write end-to-end tests for Django applications using uh PyTest and Playwright I'm Jacob. I'm from Austria and work at the University of Innsbruck. There I currently implement the main content management system with over 300 departments and for about 20,000 uh 27,000 students. And uh this system runs on uh Django CMS. In the years before I worked as a consultant on many different Django-based projects. And some of them had repeating patterns, and so I extracted their functionality and released them as third-party packages for the Django ecosystem.

Speaker 2: This made them more reusable and motivated myself to write documentation and tests. And by open sourcing them, I got a lot of valuable feedback The apps marked with an arrow are a combination of a server-side and a client-side implementation. Therefore, there's always some JavaScript involved. And since the client interacts with the server, I always was looking for a testing solution which I could use for both of them. Many of you probably used selenium for end-to-end testing, but uh how how was your experience? I never got on with it and so I stopped using it.

Speaker 2: And this changed in 2021. When I found out about the Playwright testing framework and writing end-to-end tests with Playwright now became fun. Let's do a short recap. In the early days of web development, all web applications were processed on the server. And in 2005, when Django was released, this was no exception. The content of models was stored in the database using the object relational mapper, and these models were then used to build forms, and forms were rendered into HTML and sent to the browser. And the submitted form fields were sent back to the server and validated by the form handling code and validated again by the model and then stored back to the database.

Speaker 2: And this setup was quite easy to test. Django's testing framework is based on Python unit tests, which itself is inspired by Java's JUnit. All tests are class-based. Django introduces the simple test case which inherits from the built-in test case class. The functions setup and teardown are called before each and after a test is executed. In addition, there are the functions setup class and teardown class. They are called once for each testing class instantiation. In this example, I use a third equal to check if get text translates the string into French.

Speaker 2: The test case class offers more than 30 assert methods for different kinds of comparisons. So you always have to look up and search for the proper member function. PyTest, on the other hand, only uses the built-in assert function offered by Python. PyTest was a spin-off of the PyPy project. Instead of a setup and teardown method, PyTest uses fixtures. A fixturer is a reusable function which can be injected into the testing function or into another fixtur. This allows us to nest fixtures. Here in the first fixture we override the Django settings. In the second fixture we use the yield keyword.

Speaker 2: So everything before yield is doing the setup and everything after the yield keyword is doing the teardown The second fixture then is injected into the test function. Here we use Python uh uh built in a third function instead of uh one of the comparison functions as shown on the previous slide. This in my opinion makes tests written um for PyTest easier to read And usually they also need less lines of code compared to Python unit tests. If you read blog posts about testing in Python, Almost everybody recommends PyTest over Python's unit tests. And since PyTest

Speaker 2: can also run Python's unit tests out of the box, why don't we use it to run Django, the Django test field? Well, PyTest appeared around 2008 and at that time Django was already four years old. There is a ticket, 25707 from 2020, proposing to switch to PyTest. But this simply isn't going to happen. Currently there are about 17,000 tests in Django. My attempt to run them using PyTest failed miserably. There are so many edge cases that such as Endeavor just isn't doable. And according to Adam Johnson, the Python Unitest library has implemented some features which first would have to be ported

Speaker 2: However, for Greenfield projects, I highly recommend to use Pi test. And one of the reasons is that Playwright has built-in support for its own test runner And additionally, for PyTest, but not Python's unitest. Let's do a short recap on the dependency diagram of a typical Django application. You have your application code sitting on top of the Django framework. Then you have a testing framework. This can be either Python unit test library or PyTest. And on top of this you have your testing functions. And each testing function typically emulates an HTTP request and sends it to the browser.

Speaker 2: So each emulated HTTP request gets an HTTP response, which can be examined. In addition to that, the testing function can connect to the database and check if intended size effects took place. For instance, if an object was updated. Remember, each test runs inside the same process, so while testing there is never a problem with asynchronicity. So running these tests is very predictable. But then this happened. Django was relegated to a data delivery storage. One part of the business logic was moved to the client-side implementation. And now we had two implementations of our web application in two different technologies.

Speaker 2: And each of those technologies introduced their own way of testing. As long as you have separate teams, one working on the front end and one on the back end, such an architecture can make sense. But with a shortage of developers or tight budget, such a proceeding often isn't viable. This, by the way, is where the concept of full-stack development and technologies such as HTMX start to make sense. Because when writing tests, this is what's going to happen. Instead of seeing your application as a whole, you're going to write independent tests, each mocking their interface This is an approach I don't really like because those mocking interfaces often do not represent real interactions

Speaker 2: and Recombio somber to write. And when dealing with HTMX, there isn't any good way to create purely client-side tests. In my opinion, the only alternative to test the proper functionality of HTMX applications is to write end-to-end tests. As far as I know, there currently are these end-to-end testing frameworks. Selenium, this is the oldest end-to-end testing tool. Tests can be written in Python and Django provides an interface for it. My personal experience with Selenium was not very satisfying because tests were tricky to write. And if not done properly, often were flaky. Then there's Cypress.

Speaker 2: This is another popular end-to-end testing framework. According to their documentation, tests can only be written in JavaScript. I never used it, and so I don't have any experience with it. And the newest kit on the block is playwright. Playwright has bindings for many languages, including JavaScript, Java, C sharp, and of course Python. This is a fork of Puppeteer which was created by Google and now maintained by Microsoft. Last year at DjangoCon US Avindra Fernando compared Cypress to Playwright, but he used JavaScript to write those tests. In this talk, I will focus on Playwright in combination with PyTest exclusively.

Speaker 2: There are a few things to consider when running end-to-end tests. On one side, you can access your junk application directly from inside the testing process. This has been shown on slide eight. Here we extend um PyTest by uh playwright. Then we add a headless Phantom browser. This can be any of the big fours. Now Playwright communicates bidirectionally with that browser using the DevTools protocol. The browser then invokes a real HTTP request to the listening Django server. which then answers with an HTTP response. This then alters

Speaker 2: the HTML in the browser, but can also be intercepted by playwright. Something to keep in mind is that the browser and the listening Django server both run in their own process. Therefore, be careful when accessing Django directly and simultaneously through the browser. Selenium, for instance, uses the WebDriver API, which is a kind of REST interface to interact with the browser. This means that the testing code must actively pull to see if something changed. Playwright, on the other hand, communicates over a WebSocket using the DevTools protocol. This protocol is also used by your browser development tools, so everything you can do there

Speaker 2: can also be done in Playwright. Therefore, Playwright gets immediate feedback whenever something changed in the browser. And this is a really important feature because it avoids uh having to add artificial delays into the testing code. Let's use PyTest to check if a web page renders correctly. I prefer to use BeautifulSoup for comparing rendered HTML over raw string comparison because it ignores spaces between text and the order of attributes. This makes testing of the content of HTTP responses less flaky. Here I use a fixture to convert a class-based view into a function-based view.

Speaker 2: The result of that fixture then is injected into the testing function. PyTest Django provides other useful fixtures. The request factory can be used to generate a generic request object. The given URL usually is not evaluated and can be just anything. The generated request object then is passed into the view function. There it generates a response containing HTML. Now let's check if the HTML is rendered as expected. BeautifulSoup creates a tree of nodes to represent HTML and elements. That tree then is used to check if the elements are located where anticipated and if they contain the expected attributes

Speaker 2: Instead of using the request factory as shown in the previous slide, PyTest offers a fixture to emulate a complete browser request as a testing client. This means that the request goes through all the Django URL routing and its middleware. Now we therefore must know the exact location of the rendered view to test. Testing a request-response cycle using the testing client now can easily be converted into an end-to-end test using Playwright. By installing PyTest Playwright, we get two extra fixtures which can be injected into our testing functions. One is the live server.

Speaker 2: It is provided by PyTest Django and runs a Django server with a listening socket. The other is the page fixture. It is provided by Pytest Playwright and is used to access the page through the Phantom browser. With page go to, we tell the browser to load the page from a specific URL. We then can access the elements of the DOM using page locator, just as we would do using the find method from BeautifulSoup. What we, however, should not do is to assert the properties of an element like this as it will make the test flaky. And the reason is that at the time of testing, it is not guaranteed that the DOM is in the desired state.

Speaker 2: Remember the diagram from slide 14. The Phantom browser and the listening Django server run in different processes, and so there is no guarantee that everything is in sync. Instead of using the expect keyword provided by in instead you must use the expect keyword provided by playwright. This ensures that the locator points to an element that contains the given text. This is especially important when writing web applications which modify the DOM during runtime. If the element for the given locator does not exist, playwright waits until it appears or until timeout expires. If that happens, the test will fail

Speaker 2: This example shows how to use a hybrid approach for both ways of testing. It combines an end-to-end test with a unit test. Here I try to test if an element is deleted from the database after the delete item button is clicked. First, let's create a dummy item which later will be deleted Therefore, the page has to be loaded afterwards. Now let's locate for the delete button in the DOM. Since our page is rendered using HTMX, we can use the given endpoint for deletion. We then click on that button. We expect that this deletes the item from the database. But our test sometimes fails here, and the reason is similar to the previous slide.

Speaker 2: During the assertion, the browser and the Django server might not be able to update the database. Now you might be tempted to add a sleep statement before the assertion, but this is not a good idea. Either your test delay is too short, making the test flaky again, or it's too long making the test slow. Instead, Playwright provides an expect response method. This ensures that the test does not proceed until the uh endpoint delivered the response. If the expected response is received, we can safely assume that the item has been deleted from the database. When using Playwright with Python, there currently is no way to measure the coverage of affected lines in JavaScript.

Speaker 2: To get this information, you must write your tests in Node. js. Remember, Playwright's canonical binding language is JavaScript. The Python API does not support all the features currently implemented in Playwright. However, if the context of Django applications using HTMX There is usually no JavaScript involved. And even if so, usually these parts are small functions which can be tested using the JEST testing framework. But that's a topic for another talk. The important takeaway is that we can measure the lines of affected Python code regardless if we perform a unit test

Speaker 2: , an end-to-end test, or a combination of thereof. For this talk, I prepared a live demo. So I've wrote a very simple web app for a to-do list using uh Django HTMX and Django template partials. Many thanks to Adam Johnson and Carlton Gibson for providing these libraries. And first I'm going to show how this app works, then we are going to take a short look at the HTMX code. And finally I will explain how to run the unit tests and the end to end tests with this web app So in order to I will talk and Fabian will help me to

Speaker 2: to present uh the demo demo app and now I have to switch to I have to stop mirroring, sorry. Um Okay.

Speaker 2: Wait. Where can't stop mirroring here? Oh Oh, yes. Okay So just

Speaker 2: So this to do app can you hear me Yes. This to-do app can do three amazing things. Add as many task items as you want. Add some text using the input field. Toggle each task item as completed. And Delete any task item from the list. Since this app is made in HTMX, the HTML code explains itself. Let's have a look. We have a form with an input field and a submit button. Then we have a table section.

Speaker 2: Each time we perform an HTMX action such as post, put or request, the template is re-rendered, but only partially. The table section now is replaced by the partially rendered HTML. If I add a to-do task, a new row is added to the table. In the fourth column, there is a button to toggle the task as completed. In the fifth column, there is a button to delete the task from the list. If a task is marked as completed, the text is crossed out using the HTML Dell element. Two

Speaker 2: So let's have a look how we test this using the PyTest with Beautiful Soup. Here we have different testing functions, but let's focus on testing the toggle action. First I add a dummy task to the Django model for our to-dos. With the HTMX request factory, I emulate a put request. Then I check if the task has been marked as completed. And finally, I use BeautifulSoup to check if the template is rendered partially and contains the Dell element. Let's run the test

Speaker 2: So this test took about half a second Now we look at the end-to-end test, how the end-to-end test looks like and uh again I will only focus on the testing function which taught which toggles the to-do item Here um I open the browser uh page and the phantom in the phantom browser. Then I locate the toggle button A useful feature PyTest offers, the Phantom browser can take screenshots. Here I click on the toggle button. Then I check if the task has been marked as completed.

Speaker 2: And finally, I check if the rendered HTML contains the Dell element to cross out the text. Let's run this text this test. Okay, took uh about uh two seconds. Um so it took uh considerably longer. But this test is much nearer to what users really do. Okay, thanks for listening and thanks to the organizers

Speaker 3: Thank you for the talk. Um when you're making a decision of uh whether to Would you have those um playwright tests replace the the other tests that you had if you feel like it's kind of doing the same thing, or do you like to have a a combination of both? I usually uh

Speaker 2: I usually have a combination of both. Um and I usually have them even strictly separated. I very rarely use the hybrid approach I've shown in uh one of the slides.

Speaker 4: Hello, thank you for the talk. Py test fixtures when you use them are not imported and uh so the way to uh find out that some fixtures that you need exists is um not through the usual tools where you the defa where you how the usual sorry methods you use to find uh existing pieces of code. Uh can you say something about discovering fixtures for use in PyTest-based tests.

Speaker 2: Well there's something I've I haven't shown is the uh conf test where you put the fixtures you want to um offer globally to all of your tests. And then you have the fixtures you can just use per module. And then there are the fixtures which are from external libraries like this one. And actually are always They always use dependency injection and I um don't I don't really know how to I mean I j I just look at the documentation I I don't have any way to to see which fixtures are generally available. You have to go through the conftest. py and your own module and then uh

Speaker 2: and then read the documentation. That's my advice.

Speaker 5: Um thank you for the nice talk. Um I wonder do you have any experience in testing single page applications like Angular-based applications or React-based or something like that? Is there any difference in using playwright for that kind of uh setup?

Speaker 2: For I haven't shown it this in this talk because I would not have had the time for it. But uh a nice feature of Playwright is uh it has a um it's named Playwright CodeGen and you can uh create a kind of you can record tests which you want to execute later. You can use that for single page or react. Angular -based applications as well. You can this probably is an easy way to get access to those tests. I um actually uh did not use it in that combination yet, but uh I would use it if I would

Speaker 2: write these kind of applications.

Speaker 6: Do you have any recommendations for running tests in parallel, especially if you're making use of the Django caching framework, which is not isolated between tests.

Speaker 2: No, I'm I'm not sorry. Um

Speaker 7: do you have any recommend uh recommendation on how to do authentication? Do you always go over the login page or

Speaker 2: Authentication authentication.

Speaker 7: Yeah.

Speaker 2: Well with uh uh Pi t with end to end testing you can just um do the uh do the authentication before you start uh doing the other tests. You can even write quite long tests with uh playwright code jen you can just it's like a playbook You can just have a long test which then for instance if you have an e-commerce site you could just do login , put stuff into the cart, go to checkout, pay. with your fake credit card and so on. So you could just do everything in there. Um For me personally, if I do uh uh tests, I usually bypass the authentication to

Speaker 2: Because I don't want to authenticate every time before I do the test. This is more unitest-like if you bypass the authentication.

Speaker 5: Thanks.

Speaker 8: Yeah, uh thank you for talk. Um I was wondering sometimes uh when you're testing things it's not the necessarily one user interacting with a system that is valuable to the test but also affects uh for when multiple people are uh interacting with a system for instance a jet system that if person A types a message that it will show up on the screen of person B or that while like a to-do app where Uh one person edits the uh object and the other has deleted it in the meantime that a nice error message gets shown Does Playwright have any support for multiple interacting sessions with different uh credentials or even the same credentials like if you have two tabs

Speaker 8: open?

Speaker 2: Oh good question. I uh I I never tried out uh on this, but I since you're since you probably it's better to you to do to really write a load uh a load test on this and to use uh different uh session IDs. And that's something which is um probably difficult in playwright. I would have to look at the documentation. Maybe you can open different browser pages with different session IDs or different session cookies and then do the testing like that. Uh but uh I did I haven't used it for load tests or concurrency tests yet. Hello. Uh

Speaker 6: thank you. Um we are in the process of implementing playwright within a new application that we have. And maybe this is a little bit off topic, but we were wondering if you have any opinions or advice on When to run end-to-end tests, how often, and on which platform, for example, do you have strong opinions in terms of Uh mocking the request could not be deemed a true end-to-end test or should it be done in production like servers, for example

Speaker 2: I run them usually together with my unit tests. They just they just take a little bit longer and uh I have a test suite which runs them and uh they run together with all the other Pi tests. But of course uh you can think about different matrices of t if if you if you use matrices in your setups. You can probably do the unit tests using a matrix with different Python versions and different Django versions. And for the end-to-end tests, you do it with a different with different browsers and just one Django version and one uh Python version so that you don't multiply too many uh too many different

Speaker 2: uh so that the matrix does not get too multi-dimensional So I would I I I I would recommend that approach.

Speaker 1: Okay, thank you guys. Um unfortunately the time is up.

Speaker 2: Okay.

Speaker 1: Uh I'm sure Jacob will be here to answer more questions uh if anyone has.

Speaker 2: Okay.

Speaker 1: Thank you.

Speaker 2: Thank you.