Switching to Playwright for integration tests

Speaker 1: Setting you up for YouTube Live And we're live.

Speaker 2: Oh, hello.

Speaker 1: Hey Sage.

Speaker 2: Hey Thibaut.

Speaker 1: Good to have you here. Do you have a jingle for us today?

Speaker 2: Maybe, yeah. Well, I just hope I don't embarrass myself today, but let's see.

Speaker 1: Noching the bar pretty high

Speaker 2: Are we doing a real podcast now?

Speaker 1: Oh, this is really cool. Um have you been practicing your guitar for a while now?

Speaker 2: Uh, on and off. So yeah, not as much as I should be.

Speaker 1: I am one of those people that enjoys music but has no skill playing it whatsoever. So this is already very impressive

Speaker 2: It's just a few notes.

Speaker 1: Yeah. I used to have a harmonica and

Speaker 2: like

Speaker 1: practice it enough to know how to play like five notes out of like one of the most famous like harmonica western tunes.

Speaker 2: Country boy.

Speaker 1: Uh anyway, this is a very serious uh podcast, Sage. Um thank you for making it. What are we talking about today?

Speaker 2: I think we are talking about how we are moving away from Puppeteer to playwright for our end-to-end tests in Whitetail. No

Speaker 1: , puppity, your playwright, don't you love all those uh JavaScript projects names?

Speaker 2: It's not strictly JavaScript though, right?

Speaker 1: Is it not?

Speaker 2: Well, playwright. I think you can actually run it from Python.

Speaker 1: Oh yeah, okay, yeah. Yeah, because they have different um Syntaxes then to write your tests in is that the idea?

Speaker 2: I think so. And just how you install and run the tests as well.

Speaker 1: Yeah, makes sense. And then I assume under the hood probably there's lots of code reuse in how they talk to the browser. I guess you have the same protocol and maybe some code shared on the implementation, no?

Speaker 2: I think so. I I would assume so, yeah.

Speaker 1: Yeah. Very interesting that they made that choice. Um why are we rewriting our tests from puppeteer to playwright?

Speaker 2: Well, I notice there is an existing issue for that.

Speaker 1: Is there? No, I created this and forgot promptly. Great. Oh we discussed it on Mastodon Oh yeah, that was about um adoption of uh playwrights on specific projects. Um Yeah, and how you manage having test data available for those test suits. And we were considering the Python variants back then. Did you end up uh writing your implementation with the Python variants or the JS one?

Speaker 2: Um no. I ended up just refactoring our existing Puppeteer setup to Playwright because we already had like the infrastructure and like all all the tooling working with Puppeteer, so I was Looking more about as a first step at least to to switch from puppeteer to playwright.

Speaker 1: Yeah.

Speaker 2: But I can see how we might want to switch to the Python um usage because the thing with using uh JavaScript to write the test is Well, all the test cases are written in JavaScript, so you don't really have um like the nice utilities provided by Django 's test suite like um having the database roll back at the end of the test um so that you can isolate each test data

Speaker 1: Yeah, that makes sense. I mean currently our tests are pretty basic, so I think they all run off the same uh Test shoot app is that right?

Speaker 2: Yeah, so we are running the Django server separately. It's running like a normal run server thing, uh, development. uh mode and then we spin up these browsers with um jest and puppeteer or play or playwright. So yeah, it's more like as if you are browsing um your Django project in a local testing environment. Uh well, local d development environment, not testing.

Speaker 1: Yeah, yeah. So there's no isolation between test cases of um any data manipulation that might happen. Yeah. Yeah. I remember when we set this up like I think probably five plus years ago now, feeling um feeling quite proud of having it pulled off because uh back then I felt like the norm was very much um selenium.

Speaker 2: Yeah, I remember those days.

Speaker 1: Yeah, I mean s I feel like Selenium does lots of things it does right, but at least back then it was super, super heavyweight and challenging to configure.

Speaker 2: Yeah, and it it tends to break after browser updates. Like uh there are new flags that you need to add, otherwise it would crash on smaller um systems like CI. where you don't have like you know the graphics driver or whatever.

Speaker 1: Yeah. Yeah. I don't really remember exactly Or nor do I, you know, have looked into the depths of the code enough to understand, but with those newer libraries, it felt like like whether because of the browser integration or the ease of installation you could like have the promise of air quotes just installing the package. Usually it would come bundled with a browser and it would just work with that browser. And I remember like not having to worry ever again essentially about whether you know the test suit would start. You still have to worry about whether the tests are faky. But at least you don't have to worry about whether the test suit will start at all, if that makes sense.

Speaker 2: Yeah, yeah. Sometimes it would just crash immediately and then all the tests are failing because none of them actually ran.

Speaker 1: Yeah. I think what we have set up in Magtel right now it's It's pretty solid as far as you know the tests running reliably enough. But yeah, looking at this issue definitely We're worried about the friction it creates for contributors our setup being so ad hoc, I suppose. Um maybe just because Puppeteer is much less popular these days when it comes to testing. One of the things I noticed comparing the two is obviously there's lots of more browser icons on there. Do you think eventually we'd want to run our test suite across multiple browsers?

Speaker 2: Oh, I think that would be like in an ideal world that would be what we do because we do support most modern browsers, right? So yeah

Speaker 1: Yeah. Um not necessarily sure it's worth like the cycles in CI because I don't think we have that much code anyway that is written in a way that risks being browser dependent, but for local testing, like it would speed up my life a lot. If I could um run across multiple browsers there, like when you make new new code that you haven't tested yet Um

Speaker 2: yeah, definitely not going to run it on every single commit that gets pushed to main, but maybe if it's like a manual trigger that we can run just before a release candidate, for example.

Speaker 1: Yeah. And we had discussed doing something similar for performance benchmarking of the browser rendering, as I recall. That also felt like something that again, maybe not in every commit, but um plenty valuable enough to have it set up in a way that you can write it on on demand and or as part of the release cycle. Do we still have our discussions on benchmarking in here? Yeah, so JS bundles, don't we love them? And

Speaker 1: I think further down we have those lighthouse comparison reports between releases. Yeah, so this is a synthetic lighthouse score and you can clearly see going left to right over time how our score is lowering.

Speaker 2: Oh no.

Speaker 1: I assume as we add more and more. Features and code?

Speaker 2: Is it not the browsers slowing down?

Speaker 1: What's take right there? I definitely feel like it's you know those pesky software people making more and more use of the hardware. Yeah, I mean, you know, there's probably also a case that some of it is warranted because we do have more capabilities, but we ideally want to make enough refactorings so that Uh it doesn't scale that way, I suppose, as you ship more it gets slower. So yeah, having this data to start with invaluable, then you know backing that trend, another story But I feel like we have pretty decent decent plans to refactor our way out of this, right? That's kind of what you've been up to lately doing JavaScript tooling improvements so we can move on to the world of ES modules

Speaker 1: and Fancier dynamic loading.

Speaker 2: Yeah, and maybe we can eventually remove jQuery.

Speaker 1: Do we have an issue for that?

Speaker 2: Pretty sure we do

Speaker 1: Yeah, I think this is one of many actually. There's probably a separate umbrella issue that I can't like spot right away for us to take it on for other aspects of the project as part of our stimulus adoption Anyway, let's get back to the topic. You made a pull request, Sage. Tell us more about it.

Speaker 2: Yeah. Um yeah, so I was working on upgrading our tooling, uh well, you know, the the packages that we use for things like tests and linting and the recently I was working on the front-end packages that we use. So uh this morning I think Depot just merged the upgrade of ESLint. to version 9, which I know that version 10 was just released like a couple weeks ago or so, but I think last time I checked, other packages that we use, like TypeScript ESLint, hasn't been updated to support ESLint10. So as a first step, we haven't even uh before this VR, we didn't have ESLin version 9 yet.

Speaker 2: We were still stuck on version 8 because um they migrated to a new version of the uh config format to use a flat config as they say. And yeah, we also one of the things that blocked us from upgrading this was the fact that we were using Airbnb's config or esln and it has served us very well over the years but um they also haven't been Um well they haven't updated the package to support ESLin 9, so it's one of the blockers and I had to Essentially migrate the rules that we previously adopted from Airbnb to our own.

Speaker 2: And yeah, I just publish uh an RC version of the config to test. um that the new config works well with Wagtail. And yeah, after I upgraded ESLint, I looked into the other tooling that we use and one of them is Puppeteer currently. And I I know that Tebow opened the issue about switching to playwright. So yeah, I also wondered if AI would be good at handling tasks like this. So I did try using GitHub's copilot extension in VS Code with the agent mode. And I basically just asked it, hey, can you switch

Speaker 2: our tooling setup for the integration tests from Puppeteer to Playwright? and seems to have done so quite well. Al uh though obviously I did um review the whole thing 'cause I think m you know, as always, well, not always maybe, but as expected sometimes. It's uh sometimes it gives you code that may not be necessary. Like it tends to be too defensive in writing the code. Like for example, one thing I found was uh so if we go to the playwright environ environment, I think.

Speaker 2: Oh actually maybe not here. But yeah, so uh this bit. Oh yeah, actually yes, it it was here. Um so if you scroll up a bit deepo. Um no no no still in the sorry in the playwright uh environment. js um in the teardown part Um yeah, I think it originally had all this like if this global page, this global page close Like um in most cases, I don't think um all of those variables will ever be unavailable. But I guess maybe if some for some reason the browser crash during testing.

Speaker 2: But yeah, we can just use the optional chaining, right? So I had to tell it uh can we just remove the ifs to use like simpler syntax. But overall I think it did a pretty good job at migrating the setup. I also asked it to rate the doc documentation I think Playwright has a documentation from uh for migrating from Puppeteer specifically. Um yeah, I'm not sure how much it follows that uh documentation, but Oh. Oh

Speaker 1: maybe it's in the JavaScript docs.

Speaker 2: Oh maybe, yeah. Yeah, there we go. Yeah If the AI looked at this page, I think that's that might be what happened because it it seemed to produce quite good results with the migration.

Speaker 1: Yeah, that makes a lot of sense. Um I was actually really surprised seeing your pull request for the first time um in the way you approached it. Because I felt like when we discussed um switching to playwrights in the past, it was also with the idea of adopting playwrights test runner, whereas here you've literally only looked at the I guess browser automation integration with the test runner sting being being just. Do you think we're gonna move to the playwright test runner as a future step or not so much?

Speaker 2: I think that would make sense. Um that would definitely require the test files, the assertions to to use playwrights, right, instead of using gists. So Yeah, uh I think in in this first step I just wanted to make sure that okay, at least we can run the browser with uh playwright and you know updating our dependencies to swap out puppeteer with playwright and then if we want to take it a step further from there where we actually refactor the chest test uh assertions to use Uh playwright. I I would be happy to in a future PR.

Speaker 1: Yeah. I um actually also attempted this with AI and I asked my AI to also rewrite our assertions to the playwright test runner Uh which is why I was surprised seeing your PR the first time around. Um I think it's um Interesting because when when I was initially setting up our Puppeteer plus jest setup, the idea was very much to have the same assertions style everywhere in unit tests and integration test. But in the end, yeah, it does mean having this kind of unexpected integration between the test runner

Speaker 1: and the browser automation. uh for people who come from a world where you know the two are much more linked together. And do you think we're gonna keep writing our tests in JavaScript then or also eventually move to the Python integration?

Speaker 2: Yeah, I think that's one of the reasons why I didn't look uh too deep into refactoring the tests because Um yeah, there is the possibility of actually running playwright um from Python, but I think it Currently, well at least officially it only supports running it from PyTest, which we don't use for running White Tails tests. I think at one point we did have a PyTest configuration in the repo added many years ago, but it was not updated and I don't think anyone actually ran the tests with PyTest. because none of our CI setup uses by test.

Speaker 2: And yeah, I uh a few months ago I removed it because if we really do want to use spytest, I think we we need to start fresh and make sure that it is actually maintained. But yeah, I wasn't sure if it's worth adding. This is basically yet another test infrastructure infrastructure or like test setup for our Python test. So yeah, I I'm still not sure about this to be honest.

Speaker 1: Yeah. I mean, I think there's uh there's a few considerations there. There is the ease of um writing and understanding the tests and how much coverage that might allow us to get. And then there's the capabilities of the test runner once you need to actually troubleshoot what you wrote And I think these days that probably also factor in how AI like agentique coding fits into this. making sure that because you know that's one of the big strengths of AI coding tools that they can write tests much faster than I ever will and You know, it's not like writing tests is a particularly complex thing.

Speaker 1: It's just very time consuming to write tests that have a good coverage. So I feel like Maybe that's a consideration there. Which which style of writing tests makes it the simplest for us to have the best results in a world where the tests are written with AI? I guess I guess the question there is also how much the test data like isolation we care about. It feels like if we did then Py test makes lots of sense, right? Because We already have lots of uh Python code that sets up test data for our unit tests. We could reuse that, I presume.

Speaker 2: Yeah. Yeah, the the test project, all the models, it it already has it and would be great to use it. So I imagine If we do proceed with this, the set of tests that are running with Playwright should be separate because I don't see us migrating the whole test suite to to use PyTest, right? So it's probably a small subset that you run with a separate command.

Speaker 1: But do you think the fixtures will be shared at least or not even the other thing?

Speaker 2: Oh yeah, I I think the the fixtures could be shared, yeah. Or um we could also, you know, add another fixture that is specific to to this But I think the current ones we only have like a very small set of fixtures because a bunch of tests also run uh create the objects dynamically from the test code itself.

Speaker 1: Yeah Pictures might not be the best term here because it's so overloaded, but I do feel like, you know, even in a world where AI makes it simple to write assertions uh manu ma managing the test data that you have available that is still like something that requires some human touch, whether it's uh dynamically created in um In the test cases ordered it separately. Uh don't perhaps that's an old fashioned view as well. I don't know. How do you How do you feel about it? How much what's your ideal amount of test or QA code written by AI versus without AI?

Speaker 2: It's difficult. Well, you know, I'm I'm not going to lie, I do use AI to help me write tests because most of the time lots of test code is kind of repetitive, right? Okay, you do the setup, do the assertion, and so on. Um yeah, I think if I think AI benefits a lot from having a consistent way, uh like practices to to write the tests. If there is a clear pattern of how the tests are written, it can follow uh those existing patterns quite well. So yeah, if if we have a clear

Speaker 2: maybe uh some clear examples in our code base I think it would be beneficial for the AI.

Speaker 1: Yeah In other respect, I really like um the time you spent to set up our Python unit tests with um With um

Speaker 2: beautiful soup.

Speaker 1: I mean honestly like yes, but not necessarily beautiful soup, just the idea of uh using a CSS selector library to identify elements And then be able to write assertions on their attributes. Like beautiful soup is the tool we're using for this right now. But um I guess in some respects, I think it could be used, uh it could be any other. It's just like this is this is I I'm a front-end dev. This is so much easier to understand which element this refers to than like regular expression style tests on the markup directly. And um even once the element he selected, like just the assertion, it's much clearer. What we're checking for exactly is not reliant on the markup

Speaker 1: at all.

Speaker 2: Yeah, and it's it's very portable because you can use the same selector, let's say you copy and paste it to the browser to get the same elements. Um whereas if you use like Some specific syntax like I think Beautiful Soup has like find okay this tag with these attributes as a dictionary Well, it it could be more readable in in some cases if you have a very complex selector, but for simple cases like this, it's very clear the selector. Um yeah, it's It's just easier to read, I think. And is it easier to write as well.

Speaker 1: I don't know how much you um follow like internet opinion pieces, Sage, but there's this article that I came across recently that I found very interesting, which is, you know, on I guess the challenges we face as developers, technical debts being long-term issues on projects where You make trade-offs in the interest of speed and then the trade-offs, you know, come back later to bite your ass. And how we Agentic AI, there's this sense that you know you don't necessarily have to worry about this too much anymore. But you still do have to worry about how much brain power it takes to have all of this in your head. Um so

Speaker 1: in that respect, yeah, those kinds of like syntax choices, like developer experience API choices, even for QA code, still super relevant, I think. Anyway, so I think the TLDR is move to playwrights to hopefully get more carries out of our test runner. for now move in a way that uh we can start using it without having such a big shift but then we still need to decide how we're gonna write our assertions in the long run. I feel like I'd I'd honestly prefer we went a bit further than this and already started adopting the

Speaker 1: JavaScript uh assertions from Playwright. Have an even simpler setup, I expect.

Speaker 2: Yeah, I would be happy to that for for that. Or maybe you could just push your local branch where you tested it with AI and see if it Works.

Speaker 1: Yeah. The only reason I didn't is uh you know I do feel like um It only took me five minutes with AI and it's definitely something that needs more than five minutes worth of conf consideration of what you want to do in the long run. So it's not because it's done, like the code is so cheap. uh guess correctness of um what you do is still um important to get right. Um anyway, let's try it out locally, shall we?

Speaker 2: Yeah

Speaker 1: Okay, um I have my local Wagtail Development Environment configured but I have not yet pulled that specific pull request. I thought it would be nice to go through the motion all together. We're inside cursor, but I'm not using any AI features today. It's just like any other VS code with an integrated terminal. Um and I guess I can start by checking out your pull request.

Speaker 2: C L I

Speaker 1: um terminally on GitHub, as kids say these days. So

Speaker 2: it is in the terminal and it is GitHub, so that checks out. But

Speaker 1: um a quick rebase.

Speaker 2: You're too late, I already did it

Speaker 1: Nice Quick look at the commits because don't we know that Sage loves to do very good commit messages? Okay, just a quick rename for the sake of the git history and then the actual rewriting. Next one, Sage.

Speaker 2: Yeah, so we can actually blame whoever originally wrote the code. Oh

Speaker 1: don't we love to blame? Do I need to go install in a separate folder or can I just install from the root?

Speaker 2: Oh that's a good question. I never know with SPM And OKS tooling.

Speaker 1: I think at some point we might have considered adding a way to like you know like have a post install hook that also installed in a separate folder. but decided against it just so that we don't encumber the local setup of everyone who wants to contribute to Wagtel with yet another um Browser instance?

Speaker 2: Yeah.

Speaker 1: Okay Oh sorry, overscrolling. Playwrights, nice And um then you updated the documentation, so I'll be able to refer to that to figure out how to run this, I assume.

Speaker 2: I hope so.

Speaker 1: Okay, this is just for our XG targets. So probably documentation on developing. Oh, there we are. Uh this looks very similar to what we had before.

Speaker 2: Well yeah, nothing changed in this case.

Speaker 1: Why is there changes on the file then? How do you look at it? Oh, I never know how to look at the changes in a specific commit in a specific file. Is it as simple as that?

Speaker 2: I think you can even point to a directory, I think. Oh, maybe not.

Speaker 1: I think I'm holding it wrong, Sage is the problem. Hmm. Time for the AI.

Speaker 2: I think I think you need to add a dot or something.

Speaker 1: Have you seen uh this particular agency coding tool yet?

Speaker 2: Nope.

Speaker 1: This is Vibe from Mistrol. We're running Death Stroll 2. Oh. This is the exact same code. What am I doing wrong?

Speaker 2: I think without the head.

Speaker 1: Oh, is that?

Speaker 2: Wait, what? Oh, you're currently inside the client test integration directory, so you have to finish.

Speaker 1: I'm a dummy, that's the problem.

Speaker 2: Ah, there we go. See, not even AI can help you with

Speaker 1: it because it was clever enough to move to the correct folder. It's just that I completely ignored that and um assumed that uh it was me mistyping the comment or something. There's definitely some AI brain atrophy going on

Speaker 2: like that.

Speaker 1: Okay, anyway. Let's run our local setup.

Speaker 2: Have you set the settings more too?

Speaker 1: Hmm, good point, Sage. Do you use those tools that allow you to set environment variables as you move in and out of folders?

Speaker 2: I use dear and v, however you say it.

Speaker 1: I'm not sure if I should trust them. So I've not set up that ever, but maybe I should um lean in and start using them too

Speaker 2: It's quite useful. I think it's mostly set and forget.

Speaker 1: Yeah. I just worry a bit about the forgetting part.

Speaker 2: That's true, yeah. Because you don't want to be loading the API keys and such if you don't intend to.

Speaker 1: There's some element of that and also um I think I usually quite like to focus on the lowest common denominator developer experience because

Speaker 2: Yeah, to make sure it works for everybody and not just because you did something special with these things.

Speaker 1: Yeah. I'm not entirely sure whether that's a sound way of thinking about it though, because um Like my experience isn't lowest common denominator on the MacBook Pro. That's pretty, you know, M1 processor, pretty recent. Um but I guess it's something that's good to strive for, I suppose, still.

Speaker 2: Yeah, because we don't want to have it working in one setup but not another.

Speaker 1: Yeah. Was it that fast?

Speaker 2: Well, we only have Very few test cases, so

Speaker 1: I was expecting much more drama , Sage.

Speaker 2: That's the magic.

Speaker 1: How can we configure this to um show us a headful UI? Or can we configure it?

Speaker 2: I think it's in setup. js because this is the bit where it connects to the browser with A web socket. This is the jest setup

Speaker 1: I'm questioning whether it's done anything at all. So I think the simplest way to troubleshoot this is uh introduce a failure in the in the test, right?

Speaker 2: Sure. I thought you wanted to disable headless mode.

Speaker 1: I I will, but I just like one a quick way to confirm that I'm not dreaming and it does.

Speaker 2: Yeah.

Speaker 1: Nice.

Speaker 2: That reminds me of a quote in the I think it was the Ralph Rate Me. Where it says it's so quick I had to intentionally introduce a formatting issue to make sure it works.

Speaker 1: Of course, of course Those like kids these days with their new t tools you start to worry like is it doing anything at all? Because you know it's definitely easy to make something a thousand times faster by just doing a thousand times less. But I I suppose uh it is definitely checking it. Um how how did you say we could configure it to be in um headful MOOC?

Speaker 2: I think it's setup. js. It's not here.

Speaker 1: Your princess is in another castle.

Speaker 2: Yeah. The launch server. You pass it a object. Yep. There we go.

Speaker 1: See this this is this kind of interaction in the IDE is exactly why I feel like um

Speaker 2: everything should be typed.

Speaker 1: Yes. But also like my concerns about uh using the same assertions formats um with United integration tests are a bit um dated because it's the I think the only issue still is you still only get this inside your ID I don't get this in GitHub when I'm doing code review. And if I'm vibecoding in a command line tool, I don't get this either Because like yes, this this still helps tremendously when you're changing the code, but when you're just reading it it's still tricky to remember uh the shapes of all those objects

Speaker 2: Can you imagine if GitHub has a language server?

Speaker 1: I expect it's coming on the stream.

Speaker 2: Really?

Speaker 1: Well, um I don't know like at what stage of AI adoption you're at, but suddenly if you want to cheap quality code, one step that you still have to go through is reviewing the code. Quite a bit, right?

Speaker 2: Yeah.

Speaker 1: So in some sense, like you and I working on Wagtail Core so much we spend lots of time reading code already, but I expect the trend is gonna be the same for other people on other projects where they have external contributors making the actual changes That are, you know, AI powered and then you just come in and do code review and think of the architecture decisions and um Yeah, just spend time in VS in VS Code or in GitHub. Potentially a language server would help you a fair bit You'd use it if it was set up, right?

Speaker 2: Yeah, yeah. I wouldn't say no to that.

Speaker 1: Yeah, I wonder if someone somebody else already worked on worked on it, to be honest. It wouldn't surprise me too much. um but anyway um yes let's try see what happens i'm not sure if we'll see it in the screen sharing But I'll try to like react on the spot and there was something flashing on my screen. Oh Oh wow. Oh I wish you could see this. Let me change my screen share. This is like magnificent.

Speaker 2: I believe you.

Speaker 1: I'm gonna switch uh to portion of screen. Please don't don't look at my mess too much.

Speaker 2: What's that?

Speaker 1: I'm trying to get better as my um dataves skills.

Speaker 2: Yeah, it's definitely tricky, especially if you don't have more than one monitors like just nightmare.

Speaker 1: Yep. I have uh I count five browser windows right here on the screen. Anyway, are you ready for this?

Speaker 2: Yeah

Speaker 1: It's fireworks. Don't blink.

Speaker 2: What happened? Oh Oh god.

Speaker 1: That's why it's so much faster than I expected. It makes a ton of sense, but I didn't realize how much parallelism there was in the test suit. I'm still surprised to be honest by how many windows it opens because we really only have 12 test cases, so it looks like it's opening more than one frame per window, per sorry, more than one tab per test case.

Speaker 2: I think it's because there uh there are probably two of them for each test because there's one that actually runs the browser and another one that connects to it from jest with um a web socket to do the assertions

Speaker 1: really

Speaker 2: Maybe. Maybe I'm wrong. Yeah, that was about blank.

Speaker 1: So they use some kind of um window. opener trick to isolate their tests? assertions environment from the main window.

Speaker 2: Oh I don't know you have to ask DAI for that.

Speaker 1: I don't know

Speaker 2: it's the cognitive death

Speaker 1: Or maybe it's simply when we have test cases that open up um different pages.

Speaker 2: Maybe, yeah.

Speaker 1: Anyway, very interesting, very educational, very fast. Don't necessarily feel like um speed is a big big factor here for us, but because we only have 12 cases, but I assume the idea here would be for us to greatly increase the coverage to the point that the speed definitely would add up.

Speaker 2: Yeah.

Speaker 1: Seems to work really well. Thank you, Sage.

Speaker 2: No worries Uh

Speaker 1: let's let's go back to your pull request and see if there's any other aspects of it to highlight in there. Um This makes sense to me.

Speaker 2: Oh, what is this?

Speaker 1: This looks like you didn't want to reuse A existing AX integration because of our setup, I assume it might have been too tricky. I feel like you mentioned this in your pull request description, no?

Speaker 2: Don't remember I maybe I didn't.

Speaker 1: Explain yourself.

Speaker 2: Oh. So I think in the previous setup for Puppeteer, we use a different package. That comes from WordPress of all places. Yeah, to to integrate the accessibility checker that we use, which is Axe uh into the gest assertions but that package only supports puppeteer so for playwright Um yeah, I think the AI just ended up writing the assertions. Well it's it's just extending the expect function from just so that um

Speaker 2: It can use axe for um their A API at least. But it's uh it's also f built in from Yeah, Axe has built-in support for play, right? So we just use the API to have uh like expect dot to pass axe tests, for example.

Speaker 1: Yeah. I'm a bit torn on this. I think um past Tebo would have loved to geek out over every aspect of this. and make sure that it's all written correctly, which does imply having to get up to speed with the best way to write custom assertions with this specific setup. New T ball feels like if it runs, it runs. Where do you stand safe? How much did you test this

Speaker 2: Um so originally it uh the AI wrote a bit uh the the code in a bit more complex way um for the options exclude uh assumes that so this exclude and include options they accept CSS selectors or uh an array of CSS selectors And when the AI wrote it, it split the CSS selector using uh by comma and then passing it as an array. Because it it was worried well when I asked you why did you write it this way, it said it was worried about

Speaker 2: um Whether the exclude will only match one element out of the selectors or uh multiple. But I don't know. I tested it. I removed the splitting and just passed the selector as is to axe. Seems to work well at least with our use cases and I don't think we want to do something that's too smart if it if it just works well with the default uh way of handling the selectors

Speaker 1: Yeah. That makes sense to me. This kind of defensive code is definitely um a concurrent pitfall still with AI code where you know for human written and suddenly human red code um brevity is an asset and um ai really wants to be defensive in its uh I guess coding choices in a way that really hurts Bravity. Even this, like I'm left wondering maybe there is some defensive code inside the builder that just gonna work fine even if this isn't defined.

Speaker 2: Yeah, that's the thing. I I didn't test that, but I wouldn't be surprised if that were the case. And if it does work, let's just remove it, I think.

Speaker 1: Yeah I think that's something that I might wanna try and see what happens. Um, but at the same time, you know, there's definitely a case to be said that if you can at least confirm the accessibility tests are air codes working, the specifics of the implementation really don't matter terribly. So maybe it is okay to leave uh the AI to writing it and also to trust the AI with the correctness of it.

Speaker 2: Yeah, and as we can see here, the actual Accessibility test is done by Axe and we're using the official APIs for it. So it's it's not something that the AI just wrote okay uh this uh uh this page should pass these exact checks. That's not what's happening here. So yeah

Speaker 1: Yeah, we're only trusting it with the correctness of um what is quite a slim integration. Okay, I'll just test this locally after our meeting and um Change the code a bit if uh it seems necessary. Um

Speaker 2: WordPress

Speaker 1: Yeah, I mean so maybe they should be considering a move to playwrights and maybe we should make that package the playwright version So they can depend on at Wactelslash.

Speaker 2: But if we don't end up using this in the future in favor of just writing assumptions , there's no point.

Speaker 1: Yeah, I'll look at this code a bit more, but uh this looks like a very sensible improvement. Look at that. type dot field. I'm kind of um maybe not surprised isn't the right word, but you know Puppetia says that's a fork of playwrights, so I Sorry, the other way around. Playwright side as a fork of a bit here. So I wonder at what point they decided that type isn't isn't clear enough. We we have to do that fail and encourage all of our users as part of their migration to rewrite their automation code.

Speaker 2: Machines don't actually type, they just Fill the whole thing.

Speaker 1: They just fill. Are you sure this is actually needed? Because I there could definitely be some ones where

Speaker 2: I can actually check, but it works, it works

Speaker 1: It walks, it walks.

Speaker 2: Uh look, there's a caution. Oh okay. I actually has a delay. Um

Speaker 1: I'm surprised by um that warning because personally I definitely expect in this case it's irrelevant how the field gets populated, but I think it's really rare for me to want the trade-off to have like a few milliseconds faster test suit at the expense of the risk of using the wrong method in some cases. But yeah, maybe it's the right trade-off. Hopefully we don't have any custom JavaScript for our login screen. Um thank you Sage just looks really reasonable. I think the next step will be for me to

Speaker 1: review this a bit more locally and then we need to update this to discuss what's the next step after. How much time did you spend looking at other than you know the PyTest trade-offs, other capability differences between the node and python versions?

Speaker 2: Oh I didn't spend much time with it to be honest, but I did I think I used Playwright to uh run the benchmarks that I did way back. Yeah, just to render browsers and get the results from debug toolbar. But yeah, since it uses PyTest, I was like, um, maybe I'll look into it a bit later

Speaker 1: Yeah. It's um this this specific um UI feature here that I was just screen sharing the UI mode It's something that's been mentioned by others as like a reason to use the JS version. I wonder what's their project's philosophy on like API or Kubernetes differences between the different versions of playwrights, whether their goal is that eventually all the features are available across all environments. or whether it's intentional that some of those features only make it to some environments.

Speaker 1: Kind of similar to the trade-offs we have to make with ragtail and headless support. Where you know it's good for us to aim for feature capability um that are equivalent yeah parity yeah but um in practice getting there is um is a journey Um when do we rewrite our assertions to playwright tests then? Is

Speaker 2: that something that so should we look into that or should we look into using Python Or both.

Speaker 1: Well, Python means Py test adoption.

Speaker 2: Yeah, but I was thinking if we really do want to go that way, we could make it so that Python is only used for end-to-end tests, but not the unit tests.

Speaker 1: Yep. So in this thread when we started it, um I wrote this requires consensus from core team members. I think for the pull request you've done so far, I don't see why anyone would care to object. Seems like a net improvement. Unless someone is

Speaker 2: really against Microsoft. But Puppeteer is I think it's only exclusive to Chrome, right? It doesn't support other browsers.

Speaker 1: Um didn't they yeah they added um and web driver

Speaker 2: I think that was yeah but they initially only supported yeah I think

Speaker 1: yeah yeah And um yeah, you know, I'm um sure our core team members would have their opinions in that respect, but probably for more fundamental dependencies on specific cloud providers than just library choices. Yeah, in any case, I feel like probably the switch to playwright, sorry, to play tests. Even just for integration tests might be something that requires more feedback from others. I don't know how common it is for Python projects to use a different test runner or different types of tests.

Speaker 1: Um

Speaker 2: I think by test can also run Django unit tests. But yeah, I'm not sure if there are any limitations to that, but we could possibly make it work to you know also run the tests with PyTest But but we we're not actually writing unit tests with PyTest because that would be too much of a change.

Speaker 1: Yeah. Well maybe let's report what you've done so far in that issue and then have other people reflect on the different avenues we could take this

Speaker 2: I'll probably dig a bit more and see if it makes sense for us to have the Pythest config. And if so, I'll report on this issue.

Speaker 1: Great. And maybe I'll share my draft of um switching the actual assertions to playwrights test runner.

Speaker 2: Yep, that sounds good to me.

Speaker 1: Thank you, Sage. That's been fun. Do we also have an outro music or is it just uh

Speaker 2: I think not this time

Speaker 1: Well, um hope this was interesting for others as well and um yeah we'll be back next week with another edition of the Wactail Review Podcast.

Speaker 2: Ooh. Thanks everyone