Automating Your Browser and Desktop Apps by Al Sweigart

Okay. So uh yes, this is automating your browser and desktop apps. You can follow along with the slides by downloading them from bitly slash automate talk. Um yeah, so hi, I'm Al. It's great to be back in Austin. I actually grew up in Texas. And I went to UT Austin. So even though I've lived for the last decade in San Francisco, my heart and my self-volitary code will be in Austin. Uh I'll take this off with a slightly controversial opinion. I like programming. There are several things I like about programming. Mostly that it requires like keen attention to detail. And I like programming so much that I want to teach other people to program.

So I wrote a programming book. It's called Automate the Boarding Stuff with Python. And it's available under a Creative Commons license, so you can read it for free at automatheboarding stuff. com. And I really encourage you to use Creative Commons as a license It has a lot of advantages. The main advantage I found is that uh having it online I can use analytics to look at what people are paying attention to. and which chapters are the most read. And so the chapters on web scraping and GUI automation are the most popular chapters in this book. So that's what this presentation is So uh web scraping. Um yeah, the internet is kind of a big deal. Um a lot of times you know when you're doing work on your computer, what you really are doing is

work on the internet I don't know if you've ever been to a conference where the Wi-Fi at the menu has gone down. You have about 10 minutes before people start resorting to camera. It's that important. And so when we start writing programs that can go onto websites and then start gathering information for us. That's a really useful skill to have. So uh unfortunately though it also kind of requires all this technical knowledge to have about networking protocols and connectivity issues and TC CT handshakes and whatever. But I have all those details on this next slide right here. Oh well, okay. Fortunately though, you don't actually have to know any of that information. Uh there's a module called Selenium.

It's uh a module that uh will launch a browser that you can programmatically control. So if you have to write automated tests for your web app and you don't want to just download the HTML and then run beautiful scoop on it or some other parser. They actually want it to run as it would for a user running all the JavaScript on the page and uh setting up all that state. Selenium 's great. Um you in it's really simple to install. It's just tip install Selenium. Selenium? Selenium? Probably should have figured that out for certain before this talk. But the first time I heard about this, uh my company's uh QA department was uh using it I heard about this and I was like oh wow that sounds great but it also sounds really complicated. It's actually really dead simple.

I can fit it into a 45 minute presentation. along with a talk on GUI automation. There's really only about half a dozen things that you need to know, and those are all on this slide. Hey, that worked. So I mean it's not that complicated. This is a whole bunch of wallet text. Go ahead and ignore that. But using Selenium, it's really simple. You import the module just like any other module You call this Firefox function, and this will actually launch an instance of Firefox on your computer. You'll see the window appear. everything and that gives you a browser object and you can call this get method you just send it to a website and then you can use CSS selectors to find some particular HTML element on that web page kind of like you know a link or a checkbox and You can then call the click method to click on that link or checkbox.

And then when you're done, you just call the quit method and the browser goes away. And you can continue on. doing whatever you want. So that's the entire talk. Thank you very much for coming out. Okay. So that's this is a whole bunch of telling and I I should be showing. So I'm going to do something that is incredibly ill-advised, and that is a live demo. Yes, all tabbing out of PowerPoint and running idle. Anything can go wrong. Everything will go wrong. Live demo. Okay. So also I just wanted to show that it is possible to keep all of the Selenium stuff inside your head at the same time. So let's go ahead and import this module. So from C mean

Import web driver. Uh so first bit of weirdness about setting in that you should know is that you actually have to do this exact same uh syntax you have to type from selenium import web driver you can't just have import selenium or anything like that a weird little thing that's uh about selenium that you should be aware of uh so next let's go ahead and create that browser So Firefox works out uh right out of the package. If you want to have Chrome or Safari or something else, you can uh There's a level a little bit of setup, but this will actually go ahead and launch Firefox. And you can see it appear on the screen. So I'm gonna have this off to the side. And we'll just call that get

method to send it to a website. And I kind of want to plug my books some more, so we'll just go to automoreboardstuff. com. And you can see now I have programmatic control. I'm controlling the browser from a Python script. So this is really nice. Yep, that's the website. And you know, if you're setting up an automated test, you'll have to go through and figure out what exactly your test should click on. uh where it can expect to find that so let's just do that here. I'm going to have to get an element object, so some HTML element from that page by calling. This is the next bit of weirdness for Selenium. have a really long method name, find element by CSS

Selector. Like when they were handing out words to use in method names, I think this method was at the front of the line said, Yes, I'll take them all. So kind of verbose, but uh it works. Um and then you just add in the uh CSS selector. So if you've used CSS you might be familiar with this or if you've used uh something like beautiful soup. But it can be uh it's uh a selector is kind of like a regular expression For HTML, it allows you to specify some particular part of that HTML document or that web page. And you could learn all the syntax for that to be able to write it yourself, or you can cheat by using the browser to calculate that for you. I'm gonna click on this introduction link. So I want to get the selector for that, so I'll just right-click on it and select inspect

element. That'll open up the browser tools. Every modern browser has some form of developer tools. And where did it go? There we go. So you can see it here's the HTML for this column and now I just want the selector for this particular link. So I'm gonna right-click on that and select copy selector. So in various browsers this might be copy CSS selector or copy CSS path or something, but all browsers have something like this. So we can just copy that. Paste it in there. This selector is really particular. It'll match this one link uniquely, but you can also

have more general selectors to match several different elements. So there that gives us uh this element object. And then if we want Selenium to click on that, we just call the click method. You see now we've clicked on that introduction part right there. So oh well So that's pretty handy. Um findElement by CSS selector is the method that you'll use about 80% of the time. There's also find elements plural. So if you specify a general selector that matches several different elements, it'll return a list of all the matching elements. There's also find element by uh link text

Find element by ID, but find element by CSS selector is the most useful one. So there and then there was the once you have that element object you can click on things you can also send keyboard keys so if it's uh the element for a text field you can type something in let me just push my So let's say you know I figured out the selector for that and clicked on that, and now I want to run some automated test by entering something into the search field. I'll just copy unique selector again. Copy this search element. And Yeah, so these CSS

selectors are really massively huge all the time. Oh, except for this one. I can just send it some keys. I'll just uh I want to search for the word Zophy. And that just types it right in there. And then to submit this uh search field, you know you could have a new line character that simulates pressing enter, or you can find the selector for a submit button. But uh Selenium has a nice little feature where if you have the element for some uh element inside of an HTML form, can invoke that form 's submit uh action just by calling it submit on any element inside of that So that submits this search. And now here are all the uh WordPress search results

from that So Zophy is the name of my cat, and I have a photo of her that features prominently in the image manipulation chapter of this book. She's kinda chubby, but the camera adds ten pounds and that's a lot for a hat. Oh right, I was giving a presentation. I'll just close the So I mean this is fine for you know keyboard keys that are easy to represent as single letters of a string but say if you wanted to press the page down button or the escape button or the F1 button , here's another wall of text I don't actually expect Selenium has uh constants

that you can use for those special keys, such as keys. down for the down arrow and keys. hom. They're all inside this one module. Which has an unfortunately verbose name. Let's see if I can actually remember it. From Selenium. webDriver I don't remember. Uh weber. com. keys import keys with a capital K. You have to get all this right. So yeah, just have a handy little cheats cheat uh cheat sheet off to the side. And then uh that keys uh contains all of the names for the special keyboard characters that you might want to use. So I mean yeah um this

this name when when Python has this Xenopython comb that says flat is better than nested, this is what they mean. Don't have a name like Selenium. webdriver. or dot com and dot keys uh inside the code that you write. And I have to like shrink it down so much that nobody can actually even read it just to fit it all on. on one line. But yeah, so that's you know clicking on things, typing into forms. That's basically what most web serving is and that's all your uh automated scripts will have to do. But you can also not only interact with the things inside the browser, but you can also control the browser itself. You'll probably never guess what these methods do. to click on the back forward and refresh keys. And then so once you have gone through and navigated through your web app, you'll want to check

to make sure it's actually displaying the correct things. So in order to read the text that's on the web page currently, you will have to take that element object that you find and look at its text attribute or its text uh member, which is really simple. It's nice and refreshing. So if you have some paragraph element or a div element, you can just look at dot text and as a string it'll have Uh the entire content for that page. Or for that for that one element. And if you want to get a particular attribute from that element, say you have a link element, and you want to make sure that it's pointing to the right page, in terms of Take a look at its href uh attribute, and that'll return a string with the content that it has. If you want to get a list of all of the attributes for an element, you would think He

would call get attributes plural, but it turns out that's not the case. In fact, there is no way to get a list of all the attributes. The hack around that is that you get the outer HTML attribute. And then you have to look at it returns the entire HTML tag and all the content inside of it and you I guess you can use regular expressions or some convoluted way. So that's another selenium weirdness bit Um but that's basically it. So yeah, that's Sleam is actually a surprisingly simple tool. There's not a lot to it. just uh opening the browser, getting element objects using CSS selectors, clicking on things, and then using send keys to type stuff in, and then reading what the text is.

By checking out its tense member variable. That's great. So GUI automation, uh, graphical user interface automation, that sounds really complicated also. It basically just means controlling the mouse and keyboard. from a from your program. So having a Python script that can move the mouse around on its own. So for this, I'm gonna cover uh the PyAuto GUI module, which you can install with KibInstar PyAutoGUI and it works on Python 2 and Python 3. It also works on Windows, Linux, and Mac, and it has the same simple API which is documented on py autogui. read the docs. org. So, okay, incoming wallet text, because I just want to slightly cover uh all the various functions on it but I'll just glance over this. So

mouse control. Yeah so that's huge wall attacks. Let's do something more fun. That will possibly blow up in my face, and that's always entertaining to the audience. And let's do another live demo. So, import pie autoGUI. Um first function, position. You want to find out where the mouse cursor is on the screen. So this right here just returns a simple xy coordinates. These are the same xy coordinates that uh are commonly used in programming where the origin 00 is in the upper left corner of the screen and x increases going to the right and y increases going down. If you want to find out the full size, the resolution of the screen, you can just call size

Currently PyAuto GUI doesn't really work on multi-monitor setups. Well it kind of works, but its behavior is undefined. That's my favorite euphemism in programming. And then say, okay, well, we can just have um send a click and cause uh the mouse to click itself wherever it happens to be. So I'm gonna position it Over this file menu right here, which probably nobody in the back row can see. But when I run that code, it just sends, it automatically sends that virtual mouse click right there. And so say I wanted to figure out, you know, I'm writing some script that's going to automatically test my desktop app. So I need to find figure out where exactly this file menu is going to be. So I just call position.

Okay, it's at 67 And I'll just pass those coordinates to the click function. Now it doesn't matter where the mouse is, it'll just automatically move there. click. So that's kind of nifty. You can just have the mouse move to those coordinates as well with move to. Oh, wait. I'm on automatic. And I'll just instantly zoom the mouse back over there. But if you also just want to mimic kind of what a human does and just have it gradually move. Over there you can specify a duration keyword. Let's say let's move it over there and over the course of two seconds. So starting from here, move back to the file menu and it's moving all on its own.

If you're just paying attention to the screen and not noticing that I'm not controlling controlling a mouse that um probably doesn't look that fantastic. But here we go. Okay, let's say you don't really care about moving to some absolute coordinates, you just want to move to some relative position say you know 200 pixels to the right wherever it currently is it we'll just start it right there and it ghostly moves on its own over precisely two seconds so not quite uh like a human because I sort of just like slam the mouse all over the place. But yeah, so you can move the mouse, you can click the mouse. You can also, there's a drag to and drag rel, which do the exact same thing. except hold a mouse button down. If you're trying to plot out your automated test, you're going to have to run the terminal because this doesn't display correctly in idle.

Just for your nice position. So there's a function display mouse position. So if you're plotting out what exactly you have to click on in your program, you can just run this function and it provides a handy little tool to figure out where exactly the mouse. at any given time. And it also gives you the RGB color value of whatever pixel is underneath the mouse. You can see over black it's just 000 and then over this green part the green value shoots up. And I can just exit out of that. And then uh the keyboard control, also really simple. The main function you'll use there is typewrite. Let me try this. Again, another live demo, another chance for everything to go horrifically wrong.

Hello world. Um right, this so this will send uh virtual keypresses to wherever uh whatever text field or window it currently has focus. So I'm gonna just call click. To make sure it clicks on this before typing. So I'll just move the mouse over this file editor and when I run that, it instantly types it out. Of course, if you don't want it to be instant, it you Just pass this interval keyword argument. And let's say I just wanted to pause for a fifth of a second in between each key press. And that oh, okay, so one error so far in this live demo. Position the smell scan. Okay. And it'll just slowly type everything out. Hands free. So, you know, kind of like a typewriter effect.

For those of you who don't know, a typewriter is the solution device for putting letters on pieces of paper. And if you just want to press a single key, there's pretty obvious names for them, like the page up key, just send press a string that says page up. And all of those keys are documented here in the keyboard keys list Keyboard keys. They're all just lowercase and pretty much what you expect, but if you don't get them the first time, you get just We'll look through this list. And then if you just want to do hotkey combinations, you know, there are in particular like key press down and key press up. to simulate that but for just doing like a keyboard combination

like I want to type control O to open up the open dialogue I can just say hot key Control and then O. And so here I have idle in focus so that when I press enter, it completely messes up. Alright, that is two errors so far during all my live demos. Let's see if we can keep it at least in the double digits. Okay, so hey, I completely did not do that. Hands free. Did not press Ctrl O. I had the program do it for me. So that's really nice. You can um you know this is like okay all these functions, yeah, they have pretty typical names. Uh that's kind of boring.

So let's do another live demo and see what else can go wrong. Okay, I'm gonna close this and I'm going to open the best thing that's ever come from Microsoft, Aness of Linux. And I have this program that will draw out a little square spiral shape. Um so you can see it just uses those uh drag rail to drag the mouse around I'm just gonna have a drag around here to draw out a little square spiral so make sure that's in focus I'm gonna position the mouse like right about here and then press F5 to run it. Nice.

Yeah, you can keep doing it all day. That'll be great. Uh so I mean well I mean this isn't that impressive because there are things like the pill module and pillow and you can just programmatically generate all of these you know, square spiral shapes that you want. But the nice thing about controlling the mouse to control other programs to do it is that you get to make use of all of uh and paints many many abundant features. But no, it seriously has like, oh, nice little like brushes like right here. something like that. So now using that exact same program , we just run this. And now I'm drawing, you know, a square spiral that it looks a little little bit fancier and reproducing something like that using Pillow would be a non-trivial task. So that's kind of nice.

You can actually start writing programs that control other programs, which of course is how the robots will take over. And while I'm on that subject, a slight warning. So there's this Disney movie Fantasia and there's a scene where Mickey Mouse is the sorcerer. And he's given the task to fill up this bathtub full of water so he enchants a broom to do it for him. And this broom is mindlessly carrying out this task in a loop of fill up the bucket, dump it into the the tub, but it keeps doing that even to the point where the tub starts overflowing and Mickey the Wizard can't really stop this program. And so when you're writing programs that can control the mouse and keyboard, it kind of makes it hard to move the mouse over to stop that programming.

Something is going horrifically wrong So PyAutoGUI has a few failsafes built in. One is if at any point you can just slam the mouse up to the top left corner. And it detects during any PyAuto GUI call. If it's up there at 0, 0, it'll just raise this fail stake exception and stop what it's doing. And also, after every single call, it has a tenth of a second pause Pause at the end of it so it doesn't run immediately. There's a slight pause there. You can change that if you want. I don't recommend it, but that'll probably buy you enough time to just slam the mouse up and to the left to stop it so let's do a live uh live demo of that shall we yeah so let's do a live demo of GUI automation so it takes away control of the mouse

from of a program that has an infinite loop bug. What could possibly go wrong? Okay, so I'm gonna run this and then slam the mouse up and to the left. Py out of GUI will hopefully detect that and then stop it. Otherwise, you're all gonna watch as I pull the battery out of my laptop. Let's run this. Oh, it's going crazy. I need to stop it and okay, hey, okay, yes. Just slam the mouse up there. You can see fail safe exception has been raised. So uh yeah, nice handy little trick to keep your program. This is definitely something that you have to keep in mind when you're writing these automated tests, the first few times you run that automated test, you might want to babysit it just to make sure it's you know not launching missiles at Russia or something like that.

So uh it's so PyOudaguy can control the mouse, can control the keyboard, but it's it's kind of doing this blindly. It's if you imagine a robotic welding arm in a car factory, it's just like weld here, weld here, weld here, and then repeat that over and over again. It doesn't actually know if there's a car in front of it. It just knows I should weld here, weld here, weld here. Same thing with your Pyoto GUI scripts. It's just click here, click here, click here, but that doesn't actually tell it what it's clicking on. So PyOdeguri also has a few uh screenshot and image recognition capabilities. Um they mostly rely on Pillow, which is installed as a dependency with Pyoto GUI. But if you're on Linux, you'll also have to install this program SCROT to do screenshots.

So the first one is a really simple one, it's just pixel, positive xy coordinates, and I'll return an RGB color value of what the color of that pixel is. at that particular orbit. So if your script is clicking on a gray submit button, you can add a little check to see, you know, is that actually gray right there. So that's a nice, you know, uh kind of dumb way of just having a Sanity check right there. You can also just take screenshots uh just to have a sort of visual log of what your program is doing and what the screen looks like at any given moment during your program. The main thing that uh is really nice is it's locate on screen function. So if you have an image of something you want it to click on, you can just pass that to locate on screen and that will return uh the coordinate for where it found that.

So I'm gonna do yet another live demo. Let's launch Microsoft's other great application, Calculator. And I already took a screenshot of the 7 key, so this is just its own little PNG file. And I'll just have idle Try to find that on the screen. So high aui dot locate on screen and then pass it that image file name. Calc7 key. So uh two things you should know about this function. One, it's kind of expensive to run. It takes like a full second to actually find because it's basically checking everything. pixel on the screen and there are some ways if you look in the documentation to speed that up a little bit

but you really can't use it for sort of real time like if uh you have some video game that's playing and changing the screen really or some movie or animation that's playing, it's not gonna work for that. And also this has to be an exact pixel perfect match of the image that you pass it. So if I have that seven key just even slightly uh overlapped like right there. That me oh Why do we always fight in front of groups? So if you have that just like slightly covered up, it's not going to find anything and just return none. Oh yes, so this is just a tuple of four integers. It's just the top and left, it's the um the xy coordinates of the top left corner and then the width and

the height of where it found it. But something you'll probably want to use is locate center on screen, and that'll just return the center xy coordinates of that region. So yeah, about a full second to run. What's really nice about this is then you can just pass it to Pi Auto GUI. dot click and that'll just apply it with the x and y coordinates to click on. Bam! Clicked on seven. And so this will click on seven, you know, no matter where this window is on the screen. So nice thing to have as part of your um GUI automation uh test. So yeah, what is GUI automation used for? I probably should have put this slide towards the beginning

instead of at the end. Um but s uh so Selenium is really great at automating browsers and web apps and anything beside that. So GUI automation is great for automating everything else. If you have some desktop application that you need to run You can use PyAuto GUI to control the mouse and keyboard. But even inside the browser, if you have something like Flash or an HTML5 canvas and And you need to click on somewhere inside of that and use PyAutoGUI to click on something in particular inside the browser that's not an HTML. But the best thing I think for using GUI automation is to write bots that will cheat at flash games. So last and also most ambitious live demo of this entire presentation.

Is this one coming up right next? So I have uh there's this flash game called Sushi Go Round and It it it's like a really dumb game like all Flash games are. You have customers that come in and they order sushi and you need to click on all the ingredients and serve them and They're constantly coming in and out and you need to order ingredients as they run low. And it takes a lot of practice to get good at this game, and I'm not gonna do that, so I wrote a bot. This is up on my GitHub, or you can just Google for sushi go round bot. So totally automated sushi go round player Is that

okay, yes. Let's see if this actually works. Okay, get everything set up. Sacrifice chicken behind the podium to appease the gods and let's go totally hands off it's gonna play this game for us so it uses image recognition to find where on the screen it is and I've pre-programmed the locations of all the sushi ingredients so it can detect what uh what the customer wants, just send that out. Once it starts getting low on supplies, it'll pick up the phone and start ordering all these ingredients. Um yeah, so there's it's not perfect. There's That come out. And it can c sort of get confused at times, but it it gets a high score that's way higher than anything I've ever gotten playing this game by myself.

Oh so I've I've already taken like screenshots of all the little bits of sushi that are floating above their heads, so it just checks that region for you know like oh are they ordering this like 20 giggly sushi or something, it just checks to see which image is floating in that region. And then it knows I pre-programmed all the recipes for everything. else there. Um yeah. Although I will note that the top players in the high score for this game are have like double the score of this bot, which is really impressive. Kind of pathetic also. Great, and okay, so when I want to stop this, I can just trigger the fail state by slamming the mouse up and to the left, and that'll stop this app.

But she had flash change. We've installed Selenium and PyotaGUI. The book again is automate the board and stuff with Python that you can read for free online. There's my Twitter. Tweet at me if you want to hang out. I'm in Austin until Monday. Thank you, Zerama.