RPA for Python headless (invisible) chrome usage - yes hack source file

[soorejmg]

Hi,

Does RPA for python support headless chrome? If yes, could provide the steps or documentation to do this.

Thanks
Soorej

[kensoh]

Hi Soorej, it doesn't out-of-the-box because the goal is to replicate normal human user actions on the UI frontend. Normal users will be doing stuffs directly on the webpage and not invisibly.

However, you can make a 1 line modification to an option in the source file for this tool to enable running in headless mode. See here -#109

Can you share more about your use scenario why run invisibly? Running invisibly will not be able to make use of visual automation methods to do OCR, image-matching or keyboard shortcuts on the invisible browser.

[soorejmg]

hi Kensoh,
When we were initialy introduced to tagui, our tutor said we can run it on locked system as well. SOme how we got a feel that it can run even if minimised, but then later understood that doesnot do clicks etc when chrome window is minimised.
basically our intention is to continue working on our other jobs while chrome is running in background, doing the automation process, Is it actually possible?. We take snapshot of the element etc as part of the flow

[kensoh]

Hi @soorejmg got it. Yes I think something changed along the way.

It could be Chrome browser behaviour in not actioning the click unless the window is not minimised, or could be upstream TagUI project did an enhancement for clicking using the best practice from Puppeteer by clicking on the (x,y) coordinate on the browser render window, requiring window not minimised to work.

Some ideas will be drag the Chrome browser for the automation to a corner of the screen, or another secondary screen. See if that is a usable solution for you. If apply the above hack to do headless mode, it works for the following example script -

import rpa as r
r.init()
r.url('https://google.com')
r.wait()
r.type('q','rpa')
r.click('Google Search')
r.wait(5)
r.snap('page','results.png')
r.close()

And will create the results screenshot below -

However, for some automation scenarios requiring login or 2FA it may be hard for you to automate headlessly, because you can't see the prompt for 2FA and won't be able to login.

There should be workarounds, like pausing to receive an input at the step where 2FA is needed. But it will be a much more complicated solution than running visible and hiding the window at one edge of the screen (without minimising).

[kensoh]

Btw who is your tutor? I'm keen to know if I know him or her.

[soorejmg]

It could be Chrome browser behaviour in not actioning the click unless the window is not minimised, or could be upstream TagUI project did an enhancement for clicking using the best practice from Puppeteer by clicking on the (x,y) coordinate on the browser render window, requiring window not minimised to work.

A bit unclear..
You mean, automation is possible with only window maximised?
Initially was it possible to run scripts with chrome window minimised?

Some ideas will be drag the Chrome browser for the automation to a corner of the screen, or another secondary screen. See if that is a usable solution for you. If apply the above hack to do headless mode, it works for the following example script -

You mean, even for the headless chrome there is a window required?...

However, for some automation scenarios requiring login or 2FA it may be hard for you to automate headlessly, because you can't see the prompt for 2FA and won't be able to login.

even if the 2A is part of the same web page, cannot do?.. Why is it so?..

[soorejmg]

Btw who is your tutor? I'm keen to know if I know him or her.

YU Chen Kuang is my teacher. One of the best teachers. He explains things clearly and easy to understand, If any thing I said wrong it iwill be only my mistake...

[kensoh]

The browser doesn't need to maximise, but it can't be minimised. You can leave the browser as the default window size on launch, and move it to the side of the screen or another monitor screen. To see if that helps you continue working on other things on another browser.

---

Initially yes I think a long time ago for an old Chrome and old TagUI version it is possible, but that automation method is not good and does not work as well for all websites.

---

For headless Chrome, you won't see the browser, for example the sample script above, when I run with the headless hack, the automation will run invisible without seeing any Chrome browser launched.

---

2FA can do but hard to do. If the browser is invisible and you can't see anything. How do you know when is the time to key in 2FA? Also, there is no browser window for you to type in your 2FA code. It is possible to do but hard, need to write a lot of other code to handle this.

So while it is technically possible to add an option to run headless mode for this RPA for Python tool, it is most likely going to create more headaches for users because need more effort to work through different scenarios that will not work directly in headless mode.

---

Oh great! Yes I know him since Mar 2018. He is the No #1 TagUI expert in Singapore 👍🏻👏🏻🏆

[kensoh]

👋🏻 @ck81 ! - #133 (comment)

[kensoh]

Hi @soorejmg, can you help to bring these thoughts to Chen Kuang when you see him next time? And ask for his thoughts whether good to have headless option? I feel that having the option will likely cause more headaches and confusion for users than help.

But if he feels that should be an option, then at the least I can add that option in, but perhaps unlisted in the API reference, so that only advance users know this option.

2FA can do but hard to do. If the browser is invisible and you can't see anything. How do you know when is the time to key in 2FA? Also, there is no browser window for you to type in your 2FA code. It is possible to do but hard, need to write a lot of other code to handle this.

So while it is technically possible to add an option to run headless mode for this RPA for Python tool, it is most likely going to create more headaches for users because need more effort to work through different scenarios that will not work directly in headless mode.

[ck81]

Hi @soorejmg, nice to see you here!

When we were initialy introduced to tagui, our tutor said we can run it on locked system as well. SOme how we got a feel that it can run even if minimised, but then later understood that doesnot do clicks etc when chrome window is minimised.

Yes, I still think this is one of the best features in TagUI that makes TagUI stands out from many other leading RPA tools including UiPath and Workfusion - the ability to start a robot, lock my machine, go for lunch, come back and everything done! I can also let the robot run, while simultaneously work on my other work such as outlook or Excel. Most of the other RPA tools such as UiPath or Workfusion will take control of your entire machine while the robot is running. It's also not possible to lock your machine while the robot runs. (Note: of course, this applies only to web-based apps using 100% xpath)

I think in class I only mentioned about the ability to lock the machine, and also doing other work while the TagUI robot runs. I don't think I mentioned about minimizing the robot. At work, I have 2 monitors. I like to run the robot on the 2nd monitor while I do other work on my primary monitor. I almost never minimize the robot because having written so many robots, I find that web-based apps are never 100% predictable. It could be due to some network issue, server issue, apps issue or sometimes just strange exceptions that you never encountered before - any of these will cause the robot to stop. That's why I always like to keep the robot visible on monitor 2 so that I can monitor if the robot is running ok. Any time it stops, I will investigate and keep refining the robot by adding in any new exceptions.

2FA can do but hard to do.

In real-life, I don't do 2FA. I don't even want to capture username and password. I find many users are not comfortable with this. So I always program in such a way that the robot will pause there and wait users to enter their username, password and complete the 2FA. The robot will continue automatically once they are logged in.

whether good to have headless option?

Personally I don't use headless mode. As mentioned above, I like to use the interactive and incremental approach while writing robots for web-based apps. I find it also give a lot of assurance to the users when they can see exactly what the robots are doing. The robot literally mimics the way users do the repetitive tasks, only that the robots can do it much faster. I find it helps a lot in getting the "buy-in" from the users when they can see the robots in action.

Also, I have tried headless mode on some of the websites when I started using TagUI. I find that not all websites work with headless mode. That's why after a while, I only use the chrome mode, as I find the full chrome mode most reliable and predictable.

However, interestingly, I happen to bump into this post yesterday: aisingapore/TagUI#174

As you can see in the post above, I think there are many users out there that are using the headless mode.

I also have a feel that there are quite a number of people running TagUI within Dockers in headless mode.

I feel that having the option will likely cause more headaches and confusion for users than help.
at the least I can add that option in, but perhaps unlisted in the API reference, so that only advance users know this option

@kensoh: totally agree with you on this. Think you do not need to confuse normal users with the headless option. But do keep this option open for advance users, because as mentioned above, I believe there are quite a number of TagUI users out there that are using this mode actively. But I think it's good to put a note somewhere to let people know that there is a headless option, but it's not guaranteed to work on every web page. Some will work. Some won't.

[kensoh]

Thanks @ck81 for sharing your valuable insights! Indeed No #1 TagUI expert haha

I just saw your views above after I replied your email. Sounds good, I'll be putting in an option to run in headless mode, with some caveats so that it won't confuse normal users.

My first hunch is add a r.headless() function. So that users can do a r.headless(True), before doing r.init(). This is a bit like the r.debug() function for advance users.

The other way is include the headless option as part of the r.init() function, but this can quickly get complicated when user have to provide both the chrome_browser parameter and the headless_mode parameter in order to use headless mode.

Will think more about this, and brainstorm if any other ways to do this.

What is also interesting to implement is have a burst or speed mode, where execution time can be speed up 20X by using some of the hacks here - #120 (comment)

But not straightforward to implement cuz require hacking upstream TagUI from here. See how.

This burst or speed mode is also an advance option, because many websites are not designed for superhuman speed when interacting. When automation runs super fast, will easily cause a working script to fail. Because the website is not reacting as fast as the automated sequence of actions.