Fix detection due to cdc_props removal. #986

[lukect]

#986

[fabifont]

Hi, is there a way to trigger cdc_props removal when opening an url in a new tab? Sometimes a click can open a new tab with a pre-loaded url. You can simulate that with:

driver.execute_script("window.open('https://www.google.com','_blank');")

[lukect]

Hi, is there a way to trigger cdc_props removal when opening an url in a new tab? Sometimes a click can open a new tab with a pre-loaded url. You can simulate that with:

driver.execute_script("window.open('https://www.google.com','_blank');")

You have found an edge-case which hasn't been covered yet in this project. I will update my issue (#986) and my PR (#993) to address this when we know more about it.

From my initial findings, it seems that the cdc_props are only added by ChromeDriver after you Chrome.switch_to the new window/tab. You could try manually triggering the props removal using Chrome._hook_remove_cdc_props(self._get_cdc_props()) immediately after switching the ChromeDriver to the new tab/window (Chrome.switch_to). However, there might be a short time period, between when the ChromeDriver adds the cdc_props and when we remove them, where the antibots can detect the cdc_props. This possible race condition needs to be investigated & tested.

We also need to check if cdc_props get added when switching to an iframe.

[fabifont]

Hi, is there a way to trigger cdc_props removal when opening an url in a new tab? Sometimes a click can open a new tab with a pre-loaded url. You can simulate that with:

driver.execute_script("window.open('https://www.google.com','_blank');")

You have found an edge-case which hasn't been covered yet in this project. I will update my issue (#986) and my PR (#993) to address this when we know more about it.

From my initial findings, it seems that the cdc_props are only added by ChromeDriver after you Chrome.switch_to the new window/tab. You could try manually triggering the props removal using Chrome._hook_remove_cdc_props(self._get_cdc_props()) immediately after switching the ChromeDriver to the new tab/window (Chrome.switch_to). However, there might be a short time period, between when the ChromeDriver adds the cdc_props and when we remove them, where the antibots can detect the cdc_props. This possible race condition needs to be investigated & tested.

We also need to check if cdc_props get added when switching to an iframe.

Yes, actually I am using the workaround you described above, but as you noticed there's always a short time period when antibot can detect you.

Another dirty solution could be turning off your network (with network namespaces maybe), check if the click has opened a new tab, switch to it, remove cdc_props, turn on the network and refresh the page.

[lukect]

Hi, is there a way to trigger cdc_props removal when opening an url in a new tab? Sometimes a click can open a new tab with a pre-loaded url. You can simulate that with:

driver.execute_script("window.open('https://www.google.com','_blank');")

You have found an edge-case which hasn't been covered yet in this project. I will update my issue (#986) and my PR (#993) to address this when we know more about it.
From my initial findings, it seems that the cdc_props are only added by ChromeDriver after you Chrome.switch_to the new window/tab. You could try manually triggering the props removal using Chrome._hook_remove_cdc_props(self._get_cdc_props()) immediately after switching the ChromeDriver to the new tab/window (Chrome.switch_to). However, there might be a short time period, between when the ChromeDriver adds the cdc_props and when we remove them, where the antibots can detect the cdc_props. This possible race condition needs to be investigated & tested.
We also need to check if cdc_props get added when switching to an iframe.

Yes, actually I am using the workaround you described above, but as you noticed there's always a short time period when antibot can detect you.

Another dirty solution could be turning off your network (with network namespaces maybe), check if the click has opened a new tab, switch to it, remove cdc_props, turn on the network and refresh the page.

Simplest quick fix (if the website isn't ensuring you still have the old window open):

def prevent_popups():
    prevent_popups_script: str = """
        window.open = (url, target, windowFeatures) => {
            window.location.href = url;
        }
    """
    driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument',
                           {'source': prevent_popups_script})
    driver.execute_script(prevent_popups_script)

Simply override the window.open/tab/pop-up behaviour. Run this function once on every native window object (at startup and for every window you Chrome.switch_to for the first time).

I have a more comprehensive patch in the works.

[mdonova33]

Hi, is there a way to trigger cdc_props removal when opening an url in a new tab? Sometimes a click can open a new tab with a pre-loaded url. You can simulate that with:

driver.execute_script("window.open('https://www.google.com','_blank');")

You have found an edge-case which hasn't been covered yet in this project. I will update my issue (#986) and my PR (#993) to address this when we know more about it.

From my initial findings, it seems that the cdc_props are only added by ChromeDriver after you Chrome.switch_to the new window/tab. You could try manually triggering the props removal using Chrome._hook_remove_cdc_props(self._get_cdc_props()) immediately after switching the ChromeDriver to the new tab/window (Chrome.switch_to). However, there might be a short time period, between when the ChromeDriver adds the cdc_props and when we remove them, where the antibots can detect the cdc_props. This possible race condition needs to be investigated & tested.

We also need to check if cdc_props get added when switching to an iframe.

Yes, actually I am using the workaround you described above, but as you noticed there's always a short time period when antibot can detect you.

Another dirty solution could be turning off your network (with network namespaces maybe), check if the click has opened a new tab, switch to it, remove cdc_props, turn on the network and refresh the page.

cdc_props likely get added when switching to an iFrame. I have an edge case right now where switching to an iFrame gets me detected.

[lukect]

Hi, is there a way to trigger cdc_props removal when opening an url in a new tab? Sometimes a click can open a new tab with a pre-loaded url. You can simulate that with:

driver.execute_script("window.open('https://www.google.com','_blank');")

You have found an edge-case which hasn't been covered yet in this project. I will update my issue (#986) and my PR (#993) to address this when we know more about it.

From my initial findings, it seems that the cdc_props are only added by ChromeDriver after you Chrome.switch_to the new window/tab. You could try manually triggering the props removal using Chrome._hook_remove_cdc_props(self._get_cdc_props()) immediately after switching the ChromeDriver to the new tab/window (Chrome.switch_to). However, there might be a short time period, between when the ChromeDriver adds the cdc_props and when we remove them, where the antibots can detect the cdc_props. This possible race condition needs to be investigated & tested.

We also need to check if cdc_props get added when switching to an iframe.

Yes, actually I am using the workaround you described above, but as you noticed there's always a short time period when antibot can detect you.
Another dirty solution could be turning off your network (with network namespaces maybe), check if the click has opened a new tab, switch to it, remove cdc_props, turn on the network and refresh the page.

cdc_props likely get added when switching to an iFrame. I have an edge case right now where switching to an iFrame gets me detected.

Does it work using my current PR? pip install git+https://github.com/lukect/undetected-chromedriver.git

I'm passing all the cdc_props tests in my own testing setup, even within one iframe, using my fork. I no longer believe iframe needs to be handled separately, because the Page.addScriptToEvaluateOnNewDocument CDP command seems to apply to all sub-frames/windows of the parent window.

[mdonova33]

I will be able to check a little later, but what’s interesting in my case, is that all of the iFrames seems to be blocked, even if they aren’t specifically switched to using selenium. For example, opening a normal browser window will result in the iFrame loading correctly. But opening a window using UndetectedChomedriver and navigating to the same iFrame (even without explicitly switching to it using Selenium) results in a bot detected, and sometimes a “refused to connect” error, from the iFrame.

[lukect]

I will close this PR, because after reading the Chromium/ChromeDriver source code; I realized there is a far better way to solve this by improving the ChromeDriver binary executable patcher.py. The improvement would make this PR (and some code currently in the project) redundant.

JavaScript hack

Hi, is there a way to trigger cdc_props removal when opening an url in a new tab? Sometimes a click can open a new tab with a pre-loaded url. You can simulate that with:

driver.execute_script("window.open('https://www.google.com','_blank');")

I have a more comprehensive patch in the works.

I did make a lot of progress on a JavaScript hack to override the window.open behavior. However, I decided to abandon it as it was becoming too complex and improving patcher became the better option due to excess caveats. I leave my hack/research below incase it is of future use:

def _handle_cdc_props(driver: uc.Chrome) -> bool:
    # True = cdc_props now removed | False = There were no cdc_props to remove
    cdc_props = driver.execute_script("""
        const j=[];
        for(const p in window){
           if(/^[a-z]{3}_[a-z]{22}_.*/i.test(p)){
               j.push(p);
               delete window[p];
           }
        }
        return j;
    """)
    if len(cdc_props) > 0:
        props_js_array = '[' + ','.join('"' + p + '"' for p in cdc_props) + ']'
        driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument',
                               {'source': props_js_array + '.forEach(k=>delete window[k]);'})
        return True
    else:
        return False


unlock_func_name: str = str().join(random.choices(population=string.ascii_letters, k=random.randint(16, 32)))
protect_popups_script_js: str = """
    const old_window_open = window.open;
    window.open = (url, target, windowFeatures) => {
        old_window_open("javascript:{" +
            "window.location.href = 'about:blank';" +
            "window.""" + unlock_func_name + """ = () => {" +
                "window.location.href = '" + url.replaceAll('\\\\', '\\\\\\\\') + "';" +
                "delete window['unlock_new_window'];" +
            "};" +
        "}", target, windowFeatures);
    }
"""

protected_windows: set[str] = set()


def _protect_current_window_windowOpen(driver: uc.Chrome) -> bool:
    # True = Window now protected | False = Window was already protected
    current_window: str = driver.current_window_handle
    if current_window in protected_windows:
        return False  # Already protected
    _handle_cdc_props(driver)
    driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {'source': protect_popups_script_js})
    driver.execute_script(protect_popups_script_js)
    protected_windows.add(current_window)


def unlock_new_window(driver: uc.Chrome) -> bool:  # Call after switching to tab/window opened by a page.
    # True = Window now unlocked, | False = Window wasn't locked
    if not driver.execute_script(f'return typeof window.{unlock_func_name} === "function";'):
        return False  # Not locked
    _protect_current_window_windowOpen(driver)
    driver.execute_script(f'window.{unlock_func_name}();')
    return True


def protect_current_window(driver: uc.Chrome) -> bool:  # Call at startup and after Chrome.switch_to a new tab/window.
    # True = protected, False = Already protected / wasn't necessary.
    needed_removal: bool = _handle_cdc_props(driver)
    needed_protection: bool = _protect_current_window_windowOpen(driver)
    return needed_removal or needed_protection

Call protect_current_window(driver) after starting the driver and before using it. Also call protect_current_window(driver) if you opened a new tab/window yourself (such as Chrome.window_new()). When you switch to a tab/window opened by a page, call unlock_new_window(driver).

How it works

We are preventing a new tab/window from being opened by a page, opening a new one ourselves, then protecting that window (removing the cdc_props) before unlocking the new window (overriding the window.open behavior to protect the new window and only then going to the URL the original page intended us to go to in a new tab/window).

Caveats

You might still get detected when new pop-ups/windows/tabs get opened by a website, because:

1. This only protects new new pop-ups/windows/tabs opened by JavaScript (window.open). This doesn't protect hyperlinks (<a>) or <form> submission with <form target="_blank" action="url" method="post">. These two behaviors are also overridable (add document.addEventListener('click', (e) => {/*handle logic*/}); and document.addEventListener('submit', (e) => {/*handle logic*/}); to Page.addScriptToEvaluateOnNewDocument (so it also loads on every iframe)). However, I decided not to continue with overriding these due to a lot of logic to handle while there would still the other caveats allowing for us to be detected...
2. This doesn't handle document.referrer in JavaScript. [Detectable by antibot scripts, if those scripts check this variable.] This is easily overridable (Object.defineProperty(document, "referrer", {get: () => {return "https://example.com/";}});), however we would need to detect which window opened this new window and properly handle the referrer-policy of the old window.
3. This doesn't handle the referer HTTP header. [Inconsistent/Abnormal/Missing HTTP headers could be detected by the Web Application Firewall (WAF) / Proxy.] This header cannot even be fixed due to a Chromium bug where the protection of the referer header from JavaScript manipulation overreaches into preventing manipulation from CDP.
4. This doesn't handle the Sec-Fetch-Site HTTP header correctly. [Inconsistent/Abnormal/Missing HTTP headers could be detected by the Web Application Firewall (WAF) / Proxy.]
5. This doesn't handle the Sec-Fetch-User HTTP header correctly. [Inconsistent/Abnormal/Missing HTTP headers could be detected by the Web Application Firewall (WAF) / Proxy.]

[lukect]

Use this 100% stable fix:
#1010

@fabifont @mdonova33 @mdmintz

[mdmintz]

@lukect The #1010 update worked on some sites, eg: https://nowsecure.nl/#relax, https://pixelscan.net/, and https://fingerprint.com/products/bot-detection, but not on others, such as Google Login.
(Your previous solution #993 worked on all 4 of the above, so #1010 appears to be a step in the wrong direction, so far.)

[ultrafunkamsterdam]

We have a click_safe() method for this...

[ultrafunkamsterdam]

And driver.tab_new()....
And driver.window_new()

There is even a method which removes cdc Js vars.

def _hook_remove_cdc_props(self)
    self.execute_cdp_cmd(
            "Page.addScriptToEvaluateOnNewDocument",
            {
                "source": """
                    let objectToInspect = window,
                        result = [];
                    while(objectToInspect !== null)
                    { result = result.concat(Object.getOwnPropertyNames(objectToInspect));
                      objectToInspect = Object.getPrototypeOf(objectToInspect); }
                    result.forEach(p => p.match(/.+_.+_(Array|Promise|Symbol)/ig)
                                        &&delete window[p]&&console.log('removed',p))
                   

[lukect]

We have a click_safe() method for this...

This has a race condition (the click could occur before ChromeDriver IPC disconnects).

And driver.tab_new()....
And driver.window_new()

We are talking about a page's script opening a new window without user action. window.open('url', '_blank'); isn't protected.
An antibot script could also add a <a href="https://example.com/antibot" target="_blank">F</a> HTMLElement and then .click() on it to detect the user while ChromeDriver is still connected. This is especially a problem if the new window requires interaction, like a payment pop-up. I can create a POC to demonstrate easy detection of undetected-chromedriver if you require.

def _hook_remove_cdc_props(self)
    self.execute_cdp_cmd(
            "Page.addScriptToEvaluateOnNewDocument",
            {
                "source": """
                    let objectToInspect = window,
                        result = [];
                    while(objectToInspect !== null)
                    { result = result.concat(Object.getOwnPropertyNames(objectToInspect));
                      objectToInspect = Object.getPrototypeOf(objectToInspect); }
                    result.forEach(p => p.match(/.+_.+_(Array|Promise|Symbol)/ig)
                                        &&delete window[p]&&console.log('removed',p))
                   

This implementation is flawed: #986.
Please accept #1010 to fix everything I just mentioned.