Cannot Add Prefs as Experimental Option #524

metalpacoforeal · 2022-03-02T15:03:00Z

Hi there!

I'm trying to download a PDF file using uc version 3.1.3 and I'm unable to add an experimental option to download PDF files externally.

This code works fine with selenium's webdriver, but not with UC.

Here's the code snippet...

`
from selenium import webdriver
import undetected_chromedriver as uc

if name == 'main':
# Set up a headless chromedriver
options = webdriver.ChromeOptions()

# Set pdf download preference
prefs = {"plugins.always_open_pdf_externally": True}
options.add_experimental_option("prefs", prefs)

# Initialize driver
driver = uc.Chrome(options=options, version_main=98)
driver.get('www.barchart.com')

`

And here's the traceback:

Traceback (most recent call last): File "/Users/jrodriguez/Library/Application Support/JetBrains/PyCharm2021.3/scratches/scratch_3.py", line 13, in <module> driver = uc.Chrome(options=options, version_main=98) File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/undetected_chromedriver/__init__.py", line 369, in __init__ super(Chrome, self).__init__( File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__ super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog", File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 93, in __init__ RemoteWebDriver.__init__( File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 268, in __init__ self.start_session(capabilities, browser_profile) File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/undetected_chromedriver/__init__.py", line 552, in start_session super(selenium.webdriver.chrome.webdriver.WebDriver, self).start_session( File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 359, in start_session response = self.execute(Command.NEW_SESSION, parameters) File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute self.error_handler.check_response(response) File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: cannot parse capability: goog:chromeOptions from invalid argument: unrecognized chrome option: prefs Stacktrace: 0 chromedriver 0x0000000100796ee9 chromedriver + 5013225 1 chromedriver 0x00000001007221d3 chromedriver + 4534739 2 chromedriver 0x00000001002f8a68 chromedriver + 170600 3 chromedriver 0x00000001003112b7 chromedriver + 271031 4 chromedriver 0x00000001003157c5 chromedriver + 288709 5 chromedriver 0x000000010030de9b chromedriver + 257691 6 chromedriver 0x000000010034f1c8 chromedriver + 524744 7 chromedriver 0x000000010034ec6d chromedriver + 523373 8 chromedriver 0x000000010035030a chromedriver + 529162 9 chromedriver 0x000000010034a6d3 chromedriver + 505555 10 chromedriver 0x000000010032076e chromedriver + 333678 11 chromedriver 0x0000000100321745 chromedriver + 337733 12 chromedriver 0x0000000100752efe chromedriver + 4734718 13 chromedriver 0x000000010076ca19 chromedriver + 4839961 14 chromedriver 0x00000001007721c8 chromedriver + 4862408 15 chromedriver 0x000000010076d3aa chromedriver + 4842410 16 chromedriver 0x0000000100747a01 chromedriver + 4688385 17 chromedriver 0x0000000100788538 chromedriver + 4953400 18 chromedriver 0x00000001007886c1 chromedriver + 4953793 19 chromedriver 0x000000010079e225 chromedriver + 5042725 20 libsystem_pthread.dylib 0x00007fff2071a8fc _pthread_start + 224 21 libsystem_pthread.dylib 0x00007fff20716443 thread_start + 15

The text was updated successfully, but these errors were encountered:

morjun · 2022-03-03T12:44:17Z

struggling with the same issue here

sebdelsol · 2022-03-03T18:13:18Z

Chrome is started before its driver and misses all desired capabilities pushed by the driver. Here's a solution that works for me:

import json
import os
import tempfile
from functools import reduce

import undetected_chromedriver as webdriver


class ChromeWithPrefs(webdriver.Chrome):
    def __init__(self, *args, options=None, **kwargs):
        if options:
            self._handle_prefs(options)

        super().__init__(*args, options=options, **kwargs)

        # remove the user_data_dir when quitting
        self.keep_user_data_dir = False

    @staticmethod
    def _handle_prefs(options):
        if prefs := options.experimental_options.get("prefs"):
            # turn a (dotted key, value) into a proper nested dict
            def undot_key(key, value):
                if "." in key:
                    key, rest = key.split(".", 1)
                    value = undot_key(rest, value)
                return {key: value}

            # undot prefs dict keys
            undot_prefs = reduce(
                lambda d1, d2: {**d1, **d2},  # merge dicts
                (undot_key(key, value) for key, value in prefs.items()),
            )

            # create an user_data_dir and add its path to the options
            user_data_dir = os.path.normpath(tempfile.mkdtemp())
            options.add_argument(f"--user-data-dir={user_data_dir}")

            # create the preferences json file in its default directory
            default_dir = os.path.join(user_data_dir, "Default")
            os.mkdir(default_dir)

            prefs_file = os.path.join(default_dir, "Preferences")
            with open(prefs_file, encoding="latin1", mode="w") as f:
                json.dump(undot_prefs, f)

            # pylint: disable=protected-access
            # remove the experimental_options to avoid an error
            del options._experimental_options["prefs"]


if __name__ == "__main__":
    prefs = {
        "profile.default_content_setting_values.images": 2,
        # "download.default_directory": "d:/temp",
        # "plugins.always_open_pdf_externally": True,
    }
    options = webdriver.ChromeOptions()
    options.add_experimental_option("prefs", prefs)

    # use the derived Chrome class that handles prefs
    driver = ChromeWithPrefs(options=options)

The idea is to create a "Default/Preferences" json file from the ChromeOptions().experimental_options["prefs"] key and remove it to avoid the error. A way to do it is to create and use a derived class of Chrome that handles this.

Disclaimer:

I don't know Chrome and Selenium enough to be sure it's the best way to do it and that there's no caveats.
I don't know if this would work with the others Chrome specific capabilities. Hence I limited it to the "prefs" key.

metalpacoforeal · 2022-03-03T20:49:12Z

Thanks, @sebdelsol! I'll give that a shot and report back. 🙂

metalpacoforeal · 2022-03-03T21:35:04Z

So @sebdelsol, your code works for adding prefs (yay!)... but for some reason, now I can't download the PDF due to "insufficient permissions." Much closer to my script doing what it's supposed to do, though, so thank you! 🙂

sebdelsol · 2022-03-03T21:42:49Z

Make sure your Downloads folder has proper permissions. Btw you can now change it with prefs = {"download.default_directory": "your path"}

sebdelsol · 2022-03-03T21:54:29Z

This might be a fix for some issues : #517, #495, #493, #477, #452, #338, #309, #260, #213, #211 and #194

metalpacoforeal · 2022-03-04T14:36:25Z

It works! You were absolutely right about the permissions on the folder, @sebdelsol. Thanks so much!

bhargav-sarvaria · 2022-03-24T15:43:24Z

Chrome is started before its driver and misses all desired capabilities pushed by the driver. Here's a solution that works for me:

import json
import os
import tempfile
from functools import reduce

import undetected_chromedriver as webdriver


class ChromeWithPrefs(webdriver.Chrome):
    def __init__(self, *args, options=None, **kwargs):
        if options:
            self._handle_prefs(options)

        super().__init__(*args, options=options, **kwargs)

        # remove the user_data_dir when quitting
        self.keep_user_data_dir = False

    @staticmethod
    def _handle_prefs(options):
        if prefs := options.experimental_options.get("prefs"):
            # turn a (dotted key, value) into a proper nested dict
            def undot_key(key, value):
                if "." in key:
                    key, rest = key.split(".", 1)
                    value = undot_key(rest, value)
                return {key: value}

            # undot prefs dict keys
            undot_prefs = reduce(
                lambda d1, d2: {**d1, **d2},  # merge dicts
                (undot_key(key, value) for key, value in prefs.items()),
            )

            # create an user_data_dir and add its path to the options
            user_data_dir = os.path.normpath(tempfile.mkdtemp())
            options.add_argument(f"--user-data-dir={user_data_dir}")

            # create the preferences json file in its default directory
            default_dir = os.path.join(user_data_dir, "Default")
            os.mkdir(default_dir)

            prefs_file = os.path.join(default_dir, "Preferences")
            with open(prefs_file, encoding="latin1", mode="w") as f:
                json.dump(undot_prefs, f)

            # pylint: disable=protected-access
            # remove the experimental_options to avoid an error
            del options._experimental_options["prefs"]


if __name__ == "__main__":
    prefs = {
        "profile.default_content_setting_values.images": 2,
        # "download.default_directory": "d:/temp",
        # "plugins.always_open_pdf_externally": True,
    }
    options = webdriver.ChromeOptions()
    options.add_experimental_option("prefs", prefs)

    # use the derived Chrome class that handles prefs
    driver = ChromeWithPrefs(options=options)

The idea is to create a "Default/Preferences" json file from the ChromeOptions().experimental_options["prefs"] key and remove it to avoid the error. A way to do it is to create and use a derived class of Chrome that handles this.

Disclaimer:

I don't know Chrome and Selenium enough to be sure it's the best way to do it and that there's no caveats.
I don't know if this would work with the others Chrome specific capabilities. Hence I limited it to the "prefs" key.

THANK YOU SO MUCH! I WANT TO GIVE YOU A BIG HUG!

negotiable-pride · 2022-04-13T12:45:49Z

This doesn't work for mobileEmulation though - even after replacing all the pref parts with mobileEmulation. Can anyone help?

sebdelsol · 2022-04-13T13:49:14Z

As stated it works only with prefs experimental options because they are actually stored in the Profile/Default/Preferences file.
I don't know whether this other experimental option you need is stored in Chrome profile and/or it's a mere capability... To find out you should begin with a simple Selenium driver and check how it's handled with a debugger and check thoroughly the Selenium repo.

EDIT : By quickly searching on the Selenium repo, it seems you only need to add the correct capabilities. Please check the Selenium documentation how to do that.

EDIT2 : @xerobia-test, I'm only a contributor. I needed prefs experimental options and suggested a pull request to do that and it was merged. If you have further needs, feel free to do the same.

EDIT3 : Those option are not that easy to implement since you can't rely on your driver forwarding them with a mere request when launching the browser. In undetected-chromdriver case, Chrome is spawned before the driver (that's half the trick how it remains undetectable) hence restricting further requests for experimental options. I haven't found any reliable accessible documentations for how each were actually implemented, so it was for me a trial and error process (at least faster than finding my way in Chromium code).

negotiable-pride · 2022-04-13T13:51:20Z

I see. Are there any plans to get the add_experimental_options option working again in the near future?

negotiable-pride · 2022-04-13T14:49:20Z

That's ok. Thanks for your efforts! I've been messing around with it for hours now - I think I'll just settle for now.

tmb5cg · 2022-04-29T16:57:28Z

I don't understand how/where you guys learn everything you know but @sebdelsol thank you.

@xerobia-test I am also settling. I am trying to post instagram stories, which only works by emulating mobile via chrome dev tools for some reason. I'm going to search for a chrome extension that maybe can emulate mobile, but if I understood what sebdelsol said, it seems like mobile emulation is an entirely separate operation that is done somewhere special in between the driver/sending requests - who knows

kaliiiiiiiiii · 2022-11-08T07:10:58Z

This doesn't work for mobileEmulation though - even after replacing all the pref parts with mobileEmulation. Can anyone help?

have a look at Selenium-Profiles

Repo still in progress // working on btw

sebdelsol · 2022-11-08T11:10:22Z

The prefs experimental options are not correctly handled since those could be nested dict and can't be properly merged with a mere dict.update or the ** operator in a dict comprehension as used in the code above.
Please check #869 for a workaround till the author merge the relevant pull requests.

Bohdanloz · 2022-12-25T14:43:43Z

Hi there, i try to use your code in my project - automation of google meet joining and it still asks me for micro and video camera access. How can I fix this?

from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import undetected_chromedriver as uc
import time
import os
import json
import tempfile
from functools import reduce

class ChromeWithPrefs(uc.Chrome):
    def __init__(self, *args, options=None, **kwargs):
        if options:
            self._handle_prefs(options)

        super().__init__(*args, options=options, **kwargs)

        # remove the user_data_dir when quitting
        self.keep_user_data_dir = False

    @staticmethod
    def _handle_prefs(options):
        if prefs := options.experimental_options.get("prefs"):
            # turn a (dotted key, value) into a proper nested dict
            def undot_key(key, value):
                if "." in key:
                    key, rest = key.split(".", 1)
                    value = undot_key(rest, value)
                return {key: value}

            # undot prefs dict keys
            undot_prefs = reduce(
                lambda d1, d2: {**d1, **d2},  # merge dicts
                (undot_key(key, value) for key, value in prefs.items()),
            )

            # create an user_data_dir and add its path to the options
            user_data_dir = os.path.normpath(tempfile.mkdtemp())
            options.add_argument(f"--user-data-dir={user_data_dir}")

            # create the preferences json file in its default directory
            default_dir = os.path.join(user_data_dir, "Default")
            os.mkdir(default_dir)

            prefs_file = os.path.join(default_dir, "Preferences")
            with open(prefs_file, encoding="latin1", mode="w") as f:
                json.dump(undot_prefs, f)

            # pylint: disable=protected-access
            # remove the experimental_options to avoid an error
            del options._experimental_options["prefs"]

prefs = {
	"profile.default_content_setting_values.media_stream_mic": 2,
	"profile.default_content_setting_values.media_stream_camera": 2,
	"profile.default_content_setting_values.geolocation": 2,
	"profile.default_content_setting_values.notifications": 2
}
#user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'
opt = uc.ChromeOptions()
opt.add_argument('--no-sandbox')
#opt.add_argument('--headless=chrome')
opt.add_argument("--disable-extensions")
opt.add_argument("--disable-infobars")
opt.add_argument("user_agent=DN")
#opt.add_argument(f'user-agent={user_agent}')
opt.add_argument("--proxy-server='direct://'")
opt.add_argument("--proxy-bypass-list=*")
opt.add_argument('--disable-gpu')
opt.add_argument("--window-size=1920x1080")
opt.add_argument("--allow-insecure-localhost")
opt.add_argument('--ignore-certificate-errors')
opt.add_argument('--allow-running-insecure-content')
opt.add_argument('--disable-dev-shm-usage')
opt.add_experimental_option("prefs", prefs)
opt.add_argument('--start-maximized')

def join_google(link, mail_address, password):
	driver = ChromeWithPrefs(options=opt)

Here is how my code looks like. Can you please say me what I am doing wrong? Cause all the time it asks me for micro and video access and I cannot close this popup message cause it's covered with another one.

asapsmc · 2024-01-08T15:21:34Z

I'm having a problem with the (above) @sebdelsol workaround.
When I don't have to set experimental options (thus I don't have the need to use this workaround) my code works perfectly (opens a specific chrome profile) with this

options = webdriver.ChromeOptions()
options.headless=False
options.add_argument("--log-level=DEBUG")
options.add_argument("--user-data-dir=/Users/machine/Library/Application Support/Google/Chrome")
driver = uc.Chrome(options=options)

However, when I use the above workaround code, when I try to do this

prefs = {
    "profile.default_content_setting_values.images": 2,
     # "download.default_directory": "d:/temp",
     # "plugins.always_open_pdf_externally": True,
}
options2 = webdriver.ChromeOptions()
options2.add_experimental_option("prefs", prefs)
driver2 = ChromeWithPrefs(options=options2)

I get The chromedriver version (114.0.5735.90) detected in PATH at /usr/local/bin/chromedriver might not be compatible with the detected chrome version (120.0.6099.199); currently, chromedriver 120.0.6099.109 is recommended for chrome 120.*, so it is advised to delete the driver in PATH and retry.

So, it seems that with the "regular" webdriver.ChromeOptions(), Selenium (I'm using Selenium 4.16 and undetected_chromedriver 3.5.3) finds the correct ChromeDriver version, otherwise it doesn't.

Can anyone help me identify where's the issue? Why can't this approach find the correct driver automatically?

HakanBi · 2024-04-08T10:14:35Z

Add options.add_argument('--handle_prefs') to handle it

asapsmc · 2024-04-24T11:26:17Z

Do you mean options.add_argument('--handle_prefs=prefs')? I have tried options2.add_argument('--handle_prefs') options2.add_experimental_option("prefs", prefs) driver2 = uc.Chrome(options=options2), but I get an error:

Message: invalid argument: cannot parse capability: goog:chromeOptions
from invalid argument: unrecognized chrome option: prefs

ReMiOS · 2024-12-03T20:46:11Z

i've also got the error (tried selenium 4.8/4.9/4.10/4.27.1 and various Chrome/Chromium browsers):
from invalid argument: unrecognized chrome option: prefs

Using the ChromeWithPrefs did not solve anything for me

to investigate the options set in the driver use:
print( driver.options.__dict__ )
......'_experimental_options': {}......

I've found changing "options=" to "chromeOptions=" made the error disappear, but it had no effect to the options
browser = undetected_chromedriver.Chrome( chromeOptions = options )

I've ended up with parsing the Preferences file with json and inserting the keys and values before launching the driver.

metalpacoforeal closed this as completed Mar 4, 2022

This was referenced Mar 4, 2022

experimental_option prefs not working #260

Open

specify download path #493

Open

How to change chrome download location? #517

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot Add Prefs as Experimental Option #524

Cannot Add Prefs as Experimental Option #524

metalpacoforeal commented Mar 2, 2022 •

edited

Loading

morjun commented Mar 3, 2022

sebdelsol commented Mar 3, 2022 •

edited

Loading

metalpacoforeal commented Mar 3, 2022

metalpacoforeal commented Mar 3, 2022

sebdelsol commented Mar 3, 2022 •

edited

Loading

sebdelsol commented Mar 3, 2022 •

edited

Loading

metalpacoforeal commented Mar 4, 2022

bhargav-sarvaria commented Mar 24, 2022

negotiable-pride commented Apr 13, 2022

sebdelsol commented Apr 13, 2022 •

edited

Loading

negotiable-pride commented Apr 13, 2022

negotiable-pride commented Apr 13, 2022 •

edited

Loading

tmb5cg commented Apr 29, 2022

kaliiiiiiiiii commented Nov 8, 2022

sebdelsol commented Nov 8, 2022 •

edited

Loading

Bohdanloz commented Dec 25, 2022 •

edited

Loading

asapsmc commented Jan 8, 2024 •

edited

Loading

HakanBi commented Apr 8, 2024

asapsmc commented Apr 24, 2024 •

edited

Loading

ReMiOS commented Dec 3, 2024 •

edited

Loading

Cannot Add Prefs as Experimental Option #524

Cannot Add Prefs as Experimental Option #524

Comments

metalpacoforeal commented Mar 2, 2022 • edited Loading

morjun commented Mar 3, 2022

sebdelsol commented Mar 3, 2022 • edited Loading

metalpacoforeal commented Mar 3, 2022

metalpacoforeal commented Mar 3, 2022

sebdelsol commented Mar 3, 2022 • edited Loading

sebdelsol commented Mar 3, 2022 • edited Loading

metalpacoforeal commented Mar 4, 2022

bhargav-sarvaria commented Mar 24, 2022

negotiable-pride commented Apr 13, 2022

sebdelsol commented Apr 13, 2022 • edited Loading

negotiable-pride commented Apr 13, 2022

negotiable-pride commented Apr 13, 2022 • edited Loading

tmb5cg commented Apr 29, 2022

kaliiiiiiiiii commented Nov 8, 2022

sebdelsol commented Nov 8, 2022 • edited Loading

Bohdanloz commented Dec 25, 2022 • edited Loading

asapsmc commented Jan 8, 2024 • edited Loading

HakanBi commented Apr 8, 2024

asapsmc commented Apr 24, 2024 • edited Loading

ReMiOS commented Dec 3, 2024 • edited Loading

metalpacoforeal commented Mar 2, 2022 •

edited

Loading

sebdelsol commented Mar 3, 2022 •

edited

Loading

sebdelsol commented Mar 3, 2022 •

edited

Loading

sebdelsol commented Mar 3, 2022 •

edited

Loading

sebdelsol commented Apr 13, 2022 •

edited

Loading

negotiable-pride commented Apr 13, 2022 •

edited

Loading

sebdelsol commented Nov 8, 2022 •

edited

Loading

Bohdanloz commented Dec 25, 2022 •

edited

Loading

asapsmc commented Jan 8, 2024 •

edited

Loading

asapsmc commented Apr 24, 2024 •

edited

Loading

ReMiOS commented Dec 3, 2024 •

edited

Loading