Hotfix for Linux build - some Chrome files no longer exist (#1071)

thanks @kimboslice99

CHANGELOG.md

@@ -1,6 +1,56 @@
 # Changelog
 
-## v3.3.3 (2023/08/03)
+## v3.3.14-hotfix (2024/02/17)
+
+* Hotfix for Linux build - some Chrome files no longer exist
+
+## v3.3.14 (2024/02/17)
+
+* Update Chrome downloads. Thanks @opemvbs
+
+## v3.3.13 (2024/01/07)
+
+* Fix too many open files error
+
+## v3.3.12 (2023/12/15)
+
+* Fix looping challenges and invalid cookies
+
+## v3.3.11 (2023/12/11)
+
+* Update UC 3.5.4 & Selenium 4.15.2. Thanks @txtsd
+
+## v3.3.10 (2023/11/14)
+
+* Add LANG ENV - resolves issues with YGGtorrent
+
+## v3.3.9 (2023/11/13)
+
+* Fix for Docker build, capture TypeError
+
+## v3.3.8 (2023/11/13)
+
+* Fix headless=true for Chrome 117+. Thanks @NabiKAZ
+* Support running Chrome 119 from source. Thanks @koleg and @Chris7X
+* Fix "OSError: [WinError 6] The handle is invalid" on exit. Thanks @enesgorkemgenc
+
+## v3.3.7 (2023/11/05)
+
+* Bump to rebuild. Thanks @JoachimDorchies
+
+## v3.3.6 (2023/09/15)
+
+* Update checkbox selector, again
+
+## v3.3.5 (2023/09/13)
+
+* Change checkbox selector, support languages other than English
+
+## v3.3.4 (2023/09/02)
+
+* Update checkbox selector
+
+## v3.3.3 (2023/08/31)
 
 * Update undetected_chromedriver to v3.5.3
 

Dockerfile

@@ -62,17 +62,17 @@ ENTRYPOINT ["/usr/bin/dumb-init", "--"]
 CMD ["/usr/local/bin/python", "-u", "/app/flaresolverr.py"]
 
 # Local build
-# docker build -t ngosang/flaresolverr:3.3.3 .
+# docker build -t ngosang/flaresolverr:3.3.14-hotfix .
-# docker run -p 8191:8191 ngosang/flaresolverr:3.3.3
+# docker run -p 8191:8191 ngosang/flaresolverr:3.3.14-hotfix
 
 # Multi-arch build
 # docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
 # docker buildx create --use
-# docker buildx build -t ngosang/flaresolverr:3.3.3 --platform linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8 .
+# docker buildx build -t ngosang/flaresolverr:3.3.14-hotfix --platform linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8 .
 #   add --push to publish in DockerHub
 
 # Test multi-arch build
 # docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
 # docker buildx create --use
-# docker buildx build -t ngosang/flaresolverr:3.3.3 --platform linux/arm/v7 --load .
+# docker buildx build -t ngosang/flaresolverr:3.3.14-hotfix --platform linux/arm/v7 --load .
-# docker run -p 8191:8191 --platform linux/arm/v7 ngosang/flaresolverr:3.3.3
+# docker run -p 8191:8191 --platform linux/arm/v7 ngosang/flaresolverr:3.3.14-hotfix

README.md

@@ -89,7 +89,7 @@ We provide an example Systemd unit file `flaresolverr.service` as reference. You
 
 ## Usage
 
-Example request:
+Example Bash request:
 ```bash
 curl -L -X POST 'http://localhost:8191/v1' \
 -H 'Content-Type: application/json' \
@@ -100,6 +100,32 @@ curl -L -X POST 'http://localhost:8191/v1' \
 }'
 ```
 
+Example Python request:
+```py
+import requests
+
+url = "http://localhost:8191/v1"
+headers = {"Content-Type": "application/json"}
+data = {
+    "cmd": "request.get",
+    "url": "http://www.google.com/",
+    "maxTimeout": 60000
+}
+response = requests.post(url, headers=headers, json=data)
+print(response.text)
+```
+
+Example PowerShell request:
+```ps1
+$body = @{
+    cmd = "request.get"
+    url = "http://www.google.com/"
+    maxTimeout = 60000
+} | ConvertTo-Json
+
+irm -UseBasicParsing 'http://localhost:8191/v1' -Headers @{"Content-Type"="application/json"} -Method Post -Body $body
+```
+
 ### Commands
 
 #### + `sessions.create`
@@ -113,7 +139,7 @@ This also speeds up the requests since it won't have to launch a new browser ins
 | Parameter | Notes                                                                                                                                                                                                                                                                                                            |
 |-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | session   | Optional. The session ID that you want to be assigned to the instance. If isn't set a random UUID will be assigned.                                                                                                                                                                                              |
-| proxy     | Optional, default disabled. Eg: `"proxy": {"url": "http://127.0.0.1:8888"}`. You must include the proxy schema in the URL: `http://`, `socks4://` or `socks5://`. Authorization (username/password) is supported. Eg: `"proxy": {"url": "http://127.0.0.1:8888", username": "testuser", "password": "testpass"}` |
+| proxy     | Optional, default disabled. Eg: `"proxy": {"url": "http://127.0.0.1:8888"}`. You must include the proxy schema in the URL: `http://`, `socks4://` or `socks5://`. Authorization (username/password) is supported. Eg: `"proxy": {"url": "http://127.0.0.1:8888", "username": "testuser", "password": "testpass"}` |
 
 #### + `sessions.list`
 
@@ -232,6 +258,7 @@ This is the same as `request.get` but it takes one more param:
 | LOG_HTML           | false                  | Only for debugging. If `true` all HTML that passes through the proxy will be logged to the console in `debug` level.                                          |
 | CAPTCHA_SOLVER     | none                   | Captcha solving method. It is used when a captcha is encountered. See the Captcha Solvers section.                                                            |
 | TZ                 | UTC                    | Timezone used in the logs and the web browser. Example: `TZ=Europe/London`.                                                                                   |
+| LANG               | none                   | Language used in the web browser. Example: `LANG=en_GB`.                                                                                   |
 | HEADLESS           | true                   | Only for debugging. To run the web browser in headless mode or visible.                                                                                       |
 | BROWSER_TIMEOUT    | 40000                  | If you are experiencing errors/timeouts because your system is slow, you can try to increase this value. Remember to increase the `maxTimeout` parameter too. |
 | TEST_URL           | https://www.google.com | FlareSolverr makes a request on start to make sure the web browser is working. You can change that URL if it is blocked in your country.                      |

package.json

@@ -1,6 +1,6 @@
 {
   "name": "flaresolverr",
-  "version": "3.3.3",
+  "version": "3.3.14-hotfix",
   "description": "Proxy server to bypass Cloudflare protection",
   "author": "Diego Heras (ngosang / ngosang@hotmail.es)",
   "license": "MIT"

requirements.txt

@@ -1,6 +1,6 @@
 bottle==0.12.25
 waitress==2.1.2
-selenium==4.11.2
+selenium==4.15.2
 func-timeout==4.3.5
 prometheus-client==0.17.1
 # required by undetected_chromedriver

src/build_package.py

@@ -25,7 +25,7 @@ def clean_files():
 
 def download_chromium():
     # https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Linux_x64/
-    revision = "1140001" if os.name == 'nt' else '1140000'
+    revision = "1250005" if os.name == 'nt' else '1250006'
     arch = 'Win_x64' if os.name == 'nt' else 'Linux_x64'
     dl_file = 'chrome-win' if os.name == 'nt' else 'chrome-linux'
     dl_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), os.pardir, 'dist_chrome')
@@ -59,8 +59,7 @@ def download_chromium():
         # Give executable permissions for *nix
         # file * | grep executable | cut -d: -f1
         print("Giving executable permissions...")
-        execs = ['chrome', 'chrome_crashpad_handler', 'chrome_sandbox', 'chrome-wrapper', 'nacl_helper',
+        execs = ['chrome', 'chrome_crashpad_handler', 'chrome_sandbox', 'chrome-wrapper', 'xdg-mime', 'xdg-settings']
-                 'nacl_helper_bootstrap', 'nacl_irt_x86_64.nexe', 'xdg-mime', 'xdg-settings']
         for exec_file in execs:
             exec_path = os.path.join(chrome_path, exec_file)
             os.chmod(exec_path, 0o755)

src/flaresolverr_service.py

@@ -252,11 +252,11 @@ def _resolve_challenge(req: V1RequestBase, method: str) -> ChallengeResolutionT:
 def click_verify(driver: WebDriver):
     try:
         logging.debug("Try to find the Cloudflare verify checkbox...")
-        iframe = driver.find_element(By.XPATH, "//iframe[@title='Widget containing a Cloudflare security challenge']")
+        iframe = driver.find_element(By.XPATH, "//iframe[starts-with(@id, 'cf-chl-widget-')]")
         driver.switch_to.frame(iframe)
         checkbox = driver.find_element(
             by=By.XPATH,
-            value='//*[@id="cf-stage"]//label[@class="ctp-checkbox-label"]/input',
+            value='//*[@id="challenge-stage"]/div/label/input',
         )
         if checkbox:
             actions = ActionChains(driver)
@@ -294,6 +294,8 @@ def _evil_logic(req: V1RequestBase, driver: WebDriver, method: str) -> Challenge
 
     # navigate to the page
     logging.debug(f'Navigating to... {req.url}')
+    driver.get(req.url)
+    driver.start_session() # required to bypass Cloudflare
     if method == 'POST':
         _post_request(req, driver)
     else:

src/undetected_chromedriver/__init__.py

@@ -17,7 +17,7 @@ by UltrafunkAmsterdam (https://github.com/ultrafunkamsterdam)
 from __future__ import annotations
 
 
-__version__ = "3.5.3"
+__version__ = "3.5.4"
 
 import json
 import logging
@@ -396,7 +396,7 @@ class Chrome(selenium.webdriver.chrome.webdriver.WebDriver):
         if no_sandbox:
             options.arguments.extend(["--no-sandbox", "--test-type"])
 
-        if headless or options.headless:
+        if headless or getattr(options, 'headless', None):
             #workaround until a better checking is found
             try:
                 v_main = int(self.patcher.version_main) if self.patcher.version_main else 108
@@ -464,11 +464,9 @@ class Chrome(selenium.webdriver.chrome.webdriver.WebDriver):
             )
             self.browser_pid = browser.pid
 
-        # Fix for Chrome 115
+
-        # https://github.com/seleniumbase/SeleniumBase/pull/1967
         service = selenium.webdriver.chromium.service.ChromiumService(
-            executable_path=self.patcher.executable_path,
+            self.patcher.executable_path
-            service_args=["--disable-build-check"]
         )
 
         super(Chrome, self).__init__(
@@ -493,7 +491,7 @@ class Chrome(selenium.webdriver.chrome.webdriver.WebDriver):
         else:
             self._web_element_cls = WebElement
 
-        if options.headless:
+        if headless or getattr(options, 'headless', None):
             self._configure_headless()
 
     def _configure_headless(self):
@@ -813,7 +811,11 @@ class Chrome(selenium.webdriver.chrome.webdriver.WebDriver):
                 else:
                     logger.debug("successfully removed %s" % self.user_data_dir)
                     break
-                time.sleep(0.1)
+
+                try:
+                    time.sleep(0.1)
+                except OSError:
+                    pass
 
         # dereference patcher, so patcher can start cleaning up as well.
         # this must come last, otherwise it will throw 'in use' errors

src/undetected_chromedriver/dprocess.py

@@ -41,6 +41,7 @@ def start_detached(executable, *args):
     # close pipes
     writer.close()
     reader.close()
+    process.close()
 
     return pid
 

src/undetected_chromedriver/patcher.py

@@ -62,7 +62,17 @@ class Patcher(object):
         prefix = "undetected"
         self.user_multi_procs = user_multi_procs
 
-        self.is_old_chromedriver = version_main and version_main <= 114
+        try:
+            # Try to convert version_main into an integer
+            version_main_int = int(version_main)
+            # check if version_main_int is less than or equal to e.g 114
+            self.is_old_chromedriver = version_main and version_main_int <= 114
+        except (ValueError,TypeError):
+            # If the conversion fails, print an error message
+            print("version_main cannot be converted to an integer")
+            # Set self.is_old_chromedriver to False if the conversion fails
+            self.is_old_chromedriver = False
+
         # Needs to be called before self.exe_name is accessed
         self._set_platform_name()
 
@@ -280,7 +290,7 @@ class Patcher(object):
             download_url = "%s/%s/%s" % (self.url_repo, self.version_full.vstring, zip_name)
         else:
             zip_name = zip_name.replace("_", "-", 1)
-            download_url = "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/%s/%s/%s"
+            download_url = "https://storage.googleapis.com/chrome-for-testing-public/%s/%s/%s"
             download_url %= (self.version_full.vstring, self.platform_name, zip_name)
 
         logger.debug("downloading from %s" % download_url)

src/utils.py

@@ -113,7 +113,7 @@ def create_proxy_extension(proxy: dict) -> str:
 
 
 def get_webdriver(proxy: dict = None) -> WebDriver:
-    global PATCHED_DRIVER_PATH
+    global PATCHED_DRIVER_PATH, USER_AGENT
     logging.debug('Launching web browser...')
 
     # undetected_chromedriver
@@ -136,6 +136,14 @@ def get_webdriver(proxy: dict = None) -> WebDriver:
     # https://peter.sh/experiments/chromium-command-line-switches/#use-gl
     options.add_argument('--use-gl=swiftshader')
 
+    language = os.environ.get('LANG', None)
+    if language is not None:
+        options.add_argument('--lang=%s' % language)
+
+    # Fix for Chrome 117 | https://github.com/FlareSolverr/FlareSolverr/issues/910
+    if USER_AGENT is not None:
+        options.add_argument('--user-agent=%s' % USER_AGENT)
+
     proxy_extension_dir = None
     if proxy and all(key in proxy for key in ['url', 'username', 'password']):
         proxy_extension_dir = create_proxy_extension(proxy)
@@ -145,7 +153,7 @@ def get_webdriver(proxy: dict = None) -> WebDriver:
         logging.debug("Using webdriver proxy: %s", proxy_url)
         options.add_argument('--proxy-server=%s' % proxy_url)
 
-    # note: headless mode is detected (options.headless = True)
+    # note: headless mode is detected (headless = True)
     # we launch the browser in head-full mode with the window hidden
     windows_headless = False
     if get_config_headless():
@@ -153,6 +161,8 @@ def get_webdriver(proxy: dict = None) -> WebDriver:
             windows_headless = True
         else:
             start_xvfb_display()
+    # For normal headless mode:
+    # options.add_argument('--headless')
 
     # if we are inside the Docker container, we avoid downloading the driver
     driver_exe_path = None
@@ -162,10 +172,6 @@ def get_webdriver(proxy: dict = None) -> WebDriver:
         driver_exe_path = "/app/chromedriver"
     else:
         version_main = get_chrome_major_version()
-        # Fix for Chrome 115
-        # https://github.com/seleniumbase/SeleniumBase/pull/1967
-        if int(version_main) > 114:
-            version_main = 114
         if PATCHED_DRIVER_PATH is not None:
             driver_exe_path = PATCHED_DRIVER_PATH
 
@@ -295,6 +301,8 @@ def get_user_agent(driver=None) -> str:
         if driver is None:
             driver = get_webdriver()
         USER_AGENT = driver.execute_script("return navigator.userAgent")
+        # Fix for Chrome 117 | https://github.com/FlareSolverr/FlareSolverr/issues/910
+        USER_AGENT = re.sub('HEADLESS', '', USER_AGENT, flags=re.IGNORECASE)
         return USER_AGENT
     except Exception as e:
         raise Exception("Error getting browser User-Agent. " + str(e))