mirror of
https://github.com/FlareSolverr/FlareSolverr.git
synced 2025-12-05 17:18:19 +01:00
Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0fe9958afe | ||
|
|
9f8c71131f | ||
|
|
2405c00521 | ||
|
|
ff65b7cc68 | ||
|
|
409e0844a7 | ||
|
|
368d5d4e05 | ||
|
|
c7505e3cbf | ||
|
|
5a27090abe | ||
|
|
e505ea4fe4 | ||
|
|
63b6fc53e3 | ||
|
|
8d72617219 |
11
.github/workflows/release.yml
vendored
11
.github/workflows/release.yml
vendored
@@ -19,17 +19,14 @@ jobs:
|
||||
id: github_changelog
|
||||
run: |
|
||||
changelog=$(git log $(git tag | tail -2 | head -1)..HEAD --no-merges --oneline)
|
||||
changelog="${changelog//'%'/'%25'}"
|
||||
changelog="${changelog//$'\n'/'%0A'}"
|
||||
changelog="${changelog//$'\r'/'%0D'}"
|
||||
echo "changelog=${changelog}" >> $GITHUB_ENV
|
||||
echo "changelog<<EOF" >> $GITHUB_OUTPUT
|
||||
echo "$changelog" >> $GITHUB_OUTPUT
|
||||
echo "EOF" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Create release
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
tag_name: ${{ github.ref }}
|
||||
name: ${{ github.ref }}
|
||||
body: ${{ env.changelog }}
|
||||
body: ${{ steps.github_changelog.outputs.changelog }}
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GH_PAT }}
|
||||
|
||||
|
||||
@@ -1,5 +1,14 @@
|
||||
# Changelog
|
||||
|
||||
## v3.4.6 (2025/11/29)
|
||||
* Add disable image, css, fonts option with CDP. Thanks @Ananto30
|
||||
|
||||
## v3.4.5 (2025/11/11)
|
||||
* Revert to Python v3.13
|
||||
|
||||
## v3.4.4 (2025/11/04)
|
||||
* Bump dependencies, Chromium, and some other general fixes. Thanks @flowerey
|
||||
|
||||
## v3.4.3 (2025/10/28)
|
||||
* Update proxy extension
|
||||
|
||||
|
||||
12
Dockerfile
12
Dockerfile
@@ -1,4 +1,4 @@
|
||||
FROM python:3.13-slim-bookworm as builder
|
||||
FROM python:3.13-slim-bookworm AS builder
|
||||
|
||||
# Build dummy packages to skip installing them and their dependencies
|
||||
RUN apt-get update \
|
||||
@@ -67,17 +67,17 @@ ENTRYPOINT ["/usr/bin/dumb-init", "--"]
|
||||
CMD ["/usr/local/bin/python", "-u", "/app/flaresolverr.py"]
|
||||
|
||||
# Local build
|
||||
# docker build -t ngosang/flaresolverr:3.4.3 .
|
||||
# docker run -p 8191:8191 ngosang/flaresolverr:3.4.3
|
||||
# docker build -t ngosang/flaresolverr:3.4.6 .
|
||||
# docker run -p 8191:8191 ngosang/flaresolverr:3.4.6
|
||||
|
||||
# Multi-arch build
|
||||
# docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
|
||||
# docker buildx create --use
|
||||
# docker buildx build -t ngosang/flaresolverr:3.4.3 --platform linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8 .
|
||||
# docker buildx build -t ngosang/flaresolverr:3.4.6 --platform linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8 .
|
||||
# add --push to publish in DockerHub
|
||||
|
||||
# Test multi-arch build
|
||||
# docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
|
||||
# docker buildx create --use
|
||||
# docker buildx build -t ngosang/flaresolverr:3.4.3 --platform linux/arm/v7 --load .
|
||||
# docker run -p 8191:8191 --platform linux/arm/v7 ngosang/flaresolverr:3.4.3
|
||||
# docker buildx build -t ngosang/flaresolverr:3.4.6 --platform linux/arm/v7 --load .
|
||||
# docker run -p 8191:8191 --platform linux/arm/v7 ngosang/flaresolverr:3.4.6
|
||||
|
||||
76
README.md
76
README.md
@@ -33,13 +33,14 @@ It is recommended to install using a Docker container because the project depend
|
||||
already included within the image.
|
||||
|
||||
Docker images are available in:
|
||||
* GitHub Registry => https://github.com/orgs/FlareSolverr/packages/container/package/flaresolverr
|
||||
* DockerHub => https://hub.docker.com/r/flaresolverr/flaresolverr
|
||||
|
||||
- GitHub Registry => https://github.com/orgs/FlareSolverr/packages/container/package/flaresolverr
|
||||
- DockerHub => https://hub.docker.com/r/flaresolverr/flaresolverr
|
||||
|
||||
Supported architectures are:
|
||||
|
||||
| Architecture | Tag |
|
||||
|--------------|--------------|
|
||||
| ------------ | ------------ |
|
||||
| x86 | linux/386 |
|
||||
| x86-64 | linux/amd64 |
|
||||
| ARM32 | linux/arm/v7 |
|
||||
@@ -50,6 +51,7 @@ We provide a `docker-compose.yml` configuration file. Clone this repository and
|
||||
the container.
|
||||
|
||||
If you prefer the `docker cli` execute the following command.
|
||||
|
||||
```bash
|
||||
docker run -d \
|
||||
--name=flaresolverr \
|
||||
@@ -69,28 +71,29 @@ Remember to restart the Docker daemon and the container after the update.
|
||||
> Precompiled binaries are only available for x64 architecture. For other architectures see Docker images.
|
||||
|
||||
This is the recommended way for Windows users.
|
||||
* Download the [FlareSolverr executable](https://github.com/FlareSolverr/FlareSolverr/releases) from the release's page. It is available for Windows x64 and Linux x64.
|
||||
* Execute FlareSolverr binary. In the environment variables section you can find how to change the configuration.
|
||||
|
||||
- Download the [FlareSolverr executable](https://github.com/FlareSolverr/FlareSolverr/releases) from the release's page. It is available for Windows x64 and Linux x64.
|
||||
- Execute FlareSolverr binary. In the environment variables section you can find how to change the configuration.
|
||||
|
||||
### From source code
|
||||
|
||||
> **Warning**
|
||||
> Installing from source code only works for x64 architecture. For other architectures see Docker images.
|
||||
|
||||
* Install [Python 3.13](https://www.python.org/downloads/).
|
||||
* Install [Chrome](https://www.google.com/intl/en_us/chrome/) (all OS) or [Chromium](https://www.chromium.org/getting-involved/download-chromium/) (just Linux, it doesn't work in Windows) web browser.
|
||||
* (Only in Linux) Install [Xvfb](https://en.wikipedia.org/wiki/Xvfb) package.
|
||||
* (Only in macOS) Install [XQuartz](https://www.xquartz.org/) package.
|
||||
* Clone this repository and open a shell in that path.
|
||||
* Run `pip install -r requirements.txt` command to install FlareSolverr dependencies.
|
||||
* Run `python src/flaresolverr.py` command to start FlareSolverr.
|
||||
- Install [Python 3.13](https://www.python.org/downloads/).
|
||||
- Install [Chrome](https://www.google.com/intl/en_us/chrome/) (all OS) or [Chromium](https://www.chromium.org/getting-involved/download-chromium/) (just Linux, it doesn't work in Windows) web browser.
|
||||
- (Only in Linux) Install [Xvfb](https://en.wikipedia.org/wiki/Xvfb) package.
|
||||
- (Only in macOS) Install [XQuartz](https://www.xquartz.org/) package.
|
||||
- Clone this repository and open a shell in that path.
|
||||
- Run `pip install -r requirements.txt` command to install FlareSolverr dependencies.
|
||||
- Run `python src/flaresolverr.py` command to start FlareSolverr.
|
||||
|
||||
### From source code (FreeBSD/TrueNAS CORE)
|
||||
|
||||
* Run `pkg install chromium python313 py313-pip xorg-vfbserver` command to install the required dependencies.
|
||||
* Clone this repository and open a shell in that path.
|
||||
* Run `python3.13 -m pip install -r requirements.txt` command to install FlareSolverr dependencies.
|
||||
* Run `python3.13 src/flaresolverr.py` command to start FlareSolverr.
|
||||
- Run `pkg install chromium python313 py313-pip xorg-vfbserver` command to install the required dependencies.
|
||||
- Clone this repository and open a shell in that path.
|
||||
- Run `python3.13 -m pip install -r requirements.txt` command to install FlareSolverr dependencies.
|
||||
- Run `python3.13 src/flaresolverr.py` command to start FlareSolverr.
|
||||
|
||||
### Systemd service
|
||||
|
||||
@@ -99,6 +102,7 @@ We provide an example Systemd unit file `flaresolverr.service` as reference. You
|
||||
## Usage
|
||||
|
||||
Example Bash request:
|
||||
|
||||
```bash
|
||||
curl -L -X POST 'http://localhost:8191/v1' \
|
||||
-H 'Content-Type: application/json' \
|
||||
@@ -110,6 +114,7 @@ curl -L -X POST 'http://localhost:8191/v1' \
|
||||
```
|
||||
|
||||
Example Python request:
|
||||
|
||||
```py
|
||||
import requests
|
||||
|
||||
@@ -125,6 +130,7 @@ print(response.text)
|
||||
```
|
||||
|
||||
Example PowerShell request:
|
||||
|
||||
```ps1
|
||||
$body = @{
|
||||
cmd = "request.get"
|
||||
@@ -146,7 +152,7 @@ cookies for the browser to use.
|
||||
This also speeds up the requests since it won't have to launch a new browser instance for every request.
|
||||
|
||||
| Parameter | Notes |
|
||||
|-----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| session | Optional. The session ID that you want to be assigned to the instance. If isn't set a random UUID will be assigned. |
|
||||
| proxy | Optional, default disabled. Eg: `"proxy": {"url": "http://127.0.0.1:8888"}`. You must include the proxy schema in the URL: `http://`, `socks4://` or `socks5://`. Authorization (username/password) is supported. Eg: `"proxy": {"url": "http://127.0.0.1:8888", "username": "testuser", "password": "testpass"}` |
|
||||
|
||||
@@ -160,11 +166,7 @@ Example response:
|
||||
|
||||
```json
|
||||
{
|
||||
"sessions": [
|
||||
"session_id_1",
|
||||
"session_id_2",
|
||||
"session_id_3..."
|
||||
]
|
||||
"sessions": ["session_id_1", "session_id_2", "session_id_3..."]
|
||||
}
|
||||
```
|
||||
|
||||
@@ -174,13 +176,13 @@ This will properly shutdown a browser instance and remove all files associated w
|
||||
session. When you no longer need to use a session you should make sure to close it.
|
||||
|
||||
| Parameter | Notes |
|
||||
|-----------|-----------------------------------------------|
|
||||
| --------- | --------------------------------------------- |
|
||||
| session | The session ID that you want to be destroyed. |
|
||||
|
||||
#### + `request.get`
|
||||
|
||||
| Parameter | Notes |
|
||||
|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| url | Mandatory |
|
||||
| session | Optional. Will send the request from and existing browser instance. If one is not sent it will create a temporary instance that will be destroyed immediately after the request is completed. |
|
||||
| session_ttl_minutes | Optional. FlareSolverr will automatically rotate expired sessions based on the TTL provided in minutes. |
|
||||
@@ -190,6 +192,8 @@ session. When you no longer need to use a session you should make sure to close
|
||||
| returnScreenshot | Optional, default false. Captures a screenshot of the final rendered page after all challenges and waits are completed. The screenshot is returned as a Base64-encoded PNG string in the `screenshot` field of the response. |
|
||||
| proxy | Optional, default disabled. Eg: `"proxy": {"url": "http://127.0.0.1:8888"}`. You must include the proxy schema in the URL: `http://`, `socks4://` or `socks5://`. Authorization (username/password) is not supported. (When the `session` parameter is set, the proxy is ignored; a session specific proxy can be set in `sessions.create`.) |
|
||||
| waitInSeconds | Optional, default none. Length to wait in seconds after solving the challenge, and before returning the results. Useful to allow it to load dynamic content. |
|
||||
| disableMedia | Optional, default false. When true FlareSolverr will prevent media resources (images, CSS, and fonts) from being loaded to speed up navigation. |
|
||||
| tabs_till_verify | Optional, default none. Number of times the `Tab` button is needed to be pressed to end up on the turnstile captcha, in order to verify it. After verifying the captcha, the result will be stored in the solution under `turnstile_token`. |
|
||||
|
||||
> **Warning**
|
||||
> If you want to use Cloudflare clearance cookie in your scripts, make sure you use the FlareSolverr User-Agent too. If they don't match you will see the challenge.
|
||||
@@ -216,7 +220,7 @@ Example response from running the `curl` above:
|
||||
"x-frame-options": "SAMEORIGIN",
|
||||
"set-cookie": "1P_JAR=2020-07-16-04; expires=Sat..."
|
||||
},
|
||||
"response":"<!DOCTYPE html>...",
|
||||
"response": "<!DOCTYPE html>...",
|
||||
"cookies": [
|
||||
{
|
||||
"name": "NID",
|
||||
@@ -243,7 +247,8 @@ Example response from running the `curl` above:
|
||||
"sameSite": "None"
|
||||
}
|
||||
],
|
||||
"userAgent": "Windows NT 10.0; Win64; x64) AppleWebKit/5..."
|
||||
"userAgent": "Windows NT 10.0; Win64; x64) AppleWebKit/5...",
|
||||
"turnstile_token": "03AGdBq24k3lK7JH2v8uN1T5F..."
|
||||
},
|
||||
"status": "ok",
|
||||
"message": "",
|
||||
@@ -255,18 +260,18 @@ Example response from running the `curl` above:
|
||||
|
||||
### + `request.post`
|
||||
|
||||
This is the same as `request.get` but it takes one more param:
|
||||
This works like `request.get`, with the addition of the postData parameter. Note that `tabs_till_verify` is currently supported only for GET requests and requires one extra argument.
|
||||
|
||||
| Parameter | Notes |
|
||||
|-----------|--------------------------------------------------------------------------|
|
||||
| --------- | ------------------------------------------------------------------------ |
|
||||
| postData | Must be a string with `application/x-www-form-urlencoded`. Eg: `a=b&c=d` |
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Name | Default | Notes |
|
||||
|--------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| ------------------ | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| LOG_LEVEL | info | Verbosity of the logging. Use `LOG_LEVEL=debug` for more information. |
|
||||
| LOG_FILE | none | Path to capture log to file. Example: `/config/flaresolver.log`. |
|
||||
| LOG_FILE | none | Path to capture log to file. Example: `/config/flaresolverr.log`. |
|
||||
| LOG_HTML | false | Only for debugging. If `true` all HTML that passes through the proxy will be logged to the console in `debug` level. |
|
||||
| PROXY_URL | none | URL for proxy. Will be overwritten by `request` or `sessions` proxy, if used. Example: `http://127.0.0.1:8080`. |
|
||||
| PROXY_USERNAME | none | Username for proxy. Will be overwritten by `request` or `sessions` proxy, if used. Example: `testuser`. |
|
||||
@@ -275,6 +280,7 @@ This is the same as `request.get` but it takes one more param:
|
||||
| TZ | UTC | Timezone used in the logs and the web browser. Example: `TZ=Europe/London`. |
|
||||
| LANG | none | Language used in the web browser. Example: `LANG=en_GB`. |
|
||||
| HEADLESS | true | Only for debugging. To run the web browser in headless mode or visible. |
|
||||
| DISABLE_MEDIA | false | To disable loading images, CSS, and other media in the web browser to save network bandwidth. |
|
||||
| TEST_URL | https://www.google.com | FlareSolverr makes a request on start to make sure the web browser is working. You can change that URL if it is blocked in your country. |
|
||||
| PORT | 8191 | Listening port. You don't need to change this if you are running on Docker. |
|
||||
| HOST | 0.0.0.0 | Listening interface. You don't need to change this if you are running on Docker. |
|
||||
@@ -282,15 +288,17 @@ This is the same as `request.get` but it takes one more param:
|
||||
| PROMETHEUS_PORT | 8192 | Listening port for Prometheus exporter. See the Prometheus section below. |
|
||||
|
||||
Environment variables are set differently depending on the operating system. Some examples:
|
||||
* Docker: Take a look at the Docker section in this document. Environment variables can be set in the `docker-compose.yml` file or in the Docker CLI command.
|
||||
* Linux: Run `export LOG_LEVEL=debug` and then run `flaresolverr` in the same shell.
|
||||
* Windows: Open `cmd.exe`, run `set LOG_LEVEL=debug` and then run `flaresolverr.exe` in the same shell.
|
||||
|
||||
- Docker: Take a look at the Docker section in this document. Environment variables can be set in the `docker-compose.yml` file or in the Docker CLI command.
|
||||
- Linux: Run `export LOG_LEVEL=debug` and then run `flaresolverr` in the same shell.
|
||||
- Windows: Open `cmd.exe`, run `set LOG_LEVEL=debug` and then run `flaresolverr.exe` in the same shell.
|
||||
|
||||
## Prometheus exporter
|
||||
|
||||
The Prometheus exporter for FlareSolverr is disabled by default. It can be enabled with the environment variable `PROMETHEUS_ENABLED`. If you are using Docker make sure you expose the `PROMETHEUS_PORT`.
|
||||
|
||||
Example metrics:
|
||||
|
||||
```shell
|
||||
# HELP flaresolverr_request_total Total requests with result
|
||||
# TYPE flaresolverr_request_total counter
|
||||
@@ -326,5 +334,5 @@ to the file name of one of the adapters inside the `/captcha` directory.
|
||||
|
||||
## Related projects
|
||||
|
||||
* C# implementation => https://github.com/FlareSolverr/FlareSolverrSharp
|
||||
- C# implementation => https://github.com/FlareSolverr/FlareSolverrSharp
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "flaresolverr",
|
||||
"version": "3.4.3",
|
||||
"version": "3.4.6",
|
||||
"description": "Proxy server to bypass Cloudflare protection",
|
||||
"author": "Diego Heras (ngosang / ngosang@hotmail.es)",
|
||||
"license": "MIT"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
bottle==0.13.4
|
||||
waitress==3.0.2
|
||||
selenium==4.36.0
|
||||
selenium==4.38.0
|
||||
func-timeout==4.3.5
|
||||
prometheus-client==0.23.1
|
||||
# Required by undetected_chromedriver
|
||||
@@ -9,6 +9,6 @@ certifi==2025.10.5
|
||||
websockets==15.0.1
|
||||
packaging==25.0
|
||||
# Only required for Linux and macOS
|
||||
xvfbwrapper==0.2.14; platform_system != "Windows"
|
||||
xvfbwrapper==0.2.15; platform_system != "Windows"
|
||||
# Only required for Windows
|
||||
pefile==2024.8.26; platform_system == "Windows"
|
||||
|
||||
@@ -5,7 +5,7 @@ import logging
|
||||
def logger_plugin(callback):
|
||||
"""
|
||||
Bottle plugin to use logging module
|
||||
http://bottlepy.org/docs/dev/plugindev.html
|
||||
https://bottlepy.org/docs/dev/plugindev.html
|
||||
|
||||
Wrap a Bottle request so that a log line is emitted after it's handled.
|
||||
(This decorator can be extended to take the desired logger as a param.)
|
||||
|
||||
@@ -18,7 +18,7 @@ def setup():
|
||||
def prometheus_plugin(callback):
|
||||
"""
|
||||
Bottle plugin to expose Prometheus metrics
|
||||
http://bottlepy.org/docs/dev/plugindev.html
|
||||
https://bottlepy.org/docs/dev/plugindev.html
|
||||
"""
|
||||
def wrapper(*args, **kwargs):
|
||||
actual_response = callback(*args, **kwargs)
|
||||
|
||||
@@ -25,7 +25,7 @@ def clean_files():
|
||||
|
||||
def download_chromium():
|
||||
# https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Linux_x64/
|
||||
revision = "1465706" if os.name == 'nt' else '1465706'
|
||||
revision = "1522586" if os.name == 'nt' else '1522586'
|
||||
arch = 'Win_x64' if os.name == 'nt' else 'Linux_x64'
|
||||
dl_file = 'chrome-win' if os.name == 'nt' else 'chrome-linux'
|
||||
dl_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), os.pardir, 'dist_chrome')
|
||||
|
||||
@@ -11,6 +11,7 @@ class ChallengeResolutionResultT:
|
||||
cookies: list = None
|
||||
userAgent: str = None
|
||||
screenshot: str | None = None
|
||||
turnstile_token: str = None
|
||||
|
||||
def __init__(self, _dict):
|
||||
self.__dict__.update(_dict)
|
||||
@@ -46,6 +47,10 @@ class V1RequestBase(object):
|
||||
download: bool = None # deprecated v2.0.0, not used
|
||||
returnRawHtml: bool = None # deprecated v2.0.0, not used
|
||||
waitInSeconds: int = None
|
||||
# Optional resource blocking flag (blocks images, CSS, and fonts)
|
||||
disableMedia: bool = None
|
||||
# Optional when you've got a turnstile captcha that needs to be clicked after X number of Tab presses
|
||||
tabs_till_verify : int = None
|
||||
|
||||
def __init__(self, _dict):
|
||||
self.__dict__.update(_dict)
|
||||
|
||||
@@ -81,7 +81,7 @@ if __name__ == "__main__":
|
||||
|
||||
# fix ssl certificates for compiled binaries
|
||||
# https://github.com/pyinstaller/pyinstaller/issues/7229
|
||||
# https://stackoverflow.com/questions/55736855/how-to-change-the-cafile-argument-in-the-ssl-module-in-python3
|
||||
# https://stackoverflow.com/q/55736855
|
||||
os.environ["REQUESTS_CA_BUNDLE"] = certifi.where()
|
||||
os.environ["SSL_CERT_FILE"] = certifi.where()
|
||||
|
||||
@@ -97,6 +97,20 @@ if __name__ == "__main__":
|
||||
logger_format = '%(asctime)s %(levelname)-8s %(message)s'
|
||||
if log_level == 'DEBUG':
|
||||
logger_format = '%(asctime)s %(levelname)-8s ReqId %(thread)s %(message)s'
|
||||
if log_file:
|
||||
log_file = os.path.realpath(log_file)
|
||||
log_path = os.path.dirname(log_file)
|
||||
os.makedirs(log_path, exist_ok=True)
|
||||
logging.basicConfig(
|
||||
format=logger_format,
|
||||
level=log_level,
|
||||
datefmt='%Y-%m-%d %H:%M:%S',
|
||||
handlers=[
|
||||
logging.StreamHandler(sys.stdout),
|
||||
logging.FileHandler(log_file)
|
||||
]
|
||||
)
|
||||
else:
|
||||
logging.basicConfig(
|
||||
format=logger_format,
|
||||
level=log_level,
|
||||
@@ -105,12 +119,6 @@ if __name__ == "__main__":
|
||||
logging.StreamHandler(sys.stdout)
|
||||
]
|
||||
)
|
||||
if log_file:
|
||||
log_file = os.path.realpath(log_file)
|
||||
log_path = os.path.dirname(log_file)
|
||||
os.makedirs(log_path, exist_ok=True)
|
||||
|
||||
logging.getLogger().addHandler(logging.FileHandler(log_file))
|
||||
|
||||
# disable warning traces from urllib3
|
||||
logging.getLogger('urllib3').setLevel(logging.ERROR)
|
||||
|
||||
@@ -48,6 +48,11 @@ CHALLENGE_SELECTORS = [
|
||||
# Fairlane / pararius.com
|
||||
'div.vc div.text-box h2'
|
||||
]
|
||||
|
||||
TURNSTILE_SELECTORS = [
|
||||
"input[name='cf-turnstile-response']"
|
||||
]
|
||||
|
||||
SHORT_TIMEOUT = 1
|
||||
SESSIONS_STORAGE = SessionsStorage()
|
||||
|
||||
@@ -253,12 +258,17 @@ def _resolve_challenge(req: V1RequestBase, method: str) -> ChallengeResolutionT:
|
||||
logging.debug('A used instance of webdriver has been destroyed')
|
||||
|
||||
|
||||
def click_verify(driver: WebDriver):
|
||||
def click_verify(driver: WebDriver, num_tabs: int = 1):
|
||||
try:
|
||||
logging.debug("Try to find the Cloudflare verify checkbox...")
|
||||
actions = ActionChains(driver)
|
||||
actions.pause(5).send_keys(Keys.TAB).pause(1).send_keys(Keys.SPACE).perform()
|
||||
logging.debug("Cloudflare verify checkbox found and clicked!")
|
||||
actions.pause(5)
|
||||
for _ in range(num_tabs):
|
||||
actions.send_keys(Keys.TAB).pause(0.1)
|
||||
actions.pause(1)
|
||||
actions.send_keys(Keys.SPACE).perform()
|
||||
|
||||
logging.debug(f"Cloudflare verify checkbox clicked after {num_tabs} tabs!")
|
||||
except Exception:
|
||||
logging.debug("Cloudflare verify checkbox not found on the page.")
|
||||
finally:
|
||||
@@ -281,19 +291,90 @@ def click_verify(driver: WebDriver):
|
||||
|
||||
time.sleep(2)
|
||||
|
||||
def _get_turnstile_token(driver: WebDriver, tabs: int):
|
||||
token_input = driver.find_element(By.CSS_SELECTOR, "input[name='cf-turnstile-response']")
|
||||
current_value = token_input.get_attribute("value")
|
||||
while True:
|
||||
click_verify(driver, num_tabs=tabs)
|
||||
turnstile_token = token_input.get_attribute("value")
|
||||
if turnstile_token:
|
||||
if turnstile_token != current_value:
|
||||
logging.info(f"Turnstile token: {turnstile_token}")
|
||||
return turnstile_token
|
||||
logging.debug(f"Failed to extract token possibly click failed")
|
||||
|
||||
# reset focus
|
||||
driver.execute_script("""
|
||||
let el = document.createElement('button');
|
||||
el.style.position='fixed';
|
||||
el.style.top='0';
|
||||
el.style.left='0';
|
||||
document.body.prepend(el);
|
||||
el.focus();
|
||||
""")
|
||||
time.sleep(1)
|
||||
|
||||
def _resolve_turnstile_captcha(req: V1RequestBase, driver: WebDriver):
|
||||
turnstile_token = None
|
||||
if req.tabs_till_verify is not None:
|
||||
logging.debug(f'Navigating to... {req.url} in order to pass the turnstile challenge')
|
||||
driver.get(req.url)
|
||||
|
||||
turnstile_challenge_found = False
|
||||
for selector in TURNSTILE_SELECTORS:
|
||||
found_elements = driver.find_elements(By.CSS_SELECTOR, selector)
|
||||
if len(found_elements) > 0:
|
||||
turnstile_challenge_found = True
|
||||
logging.info("Turnstile challenge detected. Selector found: " + selector)
|
||||
break
|
||||
if turnstile_challenge_found:
|
||||
turnstile_token = _get_turnstile_token(driver=driver, tabs=req.tabs_till_verify)
|
||||
else:
|
||||
logging.debug(f'Turnstile challenge not found')
|
||||
return turnstile_token
|
||||
|
||||
def _evil_logic(req: V1RequestBase, driver: WebDriver, method: str) -> ChallengeResolutionT:
|
||||
res = ChallengeResolutionT({})
|
||||
res.status = STATUS_OK
|
||||
res.message = ""
|
||||
|
||||
# optionally block resources like images/css/fonts using CDP
|
||||
disable_media = utils.get_config_disable_media()
|
||||
if req.disableMedia is not None:
|
||||
disable_media = req.disableMedia
|
||||
if disable_media:
|
||||
block_urls = [
|
||||
# Images
|
||||
"*.png", "*.jpg", "*.jpeg", "*.gif", "*.webp", "*.bmp", "*.svg", "*.ico",
|
||||
"*.PNG", "*.JPG", "*.JPEG", "*.GIF", "*.WEBP", "*.BMP", "*.SVG", "*.ICO",
|
||||
"*.tiff", "*.tif", "*.jpe", "*.apng", "*.avif", "*.heic", "*.heif",
|
||||
"*.TIFF", "*.TIF", "*.JPE", "*.APNG", "*.AVIF", "*.HEIC", "*.HEIF",
|
||||
# Stylesheets
|
||||
"*.css",
|
||||
"*.CSS",
|
||||
# Fonts
|
||||
"*.woff", "*.woff2", "*.ttf", "*.otf", "*.eot",
|
||||
"*.WOFF", "*.WOFF2", "*.TTF", "*.OTF", "*.EOT"
|
||||
]
|
||||
try:
|
||||
logging.debug("Network.setBlockedURLs: %s", block_urls)
|
||||
driver.execute_cdp_cmd("Network.enable", {})
|
||||
driver.execute_cdp_cmd("Network.setBlockedURLs", {"urls": block_urls})
|
||||
except Exception:
|
||||
# if CDP commands are not available or fail, ignore and continue
|
||||
logging.debug("Network.setBlockedURLs failed or unsupported on this webdriver")
|
||||
|
||||
# navigate to the page
|
||||
logging.debug(f'Navigating to... {req.url}')
|
||||
if method == 'POST':
|
||||
logging.debug(f"Navigating to... {req.url}")
|
||||
turnstile_token = None
|
||||
|
||||
if method == "POST":
|
||||
_post_request(req, driver)
|
||||
else:
|
||||
if req.tabs_till_verify is None:
|
||||
driver.get(req.url)
|
||||
else:
|
||||
turnstile_token = _resolve_turnstile_captcha(req, driver)
|
||||
|
||||
# set cookies if required
|
||||
if req.cookies is not None and len(req.cookies) > 0:
|
||||
@@ -387,6 +468,7 @@ def _evil_logic(req: V1RequestBase, driver: WebDriver, method: str) -> Challenge
|
||||
challenge_res.status = 200 # todo: fix, selenium not provides this info
|
||||
challenge_res.cookies = driver.get_cookies()
|
||||
challenge_res.userAgent = utils.get_user_agent(driver)
|
||||
challenge_res.turnstile_token = turnstile_token
|
||||
|
||||
if not req.returnOnlyCookies:
|
||||
challenge_res.headers = {} # todo: fix, selenium not provides this info
|
||||
|
||||
31
src/tests.py
31
src/tests.py
@@ -21,11 +21,11 @@ class TestFlareSolverr(unittest.TestCase):
|
||||
proxy_socks_url = "socks5://127.0.0.1:1080"
|
||||
google_url = "https://www.google.com"
|
||||
post_url = "https://httpbin.org/post"
|
||||
cloudflare_url = "https://nowsecure.nl"
|
||||
cloudflare_url = "https://nowsecure.nl/"
|
||||
cloudflare_url_2 = "https://idope.se/torrent-list/harry/"
|
||||
ddos_guard_url = "https://anidex.info/"
|
||||
ddos_guard_url = "https://www.litres.ru/"
|
||||
fairlane_url = "https://www.pararius.com/apartments/amsterdam"
|
||||
custom_cloudflare_url = "https://www.muziekfabriek.org"
|
||||
custom_cloudflare_url = "https://www.muziekfabriek.org/"
|
||||
cloudflare_blocked_url = "https://cpasbiens3.fr/index.php?do=search&subaction=search"
|
||||
|
||||
app = TestApp(flaresolverr.app)
|
||||
@@ -92,6 +92,29 @@ class TestFlareSolverr(unittest.TestCase):
|
||||
self.assertGreater(len(solution.cookies), 0)
|
||||
self.assertIn("Chrome/", solution.userAgent)
|
||||
|
||||
def test_v1_endpoint_request_get_disable_resources(self):
|
||||
res = self.app.post_json("/v1", {
|
||||
"cmd": "request.get",
|
||||
"url": self.google_url,
|
||||
"disableMedia": True
|
||||
})
|
||||
self.assertEqual(res.status_code, 200)
|
||||
|
||||
body = V1ResponseBase(res.json)
|
||||
self.assertEqual(STATUS_OK, body.status)
|
||||
self.assertEqual("Challenge not detected!", body.message)
|
||||
self.assertGreater(body.startTimestamp, 10000)
|
||||
self.assertGreaterEqual(body.endTimestamp, body.startTimestamp)
|
||||
self.assertEqual(utils.get_flaresolverr_version(), body.version)
|
||||
|
||||
solution = body.solution
|
||||
self.assertIn(self.google_url, solution.url)
|
||||
self.assertEqual(solution.status, 200)
|
||||
self.assertIs(len(solution.headers), 0)
|
||||
self.assertIn("<title>Google</title>", solution.response)
|
||||
self.assertGreater(len(solution.cookies), 0)
|
||||
self.assertIn("Chrome/", solution.userAgent)
|
||||
|
||||
def test_v1_endpoint_request_get_cloudflare_js_1(self):
|
||||
res = self.app.post_json('/v1', {
|
||||
"cmd": "request.get",
|
||||
@@ -162,7 +185,7 @@ class TestFlareSolverr(unittest.TestCase):
|
||||
self.assertIn(self.ddos_guard_url, solution.url)
|
||||
self.assertEqual(solution.status, 200)
|
||||
self.assertIs(len(solution.headers), 0)
|
||||
self.assertIn("<title>AniDex</title>", solution.response)
|
||||
self.assertIn("<title>Литрес", solution.response)
|
||||
self.assertGreater(len(solution.cookies), 0)
|
||||
self.assertIn("Chrome/", solution.userAgent)
|
||||
|
||||
|
||||
@@ -28,6 +28,10 @@ def get_config_headless() -> bool:
|
||||
return os.environ.get('HEADLESS', 'true').lower() == 'true'
|
||||
|
||||
|
||||
def get_config_disable_media() -> bool:
|
||||
return os.environ.get('DISABLE_MEDIA', 'false').lower() == 'true'
|
||||
|
||||
|
||||
def get_flaresolverr_version() -> str:
|
||||
global FLARESOLVERR_VERSION
|
||||
if FLARESOLVERR_VERSION is not None:
|
||||
|
||||
@@ -1 +1 @@
|
||||
WebTest==3.0.6
|
||||
WebTest==3.0.7
|
||||
|
||||
Reference in New Issue
Block a user