mirror of
https://github.com/FlareSolverr/FlareSolverr.git
synced 2025-12-05 17:18:19 +01:00
Compare commits
21 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
36226b34c1 | ||
|
|
606d84f7c0 | ||
|
|
62eb363575 | ||
|
|
345d27dd5a | ||
|
|
3b9fd0aa6a | ||
|
|
93041779fb | ||
|
|
3dbb4e65d6 | ||
|
|
23dd8f8725 | ||
|
|
9ab7ab1371 | ||
|
|
cf7e4f8749 | ||
|
|
e8328adb90 | ||
|
|
843f588859 | ||
|
|
f8462c86f2 | ||
|
|
4bc083896b | ||
|
|
c9f2d6e954 | ||
|
|
177578d5d8 | ||
|
|
efcab83f6e | ||
|
|
51b7bc3b92 | ||
|
|
e5be265026 | ||
|
|
aed54e0bb3 | ||
|
|
5046f60914 |
32
.github/ISSUE_TEMPLATE.md
vendored
Normal file
32
.github/ISSUE_TEMPLATE.md
vendored
Normal file
@@ -0,0 +1,32 @@
|
||||
**Please use the search bar** at the top of the page and make sure you are not creating an already submitted issue.
|
||||
Check closed issues as well, because your issue may have already been fixed.
|
||||
|
||||
### How to enable debug and html traces
|
||||
|
||||
[Follow the instructions from this wiki page](https://github.com/FlareSolverr/FlareSolverr/wiki/How-to-enable-debug-and-html-trace)
|
||||
|
||||
### Environment
|
||||
|
||||
* **FlareSolverr version**:
|
||||
* **Last working FlareSolverr version**:
|
||||
* **Operating system**:
|
||||
* **Are you using Docker**: [yes/no]
|
||||
* **FlareSolverr User-Agent (see log traces or / endpoint)**:
|
||||
* **Are you using a proxy or VPN?** [yes/no]
|
||||
* **Are you using Captcha Solver:** [yes/no]
|
||||
* **If using captcha solver, which one:**
|
||||
* **URL to test this issue:**
|
||||
|
||||
### Description
|
||||
|
||||
[List steps to reproduce the error and details on what happens and what you expected to happen]
|
||||
|
||||
### Logged Error Messages
|
||||
|
||||
[Place any relevant error messages you noticed from the logs here.]
|
||||
|
||||
[Make sure you attach the full logs with your personal information removed in case we need more information]
|
||||
|
||||
### Screenshots
|
||||
|
||||
[Place any screenshots of the issue here if needed]
|
||||
63
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
63
.github/ISSUE_TEMPLATE/bug_report.yml
vendored
@@ -1,63 +0,0 @@
|
||||
name: Bug report
|
||||
description: Create a report of your issue
|
||||
body:
|
||||
- type: checkboxes
|
||||
attributes:
|
||||
label: Have you checked our README?
|
||||
description: Please check the <a href="https://github.com/FlareSolverr/FlareSolverr/blob/master/README.md">README</a>.
|
||||
options:
|
||||
- label: I have checked the README
|
||||
required: true
|
||||
- type: checkboxes
|
||||
attributes:
|
||||
label: Is there already an issue for your problem?
|
||||
description: Please make sure you are not creating an already submitted <a href="https://github.com/FlareSolverr/FlareSolverr/issues">Issue</a>. Check closed issues as well, because your issue may have already been fixed.
|
||||
options:
|
||||
- label: I have checked older issues, open and closed
|
||||
required: true
|
||||
- type: checkboxes
|
||||
attributes:
|
||||
label: Have you checked the discussions?
|
||||
description: Please read our <a href="https://github.com/FlareSolverr/FlareSolverr/discussions">Discussions</a> before submitting your issue, some wider problems may be dealt with there.
|
||||
options:
|
||||
- label: I have read the Discussions
|
||||
required: true
|
||||
- type: textarea
|
||||
attributes:
|
||||
label: Environment
|
||||
description: Please provide the details of the system FlareSolverr is running on.
|
||||
value: |
|
||||
- FlareSolverr version:
|
||||
- Last working FlareSolverr version:
|
||||
- Operating system:
|
||||
- Are you using Docker: [yes/no]
|
||||
- FlareSolverr User-Agent (see log traces or / endpoint):
|
||||
- Are you using a proxy or VPN: [yes/no]
|
||||
- Are you using Captcha Solver: [yes/no]
|
||||
- If using captcha solver, which one:
|
||||
- URL to test this issue:
|
||||
render: markdown
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
attributes:
|
||||
label: Description
|
||||
description: List steps to reproduce the error and details on what happens and what you expected to happen.
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
attributes:
|
||||
label: Logged Error Messages
|
||||
description: |
|
||||
Place any relevant error messages you noticed from the logs here.
|
||||
Make sure you attach the full logs with your personal information removed in case we need more information.
|
||||
If you wish to provide debug logs, follow the instructions from this <a href="https://github.com/FlareSolverr/FlareSolverr/wiki/How-to-enable-debug-and-html-trace">wiki page</a>.
|
||||
render: text
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
attributes:
|
||||
label: Screenshots
|
||||
description: Place any screenshots of the issue here if needed
|
||||
validations:
|
||||
required: false
|
||||
8
.github/ISSUE_TEMPLATE/config.yml
vendored
8
.github/ISSUE_TEMPLATE/config.yml
vendored
@@ -1,8 +0,0 @@
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: Requesting new features or changes
|
||||
url: https://github.com/FlareSolverr/FlareSolverr/discussions
|
||||
about: Please create a new discussion topic, grouped under "Ideas".
|
||||
- name: Asking questions
|
||||
url: https://github.com/FlareSolverr/FlareSolverr/discussions
|
||||
about: Please create a new discussion topic, grouped under "Q&A".
|
||||
41
.github/workflows/autotag.yml
vendored
41
.github/workflows/autotag.yml
vendored
@@ -1,20 +1,21 @@
|
||||
name: autotag
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- "master"
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
-
|
||||
name: Checkout
|
||||
uses: actions/checkout@v2
|
||||
-
|
||||
name: Auto Tag
|
||||
uses: Klemensas/action-autotag@stable
|
||||
with:
|
||||
GITHUB_TOKEN: "${{ secrets.GH_PAT }}"
|
||||
tag_prefix: "v"
|
||||
# todo: enable in the first release
|
||||
#name: autotag
|
||||
#
|
||||
#on:
|
||||
# push:
|
||||
# branches:
|
||||
# - "master"
|
||||
#
|
||||
#jobs:
|
||||
# build:
|
||||
# runs-on: ubuntu-latest
|
||||
# steps:
|
||||
# -
|
||||
# name: Checkout
|
||||
# uses: actions/checkout@v2
|
||||
# -
|
||||
# name: Auto Tag
|
||||
# uses: Klemensas/action-autotag@stable
|
||||
# with:
|
||||
# GITHUB_TOKEN: "${{ secrets.GH_PAT }}"
|
||||
# tag_prefix: "v"
|
||||
|
||||
2
.github/workflows/release-docker.yml
vendored
2
.github/workflows/release-docker.yml
vendored
@@ -47,7 +47,7 @@ jobs:
|
||||
with:
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
platforms: linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8
|
||||
platforms: linux/amd64,linux/arm/v7,linux/arm64
|
||||
push: ${{ github.event_name != 'pull_request' }}
|
||||
tags: ${{ steps.docker_meta.outputs.tags }}
|
||||
labels: ${{ steps.docker_meta.outputs.labels }}
|
||||
|
||||
245
CHANGELOG.md
245
CHANGELOG.md
@@ -1,245 +0,0 @@
|
||||
# Changelog
|
||||
|
||||
## v3.0.0 (2023/01/04)
|
||||
|
||||
* This is the first release of FlareSolverr v3. There are some breaking changes
|
||||
* Docker images for linux/386, linux/amd64, linux/arm/v7 and linux/arm64/v8
|
||||
* Replaced Firefox with Chrome
|
||||
* Replaced NodeJS / Typescript with Python
|
||||
* Replaced Puppeter with Selenium
|
||||
* No binaries for Linux / Windows. You have to use the Docker image or install from Source code
|
||||
* No proxy support
|
||||
* No session support
|
||||
|
||||
## v2.2.10 (2022/10/22)
|
||||
|
||||
* Detect DDoS-Guard through title content
|
||||
|
||||
## v2.2.9 (2022/09/25)
|
||||
|
||||
* Detect Cloudflare Access Denied
|
||||
* Commit the complete changelog
|
||||
|
||||
## v2.2.8 (2022/09/17)
|
||||
|
||||
* Remove 30 s delay and clean legacy code
|
||||
|
||||
## v2.2.7 (2022/09/12)
|
||||
|
||||
* Temporary fix: add 30s delay
|
||||
* Update README.md
|
||||
|
||||
## v2.2.6 (2022/07/31)
|
||||
|
||||
* Fix Cloudflare detection in POST requests
|
||||
|
||||
## v2.2.5 (2022/07/30)
|
||||
|
||||
* Update GitHub actions to build executables with NodeJs 16
|
||||
* Update Cloudflare selectors and add HTML samples
|
||||
* Install Firefox 94 instead of the latest Nightly
|
||||
* Update dependencies
|
||||
* Upgrade Puppeteer (#396)
|
||||
|
||||
## v2.2.4 (2022/04/17)
|
||||
|
||||
* Detect DDoS-Guard challenge
|
||||
|
||||
## v2.2.3 (2022/04/16)
|
||||
|
||||
* Fix 2000 ms navigation timeout
|
||||
* Update README.md (libseccomp2 package in Debian)
|
||||
* Update README.md (clarify proxy parameter) (#307)
|
||||
* Update NPM dependencies
|
||||
* Disable Cloudflare ban detection
|
||||
|
||||
## v2.2.2 (2022/03/19)
|
||||
|
||||
* Fix ban detection. Resolves #330 (#336)
|
||||
|
||||
## v2.2.1 (2022/02/06)
|
||||
|
||||
* Fix max timeout error in some pages
|
||||
* Avoid crashing in NodeJS 17 due to Unhandled promise rejection
|
||||
* Improve proxy validation and debug traces
|
||||
* Remove @types/puppeteer dependency
|
||||
|
||||
## v2.2.0 (2022/01/31)
|
||||
|
||||
* Increase default BROWSER_TIMEOUT=40000 (40 seconds)
|
||||
* Fix Puppeter deprecation warnings
|
||||
* Update base Docker image Alpine 3.15 / NodeJS 16
|
||||
* Build precompiled binaries with NodeJS 16
|
||||
* Update Puppeter and other dependencies
|
||||
* Add support for Custom CloudFlare challenge
|
||||
* Add support for DDoS-GUARD challenge
|
||||
|
||||
## v2.1.0 (2021/12/12)
|
||||
|
||||
* Add aarch64 to user agents to be replaced (#248)
|
||||
* Fix SOCKSv4 and SOCKSv5 proxy. resolves #214 #220
|
||||
* Remove redundant JSON key (postData) (#242)
|
||||
* Make test URL configurable with TEST_URL env var. resolves #240
|
||||
* Bypass new Cloudflare protection
|
||||
* Update donation links
|
||||
|
||||
## v2.0.2 (2021/10/31)
|
||||
|
||||
* Fix SOCKS5 proxy. Resolves #214
|
||||
* Replace Firefox ERS with a newer version
|
||||
* Catch startup exceptions and give some advices
|
||||
* Add env var BROWSER_TIMEOUT for slow systems
|
||||
* Fix NPM warning in Docker images
|
||||
|
||||
## v2.0.1 (2021/10/24)
|
||||
|
||||
* Check user home dir before testing web browser installation
|
||||
|
||||
## v2.0.0 (2021/10/20)
|
||||
|
||||
FlareSolverr 2.0.0 is out with some important changes:
|
||||
|
||||
* It is capable of solving the automatic challenges of Cloudflare. CAPTCHAs (hCaptcha) cannot be resolved and the old solvers have been removed.
|
||||
* The Chrome browser has been replaced by Firefox. This has caused some functionality to be removed. Parameters: `userAgent`, `headers`, `rawHtml` and `downloadare` no longer available.
|
||||
* Included `proxy` support without user/password credentials. If you are writing your own integration with FlareSolverr, make sure your client uses the same User-Agent header and Proxy that FlareSolverr uses. Those values together with the Cookie are checked and detected by Cloudflare.
|
||||
* FlareSolverr has been rewritten from scratch. From now on it should be easier to maintain and test.
|
||||
* If you are using Jackett make sure you have version v0.18.1041 or higher. FlareSolverSharp v2.0.0 is out too.
|
||||
|
||||
Complete changelog:
|
||||
|
||||
* Bump version 2.0.0
|
||||
* Set puppeteer timeout half of maxTimeout param. Resolves #180
|
||||
* Add test for blocked IP
|
||||
* Avoid reloading the page in case of error
|
||||
* Improve Cloudflare detection
|
||||
* Fix version
|
||||
* Fix browser preferences and proxy
|
||||
* Fix request.post method and clean error traces
|
||||
* Use Firefox ESR for Docker images
|
||||
* Improve Firefox start time and code clean up
|
||||
* Improve bad request management and tests
|
||||
* Build native packages with Firefox
|
||||
* Update readme
|
||||
* Improve Docker image and clean TODOs
|
||||
* Add proxy support
|
||||
* Implement request.post method for Firefox
|
||||
* Code clean up, remove returnRawHtml, download, headers params
|
||||
* Remove outdated chaptcha solvers
|
||||
* Refactor the app to use Express server and Jest for tests
|
||||
* Fix Cloudflare resolver for Linux ARM builds
|
||||
* Fix Cloudflare resolver
|
||||
* Replace Chrome web browser with Firefox
|
||||
* Remove userAgent parameter since any modification is detected by CF
|
||||
* Update dependencies
|
||||
* Remove Puppeter steath plugin
|
||||
|
||||
## v1.2.9 (2021/08/01)
|
||||
|
||||
* Improve "Execution context was destroyed" error handling
|
||||
* Implement returnRawHtml parameter. resolves #172 resolves #165
|
||||
* Capture Docker stop signal. resolves #158
|
||||
* Reduce Docker image size 20 MB
|
||||
* Fix page reload after challenge is solved. resolves #162 resolves #143
|
||||
* Avoid loading images/css/fonts to speed up page load
|
||||
* Improve Cloudflare IP ban detection
|
||||
* Fix vulnerabilities
|
||||
|
||||
## v1.2.8 (2021/06/01)
|
||||
|
||||
* Improve old JS challenge waiting. Resolves #129
|
||||
|
||||
## v1.2.7 (2021/06/01)
|
||||
|
||||
* Improvements in Cloudflare redirect detection. Resolves #140
|
||||
* Fix installation instructions
|
||||
|
||||
## v1.2.6 (2021/05/30)
|
||||
|
||||
* Handle new Cloudflare challenge. Resolves #135 Resolves #134
|
||||
* Provide reference Systemd unit file. Resolves #72
|
||||
* Fix EACCES: permission denied, open '/tmp/flaresolverr.txt'. Resolves #120
|
||||
* Configure timezone with TZ env var. Resolves #109
|
||||
* Return the redirected URL in the response (#126)
|
||||
* Show an error in hcaptcha-solver. Resolves #132
|
||||
* Regenerate package-lock.json lockfileVersion 2
|
||||
* Update issue template. Resolves #130
|
||||
* Bump ws from 7.4.1 to 7.4.6 (#137)
|
||||
* Bump hosted-git-info from 2.8.8 to 2.8.9 (#124)
|
||||
* Bump lodash from 4.17.20 to 4.17.21 (#125)
|
||||
|
||||
## v1.2.5 (2021/04/05)
|
||||
|
||||
* Fix memory regression, close test browser
|
||||
* Fix release-docker GitHub action
|
||||
|
||||
## v1.2.4 (2021/04/04)
|
||||
|
||||
* Include license in release zips. resolves #75
|
||||
* Validate Chrome is working at startup
|
||||
* Speedup Docker image build
|
||||
* Add health check endpoint
|
||||
* Update issue template
|
||||
* Minor improvements in debug traces
|
||||
* Validate environment variables at startup. resolves #101
|
||||
* Add FlareSolverr logo. resolves #23
|
||||
|
||||
## v1.2.3 (2021/01/10)
|
||||
|
||||
* CI/CD: Generate release changelog from commits. resolves #34
|
||||
* Update README.md
|
||||
* Add donation links
|
||||
* Simplify docker-compose.yml
|
||||
* Allow to configure "none" captcha resolver
|
||||
* Override docker-compose.yml variables via .env resolves #64 (#66)
|
||||
|
||||
## v1.2.2 (2021/01/09)
|
||||
|
||||
* Add documentation for precompiled binaries installation
|
||||
* Add instructions to set environment variables in Windows
|
||||
* Build Windows and Linux binaries. resolves #18
|
||||
* Add release badge in the readme
|
||||
* CI/CD: Generate release changelog from commits. resolves #34
|
||||
* Add a notice about captcha solvers
|
||||
* Add Chrome flag --disable-dev-shm-usage to fix crashes. resolves #45
|
||||
* Fix Docker CLI documentation
|
||||
* Add traces with captcha solver service. resolves #39
|
||||
* Improve logic to detect Cloudflare captcha. resolves #48
|
||||
* Move Cloudflare provider logic to his own class
|
||||
* Simplify and document the "return only cookies" parameter
|
||||
* Show message when debug log is enabled
|
||||
* Update readme to add more clarifications. resolves #53 (#60)
|
||||
* issue_template: typo fix (#52)
|
||||
|
||||
## v1.2.1 (2020/12/20)
|
||||
|
||||
* Change version to match release tag / 1.2.0 => v1.2.0
|
||||
* CI/CD Publish release in GitHub repository. resolves #34
|
||||
* Add welcome message in / endpoint
|
||||
* Rewrite request timeout handling (maxTimeout) resolves #42
|
||||
* Add http status for better logging
|
||||
* Return an error when no selectors are found, #25
|
||||
* Add issue template, fix #32
|
||||
* Moving log.html right after loading the page and add one on reload, fix #30
|
||||
* Update User-Agent to match chromium version, ref: #15 (#28)
|
||||
* Update install from source code documentation
|
||||
* Update readme to add Docker instructions (#20)
|
||||
* Clean up readme (#19)
|
||||
* Add docker-compose
|
||||
* Change default log level to info
|
||||
|
||||
## v1.2.0 (2020/12/20)
|
||||
|
||||
* Fix User-Agent detected by CouldFlare (Docker ARM) resolves #15
|
||||
* Include exception message in error response
|
||||
* CI/CD: Rename GitHub Action build => publish
|
||||
* Bump version
|
||||
* Fix TypeScript compilation and bump minor version
|
||||
* CI/CD: Bump minor version
|
||||
* CI/CD: Configure GitHub Actions
|
||||
* CI/CD: Configure GitHub Actions
|
||||
* CI/CD: Bump minor version
|
||||
* CI/CD: Configure Build GitHub Action
|
||||
* CI/CD: Configure AutoTag GitHub Action (#14)
|
||||
* CI/CD: Build the Docker images with GitHub Actions (#13)
|
||||
* Update dependencies
|
||||
* Backport changes from Cloudproxy (#11)
|
||||
22
Dockerfile
22
Dockerfile
@@ -1,4 +1,4 @@
|
||||
FROM python:3.11-slim-bullseye as builder
|
||||
FROM python:3.10-slim-bullseye as builder
|
||||
|
||||
# Build dummy packages to skip installing them and their dependencies
|
||||
RUN apt-get update \
|
||||
@@ -12,24 +12,28 @@ RUN apt-get update \
|
||||
&& equivs-build adwaita-icon-theme \
|
||||
&& mv adwaita-icon-theme_*.deb /adwaita-icon-theme.deb
|
||||
|
||||
FROM python:3.11-slim-bullseye
|
||||
FROM python:3.10-slim-bullseye
|
||||
|
||||
# Copy dummy packages
|
||||
COPY --from=builder /*.deb /
|
||||
|
||||
# Install dependencies and create flaresolverr user
|
||||
# We have to install and old version of Chromium because its not working in Raspberry Pi / ARM
|
||||
# You can test Chromium running this command inside the container:
|
||||
# xvfb-run -s "-screen 0 1600x1200x24" chromium --no-sandbox
|
||||
# The error traces is like this: "*** stack smashing detected ***: terminated"
|
||||
# To check the package versions available you can use this command:
|
||||
# apt-cache madison chromium
|
||||
WORKDIR /app
|
||||
RUN echo "\ndeb http://snapshot.debian.org/archive/debian/20210519T212015Z/ bullseye main" >> /etc/apt/sources.list \
|
||||
&& echo 'Acquire::Check-Valid-Until "false";' | tee /etc/apt/apt.conf.d/00snapshot \
|
||||
# Install dummy packages
|
||||
RUN dpkg -i /libgl1-mesa-dri.deb \
|
||||
&& dpkg -i /libgl1-mesa-dri.deb \
|
||||
&& dpkg -i /adwaita-icon-theme.deb \
|
||||
# Install dependencies
|
||||
&& apt-get update \
|
||||
&& apt-get install -y --no-install-recommends chromium chromium-common chromium-driver xvfb \
|
||||
&& apt-get install -y --no-install-recommends chromium=89.0.4389.114-1 chromium-common=89.0.4389.114-1 \
|
||||
chromium-driver=89.0.4389.114-1 xvfb \
|
||||
# Remove temporary files and hardware decoding libraries
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& rm -f /usr/lib/x86_64-linux-gnu/libmfxhw* \
|
||||
@@ -43,7 +47,8 @@ RUN dpkg -i /libgl1-mesa-dri.deb \
|
||||
COPY requirements.txt .
|
||||
RUN pip install -r requirements.txt \
|
||||
# Remove temporary files
|
||||
&& rm -rf /root/.cache
|
||||
&& rm -rf /root/.cache \
|
||||
&& find / -name '*.pyc' -delete
|
||||
|
||||
USER flaresolverr
|
||||
|
||||
@@ -55,11 +60,10 @@ EXPOSE 8191
|
||||
CMD ["/usr/local/bin/python", "-u", "/app/flaresolverr.py"]
|
||||
|
||||
# Local build
|
||||
# docker build -t ngosang/flaresolverr:3.0.0 .
|
||||
# docker run -p 8191:8191 ngosang/flaresolverr:3.0.0
|
||||
# docker build -t ngosang/flaresolverr:3.0.0.beta2 .
|
||||
# docker run -p 8191:8191 ngosang/flaresolverr:3.0.0.beta2
|
||||
|
||||
# Multi-arch build
|
||||
# docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
|
||||
# docker buildx create --use
|
||||
# docker buildx build -t ngosang/flaresolverr:3.0.0 --platform linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8 .
|
||||
# docker buildx build -t ngosang/flaresolverr:3.0.0.beta2 --platform linux/386,linux/amd64,linux/arm/v7,linux/arm64/v8 .
|
||||
# add --push to publish in DockerHub
|
||||
|
||||
2
LICENSE
2
LICENSE
@@ -1,6 +1,6 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2023 Diego Heras (ngosang / ngosang@hotmail.es)
|
||||
Copyright (c) 2022 Diego Heras (ngosang / ngosang@hotmail.es)
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "flaresolverr",
|
||||
"version": "3.0.0",
|
||||
"version": "3.0.0.beta2",
|
||||
"description": "Proxy server to bypass Cloudflare protection",
|
||||
"author": "Diego Heras (ngosang / ngosang@hotmail.es)",
|
||||
"license": "MIT"
|
||||
|
||||
@@ -7,7 +7,7 @@ from selenium.common import TimeoutException
|
||||
from selenium.webdriver.chrome.webdriver import WebDriver
|
||||
from selenium.webdriver.common.by import By
|
||||
from selenium.webdriver.support.wait import WebDriverWait
|
||||
from selenium.webdriver.support.expected_conditions import presence_of_element_located, staleness_of, title_is
|
||||
from selenium.webdriver.support.expected_conditions import presence_of_element_located, staleness_of
|
||||
|
||||
from dtos import V1RequestBase, V1ResponseBase, ChallengeResolutionT, ChallengeResolutionResultT, IndexResponse, \
|
||||
HealthResponse, STATUS_OK, STATUS_ERROR
|
||||
@@ -15,21 +15,17 @@ import utils
|
||||
|
||||
ACCESS_DENIED_SELECTORS = [
|
||||
# Cloudflare
|
||||
'div.cf-error-title span.cf-code-label span'
|
||||
]
|
||||
CHALLENGE_TITLE = [
|
||||
# Cloudflare
|
||||
'Just a moment...',
|
||||
# DDoS-GUARD
|
||||
'DDOS-GUARD',
|
||||
'div.main-wrapper div.header.section h1 span.code-label span'
|
||||
]
|
||||
CHALLENGE_SELECTORS = [
|
||||
# Cloudflare
|
||||
'#cf-challenge-running', '.ray_id', '.attack-box', '#cf-please-wait', '#challenge-spinner', '#trk_jschal_js',
|
||||
'#cf-challenge-running', '.ray_id', '.attack-box', '#cf-please-wait', '#trk_jschal_js',
|
||||
# DDoS-GUARD
|
||||
'#link-ddg',
|
||||
# Custom CloudFlare for EbookParadijs, Film-Paleis, MuziekFabriek and Puur-Hollands
|
||||
'td.info #js_info'
|
||||
]
|
||||
SHORT_TIMEOUT = 10
|
||||
SHORT_TIMEOUT = 5
|
||||
|
||||
|
||||
def test_browser_installation():
|
||||
@@ -180,31 +176,18 @@ def _evil_logic(req: V1RequestBase, driver: WebDriver, method: str) -> Challenge
|
||||
raise Exception('Cloudflare has blocked this request. '
|
||||
'Probably your IP is banned for this site, check in your web browser.')
|
||||
|
||||
# find challenge by title
|
||||
# find challenge selectors
|
||||
challenge_found = False
|
||||
page_title = driver.title
|
||||
for title in CHALLENGE_TITLE:
|
||||
if title == page_title:
|
||||
for selector in CHALLENGE_SELECTORS:
|
||||
found_elements = driver.find_elements(By.CSS_SELECTOR, selector)
|
||||
if len(found_elements) > 0:
|
||||
challenge_found = True
|
||||
logging.info("Challenge detected. Title found: " + title)
|
||||
logging.info("Challenge detected. Selector found: " + selector)
|
||||
break
|
||||
if not challenge_found:
|
||||
# find challenge by selectors
|
||||
for selector in CHALLENGE_SELECTORS:
|
||||
found_elements = driver.find_elements(By.CSS_SELECTOR, selector)
|
||||
if len(found_elements) > 0:
|
||||
challenge_found = True
|
||||
logging.info("Challenge detected. Selector found: " + selector)
|
||||
break
|
||||
|
||||
if challenge_found:
|
||||
while True:
|
||||
try:
|
||||
# wait until the title change
|
||||
for title in CHALLENGE_TITLE:
|
||||
logging.debug("Waiting for title: " + title)
|
||||
WebDriverWait(driver, SHORT_TIMEOUT).until_not(title_is(title))
|
||||
|
||||
# then wait until all the selectors disappear
|
||||
for selector in CHALLENGE_SELECTORS:
|
||||
logging.debug("Waiting for selector: " + selector)
|
||||
@@ -237,11 +220,11 @@ def _evil_logic(req: V1RequestBase, driver: WebDriver, method: str) -> Challenge
|
||||
challenge_res.url = driver.current_url
|
||||
challenge_res.status = 200 # todo: fix, selenium not provides this info
|
||||
challenge_res.cookies = driver.get_cookies()
|
||||
challenge_res.userAgent = utils.get_user_agent(driver)
|
||||
|
||||
if not req.returnOnlyCookies:
|
||||
challenge_res.headers = {} # todo: fix, selenium not provides this info
|
||||
challenge_res.response = driver.page_source
|
||||
challenge_res.userAgent = utils.get_user_agent(driver)
|
||||
|
||||
res.result = challenge_res
|
||||
return res
|
||||
|
||||
23
src/tests.py
23
src/tests.py
@@ -1,4 +1,5 @@
|
||||
import unittest
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from webtest import TestApp
|
||||
|
||||
@@ -19,12 +20,12 @@ class TestFlareSolverr(unittest.TestCase):
|
||||
proxy_url = "http://127.0.0.1:8888"
|
||||
proxy_socks_url = "socks5://127.0.0.1:1080"
|
||||
google_url = "https://www.google.com"
|
||||
post_url = "https://httpbin.org/post"
|
||||
post_url = "https://ptsv2.com/t/qv4j3-1634496523"
|
||||
cloudflare_url = "https://nowsecure.nl"
|
||||
cloudflare_url_2 = "https://idope.se/torrent-list/harry/"
|
||||
ddos_guard_url = "https://anidex.info/"
|
||||
custom_cloudflare_url = "https://www.muziekfabriek.org"
|
||||
cloudflare_blocked_url = "https://cpasbiens3.fr/index.php?do=search&subaction=search"
|
||||
cloudflare_blocked_url = "https://avistaz.to/api/v1/jackett/torrents?in=1&type=0&search="
|
||||
|
||||
app = TestApp(flaresolverr.app)
|
||||
|
||||
@@ -232,7 +233,7 @@ class TestFlareSolverr(unittest.TestCase):
|
||||
self.assertIsNone(solution.headers)
|
||||
self.assertIsNone(solution.response)
|
||||
self.assertGreater(len(solution.cookies), 0)
|
||||
self.assertIn("Chrome/", solution.userAgent)
|
||||
self.assertIsNone(solution.userAgent)
|
||||
|
||||
# todo: test Cmd 'request.get' should return OK with HTTP 'proxy' param
|
||||
# todo: test Cmd 'request.get' should return OK with HTTP 'proxy' param with credentials
|
||||
@@ -280,7 +281,7 @@ class TestFlareSolverr(unittest.TestCase):
|
||||
def test_v1_endpoint_request_post_no_cloudflare(self):
|
||||
res = self.app.post_json('/v1', {
|
||||
"cmd": "request.post",
|
||||
"url": self.post_url,
|
||||
"url": self.post_url + '/post',
|
||||
"postData": "param1=value1¶m2=value2"
|
||||
})
|
||||
self.assertEqual(res.status_code, 200)
|
||||
@@ -296,10 +297,22 @@ class TestFlareSolverr(unittest.TestCase):
|
||||
self.assertIn(self.post_url, solution.url)
|
||||
self.assertEqual(solution.status, 200)
|
||||
self.assertIs(len(solution.headers), 0)
|
||||
self.assertIn('"form": {\n "param1": "value1", \n "param2": "value2"\n }', solution.response)
|
||||
self.assertIn("I hope you have a lovely day!", solution.response)
|
||||
self.assertEqual(len(solution.cookies), 0)
|
||||
self.assertIn("Chrome/", solution.userAgent)
|
||||
|
||||
# check that we sent the post data
|
||||
res2 = self.app.post_json('/v1', {
|
||||
"cmd": "request.get",
|
||||
"url": self.post_url
|
||||
})
|
||||
self.assertEqual(res2.status_code, 200)
|
||||
|
||||
body2 = V1ResponseBase(res2.json)
|
||||
self.assertEqual(STATUS_OK, body2.status)
|
||||
date_hour = datetime.now(timezone.utc).isoformat().split(':')[0].replace('T', ' ')
|
||||
self.assertIn(date_hour, body2.solution.response)
|
||||
|
||||
def test_v1_endpoint_request_post_cloudflare(self):
|
||||
res = self.app.post_json('/v1', {
|
||||
"cmd": "request.post",
|
||||
|
||||
Reference in New Issue
Block a user