https://github.com/unclecode/crawl4ai
Score: 30.017380523574662
Last synced: about 1 hour ago
JSON representation
Repository metadata:
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
- Host: GitHub
- URL: https://github.com/unclecode/crawl4ai
- Owner: unclecode
- License: apache-2.0
- Created: 2024-05-09T09:48:50.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2026-09-01T07:58:26.000Z (7 days ago)
- Last Synced: 2026-09-05T05:34:35.450Z (3 days ago)
- Language: Python
- Homepage: https://crawl4ai.com
- Size: 144 MB
- Stars: 81,348
- Watchers: 414
- Forks: 8,389
- Open Issues: 172
-
Metadata Files:
- Readme: README-first.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Security: SECURITY-CREDITS.md
- Roadmap: ROADMAP.md
-
Funding:
- Github: unclecode
Owner metadata:
- Name: UncleCode
- Login: unclecode
- Email: unclecode@kidocode.com
- Kind: user
- Description: Author of Crawl4AI (#1 GitHub Trending). Founder of Kidocode, SE Asia's largest tech & biz school. Leading AI research on synthetic data & coffee lover.
- Website: http://kidocode.com
- Location: Singapore
- Twitter: unclecode
- Company: AlephNul, and KidoCode
- Icon url: https://avatars.githubusercontent.com/u/12494079?u=413d68156031fdfe74861491b75ffe6d426d8228&v=4
- Repositories: 134
- Last Synced at: 2026-09-03T11:16:05.776Z
- Profile URL: https://github.com/unclecode
Committers metadata
Last synced: about 9 hours ago
Total Commits: 1,348
Total Committers: 90
Avg Commits per committer: 14.978
Development Distribution Score (DDS): 0.335
Commits in past year: 412
Committers in past year: 36
Avg Commits per committer in past year: 11.444
Development Distribution Score (DDS) in past year: 0.541
| Name | Commits | |
|---|---|---|
| UncleCode | u****e@k****m | 896 |
| ntohidi | n****n@k****m | 151 |
| Aravind Karnam | a****m@g****m | 70 |
| Soham Kukreti | k****m@g****m | 33 |
| AHMET YILMAZ | t****k@k****m | 30 |
| hafezparast | m****m@k****m | 18 |
| nightcityblade | n****e@g****m | 14 |
| Claude | n****y@a****m | 8 |
| prokopis3 | p****3@g****m | 6 |
| Yurii Chukhlib | y****u@g****m | 6 |
| Chris Murphy | c****y@k****m | 6 |
| bizrockman | d****t@b****e | 5 |
| Br1an67 | 9****0@q****m | 4 |
| datehoer | d****r@g****m | 4 |
| ntohidikplay | “****n@k****m | 3 |
| Aravind Karnam | a****m@A****l | 3 |
| Harsh Mathur | h****x@g****m | 3 |
| NhatNguyen | a****1@s****i | 3 |
| CapSolver | 1****r | 2 |
| Martin Sjöborg | m****g@q****e | 2 |
| wakaka6 | 4****6 | 2 |
| nitesh-77 | n****l@g****m | 2 |
| lizhuxiong | p****z@1****m | 2 |
| bibi-creator | 5****r | 2 |
| Rajan Chavada | j****2@g****m | 2 |
| Otman404 | o****a@g****m | 2 |
| Mark Jan van Kampen | m****n@f****i | 2 |
| Krzysztof Olipra | c****t@k****m | 2 |
| José | j****o@g****m | 2 |
| Atharva Jaiswal | 9****5 | 2 |
| and 60 more... | ||
Issue and Pull Request metadata
Last synced: 1 day ago
Total issues: 810
Total pull requests: 502
Average time to close issues: 17 days
Average time to close pull requests: 12 days
Total issue authors: 623
Total pull request authors: 276
Average comments per issue: 1.69
Average comments per pull request: 0.73
Merged pull request: 123
Bot issues: 1
Bot pull requests: 2
Past year issues: 109
Past year pull requests: 215
Past year average time to close issues: about 1 month
Past year average time to close pull requests: 12 days
Past year issue authors: 91
Past year pull request authors: 93
Past year average comments per issue: 1.96
Past year average comments per pull request: 0.67
Past year merged pull request: 48
Past year bot issues: 0
Past year bot pull requests: 0
Top Issue Authors
- aravindkarnam (10)
- QuangTQV (8)
- ntohidi (7)
- SohamKukreti (7)
- 1933211129 (6)
- stevenh (5)
- truonghoangnguyen (5)
- Joorrit (4)
- wakaka6 (4)
- pbtsrc (4)
- vikaskookna (4)
- b-sai (4)
- Olliejp (4)
- betterthanever2 (4)
- FractalMind (4)
Top Pull Request Authors
- ntohidi (33)
- Battleplus (33)
- aravindkarnam (21)
- nightcityblade (21)
- SohamKukreti (18)
- unclecode (13)
- hafezparast (8)
- Ahmed-Tawfik94 (8)
- stevenh (7)
- wakaka6 (5)
- jichaowang02-lang (5)
- devteamaegis (5)
- Sjoeborg (4)
- leoric-crown (4)
- Aitosoft (4)
Top Issue Labels
- 🐞 Bug (420)
- 🩺 Needs Triage (297)
- question (59)
- ⚙ Done (48)
- 📌 Root caused (44)
- ⚙️ In-progress (37)
- bug (30)
- ❓ Question (30)
- enhancement (19)
- 📖 Documentation (12)
- ✨ Enhancement (11)
- 🐳 Docker (10)
- ⚙️ Under Test (9)
- ❗Invalid (8)
- ⚡ High (8)
- 🪞 Duplicate (7)
- 💪 - Beginner (6)
- ☕ Low (6)
- ⚙️ Under Review (4)
- 💪 - Intermediate (4)
- ⚰️ Wontfix (4)
- duplicate (3)
- ⏰-spill-over (3)
- 💪 - Advanced (3)
- ✅ Released (2)
- resolved (2)
- 🪶 Typo-fix (1)
- help wanted (1)
- documentation (1)
Top Pull Request Labels
- codex (5)
- 🪶 Typo-fix (2)
- 🔌 Integration (1)
- bug (1)
- ⚙️ Under Review (1)
- 📖 Documentation (1)
- ✊🏼 On-hold (1)
- enhancement (1)
- 🪞 Duplicate (1)
- 🏗️ Workflow (1)
- question (1)
Package metadata
- Total packages: 8
-
Total downloads:
- pypi: 1,482,074 last-month
- npm: 16 last-month
- Total dependent packages: 0 (may contain duplicates)
- Total dependent repositories: 0 (may contain duplicates)
- Total versions: 126
- Total maintainers: 5
- Total advisories: 12
proxy.golang.org: github.com/unclecode/crawl4ai
- Homepage:
- Documentation: https://pkg.go.dev/github.com/unclecode/crawl4ai#section-documentation
- Licenses: apache-2.0
- Latest release: v0.9.3 (published 8 days ago)
- Last Synced: 2026-09-07T17:13:57.629Z (about 22 hours ago)
- Versions: 40
- Dependent Packages: 0
- Dependent Repositories: 0
-
Rankings:
- Dependent packages count: 5.785%
- Average: 5.978%
- Dependent repos count: 6.172%
pypi.org: crawl4ai-nightly
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & scraper (Nightly Build)
- Homepage: https://github.com/unclecode/crawl4ai
- Documentation: https://crawl4ai-nightly.readthedocs.io/
- Licenses: apache-2.0
- Latest release: 25.7.11.125807 (published about 1 year ago)
- Last Synced: 2026-09-07T14:02:19.770Z (1 day ago)
- Versions: 3
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 20 Last month
-
Rankings:
- Stargazers count: 0.138%
- Forks count: 0.279%
- Dependent packages count: 8.882%
- Average: 14.837%
- Dependent repos count: 50.049%
- Maintainers (1)
pypi.org: crawl4ai-ctf
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & scraper (CTF fork with Docker cgroup memory + BFS/Best-First max_pages fixes)
- Homepage: https://github.com/unclecode/crawl4ai
- Documentation: https://crawl4ai-ctf.readthedocs.io/
- Licenses: Apache-2.0
- Latest release: 0.7.6.post3 (published 10 months ago)
- Last Synced: 2026-09-07T14:02:17.318Z (1 day ago)
- Versions: 2
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 42 Last month
-
Rankings:
- Dependent packages count: 8.364%
- Average: 27.819%
- Dependent repos count: 47.275%
- Maintainers (1)
npmjs.org: @flyrank/crawl-js
Crawl4AI JavaScript SDK that wraps the Python crwl CLI for robust server-side crawling
- Homepage: https://github.com/unclecode/crawl4ai#readme
- Licenses: MIT
- Latest release: 0.1.0 (published 10 months ago)
- Last Synced: 2026-09-07T14:46:28.145Z (1 day ago)
- Versions: 1
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 16 Last month
-
Rankings:
- Dependent repos count: 23.39%
- Average: 28.563%
- Dependent packages count: 33.736%
- Maintainers (1)
pypi.org: crawl4ai-gautamp8
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & scraper
- Homepage: https://github.com/unclecode/crawl4ai
- Documentation: https://crawl4ai-gautamp8.readthedocs.io/
- Licenses: Apache-2.0
- Latest release: 0.7.7 (published about 1 year ago)
- Last Synced: 2026-09-07T14:04:55.846Z (1 day ago)
- Versions: 7
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 33 Last month
-
Rankings:
- Dependent packages count: 8.827%
- Average: 29.282%
- Dependent repos count: 49.737%
- Maintainers (1)
pypi.org: crawl4ai
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & scraper
- Homepage: https://github.com/unclecode/crawl4ai
- Documentation: https://crawl4ai.readthedocs.io/
- Licenses: Apache-2.0
- Latest release: 0.9.3 (published 8 days ago)
- Last Synced: 2026-09-08T09:08:05.427Z (about 6 hours ago)
- Versions: 71
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 1,481,979 Last month
-
Rankings:
- Dependent packages count: 10.288%
- Average: 34.095%
- Dependent repos count: 57.902%
- Maintainers (1)
-
Advisories:
- Crawl4AI: Unauthenticated SSRF on the Docker server streaming crawl path (/crawl/stream)
- Crawl4AI: Unauthenticated RCE via Chromium launch-argument injection in browser_config.extra_args
- Crawl4AI: Arbitrary file write (path traversal) in crawler downloads can lead to RCE
- Crawl4AI: SSRF via proxy settings in the Docker server bypasses the crawl-URL SSRF check
- Crawl4AI: Arbitrary file write (symlink/TOCTOU) plus log and webhook-header injection in Docker server
- Crawl4AI: LLM credential exfiltration in Docker server via request base_url and env: token resolution
- Crawl4AI: SSRF filter bypass in Docker server via IPv6 transition forms (NAT64 / 6to4 / unspecified / v4-mapped)
- Crawl4AI: Multiple Docker API Vulnerabilities - File Write, SSRF, Auth Bypass, XSS, JS Execution
- Crawl4AI: AST Sandbox Escape via gi_frame.f_back Chain - Pre-Auth RCE in Docker API
- Crawl4AI is Vulnerable to Remote Code Execution in Docker API via Hooks Parameter
- Crawl4AI Has Local File Inclusion in Docker API via file:// URLs
- Crawl4AI SSRF vulnerability
nixpkgs-unstable: python313Packages.crawl4ai
LLM Friendly Web Crawler & Scraper
- Homepage: https://github.com/unclecode/crawl4ai
- Documentation: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/python-modules/crawl4ai/default.nix#L263
- Licenses: Apache-2.0
- Latest release: 0.9.0 (published 2 months ago)
- Last Synced: 2026-07-06T00:03:08.058Z (2 months ago)
- Versions: 1
- Dependent Packages: 0
- Dependent Repositories: 0
-
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%
- Maintainers (1)
nixpkgs-unstable: python314Packages.crawl4ai
LLM Friendly Web Crawler & Scraper
- Homepage: https://github.com/unclecode/crawl4ai
- Documentation: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/python-modules/crawl4ai/default.nix#L263
- Licenses: Apache-2.0
- Latest release: 0.9.0 (published 2 months ago)
- Last Synced: 2026-07-06T00:03:45.292Z (2 months ago)
- Versions: 1
- Dependent Packages: 0
- Dependent Repositories: 0
-
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%
- Maintainers (1)
Dependencies
requirements.txt
pypi
- aiohttp *
- aiosqlite *
- bs4 *
- chromedriver_autoinstaller *
- fastapi *
- html2text *
- httpx *
- litellm *
- pydantic *
- python-dotenv *
- requests *
- selenium *
- uvicorn *
setup.py
pypi
Dockerfile
docker
- python ${PYTHON_VERSION}-slim build
.github/workflows/test-release.yml.disabled
actions
- actions/checkout v4 composite
- actions/setup-python v5 composite
- docker/build-push-action v5 composite
- docker/login-action v3 composite
- docker/setup-buildx-action v3 composite
docker-compose.yml
docker
docs/examples/website-to-api/requirements.txt
pypi
- crawl4ai *
- fastapi *
- litellm *
- pydantic *
- uvicorn *
uv.lock
pypi
- 128 dependencies
docs/md_v2/apps/c4a-script/requirements.txt
pypi
- flask >=2.3.0
- flask-cors >=4.0.0
.github/workflows/release.yml
actions
- actions/checkout v4 composite
- actions/setup-python v5 composite
- softprops/action-gh-release v2 composite
deploy/docker/requirements.txt
pypi
- PyJWT ==2.10.1
- anyio ==4.9.0
- dnspython >=2.7.0
- email-validator ==2.2.0
- fastapi >=0.115.12
- gunicorn >=23.0.0
- httpx >=0.27.2
- jwt >=1.3.1
- mcp >=1.18.0
- prometheus-fastapi-instrumentator >=7.1.0
- pydantic >=2.11
- rank-bm25 ==0.2.2
- redis >=5.2.1
- slowapi ==0.1.9
- sse-starlette ==2.2.1
- uvicorn >=0.34.2
- websockets >=15.0.1
tests/memory/requirements.txt
pypi
- matplotlib >=3.5.0 test
- pandas >=1.5.0 test
- rich >=12.0.0 test
- seaborn >=0.12.0 test
.github/workflows/main.yml
actions
- Ilshidur/action-discord master composite
deploy/docker/tests/requirements.txt
pypi
- docker >=7.0.0 test
- httpx >=0.25.0 test
pyproject.toml
pypi
- aiofiles >=24.1.0
- aiohttp >=3.11.11
- aiosqlite ~=0.20
- alphashape >=1.3.1
- anyio >=4.0.0
- beautifulsoup4 ~=4.12
- brotli >=1.1.0
- chardet >=5.2.0
- click >=8.1.7
- cssselect >=1.2.0
- fake-useragent >=2.0.3
- httpx [http2]>=0.27.2
- httpx >=0.27.2
- humanize >=4.10.0
- lark >=1.2.2
- litellm >=1.53.1
- lxml ~=5.3
- nltk >=3.9.1
- numpy >=1.26.0,<3
- patchright >=1.49.0
- pillow >=10.4
- playwright >=1.49.0
- psutil >=6.1.1
- pydantic >=2.10
- pyopenssl >=25.3.0
- python-dotenv ~=1.0
- pyyaml >=6.0
- rank-bm25 ~=0.2
- requests ~=2.26
- rich >=13.9.4
- shapely >=2.0.0
- snowballstemmer ~=2.2
- tf-playwright-stealth >=1.1.0
- xxhash ~=3.4
.github/workflows/release.yml.backup
actions
- actions/checkout v4 composite
- actions/setup-python v5 composite
- docker/build-push-action v5 composite
- docker/login-action v3 composite
- docker/setup-buildx-action v3 composite
- softprops/action-gh-release v2 composite
.github/workflows/docker-release.yml
actions
- actions/checkout v4 composite
- docker/build-push-action v5 composite
- docker/login-action v3 composite
- docker/setup-buildx-action v3 composite
docs/examples/c4a_script/tutorial/requirements.txt
pypi
- flask >=2.3.0
- flask-cors >=4.0.0
docs/md_v2/marketplace/backend/requirements.txt
pypi
- fastapi *
- python-dotenv *
- python-multipart *
- pyyaml *
- uvicorn *