https://github.com/scrapy/scrapy
crawler crawling framework hacktoberfest python scraping web-scraping web-scraping-python
Score: 33.03131967006679
Last synced: about 2 hours ago
JSON representation
Repository metadata:
Scrapy, a fast high-level web crawling & scraping framework for Python.
- Host: GitHub
- URL: https://github.com/scrapy/scrapy
- Owner: scrapy
- License: bsd-3-clause
- Created: 2010-02-22T02:01:14.000Z (over 16 years ago)
- Default Branch: master
- Last Pushed: 2026-08-21T12:56:41.000Z (2 days ago)
- Last Synced: 2026-08-21T19:36:18.605Z (2 days ago)
- Topics: crawler, crawling, framework, hacktoberfest, python, scraping, web-scraping, web-scraping-python
- Language: Python
- Homepage: https://scrapy.org
- Size: 30.5 MB
- Stars: 63,987
- Watchers: 1,752
- Forks: 11,917
- Open Issues: 462
-
Metadata Files:
- Readme: README.rst
- Changelog: NEWS
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Citation: CITATION.cff
- Security: SECURITY.md
- Authors: AUTHORS
Owner metadata:
- Name: Scrapy project
- Login: scrapy
- Email:
- Kind: organization
- Description: An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.
- Website: https://scrapy.org
- Location:
- Twitter:
- Company:
- Icon url: https://avatars.githubusercontent.com/u/733635?v=4
- Repositories: 26
- Last Synced at: 2024-03-25T19:52:43.990Z
- Profile URL: https://github.com/scrapy
GitHub Events
Total
- Commit comment event: 1
- Create event: 32
- Delete event: 25
- Discussion event: 3
- Fork event: 758
- Gollum event: 2
- Issue comment event: 1472
- Issues event: 353
- Pull request event: 792
- Pull request review comment event: 565
- Pull request review event: 828
- Push event: 403
- Release event: 7
- Watch event: 5675
- Total: 10916
Last Year
- Create event: 9
- Delete event: 7
- Discussion event: 3
- Fork event: 168
- Gollum event: 1
- Issue comment event: 316
- Issues event: 67
- Pull request event: 212
- Pull request review comment event: 117
- Pull request review event: 155
- Push event: 115
- Release event: 3
- Watch event: 1268
- Total: 2441
Committers metadata
Last synced: 4 days ago
Total Commits: 9,252
Total Committers: 716
Avg Commits per committer: 12.922
Development Distribution Score (DDS): 0.801
Commits in past year: 459
Committers in past year: 65
Avg Commits per committer in past year: 7.062
Development Distribution Score (DDS) in past year: 0.558
| Name | Commits | |
|---|---|---|
| Pablo Hoffman | p****o@p****m | 1843 |
| Andrey Rakhmatullin | w****r@w****e | 948 |
| Daniel Graña | d****a@g****m | 743 |
| Adrián Chaves | a****n@c****o | 710 |
| Eugenio Lacuesta | e****a@g****m | 455 |
| elpolilla | n****e@n****e | 420 |
| Paul Tremberth | p****h@g****m | 392 |
| Mikhail Korobov | k****4@g****m | 339 |
| Ismael Carnales | i****s@g****m | 248 |
| Julia Medina | w****a@g****m | 174 |
| Konstantin Lopuhin | k****n@g****m | 133 |
| Laerte Pereira | l****k@g****m | 124 |
| Elias Dorneles | e****s@g****m | 116 |
| Vostretsov Nikita | w****n@g****m | 68 |
| Aditya | k****0@g****m | 68 |
| nyov | n****v@n****t | 60 |
| Jakob de Maeyer | j****1@g****m | 59 |
| Rolando Espinoza La fuente | d****o@g****m | 54 |
| Jalil SA | 6****l | 41 |
| Victor Torres | v****s@g****m | 38 |
| BroodingKangaroo | j****3@g****m | 35 |
| Edwin O Marshall | e****5@g****m | 32 |
| nramirezuy | n****y@g****m | 31 |
| Pawel Miech | p****m@g****m | 31 |
| Νικόλαος-Διγενής Καραγιάννης | d****l@g****m | 30 |
| Anubhav Patel | a****8@g****m | 30 |
| Georgiy Zatserklianyi | g****y@g****m | 30 |
| Alex Cepoi | a****i@g****m | 29 |
| Valdir Stumm Junior | s****r@g****m | 29 |
| Maram Sumanth | m****h@g****m | 29 |
| and 686 more... | ||
Issue and Pull Request metadata
Last synced: 1 day ago
Total issues: 762
Total pull requests: 1,756
Average time to close issues: over 2 years
Average time to close pull requests: 8 months
Total issue authors: 413
Total pull request authors: 495
Average comments per issue: 4.58
Average comments per pull request: 2.22
Merged pull request: 799
Bot issues: 0
Bot pull requests: 0
Past year issues: 38
Past year pull requests: 389
Past year average time to close issues: about 2 months
Past year average time to close pull requests: 8 days
Past year issue authors: 24
Past year pull request authors: 114
Past year average comments per issue: 3.47
Past year average comments per pull request: 1.77
Past year merged pull request: 154
Past year bot issues: 0
Past year bot pull requests: 0
Top Issue Authors
- wRAR (142)
- Gallaecio (68)
- mohmad-null (30)
- kmike (18)
- Prometheus3375 (12)
- Ehsan-U (11)
- pawelmhm (7)
- elacuesta (6)
- nyov (5)
- Urahara (5)
- GeorgeA92 (5)
- lopuhin (5)
- ghost (4)
- Laerte (4)
- NewUserHa (4)
Top Pull Request Authors
- wRAR (434)
- AdrianAtZyte (185)
- Gallaecio (126)
- Laerte (54)
- mlmsmith (14)
- jxlil (14)
- Rotzbua (14)
- GeorgeA92 (13)
- Sintivrousai (10)
- Rohitkr117 (10)
- kumar-sanchay (9)
- MehrazRumman (9)
- mery16q (9)
- thalissonvs (9)
- elramen (9)
Top Issue Labels
- enhancement (301)
- bug (126)
- docs (86)
- discuss (78)
- good first issue (68)
- cleanup (49)
- CI (48)
- asyncio (34)
- needs more info (21)
- help wanted (21)
- logging (19)
- media pipelines (17)
- upstream issue (15)
- performance (14)
- http (11)
- https (10)
- HTTP/2 (8)
- link extraction (8)
- security (7)
- Windows (6)
- contracts (6)
- not reproducible (5)
- typing (5)
- stale (3)
- backward-incompatible (2)
- gsoc-candidate (2)
- spam (2)
- patch available (2)
- S3 (1)
- waiting feedback (1)
Top Pull Request Labels
- enhancement (215)
- CI (178)
- cleanup (150)
- bug (78)
- typing (72)
- spam (65)
- docs (60)
- asyncio (52)
- discuss (14)
- media pipelines (9)
- needs more info (8)
- hacktoberfest-accepted (7)
- backward-incompatible (6)
- logging (5)
- performance (4)
- security (4)
- S3 (3)
- Windows (3)
- link extraction (3)
- in progress (3)
- https (3)
- upstream issue (3)
- contracts (2)
- help wanted (2)
- http (1)
- macOS (1)
- HTTP/2 (1)
- stale (1)
- item loaders (1)
- waiting feedback (1)
Package metadata
- Total packages: 9
-
Total downloads:
- conda: 61,447
- pypi: 4,146,392 last-month
- Total docker downloads: 588,557
- Total dependent packages: 142 (may contain duplicates)
- Total dependent repositories: 2,831 (may contain duplicates)
- Total versions: 179
- Total maintainers: 8
- Total advisories: 16
pypi.org: scrapy
A high-level Web Crawling and Web Scraping framework
- Homepage: https://scrapy.org/
- Documentation: https://docs.scrapy.org/
- Licenses: BSD-3-Clause
- Latest release: 2.18.0 (published 3 days ago)
- Last Synced: 2026-08-23T09:02:44.915Z (about 14 hours ago)
- Versions: 114
- Dependent Packages: 136
- Dependent Repositories: 2,753
- Downloads: 4,146,304 Last month
- Docker Downloads: 588,557
-
Rankings:
- Stargazers count: 0.021%
- Forks count: 0.05%
- Dependent packages count: 0.154%
- Dependent repos count: 0.202%
- Downloads: 0.273%
- Average: 0.358%
- Docker downloads count: 1.447%
- Maintainers (4)
-
Advisories:
- Scrapy: Arbitrary Module Import via Referrer-Policy Header in RefererMiddleware
- Duplicate Advisory: Scrapy leaks the authorization header on same-domain but cross-origin redirects
- Scrapy allows redirect following in protocols other than HTTP
- Scrapy's redirects ignoring scheme-specific proxy settings
- Scrapy leaks the authorization header on same-domain but cross-origin redirects
- Duplicate Advisory: Scrapy authorization header leakage on cross-domain redirect
- Duplicate Advisory: Scrapy decompression bomb vulnerability
- Duplicate Advisory: ReDos vulnerability of XMLFeedSpider
- Scrapy decompression bomb vulnerability
- Scrapy authorization header leakage on cross-domain redirect
- Scrapy vulnerable to ReDoS via XMLFeedSpider
- Scrapy before 2.6.2 and 1.8.3 vulnerable to one proxy sending credentials to another
- Scrapy denial of service vulnerability
- Scrapy cookie-setting is not restricted based on the public suffix list
- Incorrect Authorization and Exposure of Sensitive Information to an Unauthorized Actor in scrapy
- Scrapy HTTP authentication credentials potentially leaked to target websites
proxy.golang.org: github.com/scrapy/scrapy
- Homepage:
- Status: removed
- Documentation: https://pkg.go.dev/github.com/scrapy/scrapy#section-documentation
- Licenses: bsd-3-clause
- Latest release: v0.0.0-20260812194600-be514d8c5df4 (published 11 days ago)
- Last Synced: 2026-08-22T20:13:12.336Z (1 day ago)
- Versions: 1
- Dependent Packages: 0
- Dependent Repositories: 0
-
Rankings:
- Dependent packages count: 4.482%
- Average: 4.633%
- Dependent repos count: 4.783%
conda-forge.org: scrapy
Scrapy is an open source and collaborative framework for extracting the data you need from websites in a fast, simple, yet extensible way.
- Homepage: https://scrapy.org/
- Licenses: BSD-3-Clause-Clear
- Latest release: 2.7.1 (published almost 4 years ago)
- Last Synced: 2026-08-17T00:30:11.327Z (7 days ago)
- Versions: 31
- Dependent Packages: 4
- Dependent Repositories: 38
-
Rankings:
- Stargazers count: 0.245%
- Forks count: 0.903%
- Average: 4.883%
- Dependent repos count: 5.884%
- Dependent packages count: 12.501%
pypi.org: pylab-utils
python utility tools
- Homepage: https://github.com/scrapy/scrapy
- Documentation: https://pylab-utils.readthedocs.io/
- Licenses: BSD
- Latest release: 0.5 (published almost 5 years ago)
- Last Synced: 2026-08-22T20:03:58.038Z (1 day ago)
- Versions: 2
- Dependent Packages: 0
- Dependent Repositories: 1
- Downloads: 16 Last month
-
Rankings:
- Stargazers count: 0.02%
- Forks count: 0.049%
- Dependent packages count: 7.373%
- Average: 9.527%
- Downloads: 17.961%
- Dependent repos count: 22.233%
- Maintainers (1)
anaconda.org: scrapy
Scrapy is an open source and collaborative framework for extracting the data you need from websites in a fast, simple, yet extensible way.
- Homepage: https://www.scrapy.org
- Licenses: BSD-3-Clause
- Latest release: 2.14.1 (published 6 months ago)
- Last Synced: 2026-03-10T19:20:18.809Z (6 months ago)
- Versions: 16
- Dependent Packages: 2
- Dependent Repositories: 38
- Downloads: 61,447 Total
-
Rankings:
- Stargazers count: 0.666%
- Forks count: 2.511%
- Average: 12.511%
- Dependent packages count: 20.454%
- Dependent repos count: 26.415%
pypi.org: scrapy-hls
scrapy integration for m3u8 files
- Homepage: https://github.com/scrapy/scrapy
- Documentation: https://scrapy-hls.readthedocs.io/
- Licenses: BSD
- Latest release: 0.1 (published over 5 years ago)
- Last Synced: 2026-08-22T20:03:48.033Z (1 day ago)
- Versions: 1
- Dependent Packages: 0
- Dependent Repositories: 1
- Downloads: 11 Last month
-
Rankings:
- Stargazers count: 0.02%
- Forks count: 0.049%
- Dependent packages count: 7.373%
- Average: 18.879%
- Dependent repos count: 22.233%
- Downloads: 64.72%
- Maintainers (1)
pypi.org: bf-scrapy-base
基于scrapy的二次开发
- Homepage: https://scrapy.org/
- Documentation: https://docs.scrapy.org/
- Licenses: BSD-3-Clause
- Latest release: 0.0.6 (published 12 months ago)
- Last Synced: 2026-08-22T20:03:48.562Z (1 day ago)
- Versions: 6
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 18 Last month
-
Rankings:
- Dependent packages count: 8.63%
- Average: 28.63%
- Dependent repos count: 48.631%
- Maintainers (1)
pypi.org: scrapy-qfm
A high-level Web Crawling and Web Scraping framework
- Homepage: https://scrapy.org
- Documentation: https://docs.scrapy.org/
- Licenses: BSD
- Latest release: 2.11.2 (published almost 2 years ago)
- Last Synced: 2026-08-22T20:03:59.772Z (1 day ago)
- Versions: 2
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 17 Last month
-
Rankings:
- Dependent packages count: 10.019%
- Average: 33.204%
- Dependent repos count: 56.388%
- Maintainers (1)
pypi.org: aminer-scrapy
A high-level Web Crawling and Web Scraping framework
- Homepage: https://scrapy.org
- Documentation: https://docs.scrapy.org/
- Licenses: BSD
- Latest release: 2.11.1 (published over 2 years ago)
- Last Synced: 2026-08-22T20:05:00.160Z (1 day ago)
- Versions: 6
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 26 Last month
-
Rankings:
- Dependent packages count: 10.037%
- Average: 38.143%
- Dependent repos count: 66.249%
- Maintainers (1)
Dependencies
- actions/checkout v3 composite
- actions/setup-python v4 composite
- pre-commit/action v3.0.0 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- pypa/gh-action-pypi-publish v1.6.4 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- sphinx ==5.0.2
- sphinx-hoverxref ==1.1.1
- sphinx-notfound-page ==0.8
- sphinx-rtd-theme ==1.0.0
- cryptography >=37.0.0
- cssselect >=0.9.1
- defusedxml >=0.7.1
- itemadapter >=0.1.0
- itemloaders >=1.0.1
- lxml >=4.6.4
- packaging *
- parsel >=1.5.0
- protego >=0.1.15
- pydispatcher >=2.0.5; platform_python_implementation == "CPython"
- pyopenssl >=22.0.0
- pypydispatcher >=2.1.0; platform_python_implementation == "PyPy"
- queuelib >=1.4.2
- service-identity >=18.1.0
- tldextract *
- twisted >=21.7.0,<=25.5.0
- w3lib >=1.17.0
- zope-interface >=5.1.0
- CodSpeedHQ/action 0ca9cbbf4623b599a6c3ed4fc8a922942705d9f1 composite
- actions/checkout 3d3c42e5aac5ba805825da76410c181273ba90b1 composite
- actions/setup-python 5fda3b95a4ea91299a34e894583c3862153e4b97 composite
- astral-sh/setup-uv c771a70e6277c0a99b617c7a806ffedaca235ff9 composite
- actions/github-script 3a2844b7e9c422d3c10d287c895573f7108da1b3 composite
- actions/checkout 3d3c42e5aac5ba805825da76410c181273ba90b1 composite
- actions/setup-python 5fda3b95a4ea91299a34e894583c3862153e4b97 composite
- astral-sh/setup-uv c771a70e6277c0a99b617c7a806ffedaca235ff9 composite
- h2 *
- pydantic *
- scrapy-spider-metadata *
- sphinx *
- sphinx-notfound-page *
- sphinx-reredirects *
- sphinx-rtd-dark-mode *
- sphinx-rtd-theme *