https://github.com/adbar/trafilatura
article-extractor corpus-builder corpus-tools crawler html-to-markdown html2text llm news-aggregator news-crawler nlp rag readability rss-feed scraping tei text-cleaning text-extraction text-mining text-preprocessing web-scraping
Score: 29.37687952660653
Last synced: about 12 hours ago
JSON representation
Repository metadata:
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- Host: GitHub
- URL: https://github.com/adbar/trafilatura
- Owner: adbar
- License: apache-2.0
- Created: 2019-04-08T11:38:48.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2026-08-04T15:33:35.000Z (8 days ago)
- Last Synced: 2026-08-06T01:56:17.802Z (7 days ago)
- Topics: article-extractor, corpus-builder, corpus-tools, crawler, html-to-markdown, html2text, llm, news-aggregator, news-crawler, nlp, rag, readability, rss-feed, scraping, tei, text-cleaning, text-extraction, text-mining, text-preprocessing, web-scraping
- Language: Python
- Homepage: https://trafilatura.readthedocs.io
- Size: 34.3 MB
- Stars: 6,412
- Watchers: 35
- Forks: 402
- Open Issues: 66
-
Metadata Files:
- Readme: README.md
- Changelog: HISTORY.md
- Contributing: CONTRIBUTING.md
- Funding: .github/FUNDING.yml
- License: LICENSE
- Citation: CITATION.cff
-
Funding:
- Github: adbar
- Ko fi: adbarbaresi
Owner metadata:
- Name: Adrien Barbaresi
- Login: adbar
- Email:
- Kind: user
- Description: Research scientist – natural language processing, web scraping and text analytics. Mostly with Python.
- Website: adrien.barbaresi.eu
- Location: Berlin
- Twitter: adbarbaresi
- Company: Berlin-Brg. Academy of Sciences (BBAW)
- Icon url: https://avatars.githubusercontent.com/u/2125866?u=e2eeeb3384ab9391f598ade37d4388ad23199ebd&v=4
- Repositories: 37
- Last Synced at: 2024-06-11T15:59:25.206Z
- Profile URL: https://github.com/adbar
GitHub Events
Total
- Create event: 24
- Delete event: 26
- Fork event: 73
- Issue comment event: 160
- Issues event: 70
- Pull request event: 72
- Pull request review comment event: 27
- Pull request review event: 25
- Push event: 99
- Release event: 1
- Watch event: 1295
- Total: 1872
Last Year
- Create event: 4
- Delete event: 6
- Fork event: 22
- Issue comment event: 4
- Issues event: 9
- Pull request event: 6
- Push event: 22
- Watch event: 416
- Total: 489
Committers metadata
Last synced: 3 days ago
Total Commits: 1,587
Total Committers: 66
Avg Commits per committer: 24.045
Development Distribution Score (DDS): 0.125
Commits in past year: 37
Committers in past year: 12
Avg Commits per committer in past year: 3.083
Development Distribution Score (DDS) in past year: 0.514
| Name | Commits | |
|---|---|---|
| Adrien Barbaresi | b****i@b****e | 1389 |
| Felipe Hertzer | f****r@g****m | 82 |
| Vincent Barbaresi | v****i@g****m | 12 |
| Luise K | l****r@g****m | 11 |
| Cody | e****e@g****m | 7 |
| LukasBBAW | 6****W | 5 |
| Sebastian Cao | c****l@g****m | 5 |
| Phong Nguyen | p****t@g****m | 4 |
| André Mácola | a****a@g****m | 3 |
| Ido Shamun | i****1@g****m | 3 |
| Igor Saevets | i****s | 3 |
| Denis Moklaf | r****i@g****m | 2 |
| deedy5 | 6****5 | 2 |
| Vincent Gao | g****0@g****m | 2 |
| Tony Yang | t****a@g****m | 2 |
| Ruggero Marchei | r****i@d****t | 2 |
| Gregor Middell | g****r@m****t | 2 |
| Ellie Lockhart | 6****t | 2 |
| Edward Krueger | e****r@g****m | 2 |
| Eyüp Can Akman | e****n@g****m | 1 |
| François Schmidts | f****s@g****m | 1 |
| Elwin | a****5@1****m | 1 |
| David Hall | d****l@g****m | 1 |
| Hergen Lehmann | 3****m | 1 |
| Jan-Philipp Igla | h****o@j****e | 1 |
| Kento Nozawa | k****w@k****p | 1 |
| Chris | c****l@g****m | 1 |
| Ashik Paul | a****7@g****m | 1 |
| Andrzej Witecki | a****i@g****m | 1 |
| Aleks Bunin | s****b | 1 |
| and 36 more... | ||
Issue and Pull Request metadata
Last synced: 1 day ago
Total issues: 293
Total pull requests: 492
Average time to close issues: 6 months
Average time to close pull requests: 11 days
Total issue authors: 169
Total pull request authors: 69
Average comments per issue: 2.31
Average comments per pull request: 1.87
Merged pull request: 365
Bot issues: 0
Bot pull requests: 52
Past year issues: 33
Past year pull requests: 45
Past year average time to close issues: 3 months
Past year average time to close pull requests: 9 days
Past year issue authors: 28
Past year pull request authors: 19
Past year average comments per issue: 0.58
Past year average comments per pull request: 1.91
Past year merged pull request: 28
Past year bot issues: 0
Past year bot pull requests: 0
Top Issue Authors
- adbar (64)
- alroythalus (12)
- fortyfourforty (9)
- andremacola (7)
- Honesty-of-the-Cavernous-Tissue (5)
- zirkelc (4)
- felipehertzer (4)
- pieterhartel (3)
- naktinis (3)
- edkrueger (3)
- krstp (3)
- yerinNam (3)
- DesBw (3)
- TheRabidWolverine (2)
- hamsarajan (2)
Top Pull Request Authors
- adbar (281)
- dependabot[bot] (51)
- felipehertzer (20)
- unsleepy22 (13)
- knit-bee (10)
- vbarbaresi (6)
- cycsmail (5)
- andremacola (4)
- gremid (4)
- bobtheskull-flameeyes (3)
- eyupcanakman (3)
- zirkelc (3)
- idoshamun (3)
- igorsaevets (3)
- dmoklaf (2)
Top Issue Labels
- bug (77)
- enhancement (58)
- question (40)
- feedback (20)
- documentation (18)
- maintenance (17)
- wontfix (4)
- duplicate (3)
- up for grabs (3)
- evaluation (3)
- good first issue (2)
- dependencies (1)
Top Pull Request Labels
- dependencies (51)
- feedback (2)
Package metadata
- Total packages: 2
-
Total downloads:
- pypi: 13,394,118 last-month
- Total docker downloads: 9,653
- Total dependent packages: 71 (may contain duplicates)
- Total dependent repositories: 63 (may contain duplicates)
- Total versions: 93
- Total maintainers: 1
pypi.org: trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML.
- Homepage: https://trafilatura.readthedocs.io
- Documentation: https://trafilatura.readthedocs.io/
- Licenses: Apache-2.0
- Latest release: 2.2.0 (published 12 days ago)
- Last Synced: 2026-08-11T13:05:52.473Z (1 day ago)
- Versions: 52
- Dependent Packages: 71
- Dependent Repositories: 63
- Downloads: 13,394,118 Last month
- Docker Downloads: 9,653
-
Rankings:
- Dependent packages count: 0.24%
- Downloads: 0.677%
- Stargazers count: 1.491%
- Average: 1.739%
- Dependent repos count: 1.867%
- Docker downloads count: 2.374%
- Forks count: 3.787%
- Maintainers (1)
proxy.golang.org: github.com/adbar/trafilatura
- Homepage:
- Documentation: https://pkg.go.dev/github.com/adbar/trafilatura#section-documentation
- Licenses: apache-2.0
- Latest release: v2.2.0+incompatible (published 7 days ago)
- Last Synced: 2026-08-11T13:05:53.963Z (1 day ago)
- Versions: 41
- Dependent Packages: 0
- Dependent Repositories: 0
-
Rankings:
- Dependent packages count: 4.568%
- Average: 4.721%
- Dependent repos count: 4.875%
Dependencies
- docutils >=0.18.1
- pydata-sphinx-theme >=0.9.0
- sphinx >=5.0.2
- trafilatura *
- certifi *
- charset_normalizer *
- courlan *
- htmldate *
- justext *
- lxml *
- urllib3 *
- beautifulsoup4 ==4.11.1 test
- boilerpy3 ==1.0.6 test
- goose3 ==3.1.11 test
- html-text ==0.5.2 test
- html2text ==2020.1.16 test
- inscriptis ==2.2.0 test
- justext ==3.0.0 test
- news-please ==1.5.22 test
- newspaper3k ==0.2.8 test
- readabilipy ==0.2.0 test
- readability-lxml ==0.8.1 test
- trafilatura ==1.2.2 test
- actions/checkout v3 composite
- github/codeql-action/analyze v2 composite
- github/codeql-action/autobuild v2 composite
- github/codeql-action/init v2 composite
- actions/cache v2 composite
- actions/checkout v3 composite
- actions/setup-python v4 composite
- codecov/codecov-action v3 composite