Common Crawl - Blog - Announcing the First Stable Release of CC-Downloader The Data OverviewCDXJ IndexURL IndexWeb GraphsLatest CrawlCrawl StatsGraph StatsErrata Resources Get StartedAI AgentBlogExamplesCCBotInfra StatusOpt-Out LedgerFAQ Community Research PapersMailing List ArchiveHugging FaceDiscordCollaborators About AboutTeamJobsPrivacy PolicyTerms of Use Search AI Agent Contact Us ←  Back to Blog August 10, 2026 Announcing the First Stable Release of CC-Downloader Over a year ago we released cc-downloader, an experimental tool to politely download Common Crawl data. Today we're releasing its first stable version, with a Rust library and Python bindings. Pedro Ortiz Suarez Pedro is a Principal Research Scientist at the Common Crawl Foundation. Over a year ago we released an experimental version of cc-downloader, a command-line tool for politely downloading Common Crawl data via HTTPS. We introduced cc-downloader in response to a significant increase in downloads and requests for our datasets, which made it harder for many users to successfully access the data in our bucket due to the high traffic. cc-downloader, which was initially meant to be an experimental tool, used a polite retry mechanism that implemented jitter and exponential backoff strategies to avoid overwhelming our infrastructure. It was also developed in Rust, allowing us to enable parallel downloads easily, while also keeping our tool memory-safe. Over the course of a year, we monitored the usage of cc-downloader and saw significant adoption from the community. Number of successful requests made with cc-downloader per month and version, and cc-downloader traffic share in bytes and requests among top 1K User Agents per month. cc-downloader is now one of the top 10 most used clients to download Common Crawl data through CloudFront, accounting for almost 14 PiB downloaded in one and a half years. User Agent Family Requests (M) Downloaded (PiB) python-requests 3027.8 138.01 Wget 272.8 73.02 aiohttp 548.4 49.67 Mozilla 2103.8 40.49 aria2 403.9 29.28 (unknown) 75.9 21.59 Go-http-client 99.5 18.44 curl 89.8 14.32 cc-downloader 74.9 13.83 rclone 13.4 12.22 Top 10 User Agents by downloaded data from Feb 2025 to Jun 2026. As such, we decided to completely refactor cc-downloader, turning it into a library with its own documentation. And seeing that python-requests is by far the most used client to download our data, we also decided to release Python bindings, in the hope that this will encourage the community to adopt cc-downloader more widely, instead of using potentially impolite implementations of python-requests. Finally, we added support for downloading contrib content in our bucket, as well as for filtering the URL index by subset. We also fixed various issues opened by the community over the course of last year, and we completely rewrote the documentation for the standalone binary. We hope to continue developing and supporting cc-downloader in the future, so if you find any issues or would like to submit a feature request, please visit its GitHub repository, where you will find our new documentation as well as easy instructions to install and start using cc-downloader right away. This release was authored by: Pedro Ortiz Suarez Pedro is a Principal Research Scientist at the Common Crawl Foundation. Erratum:  Content is truncated Originally reported by:  More details Some archived content is truncated due to fetch size limits imposed during crawling. This is necessary to handle infinite or exceptionally large data streams (e.g., radio streams). Prior to March 2025 (CC-MAIN-2025-13), the truncation threshold was 1 MiB. From the March 2025 crawl onwards, this limit has been increased to 5 MiB. The Data Overview CDXJ Index URL Index Web Graphs Latest Crawl Crawl Stats Graph Stats Errata Resources Get Started AI Agent Blog Examples CCBot Infra Status Opt-Out Ledger FAQ Community Research Papers Mailing List Archive Hugging Face Discord Collaborators About About Team Jobs Privacy Policy Terms of Use © 2026 Common Crawl