Robot policy - Wikitech Jump to content Main menu Main menu move to sidebar hide Navigation Main page Server admin log: Prod Admin log: RelEng Incident status Deployments SRE Team Help Cloud VPS & Toolforge Cloud VPS portal Toolforge portal Request VPS project Admin log: Cloud VPS wikitech.wikimedia.org Recent changes Special pages Village pump Search Search English Appearance Donate Create account Log in Personal tools Donate Create account Log in Contents move to sidebar hide Beginning 1 Generally applicable rules 2 Website rules 3 API rules Toggle API rules subsection 3.1 REST API rules 3.2 Action API rules 3.3 Media API rules 4 Rules for other resources 5 What to do if these limits are too strict for me? 6 See also Toggle the table of contents Robot policy Page Discussion English Read View source View history Tools Tools move to sidebar hide Actions Read View source View history General What links here Related changes Permanent link Page information Cite this page Get shortened URL Print/export Create a book Download as PDF Printable version Appearance move to sidebar hide From Wikitech As an Operator of a program automatically consuming the content of the wikis (robot), certain rules apply which depend on the type of content you’re accessing and how you access it. Bots that don't follow these guidelines may be rate-limited and informed of the reason, and pointed to ways to request higher quotas or exemptions. Please note: while the guidelines apply to any bot that connects to our environment, rate limiting is not enforced on bots in Toolforge. Bots that repeatedly try to get around these guidelines or rate limits, and/or threaten the stability of the sites may be blocked. The following rules are an evolution of the previous version of this policy which was originally published in 2009. The updates include coverage of more systems, better defined limits, and clarifications to help any Bot Operator who acts in good faith to limit their impact on our systems. Generally applicable rules The following rules apply to any activity on our websites; rules in the following sections will be specific to sites/URLs instead. Consider if dumps are more efficient than live requests. Check if you can use Wikimedia Dumps or other forms of offline collection of our data instead of making live requests. If dumps are a viable option for your use case it will reduce the strain on our very limited resources and make your life easier. Accurately identify your User-Agent. Always identify your bot clearly via its User-Agent HTTP header by following the Wikimedia Foundation User-Agent Policy. When you are making a significant number of requests avoid user-agent impersonation and possible collateral damage blocks by either: Provide a URL where we can download a JSON formatted list of CIDRs from which your requests will originate. See the "what to do if the limits are too strict for me" section. Authenticate your requests using an on-wiki account when you are making API requests. An OAuth 2.0 access token is the preferred authentication method, but session cookies are also supported. Honor Robots.txt. Honor every directive in our robots.txt file. Default to gzip. Always request content with an Accept-Encoding: gzip HTTP header to reduce bandwidth usage. This is not necessary when you are directly requesting media files (i.e. images or video) which are already in a compressed format. Respect our HTTP status codes. When we reply with a 429 Too Many Requests status code respect the delay specified by the Retry-After header sent with the response. Cached interfaces are preferred and more efficient. If you need the HTML content of pages use either the /wiki/Article_name URL or the corresponding Wikimedia REST API /api/rest_v1/page/html/Article_name endpoint. These requests will be cheaper for us and faster for you because they can be cached in our content delivery network. More information about these request methods can be found below. Website rules i.e. https://en.wikipedia.org/wiki/Main_Page Always crawl the website via the /wiki/Article_name URLs, with no query parameters. This will ensure that if the content is CDN-cached, you’ll get a faster response allowing you to crawl the site faster and more efficiently. If you're making read requests, do not emulate a browser - do not store cookies or execute javascript, unless you're not crawling the sites at high volume (so, more than 5 requests per second). Assuming you are following all of our best practices ideally, still ensure that the maximum concurrent number of requests is fewer than 10 overall, and keep the average requests per second below 20. Avoid accessing content that is not current, or via non-canonical URLs: do not crawl the site using the oldId or curid parameters, and only use the /wiki/Title format URLs. API rules All activity is subject to cross-API rate limiting that is global for all Wikimedia sites. See the documentation on rate limits for full details and best practices. In summary: API rate limits take into account client identity to determine the level of access. Stronger forms of identification result in a higher limit, such as running in Wikimedia Cloud Services (WMCS) or authenticating requests. The highest limits require running in WMCS, community bot approval, or being well-known to the Wikimedia Foundation. Further limits or additional best practices may apply to specific APIs. REST API rules i.e. https://en.wikipedia.org/api/… You can use this interface to fetch the HTML content of the pages, or their summary. If unauthenticated, keep the concurrency of your requests to 3 at a time, and below 5 requests per second overall. If authenticated, you can raise the number of requests per second to 10. Avoid accessing content that is not current, so avoid requesting a specific revision in your URL. Action API rules i.e. https://en.wikipedia.org/w/api.php?… Avoid using the action API for HTML content of pages. Use the website and/or the REST API instead. If unauthenticated, keep the concurrency of your requests to 1 at a time, and below 5 requests per second overall. If authenticated, you can raise the concurrency to 3 overall, and the number of requests per second to 10. Avoid using expensive API endpoints: if your request takes more than 1 second to serve, please wait 5 seconds before making another request. Where supported, use batch requests. Media API rules i.e. https://upload.wikimedia.org/… Always keep a total concurrency of at most 2, and limit your total download speed to 25 Mbps (as measured over 10 second intervals). Only use originals or one of our standard thumbnail sizes, which you can see in mediawiki.org. Prefer thumbnails to downloading of originals if possible. Rules for other resources i.e. Gerrit, GitLab, Phabricator, and other wikimedia.org services. Always keep a total concurrency of at most 1, and use a delay between requests of at least 1 second. Pause crawling for at least 15 minutes if you receive a 5xx status code. What to do if these limits are too strict for me? These limits are not per-domain but global for all Wikimedia properties, with an exception for community projects of the kind which would be eligible to run in Wikimedia Foundation’s hosted infrastructure (Wikimedia Cloud Services offerings). Specifically, bots running in Toolforge and in any other Wikimedia Cloud Services offering are explicitly exempted from such limits. We will still reserve the right to temporarily rate-limit or block individual bots that might be compromising the stability of the websites. Community bots that need a higher volume of API requests should run in WMCS or authenticate and request the bot flag from your local wiki community. Community-approved bots get higher rate limits. If you are an external entity and you need a higher volume of requests, you should use Wikimedia Enterprise APIs instead. This is the preferred choice for commercial, high-volume users of the APIs. Bot operators who are unsure how to get the access they need can contact the Wikimedia Foundation at [email protected]. See also Policy:Wikimedia Foundation User-Agent Policy Policy:Wikimedia Foundation API Usage Guidelines Wikimedia APIs/Rate limits API:Etiquette Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Robot_policy&oldid=2391151" This page was last edited on 16 March 2026, at 14:19. Text is available under the Creative Commons Attribution-ShareAlike License; additional terms may apply. See Terms of Use for details. Privacy policy About Wikitech Disclaimers Code of Conduct Developers Statistics Cookie statement Mobile view Search Search Toggle the table of contents Robot policy Add topic