Executive brief
Protego is a Python library that parses robots.txt files to determine which web URLs a crawler may access. The library constructs regular expressions to match URLs against permission directives. A malicious robots.txt file with many wildcard characters can cause the regex engine to perform exponential backtracking, freezing the application indefinitely and preventing normal web crawling operations.
Technical details
This is a Regular Expression Denial of Service (ReDoS) vulnerability in the wildcard matching logic of protego._urlpattern._URLPattern._prepare_pattern_for_regex(). The vulnerable code translates each asterisk (*) in robots.txt directives into a lazy regex piece (.*?), so a directive like "/\*1\*1\*1...\*Z" generates a regex with many overlapping lazy quantifiers. When the can_fetch() method attempts to match a URL against this regex, the regex engine exhibits exponential backtracking—trying all possible combinations before determining non-match—causing the application to hang. The vulnerability requires an attacker to control or influence the robots.txt content served to the application. No authentication or user interaction is strictly required beyond passive acceptance of the malicious robots.txt. The issue is fixed in version 0.6.2 by replacing regex-based wildcard matching with a simpler non-regex approach.
Affected products
- scrapy Protego <= 0.6.1
Timeline
- 2026-06-25: disclosed: GHSA-wjmf-p669-5m5p published
- 2026-06-25: patched: Version 0.6.2 released with fix
- 2026-08-28: advisory: Advisory updated on GitHub