Executive brief
Scrapy, a popular web scraping framework, incorrectly follows HTTP redirects for non-HTTP protocols like file://, ftp://, and s3://. An attacker with the ability to control a spider's start URLs could exploit this to redirect requests to local files, malicious FTP servers, or cloud storage, potentially exposing sensitive credentials or local data that the spider processes.
Technical details
Scrapy's redirect middleware (RedirectMiddleware and MetaRefreshMiddleware) follows HTTP redirects indiscriminately across all URL protocols (file://, ftp://, s3://, data://, etc.) instead of restricting them to http:// and https:// schemes. An attacker with write access to spider start_urls and read access to spider output can chain redirects to exfiltrate sensitive data: file:// redirects expose local files, ftp:// redirects can capture FTP credentials, and s3:// redirects can access cloud storage using configured S3 credentials. The vulnerability is network-reachable and requires low privilege (ability to define start URLs). The fix, available in Scrapy 2.11.2, restricts redirect following to HTTP/HTTPS only.
Affected products
- Scrapy Scrapy < 2.11.2
Timeline
- 2024-05-14: disclosed
- 2024-05-14: patched: Fixed in Scrapy 2.11.2