Executive brief
Scrapy is a popular Python web scraping framework used to download and process content from the internet. Its S3DownloadHandler component, which facilitates downloading objects from Amazon S3 storage, has a critical flaw: it transmits AWS authentication credentials and S3 objects over unencrypted HTTP by default instead of HTTPS. An attacker on the same network (such as on public Wi-Fi, a compromised router, or via ISP observation) can eavesdrop on this traffic to steal AWS credentials and access sensitive data stored in S3, or manipulate responses to poison scraped data.
Technical details
The vulnerability stems from insecure default configuration in scrapy/core/downloader/handlers/s3.py. The code converts s3:// scheme requests to HTTP instead of HTTPS unless request.meta["is_secure"] is explicitly set. After rewriting the URL, the handler signs the request with AWS credentials (via add_auth) and sends it over plaintext HTTP. This violates CWE-319 (Cleartext Transmission of Sensitive Information). The attack vector is network-based; any attacker positioned on the network path between Scrapy and AWS S3 can passively eavesdrop to read bucket paths, AWS Authorization headers, temporary security tokens, and S3 object contents. Active MITM attackers can modify response bodies, status codes, and headers, leading to data poisoning and cache poisoning. Exploitation requires only that a user runs Scrapy with S3 requests and AWS credentials configured—no privileges or user interaction needed. The vulnerability affects Scrapy versions prior to 2.17.0, which patches the issue.
Affected products
- Scrapy Project Scrapy < 2.17.0
Timeline
- 2026-07-07: disclosed: Vulnerability published to GitHub Advisory Database
- 2026-09-02: patched: Patched in Scrapy 2.17.0