CVE-2017-14158: Scrapy uncontrolled resource consumption in S3FilesStore

Executive brief

Scrapy, a popular web crawling and data extraction framework, is vulnerable to a denial of service attack. An attacker can provide large files that exhaust the system's memory, potentially causing the application to crash or become unresponsive. This is particularly impactful when the framework is configured to save data to slow storage services like Amazon S3.

Technical details

Scrapy (versions 0.7 through 2.15.0) contains an uncontrolled resource consumption vulnerability (CWE-400). The issue stems from the interaction between the `dataReceived` handler in `core/downloader/handlers/http11.py` and the `S3FilesStore` component. When crawling, Scrapy may read an arbitrary number of large files into memory simultaneously. If these files are being written to a slow storage backend in a separate thread, the memory consumption can grow until the process crashes. This can be triggered by a remote attacker serving large files to the crawler. No patch is currently specified in the advisory for the affected version range.

Affected products

• Scrapy Scrapy >= 0.7, <= 2.15.0

Timeline

• 2017-09-05: advisory: NVD published CVE-2017-14158
• 2022-05-17: advisory: GitHub Advisory published

References

• https://api.github.com/users/jhutchings1
• https://github.com/jhutchings1
• https://api.github.com/users/jhutchings1/gists%7B/gist_id%7D
• https://api.github.com/users/jhutchings1/repos
• https://avatars.githubusercontent.com/u/12853539?v=4
• https://api.github.com/users/jhutchings1/events%7B/privacy%7D