PYSEC-2019-44 - Prior to Spark 2.3.3, in certain situations Spark would write user data to local disk unencrypted, even if spark.io.encryption.enabled=true.

Severity: info · Published 2019-08-07

Prior to Spark 2.3.3, in certain situations Spark would write user data to local disk unencrypted, even if spark.io.encryption.enabled=true. This includes cached blocks that are fetched to disk (controlled by spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in Pyspark, using broadcast and parallelize; and use of python udfs.

Technologies: pyspark (PyPI). Vendors: PyPI.

Quick answers

Is This vulnerability exploited in the wild?
No. Junglewise Threat Intelligence has no report of This vulnerability being exploited in the wild.
How severe is This vulnerability?
Informational severity, published 7 August 2019.
Which products and versions are affected?
pyspark (PyPI) ECOSYSTEM:introduced=0,fixed=2.3.3.
Is there a fix for This vulnerability?
Yes. pyspark (PyPI): upgrade to 2.3.3 or later. Junglewise finds every place you run it, drafts the upgrade and applies it once you approve.
Am I affected?
Check whether you run pyspark (PyPI) at an affected version. Junglewise does that check for you around the clock: connect your code, cloud and domains, and it matches new vulnerabilities like this one against what you actually run and tells you when one reaches you.

How to fix

  • pyspark (PyPI): upgrade to 2.3.3 or later
    pip install "pyspark>=2.3.3"

Affected products

  • pyspark (PyPI) ECOSYSTEM:introduced=0,fixed=2.3.3

This page as Markdown. Free to reuse under CC BY 4.0 with a link back.