PYSEC-2019-44 - Prior to Spark 2.3.3, in certain situations Spark would write user data to local disk unencrypted, even if spark.io.encryption.enabled=true.

Severity: info · Published 2019-08-07

Technologies: pyspark (PyPI). Vendors: PyPI.

Executive brief

Prior to Spark 2.3.3, in certain situations Spark would write user data to local disk unencrypted, even if spark.io.encryption.enabled=true. This includes cached blocks that are fetched to disk (controlled by spark.maxRemoteBlockSizeFetchToMem); in SparkR, using parallelize; in Pyspark, using broadcast and parallelize; and use of python udfs.

Affected products

PyPI pyspark

Related threats

Apache Spark History Server XSS in job names
Apache Spark Inadequate Encryption Strength in RPC Communication
PYSEC-2023-72 - ** UNSUPPORTED WHEN ASSIGNED ** The Apache Spark UI offers the possibility to enable ACLs via the configuration option spark.acls.enable. Wi
PYSEC-2023-44 - In Apache Spark versions prior to 3.4.0, applications using spark-submit can specify a 'proxy-user' to run as, limiting privileges. The appl
PYSEC-2022-42976 - A stored cross-site scripting (XSS) vulnerability in Apache Spark 3.2.1 and earlier, and 3.3.0, allows remote attackers to execute arbitrary