[spark] Spark 3.3 throws write exceptions during partial-update engine operations

[Aiden-Dong]

Purpose

Linked issue: #5932

fix bug with Spark 3.3 throws write exceptions during partial-update engine operations

CREATE TABLE T( 
    f1 int, 
    f2 string, 
    f3 string, 
    f4 string) 
 TBLPROPERTIES (    
  'bucket'='1', 
  'primary-key'='f1', 
  'write-only'='true', 
  'merge-engine'='partial-update',   
  'file.format'='parquet', 
  'file.compression' = 'zstd'  
 )

INSERT INTO T(f1, f4) VALUES(1, 'test')

Tests

spark.sql(
      "CREATE TABLE T( f1 int, f2 string, f3 string, f4 string)\n" +
        "TBLPROPERTIES (\n" + "  'bucket'='1',\n" +
        "  'primary-key'='f1',\n" +
        "  'write-only'='true',\n" +
        "  'merge-engine'='partial-update',\n" + "" +
        "  'file.format'='parquet',\n" +
        "  'file.compression' = 'zstd'\n" + ")")

    spark.sql("INSERT INTO T(f1, f4) VALUES(1, 'test')")

    checkAnswer(
      spark.sql("SELECT * FROM T"),
      Row(1, null, null, "test")
    )

API and Format

Documentation

[cxzl25]

Interesting function, so that we don't need a higher version of Spark and we can only specify some columns to update.

---

[SPARK-38795][SQL] Support INSERT INTO user specified column lists with DEFAULT values

https://issues.apache.org/jira/browse/SPARK-38795