[Bug] [spark] Schema evolution "write.merge-schema" not working as expected.

[zukowskilz]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

1.2.0

Compute Engine

Spark 3.4.4

Minimal reproduce step

use schema evolution

data.write.format("paimon")
.mode("append")
.option("write.merge-schema", "true")
.save(location)

What doesn't meet your expectations?

When I added new field I recived exception from here

paimon/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonAnalysis.scala

Line 86 in c0f1ffb

if (inputCols.size > expectedCols.size) {

if (inputCols.size > expectedCols.size) {
  throw new RuntimeException(
    s"Cannot write incompatible data for the table `${table.name}`, " +
      "the number of data columns don't match with the table schema's.")
}

This check is executed from PaimonSparkSessionExtensions before write, without taking into account "write.merge-schema" option set as true

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!