add flag

Author: zhengruifeng

python/docs/source/migration_guide/pyspark_upgrade.rst

@@ -22,7 +22,7 @@ Upgrading PySpark
 Upgrading from PySpark 4.0 to 4.1
 ---------------------------------
 
-* In Spark 4.1, ``DataFrame['column_name']`` on Spark Connect Python Client will not eagerly validate the column name which might need an RPC. To restore the legacy behavior, set ``spark.sql.pyspark.legacy.eagerColumnNameValidation.enabled`` to ``true``.
+* In Spark 4.1, ``DataFrame['name']`` and ``DataFrame.name`` on Spark Connect Python Client no longer eagerly validate the column name. To restore the legacy behavior, set ``spark.sql.pyspark.legacy.eagerColumnNameValidation.enabled`` to ``true``.
 * In Spark 4.1, Arrow-optimized Python UDF supports UDT input / output instead of falling back to the regular UDF. To restore the legacy behavior, set ``spark.sql.execution.pythonUDF.arrow.legacy.fallbackOnUDT`` to ``true``.
 * In Spark 4.1, unnecessary conversion to pandas instances is removed when ``spark.sql.execution.pythonUDTF.arrow.enabled`` is enabled. As a result, the type coercion changes when the produced output has a schema different from the specified schema. To restore the previous behavior, enable ``spark.sql.legacy.execution.pythonUDTF.pandas.conversion.enabled``.
 

python/pyspark/sql/connect/dataframe.py

@@ -1704,7 +1704,10 @@ def __getattr__(self, name: str) -> "Column":
                 errorClass="JVM_ATTRIBUTE_NOT_SUPPORTED", messageParameters={"attr_name": name}
             )
 
-        if name not in self.columns:
+        if (
+            os.environ.get("spark.sql.pyspark.legacy.eagerColumnNameValidation.enabled") == "true"
+            and name not in self.columns
+        ):
             raise PySparkAttributeError(
                 errorClass="ATTRIBUTE_NOT_SUPPORTED", messageParameters={"attr_name": name}
             )