df.show() would cause multi runs of operations

[xy-xin]

Suppose we have two UDFs: udf1 and udf2, and the following code:

df = xxx

df = df.with_column("output1", udf1(col("input1")))
**df.show()**

df = df.with_column("output2", udf2(col("input2")))
**df.show()**

In this case, udf1 will be triggered twice, which is not what the user expects.

We can work around this by using collect():

df = xxx

df = df.with_column("output1", udf1(col("input1"))
**df.collect()**
**df.show()**

df = df.with_column("output2", udf2(col("input2"))
**df.show()**

However, this requires the user to have experience with how Daft works, which can be challenging for algorithm developers.

In conclusion, users must understand how Daft works and what happens when calling show(). Otherwise, this behavior may confuse users.

[srilman]

I think the misunderstanding here is that df.show() does more than just run df. It also applies a limit to the end of the query plan.

So if df represents a plan like

ParquetScan -> Projection -> UDF

then df.show will actually run

ParquetScan -> Limit -> Project -> UDF

The limit it in the front because of limit pushdown. And so it is not always possible to modify df and run df.show() again and expect it to do the same thing. For example, if we added a groupby, we can't push the limit all the way though and the plan is now too different.

In your example, it is probably fine, but we need to somehow save the plan and showed output together, which we don't do for memory purposes. Wdyt?