[SPARK-51415][SQL] Make timestamp from date and time #51179

MaxGekk · 2025-06-14T14:36:34Z

What changes were proposed in this pull request?

In the PR, I propose to extend the make_timestamp function, and accept a date and time field + optional time zone.

Syntax

make_timestamp(date[, time[, timezone]])

Arguments

date: A date expression
time: A time expression

Returns

A TIMESTAMP.

Examples

> SELECT make_timestamp(DATE'2014-12-28', TIME'6:30:45.887');
 2014-12-28 06:30:45.887

Why are the changes needed?

Users will be able to create a timestamp by combining a time and a date.

Does this PR introduce any user-facing change?

No, it just extends the existing API.

How was this patch tested?

By running the affected test suites:

$ build/sbt "test:testOnly *ExpressionInfoSuite"
$ build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z timestamp.sql"

Was this patch authored or co-authored using generative AI tooling?

No.

mihailom-db · 2025-06-16T08:30:14Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala

@@ -2746,7 +2746,11 @@ object TryMakeTimestampLTZExpressionBuilder extends ExpressionBuilder {

 // scalastyle:off line.size.limit
 @ExpressionDescription(
-  usage = "_FUNC_(year, month, day, hour, min, sec[, timezone]) - Create timestamp from year, month, day, hour, min, sec and timezone fields. The result data type is consistent with the value of configuration `spark.sql.timestampType`. If the configuration `spark.sql.ansi.enabled` is false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.",
+  usage = """
+    _FUNC_(year, month, day, hour, min, sec[, timezone]) - Create timestamp from year, month, day, hour, min, sec and timezone fields. The result data type is consistent with the value of configuration `spark.sql.timestampType`. If the configuration `spark.sql.ansi.enabled` is false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.


nit: Should we move the output type and error out comments after the second declaration of the function?

MaxGekk · 2025-06-17T12:26:23Z

@srielau @uros-db @mihailom-db Please, review the PR.

srielau · 2025-06-17T14:28:22Z

What is the exact type? Since both DATE and TIME are WITHOUT TIMEZONE , presumably the result should be without timezone.
Do we have a separate make_timestamp_ntz() version?

uros-db · 2025-06-17T17:22:45Z

sql/core/src/test/resources/sql-tests/analyzer-results/datetime-legacy.sql.out

+-- !query
+SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
+-- !query analysis
+[Analyzer test output redacted due to nondeterminism]


What's exactly non-deterministic here?

Not clear to me too. I have leaved a question in the PR: https://github.com/apache/spark/pull/40496/files#r2154235184

I would assume that we might call current_timestamp or similar functions that could produce different literals. But if this is the case, I would say this is the bug in the new system where we do not filter on the specific expressions, but on literals in general. cc: @dtenedor on this PR, for context

uros-db · 2025-06-17T17:23:19Z

sql/core/src/test/resources/sql-tests/inputs/timestamp.sql

@@ -169,3 +169,8 @@ select timediff(SECOND, date'2022-02-15', timestamp'2022-02-14 23:59:59');

 select timediff('MINUTE', timestamp'2023-02-14 01:02:03', timestamp'2023-02-14 02:00:03');
 select timediff('YEAR', date'2020-02-15', date'2023-02-15');
+
+-- Construct timestamp from date and time


How do we handle null values? Can we add tests for this?

I added tests. Thanks.

mihailom-db · 2025-06-17T20:08:19Z

@srielau We do have make_timestamp_ntz(). This is the base function which takes timezone from the config or from the input. When it comes to date/time not having zone, I would say neither do hour, minute, second, day, year, month from which the original make_timestamp was done. Should we maybe just improve the docs around this?

mihailom-db

LGTM, apart from few comments from @uros-db

MaxGekk · 2025-06-18T10:31:22Z

presumably the result should be without timezone

It depends on the config as the function doc says: "The result data type is consistent with the value of configuration spark.sql.timestampType.". If the result type is TIMESTAMP_NTZ, return local timestamp otherwise non-local using provided time zone or from the SQL config.

New behaviour of two parameters date and time is consistent to existing one with the year, month, ... seconds.

MaxGekk · 2025-06-18T10:44:20Z

sql/core/src/test/resources/sql-tests/analyzer-results/datetime-legacy.sql.out

+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+  "errorClass" : "FAILED_FUNCTION_CALL",


Seems like we don't properly handle NULLs in time expressions, at least by HoursOfTime. I will recheck this.

uros-db

Thanks for addressing the comments Max!

MaxGekk · 2025-06-18T17:26:47Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/timeExpressions.scala

    StaticInvoke(
      classOf[DateTimeUtils.type],
      DecimalType(8, 6),
      "getSecondsOfTimeWithFraction",
      Seq(child, Literal(precision)),
      Seq(child.dataType, IntegerType))
  }
-  private val precision: Int = child.dataType.asInstanceOf[TimeType].precision


I made the fix here in the PR because the bug is triggered when the expression is invoked from MakeTimestamp with NullType directly (without checking inputTypes).

MaxGekk · 2025-06-19T07:28:56Z

@LuciferYang @yaooqinn @dongjoon-hyun Could you review this PR, please.

Make timestamp from date and time

6d17c5c

github-actions bot added the SQL label Jun 14, 2025

MaxGekk changed the title ~~[WIP][SQL] Make timestamp from date and time~~ [WIP][SPARK-51415][SQL] Make timestamp from date and time Jun 14, 2025

mihailom-db reviewed Jun 16, 2025

View reviewed changes

Address a review comment

6ba1ce6

MaxGekk changed the title ~~[WIP][SPARK-51415][SQL] Make timestamp from date and time~~ [SPARK-51415][SQL] Make timestamp from date and time Jun 17, 2025

MaxGekk marked this pull request as ready for review June 17, 2025 12:25

uros-db reviewed Jun 17, 2025

View reviewed changes

mihailom-db approved these changes Jun 17, 2025

View reviewed changes

Add tests for NULLs

24fddf8

MaxGekk commented Jun 18, 2025

View reviewed changes

uros-db approved these changes Jun 18, 2025

View reviewed changes

MaxGekk added 2 commits June 18, 2025 20:22

Fix the expression seconds with fractions

ee89892

Merge remote-tracking branch 'origin/master' into make_timestamp

a751d60

MaxGekk commented Jun 18, 2025

View reviewed changes

MaxGekk requested a review from cloud-fan June 18, 2025 20:09

[SPARK-51415][SQL] Make timestamp from date and time #51179

Are you sure you want to change the base?

[SPARK-51415][SQL] Make timestamp from date and time #51179

Conversation

MaxGekk commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Syntax

Arguments

Returns

Examples

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented Jun 17, 2025

Uh oh!

srielau commented Jun 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGekk Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mihailom-db commented Jun 17, 2025

Uh oh!

mihailom-db left a comment

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented Jun 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uros-db left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented Jun 19, 2025

Uh oh!

Uh oh!

MaxGekk commented Jun 14, 2025 •

edited

Loading

MaxGekk Jun 18, 2025 •

edited

Loading