Skip to content

[SPARK-51415][SQL] Make timestamp from date and time #51179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
+462 −5
Open

[SPARK-51415][SQL] Make timestamp from date and time #51179

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6d17c5c
Make timestamp from date and time
MaxGekk Jun 14, 2025
6ba1ce6
Address a review comment
MaxGekk Jun 16, 2025
24fddf8
Add tests for NULLs
MaxGekk Jun 18, 2025
ee89892
Fix the expression seconds with fractions
MaxGekk Jun 18, 2025
a751d60
Merge remote-tracking branch 'origin/master' into make_timestamp
MaxGekk Jun 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion ...talyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2746,7 +2746,13 @@ object TryMakeTimestampLTZExpressionBuilder extends ExpressionBuilder {

// scalastyle:off line.size.limit
@ExpressionDescription(
usage = "_FUNC_(year, month, day, hour, min, sec[, timezone]) - Create timestamp from year, month, day, hour, min, sec and timezone fields. The result data type is consistent with the value of configuration `spark.sql.timestampType`. If the configuration `spark.sql.ansi.enabled` is false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.",
usage = """
_FUNC_(year, month, day, hour, min, sec[, timezone]) - Create timestamp from year, month, day, hour, min, sec and timezone fields.
_FUNC_(date, time[, timezone]) - Create timestamp from date, time and time zone.
The result data type is consistent with the value of configuration `spark.sql.timestampType`. If the configuration `spark.sql.ansi.enabled` is false, the function returns NULL on invalid inputs. Otherwise, it will throw an error instead.
""",
arguments = """
Arguments:
* year - the year to represent, from 1 to 9999
Expand All @@ -2758,6 +2764,8 @@ object TryMakeTimestampLTZExpressionBuilder extends ExpressionBuilder {
The value can be either an integer like 13 , or a fraction like 13.123.
If the sec argument equals to 60, the seconds field is set
to 0 and 1 minute is added to the final timestamp.
* date - a date to represent, from 0001-01-01 to 9999-12-31
* time - a local time to represent, from 00:00:00 to 23:59:59.999999
* timezone - the time zone identifier. For example, CET, UTC and etc.
""",
examples = """
Expand All @@ -2772,6 +2780,10 @@ object TryMakeTimestampLTZExpressionBuilder extends ExpressionBuilder {
2019-06-30 23:59:01
> SELECT _FUNC_(null, 7, 22, 15, 30, 0);
NULL
> SELECT _FUNC_(make_date(2014, 12, 28), make_time(6, 30, 45.887));
2014-12-28 06:30:45.887
> SELECT _FUNC_(DATE'2014-12-28', TIME'6:30:45.887', 'CET');
2014-12-27 21:30:45.887
""",
group = "datetime_funcs",
since = "3.0.0")
Expand Down Expand Up @@ -2813,6 +2825,17 @@ case class MakeTimestamp(
SQLConf.get.timestampType)
}

def this(date: Expression, time: Expression) = {
this(Year(date), Month(date), DayOfMonth(date),
HoursOfTime(time), MinutesOfTime(time), SecondsOfTimeWithFraction(time))
}

def this(date: Expression, time: Expression, timezone: Expression) = {
this(Year(date), Month(date), DayOfMonth(date),
HoursOfTime(time), MinutesOfTime(time), SecondsOfTimeWithFraction(time),
timezone)
}

override def children: Seq[Expression] = Seq(year, month, day, hour, min, sec) ++ timezone
// Accept `sec` as DecimalType to avoid loosing precision of microseconds while converting
// them to the fractional part of `sec`. For accepts IntegerType as `sec` and integer can be
Expand Down
18 changes: 18 additions & 0 deletions sql/core/src/test/resources/sql-tests/analyzer-results/datetime-legacy.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1899,3 +1899,21 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's exactly non-deterministic here?

Copy link
Member Author

@MaxGekk MaxGekk Jun 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not clear to me too. I have leaved a question in the PR: https://github.com/apache/spark/pull/40496/files#r2154235184

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would assume that we might call current_timestamp or similar functions that could produce different literals. But if this is the case, I would say this is the bug in the new system where we do not filter on the specific expressions, but on literals in general. cc: @dtenedor on this PR, for context



-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]
18 changes: 18 additions & 0 deletions sql/core/src/test/resources/sql-tests/analyzer-results/nonansi/timestamp.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1015,3 +1015,21 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]
18 changes: 18 additions & 0 deletions sql/core/src/test/resources/sql-tests/analyzer-results/timestamp.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -943,3 +943,21 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]
18 changes: 18 additions & 0 deletions sql/core/src/test/resources/sql-tests/analyzer-results/timestampNTZ/timestamp-ansi.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -964,3 +964,21 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]
18 changes: 18 additions & 0 deletions sql/core/src/test/resources/sql-tests/analyzer-results/timestampNTZ/timestamp.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1024,3 +1024,21 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query analysis
[Analyzer test output redacted due to nondeterminism]
5 changes: 5 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/timestamp.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,3 +169,8 @@ select timediff(SECOND, date'2022-02-15', timestamp'2022-02-14 23:59:59');

select timediff('MINUTE', timestamp'2023-02-14 01:02:03', timestamp'2023-02-14 02:00:03');
select timediff('YEAR', date'2020-02-15', date'2023-02-15');

-- Construct timestamp from date and time
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we handle null values? Can we add tests for this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added tests. Thanks.

SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0');
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET');
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456');
24 changes: 24 additions & 0 deletions sql/core/src/test/resources/sql-tests/results/datetime-legacy.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2495,3 +2495,27 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query schema
struct<make_timestamp(year(DATE '0001-01-01'), month(DATE '0001-01-01'), dayofmonth(DATE '0001-01-01'), hour(TIME '00:00:00'), minute(TIME '00:00:00'), secondsoftimewithfraction(TIME '00:00:00')):timestamp>
-- !query output
0001-01-01 00:00:00


-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query schema
struct<make_timestamp(year(DATE '2025-06-14'), month(DATE '2025-06-14'), dayofmonth(DATE '2025-06-14'), hour(TIME '17:25:30.5'), minute(TIME '17:25:30.5'), secondsoftimewithfraction(TIME '17:25:30.5'), CET):timestamp>
-- !query output
2025-06-14 08:25:30.5


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query schema
struct<make_timestamp(year(DATE '9999-12-31'), month(DATE '9999-12-31'), dayofmonth(DATE '9999-12-31'), hour(TIME '23:59:59.123456'), minute(TIME '23:59:59.123456'), secondsoftimewithfraction(TIME '23:59:59.123456')):timestamp>
-- !query output
9999-12-31 23:59:59.123456
24 changes: 24 additions & 0 deletions sql/core/src/test/resources/sql-tests/results/nonansi/timestamp.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1213,3 +1213,27 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query schema
struct<make_timestamp(year(DATE '0001-01-01'), month(DATE '0001-01-01'), dayofmonth(DATE '0001-01-01'), hour(TIME '00:00:00'), minute(TIME '00:00:00'), secondsoftimewithfraction(TIME '00:00:00')):timestamp>
-- !query output
0001-01-01 00:00:00


-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query schema
struct<make_timestamp(year(DATE '2025-06-14'), month(DATE '2025-06-14'), dayofmonth(DATE '2025-06-14'), hour(TIME '17:25:30.5'), minute(TIME '17:25:30.5'), secondsoftimewithfraction(TIME '17:25:30.5'), CET):timestamp>
-- !query output
2025-06-14 08:25:30.5


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query schema
struct<make_timestamp(year(DATE '9999-12-31'), month(DATE '9999-12-31'), dayofmonth(DATE '9999-12-31'), hour(TIME '23:59:59.123456'), minute(TIME '23:59:59.123456'), secondsoftimewithfraction(TIME '23:59:59.123456')):timestamp>
-- !query output
9999-12-31 23:59:59.123456
24 changes: 24 additions & 0 deletions sql/core/src/test/resources/sql-tests/results/timestamp.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1220,3 +1220,27 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query schema
struct<make_timestamp(year(DATE '0001-01-01'), month(DATE '0001-01-01'), dayofmonth(DATE '0001-01-01'), hour(TIME '00:00:00'), minute(TIME '00:00:00'), secondsoftimewithfraction(TIME '00:00:00')):timestamp>
-- !query output
0001-01-01 00:00:00


-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query schema
struct<make_timestamp(year(DATE '2025-06-14'), month(DATE '2025-06-14'), dayofmonth(DATE '2025-06-14'), hour(TIME '17:25:30.5'), minute(TIME '17:25:30.5'), secondsoftimewithfraction(TIME '17:25:30.5'), CET):timestamp>
-- !query output
2025-06-14 08:25:30.5


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query schema
struct<make_timestamp(year(DATE '9999-12-31'), month(DATE '9999-12-31'), dayofmonth(DATE '9999-12-31'), hour(TIME '23:59:59.123456'), minute(TIME '23:59:59.123456'), secondsoftimewithfraction(TIME '23:59:59.123456')):timestamp>
-- !query output
9999-12-31 23:59:59.123456
24 changes: 24 additions & 0 deletions sql/core/src/test/resources/sql-tests/results/timestampNTZ/timestamp-ansi.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1216,3 +1216,27 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query schema
struct<make_timestamp(year(DATE '0001-01-01'), month(DATE '0001-01-01'), dayofmonth(DATE '0001-01-01'), hour(TIME '00:00:00'), minute(TIME '00:00:00'), secondsoftimewithfraction(TIME '00:00:00')):timestamp_ntz>
-- !query output
0001-01-01 00:00:00


-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query schema
struct<make_timestamp(year(DATE '2025-06-14'), month(DATE '2025-06-14'), dayofmonth(DATE '2025-06-14'), hour(TIME '17:25:30.5'), minute(TIME '17:25:30.5'), secondsoftimewithfraction(TIME '17:25:30.5'), CET):timestamp_ntz>
-- !query output
2025-06-14 17:25:30.5


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query schema
struct<make_timestamp(year(DATE '9999-12-31'), month(DATE '9999-12-31'), dayofmonth(DATE '9999-12-31'), hour(TIME '23:59:59.123456'), minute(TIME '23:59:59.123456'), secondsoftimewithfraction(TIME '23:59:59.123456')):timestamp_ntz>
-- !query output
9999-12-31 23:59:59.123456
24 changes: 24 additions & 0 deletions sql/core/src/test/resources/sql-tests/results/timestampNTZ/timestamp.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1192,3 +1192,27 @@ org.apache.spark.sql.catalyst.parser.ParseException
"fragment" : "timediff('YEAR', date'2020-02-15', date'2023-02-15')"
} ]
}


-- !query
SELECT make_timestamp(DATE'0001-01-01', TIME'0:0:0')
-- !query schema
struct<make_timestamp(year(DATE '0001-01-01'), month(DATE '0001-01-01'), dayofmonth(DATE '0001-01-01'), hour(TIME '00:00:00'), minute(TIME '00:00:00'), secondsoftimewithfraction(TIME '00:00:00')):timestamp_ntz>
-- !query output
0001-01-01 00:00:00


-- !query
SELECT make_timestamp(DATE'2025-06-14', TIME'17:25:30.5', 'CET')
-- !query schema
struct<make_timestamp(year(DATE '2025-06-14'), month(DATE '2025-06-14'), dayofmonth(DATE '2025-06-14'), hour(TIME '17:25:30.5'), minute(TIME '17:25:30.5'), secondsoftimewithfraction(TIME '17:25:30.5'), CET):timestamp_ntz>
-- !query output
2025-06-14 17:25:30.5


-- !query
SELECT make_timestamp(DATE'9999-12-31', TIME'23:59:59.123456')
-- !query schema
struct<make_timestamp(year(DATE '9999-12-31'), month(DATE '9999-12-31'), dayofmonth(DATE '9999-12-31'), hour(TIME '23:59:59.123456'), minute(TIME '23:59:59.123456'), secondsoftimewithfraction(TIME '23:59:59.123456')):timestamp_ntz>
-- !query output
9999-12-31 23:59:59.123456