Skip to content

Commit 8e02a64

Browse files
committed
[SPARK-48695][PYTHON] TimestampNTZType.fromInternal not use the deprecated methods
### What changes were proposed in this pull request? `TimestampNTZType.fromInternal` not use the deprecated methods ### Why are the changes needed? ``` In [2]: ts = 111111111111 In [3]: datetime.datetime.utcfromtimestamp(ts // 1000000).replace( ...: microsecond=ts % 1000000 ...: ) <ipython-input-3-166450e5ec26>:1: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC). datetime.datetime.utcfromtimestamp(ts // 1000000).replace( Out[3]: datetime.datetime(1970, 1, 2, 6, 51, 51, 111111) ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? new tests ### Was this patch authored or co-authored using generative AI tooling? No Closes #47068 from zhengruifeng/fix_ntz_conversion. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
1 parent 09cb592 commit 8e02a64

File tree

2 files changed

+10
-2
lines changed

2 files changed

+10
-2
lines changed

python/pyspark/sql/tests/test_serde.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,14 @@ def test_time_with_timezone(self):
9595
self.assertEqual(now, now1)
9696
self.assertEqual(now, utcnow1)
9797

98+
def test_ntz_from_internal(self):
99+
for ts in [1, 22, 333, 44444444, 5555555555]:
100+
t1 = datetime.datetime.utcfromtimestamp(ts // 1000000).replace(microsecond=ts % 1000000)
101+
t2 = datetime.datetime.fromtimestamp(ts // 1000000, datetime.timezone.utc).replace(
102+
microsecond=ts % 1000000, tzinfo=None
103+
)
104+
self.assertEqual(t1, t2)
105+
98106
# regression test for SPARK-19561
99107
def test_datetime_at_epoch(self):
100108
epoch = datetime.datetime.fromtimestamp(0)

python/pyspark/sql/types.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -434,8 +434,8 @@ def toInternal(self, dt: datetime.datetime) -> int:
434434
def fromInternal(self, ts: int) -> datetime.datetime:
435435
if ts is not None:
436436
# using int to avoid precision loss in float
437-
return datetime.datetime.utcfromtimestamp(ts // 1000000).replace(
438-
microsecond=ts % 1000000
437+
return datetime.datetime.fromtimestamp(ts // 1000000, datetime.timezone.utc).replace(
438+
microsecond=ts % 1000000, tzinfo=None
439439
)
440440

441441

0 commit comments

Comments
 (0)