Refactor date parsing #1372

Isira-Seneviratne · 2025-09-12T02:54:27Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

Update the DateWrapper class to store an Instant, and remove the deprecated methods.
[YouTube] Improve the upload date parsing so that non-approximate dates can be returned.
[SoundCloud] Return the original instant string in the stream extractor.

…actor

TobiGr

thanks

TobiGr · 2025-09-12T17:04:09Z

...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java

+        final String dateStr = playerMicroFormatRenderer.getString("uploadDate",
+                playerMicroFormatRenderer.getString("publishDate", ""));
+        if (!dateStr.isEmpty()) {
+            return dateStr;
        }

-        final JsonObject liveDetails = playerMicroFormatRenderer.getObject(
-                "liveBroadcastDetails");
-        if (!liveDetails.getString("endTimestamp", "").isEmpty()) {
-            // an ended live stream
-            return liveDetails.getString("endTimestamp");
-        } else if (!liveDetails.getString("startTimestamp", "").isEmpty()) {
-            // a running live stream
-            return liveDetails.getString("startTimestamp");
+        final var liveDetails = playerMicroFormatRenderer.getObject("liveBroadcastDetails");
+        final String timestamp = liveDetails.getString("endTimestamp", // an ended live stream
+                liveDetails.getString("startTimestamp", "")); // a running live stream
+
+        if (!timestamp.isEmpty()) {
+            return timestamp;


this makes it harder to understand what's going on. I'd prefer to keep it readable and revert this change.

I feel like the code below line 180 as a whole could be removed. I checked how the date is being extracted, and it's always taken from uploadDate or publishDate.

YouTube might return different layouts in different cases, so it's better to keep all cases

...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java

TobiGr · 2025-09-17T06:30:37Z

Please list all the breaking changes in the PR description so they can be copied to the release notes.

TobiGr · 2025-09-17T06:42:41Z

The tests are now dependent on the runner's time zone, which is not good. While the automated tests pass (server is set to UTC), tests on my computer fail:

results


YoutubeStreamExtractorControversialTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2010-09-09T15:40:44> but was: <2010-09-09T17:40:44>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$DescriptionTestPewdiepie > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2019-08-24T15:39:57> but was: <2019-08-24T17:39:57>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$DescriptionTestUnboxing > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2018-06-19T19:41:34> but was: <2018-06-19T21:41:34>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$NoVisualMetadataVideoTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2017-05-16T14:50:53> but was: <2017-05-16T16:50:53>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$PublicBroadcasterTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2023-07-07T15:30:08> but was: <2023-07-07T17:30:08>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$RatingsDisabledTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2023-01-13T21:53:57> but was: <2023-01-13T22:53:57>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$StreamSegmentsTestMaiLab > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2020-11-19T05:30:01> but was: <2020-11-19T06:30:01>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorDefaultTest$StreamSegmentsTestMaiLab > testName() SKIPPED

YoutubeStreamExtractorDefaultTest$StreamSegmentsTestMaiLab > testTags() SKIPPED

YoutubeStreamExtractorDefaultTest$StreamSegmentsTestTagesschau > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2021-03-17T19:56:59> but was: <2021-03-17T20:56:59>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorLivestreamTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2022-07-12T12:12:29> but was: <2022-07-12T14:12:29>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorRelatedMixTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2015-07-09T16:34:35> but was: <2015-07-09T18:34:35>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

YoutubeStreamExtractorUnlistedTest > testUploadDate() FAILED
    org.opentest4j.AssertionFailedError: expected: <2017-09-22T12:15:21> but was: <2017-09-22T14:15:21>
        at app//org.schabi.newpipe.extractor.services.DefaultStreamExtractorTest.testUploadDate(DefaultStreamExtractorTest.java:206)
        at java.base@17.0.16/java.lang.reflect.Method.invoke(Method.java:569)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)
        at java.base@17.0.16/java.util.ArrayList.forEach(ArrayList.java:1511)

Stypox · 2025-09-17T08:38:52Z

Mmmh, what's the advantage of using Instant instead of OffsetDateTime? I see this PR does many changes but I don't see clear advantages. I mean, sure, some code is shorter, but the longer code we had before worked just fine, or was there some limitation I am not getting?

Isira-Seneviratne · 2025-09-18T00:27:22Z

Mmmh, what's the advantage of using Instant instead of OffsetDateTime? I see this PR does many changes but I don't see clear advantages. I mean, sure, some code is shorter, but the longer code we had before worked just fine, or was there some limitation I am not getting?

Instant represents a single point of time in the UTC time zone, while OffsetDateTime stores a zone offset. As such, I felt that storing an Instant would be better as it avoids having to convert between different time zones in order to obtain it, and a local date/time value can be obtained via their factory methods.

Isira-Seneviratne added 5 commits September 12, 2025 08:16

Update DateWrapper to store an Instant

953189f

Allow retrieval of a non-approximate DateWrapper in YoutubeStreamExtr…

758c31b

…actor

Improve tests

2dc17d4

Fix checkstyle issues, added more refactorings

6bc9d32

Refactor PeerTube date parsing method

380c303

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from cadda5e to 380c303 Compare September 12, 2025 16:46

Rm substring call

e9dd84f

TobiGr reviewed Sep 13, 2025

View reviewed changes

Apply code review suggestions

c79afc0

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from ddaef9a to c79afc0 Compare September 13, 2025 01:19

Isira-Seneviratne mentioned this pull request Sep 13, 2025

Refactor codebase to use Instant instead of OffsetDateTime TeamNewPipe/NewPipe#12633

Open

5 tasks

Isira-Seneviratne added 3 commits September 16, 2025 06:40

Do additional refactoring

f8d6f94

Use LocalDateTime instead of LocalDate

8727659

Fix some issues

f9f71cb

Fix tests

59e78c2

Avoid potential NPEs in Soundcloud extractors

f38b72c

Isira-Seneviratne force-pushed the Refactor-date-parsing branch from 0411b8a to f38b72c Compare September 18, 2025 00:54

Fix nested tests

1a5ac2f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Refactor date parsing #1372

Refactor date parsing #1372

Uh oh!

Isira-Seneviratne commented Sep 12, 2025 •

edited

Loading

Uh oh!

TobiGr left a comment

Uh oh!

TobiGr Sep 12, 2025

Uh oh!

Isira-Seneviratne Sep 13, 2025

Uh oh!

Stypox Sep 17, 2025

Uh oh!

Uh oh!

TobiGr commented Sep 17, 2025

Uh oh!

TobiGr commented Sep 17, 2025

Uh oh!

Stypox commented Sep 17, 2025

Uh oh!

Isira-Seneviratne commented Sep 18, 2025

Uh oh!

Uh oh!

Uh oh!

Refactor date parsing #1372

Are you sure you want to change the base?

Refactor date parsing #1372

Uh oh!

Conversation

Isira-Seneviratne commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiGr left a comment

Choose a reason for hiding this comment

Uh oh!

TobiGr Sep 12, 2025

Choose a reason for hiding this comment

Uh oh!

Isira-Seneviratne Sep 13, 2025

Choose a reason for hiding this comment

Uh oh!

Stypox Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TobiGr commented Sep 17, 2025

Uh oh!

TobiGr commented Sep 17, 2025

Uh oh!

Stypox commented Sep 17, 2025

Uh oh!

Isira-Seneviratne commented Sep 18, 2025

Uh oh!

Uh oh!

Isira-Seneviratne commented Sep 12, 2025 •

edited

Loading