[enhancement] "What's New" sorting improvement

[M86xKC]

So now the "What's New" page shows the latest video from each channel you subscribed to, but that's not how YouTube subscriptions work, youtubes' subscription list sorts videos by the publish date.

Now in NewPipe each channel have only one video in the "What's New" feed even if they published 10 videos recently "say 1 day ago" at the same time only one will show up in "What's New", and some other channels that you may subscribed to, haven't published in months and you have watched their videos it still shows them every time you go to "What's New".

So it will be nice to sort the "What's New" videos by the publish date.

[theScrabi]

It would be nice if we could have a proper feed manager like typical rss managers that shows how much videos of a channel you didn't see, and which one they are.

[karyogamy]

This problem is noted and discussed in more detail in #739, but may take some time before a good implementation surfaces. In short, the timestamp available on the webpage is insufficient in determining the time of publishing, so the latest video is used as a replacement for getting the feed. Also, in order to implement a proper feed, we have to consider how to deal with other services in the future.

[M86xKC]

@theScrabi @karyogamy @epitron @Generator @timofonic
I found this "https://github.com/ram-on/SkyTube" and it is not feature-rich as NewPipe but its "feed tab" is awesome, (It sorts the videos by time published) I hope you can fork it and get it workin' in NewPipe

[karyogamy]

@pupyc SkyTube uses Youtube API for retrieving video's publish date, something NewPipe will not support.

[jfwerner]

@karyogamy It still knows when the video was published though....

Just make it like YouTube. Just a normal feed. I don't want to click on "What's new" to see the videos. Wouldn't it be much more logical to make that "My subscribed channels" and show the Feed directly?

[epistularum]

In my opinion a subscription feed would be a major addition for anyone wishing to use newpipe as a replacement of the YouTube app.

[theScrabi]

Yes I see it like you. I wished NewPipe had this. Just someone had to do it :)

[wojcik-online]

I've been using NewPipe as a replacement for the YouTube app, and this issue has been bothering me the most. So I've started to work on an implementation, but before adding a PR, I would like to get some input.

How I imagine the feed to work:

• When the feed is first opened, it loads the ChannelInfos of all subscribed channels. This takes some time.
• In the meantime it could display cached items (saved in the db?).
• When loading is finished, it discards all videos older then a month (?) and sorts the remaining ones.
• The textual publication date is parsed. The translated resource strings would have to provide format patterns.

@theScrabi What do you think about it?

[theScrabi]

Sounds good. But isn't the date already parsed in the extractor? If not than the parsing would belong there anyway.

[karyogamy]

@wojcik-online Looks good to me too, though like @theScrabi said, we do have to take into account that different services may have different format patterns on the upload date strings. Also, I'd like request you to not change the database at the moment as it is undergoing a schema overhaul in #1004. By the way, great job on the change you made on feed fragment, I didn't know you can use Rx buffer like that.

[mauriciocolli]

@theScrabi Take a look here: YoutubeStreamInfoItemExtractor.java#L118, we are simply taking what the streaming service is displaying.

---

I think all translation stuff should happen in the front end (NewPipe) but here's some ups and downs on downloading the page in only one language:

Ups:

• We have to handle several languages and changes that the streaming service may be doing, so it would make things easier to parse
• Less testing to check support of another languages (the translation would be handled by the front end)

Downs:

• The content will be in only one language, even if the original content is in another
  • For example, the title and description that the author may be translating (and it is rather common to multiple uploaders translate it to english)
  • I don't know if there's a way to get the original when we are downloading the page accepting only one language (except the feed page, but that would only work in YouTube)

[wojcik-online]

@mauriciocolli I think the page could be downloaded in the language selected by the user and the parsing could take place in the front end. To parse every supported language, format patterns would have to be added to the resource strings.

<?xml version='1.0' encoding='UTF-8'?> 
<resources>
    <string name="service_yt_http_accept_language">en-US</string>
    <string name="service_yt_date_format_month">(\d)+ months ago</string>
</resources>

[mauriciocolli]

@wojcik-online that's the problem, it's just too many and who knows for how long they will be valid.

Even if we did, people would have to actively maintain the multitude of languages (someone who knows, at least, regexp). And then, even if the last point wasn't a problem, there's multiple services that could be using different words/phrases that would amplify the maintenance.

For these reasons, I think the cost of maintenance it's just too high.

[wojcik-online]

@mauriciocolli Do you think that the cost of keeping the following format up to date (in every language) would be too high as well? I doubt that the phrases are likely to change any time soon and the format doesn't require any knowledge of regexp.

<resources>
    <string name="upload_date_seconds_ago_phrase">sec</string>
    <string name="upload_date_minutes_ago_phrase">min</string> 
    <string name="upload_date_hours_ago_phrase">hour</string> 
    <string name="upload_date_days_ago_phrase">day</string> 
    <string name="upload_date_weeks_ago_phrase">week</string> 
    <string name="upload_date_months_ago_phrase">month</string> 
    <string name="upload_date_years_ago_phrase">year</string> 
</resources>

In the code, we could check which time unit is meant by calling: textualDateFromYoutube.contains(R.string.upload_date_hours_ago_phrase)

I have implemented a parser TimeAgoParser.java that works nicely with other services that provide exact upload times (such as SoundCloud or podcast RSS).

[mauriciocolli]

Yes, I was wondering of doing something like this, and in fact, there's a whole subject to this problem, check "natural language processing" (but parsing a "static" server generated dates wouldn't be as complicated as, for example, a real human talk).

If we could find a library good enough for that, we could use it, and if none meets our requirements, that would be the perfect opportunity to start a new project. I think it fits more as a dedicated one that could support multiple languages, and therefore, multiple services. What do you think?

Then the only thing left is turn the time to relative again, for that, there's some libraries out there.
PrettyTime, for example (it even has a parser, but it's not updated for some time now).

[wojcik-online]

@mauriciocolli There also is DateUtils.getRelativeTimeSpanString() in the Android API. It seems to automatically take the user locale into account.
I will look into PrettyTime and other parsers.

A lot of other services (e.g. SoundCloud) provide the exact upload date instead of a '2 months ago' type of thing. Do you think that the dates of other services should be displayed in the same way as the YouTube dates? This would certainly create a consistent UI.

[mauriciocolli]

@wojcik-online definitely, other than consistency, a person can identify/process a relative time much more quickly than a exact one, and for those who want it, it'll still be there in the detail page.

[wojcik-online]

@mauriciocolli I think that at the end of the day it should be the responsibility of the streaming service to parse its dates properly. I don't think it is possible to create a parser that will support all the formats of all future services in every language.
Therefore I think we should concentrate on a simple parser of YouTube dates and if future services use a similar format, this parser can be generalized. This will ensure the most gain for the least effort.
I will finish my implementation and add a Pull Request to discuss it further.

PrettyTime seems like an overkill to me. It supports centuries and millennia. (I wonder what the Romans watched on YouTube.) And the natural language parser unfortunately only supports English.

[mauriciocolli]

@wojcik-online The idea of a separate project was to create a library that could be used for other projects (specifically for parsing websites) that seems to be nonexistent, and better utilize the key words to support, at least, most services, but I see that it's a bit hard to do it correctly to every single one. For now, let's do it as you said.

PrettyTime seems like an overkill to me.

Now it definitely is, only suggested PrettyTime because of the relative strings, as I didn't know Android had it built in (and it seems to have a great support for pretty much all languages).

I will finish my implementation and add a Pull Request to discuss it further.

I'm already building/gathering the phrases and key words for YouTube for all supported languages, so you're more than welcome. (Also there's some strings that don't have an number in there (e.g just now, moments ago...), but that's an issue that we can discuss in the pr).

[theScrabi]

Android had it built

I assume you would like to put the parser into the fronted and not into the extractor then?
Could it still be considered to put it in the extractor?

[mauriciocolli]

Here's a gist of it:

• Trying to make the extractor return a normal and predefined date pattern for all services
• Show the user relative times (e.g. 5 minutes ago) for reasons mentioned above, and this is the job of the front end (conveniently, Android already have it translated and implemented)

[theScrabi]

I see. Yea thats awesome. :)

[justanidea]

Is this option going to be applied for soundcloud? :D

[wojcik-online]

@justanidea Yes, it's independent of the underlying service. There's one feed with the newest items of all services.

[justanidea]

And are we going to be able to separate them if we want?

[justanidea]

This topic makes me think we just could do a section of Newpipe with a big feed for all the services, and a specific feed for each section (just an idea hehe)
Sorry if its makes you work more (what you have already done is awesome :D)

[theScrabi]

big feed for all the services, and a specific feed for each section (just an idea hehe)

Possible :)

[532910]

It's not an "enhancement". It's the second major bug (after 1080p video, that's already fixed and allows use NewPipe as youtube player. (I'll better see youtube via firefox than 720p resolution.))

Current sorting doesn't answer the question: "Is there something new I’m subscribed to?"
This bug doesn't allow to use NewPipe to receive new media as it appears.

[Philipp0205]

Is there any update on this? :)

[Vldm2r]

Wow, this requestion has been hanging for a year now and it has never been added to the program ... Will it be implemented in the future?

[M86xKC]

@mauriciocolli I was just going to mention that we can use a mini rss reader to get accurate date of publishing, like any rss reader. That is what I use now and it is pretty accurate !

Thank you dude for the pr 👌
Hope it gets reviewed and merged soon

[Philipp0205]

Thanks for the heads-up that there is already a pull request!

[rakshazi]

Waiting it for years
@mauriciocolli thank you for your effort :)

[Denyerec]

Has this been furthered? Would be awesome for subscriptions to show unwatched videos - Bites to come back after a while and see only one, as you miss out on anything in between!

[T-vK]

Hey, can anyone give me some input on what exactly the problem is? Maybe I can help.
It sounds to me as if you just don't know how to retrieve the publishing date (instead of the upload- or update date).
That's actually a super trivial task. You just take the rss URLs for all your channels and hit them with an HTTP GET request. For example: https://www.youtube.com/feeds/videos.xml?channel_id=UCXuqSBlHAE6Xw-yeJA0Tunw (Will return all the information on recently published LinusTechTips videos)

Then you simply use a feed parsing library and extract the publish date for each video. The date is in a reliably parsable format too:

<published>2019-12-07T19:29:05+00:00</published>

(it's called ISO8601 which is very commonly used and supported by most date/time libraries)

[Stypox]

@T-vK see the discussion about feeds in #2309
To sum up the problem with feeds is that they do not provide live streams, and normal videos only provide a localization-based rounded publish date.

[T-vK]

To sum up the problem with feeds is that they do not provide live streams

Having live streams in the subscription feed seems like a really low priority compared to having it actually work at all with normal videos.

normal videos only provide a localization-based rounded publish date.

Yes, they are rounded to the second if you take the data from the feed. That seems good enough to me.
And no, the date is not localized, it's an ISO8601 date/time that doesn't change. I've tried it with several different IP addresses and several different "Accept-Language" HTTP request headers. It always returned the ISO8601 date/time with a +00:00 timezone.

Correct me if I'm wrong.

[Stypox]

What I meant by "normal videos" is videos in the channel page (without feed).
Again, see #2309