Automatically import videos from other platforms

[roipoussiere]

In order to simplify the PeerTube adoption from current Youtubers, it's important to simplify the importing process.

The import script is a good step in this direction, but it is possible to do better: automatically import Youtube videos, so the PeerTube import can be totally transparent for the Youtuber.

We can do this without the Youtube API, by using atom feeds: https://www.youtube.com/feeds/videos.xml?channel_id=<channel_id>.

I'm not used with TypeScript but I wrote a small Python script as a proof-of-concept example, available here.

Used with the import script and a 5min-cronjob, a PeerTube user could feed its instance by only specifying its Youtube channel name.

Note that we also need a way to check if the video is not already on the PeerTube instance, can we provide the youtube video id in some video metadata?

[rigelk]

Since it's a auto import, we can assume videos previously imported have the same title as on YT. We could just check against titles for a first implementation.

[rezonant]

Maybe for a POC, but speaking as someone who does these sort of automated import ops at my day job (I work for a large YouTube channel) I can say that titles can and are modified after publishing, especially within the first hour after publishing. Can't we store the YouTube video ID in some form of metadata entry on the PeerTube video and look it up to see if we've already imported it?

[rigelk]

Ah, right :/

I guess it's not a problem to add this metadata, since it's something that doesn't need to be federated (meaning we don't break things if we add a field to the video model).

My concern is more as to how to make that metadata structure broad enough to be reused for imports from the other platforms supported by youtube-dl. I guess a HashMap is fine there.

[roipoussiere]

I can say that titles can and are modified after publishing

fyi Youtube feed also provides an updated field, so I guess if the video is renamed, it will appears in the feed for the second time, with the same published date (and youtube id) but an other updated date.

---

My concern is more as to how to make that metadata structure broad enough to be reused for imports from the other platforms supported by youtube-dl. I guess a HashMap is fine there.

But not all platforms supported by youtube-dl provides an Atom/RSS feed.

Note that a unique item id is mandatory for Atom feed:

atom:entry elements MUST contain exactly one atom:id element.

... but is optional for RSS feed:

<guid> is an optional sub-element of . guid stands for globally unique identifier. It's a string that uniquely identifies the item.

Related: The RSS bridge project provides Atom/RSS for many websites, including video providers.

---

In fact we could simply use the video URL as a unique identifier, it is supported by all platforms and supposed to be unique.

[rezonant]

URL is an elegant way to do it but canonization is a factor. Though you wouldn't run into it with auto YT import you may run into it in other components. An example is the 4-5 different ways to express a YouTube URL (YouTube.be, mobile.youtube.com/watch, gaming.youtube.com/watch, music.youtube.com/watch, YouTube.com/watch). For this reason I would recommend just having a provider and ID pair, or just "provider:providerID" format. Use .split(':', 2) to separate the two

[roipoussiere]

An example is the 4-5 different ways to express a YouTube URL (YouTube.be, mobile.youtube.com/watch, gaming.youtube.com/watch, music.youtube.com/watch, YouTube.com/watch)

Hmm yes, good point. provider and providerID could be fine.

---

Note that Vimeo and DailyMotion also officially provide feeds, where we can find video unique ids:

• <feed> / <entry> / <yt:videoId> for Youtube (example);
• <rss> / <channel> / <item> / <dm:id> for DailyMoton (example);
• <rss> / <channel> / <item> / <guid> for Vimeo (example).

So we could easily implement auto-download also for these platforms.

[Bugsbane]

This is an important idea for boosting the amount of content on the Peertube network. Once included, I would go as far as to as to ask for new users channels elsewhere that they'd like to set up for auto-import, when they're registering.

That way, creators from other platforms who make an account on a PeerTube instance "just to try it out" will automatically get up and running quickly and the network gets a ton of ongoing new content, even if they forget about their PeerTube channel and don't touch it again.

Beyond the scope of this issue, but worth considering later, would be actually doing a full import of someone's channel (not just videos), ie their channel description, avatar etc.

[McFlat]

I don't think that checking against the video title/name would be good, because users can upload different videos having the same name. it's better to store the original url for the imported video and check against the imported video url.

This is how it's done for the frontend when a user imports a video, but for the video-import script it doesn't set the original video url that's imported(targetUrl). So really it is a bug in the ./server/tools/peertube-import-videos.ts because it's not performing the same steps as the video import from the frontend, it also doesn't create entries in the videoImport table when using the script, but it does do it when done by a user from the frontend.

[McFlat]

Next I want to work on the videos import script to be like the frontend one because, the difference really affects me in a big way. I need it to check the targetUrl instead of a name/title of the video to keep out any duplicates.

[McFlat]

I think using a generic target url, converted from the many different possibilities would be best instead of doing provider:id, that way you can just use the url the way it is, instead of having to do a conversion all the time to fetch it

[gnouts]

(Maybe out of scope but) In the meantime I personnaly use @roipoussiere's Python script with small improvement and a cron job.
Here is my code : https://taboulisme.com/git/nouts/peertube-import
Though, it can only be set by the instance admin. I'm using it for duplicating some YouTube channels at https://alttube.fr/videos/local

[aliceinwire]

@gnouts I like your project but it would be more intuitive to have it on peertube anyway
It would improve discovery and usage of such tool

[fflorent]

@gnouts I would like to report an issue (in node arguments, -l is for license whereas your intent is to specify language, so this should be -L or --language). Where may I report a bug? :)

[gnouts]

@fflorent Thanks, I set up a github clone here : https://github.com/gnouts/peertube-import
@aliceinwire Sure, I agree. In the meantime it works for my usecase and actually I can't do better with my time and knowledge.

[fflorent]

While I remember having used this script to import a whole YT channel, I recently tried again and failed to run it for that purpose.

But the documentation seems to tell that we can pass the ID of a channel: https://github.com/Chocobozzz/PeerTube/blob/develop/support/doc/tools.md#peertube-import-videosjs

I also see that youtube-dl supports downloading a whole channel. I wonder if that's possible to use a cron script to automatically synchronize a YT channel. I'll investigate that but if anyone can tell me more about it, I would be very grateful! :)

[fflorent]

OK, so I could run the peertube-import-video script in order to upload a whole YT channel to Peertube. Also I figured out that we cannot rely on video names in order to detect whether a video has already been uploaded (if the YT owner renames a video, that creates duplicates…). Rather than that, in order to synchronize a YT channel with a PT one, I propose to rely on a --since parameter and use a crontab job.

For that purpose, I opened this PR: #1991

Even if that's WIP, feedback welcome :).

Florent

[Bugsbane]

Aren't all YouTube video ID's unique? If so, couldn't the import just store the YouTube video ID and then check there aren't already any videos imported with that same ID?

[fflorent]

@Bugsbane Yes, it has already been suggested here: #1991 (comment)

But it requires more efforts for implementing this.

If you are willing to contribute, please do so. I would personally appreciate this improvement :).

[Bugsbane]

If you are willing to contribute, please do so.

I'm more than happy to contribute... however my personal skills though lie in design. UX, communications and marketing rather than coding. :P

[mister-monster]

I actually found this issue (and multiple others) after i had almost completed a Python tool to do this, so i figured i would comment about it here.

https://github.com/mister-monster/YouTube2PeerTube

This is a tool that watches YouTube channels, and when new videos are found it mirrors them to a PeerTube channel.

[Chocobozzz]

@mister-monster Don't hesitate to make a MR to add your script in the documentation website: https://docs.joinpeertube.org/#/use-third-party-application

[Seyferto]

So, this functions now and available for instances (presumably after updates)?

[fflorent]

So, this functions now and available for instances (presumably after updates)?

No. It will be available once v5 is out

Edit : or if admin checkout the develop branch, but that's not something advised.

[Seyferto]

When will it be released?

[fflorent]

Expected by the end of 2022
https://joinpeertube.org/news#ideas-jpt

[Seyferto]

Why not something advised?

[fflorent]

Why not something advised?

In production, because of some bugs or unstable / broken features in develop.

[Seyferto]

Testing takes 5 months?

[Booteille]

Testing takes 5 months?

PeerTube minor releases are made every 3 months or so. Major releases are made every year.

[Seyferto]

Is this minor? It should have really been there since the beginning, though, as doing them one by one is so tedious...

[Chocobozzz]

@Seyferto Next minor release is planned for September. If you're not happy with this ETA you can use nightly builds at your own risks: https://builds.joinpeertube.org/nightly/

Complaining about the time a feature is released in a free software project without a thank you when @fflorent took a lot of its own free time to develop is quite annoying. See also the section of our FAQ regarding this kind of remark: https://joinpeertube.org/faq#peertube-does-not-contain-all-the-tools-i-need-to-manage-my-instance

[Seyferto]

I did actually like it when I noticed it was being done, but was disappointed if it was going to be released by the end of the year... if it's next month then that is better... so, nightly has this feature? Do you suppose others' instances usually use nightlies, though, or do most people typically wait for a release?

[Booteille]

I did actually like it when I noticed it was being done, but was disappointed if it was going to be released by the end of the year... if it's next month then that is better... so, nightly has this feature? Do you suppose others' instances usually use nightlies, though, or do most people typically wait for a release?

Actually, @fflorent told you using nightly builds are not recommended since it's considered unstable. Some admins use it, others don't. That's up to you to do so.

Nightly has every commit merged. It's the "dev" branch.

[emansom]

In production, because of some bugs or unstable / broken features in develop.

@fflorent For people wanting to help test this new feature; are you referring to active bugs with this new functionality or more as a general statement/warning?

[fflorent]

@emansom That's a general statement, yes :)

[Seyferto]

@fflorent Is this being tested anywhere as of now?