Channel sync

[fflorent]

Description

Implement channel synchronization, but with some lacks detailed below.

Related issues

Fixed #754
Also it paves the way for the importation of channel videos from other platforms in the UI (like peertube-import-videos).

Has this been tested?

• 👍 Manual tests done
• 👍 Unit tests done

Screenshots

EDIT1 : Screencast of admin configuration

recording-optimized-2.mp4

[Chocobozzz]

Thanks a lot for this wonderful work!

Just made a first "quick" review of the code. Will do other ones in the week!

[fflorent]

In another task: UI for managing other users synchronizations.

[fflorent]

Issue reported here #5181

[fflorent]

I took into account your remarks

[fflorent]

I need this attribute for ensureCanManageChannel which picks specifically res.locals.videoChannel.

[fflorent]

I improved a little bit the UI of the list of the synchronizations:

[fflorent]

I wonder if there could not be some issues with video orders when:

• the user creates a synchronization, requesting a channel import
• the channel import runs while a sync is run

I wonder if we should no passe an optional syncId in the payload to flag the synchronization as being processed, and skip any of sync in PROCESSING.

What do you think @Chocobozzz?

[Chocobozzz]

I'll check how we can handle this case and update the code

[Chocobozzz]

Thanks, I'll rebase from develop and merge this PR

[Chocobozzz]

🎉 🎉 🎉 🙏 :)

[emansom]

I have imported a YouTube channel to my PeerTube instances a few weeks ago, when this feature wasn't available. Is there any way I can flag the existing videos as being "imported" already?

e.g. by executing queries directly on the database?

[fflorent]

I have migrated a YouTube channel to my PeerTube instances a few weeks ago, when this feature wasn't available. Is there any way I can flag the existing videos as being "imported" already?

@emansom If these videos were imported through the Import with URL feature, you should have nothing to do, these videos should already be recognized as imported.

[emansom]

@emansom If these videos were imported through the Import with URL feature, you should have nothing to do, these videos should already be recognized as imported.

They were imported using the peertube-import-videos.js CLI and don't show up in the imported tab.
https://docs.joinpeertube.org/maintain-tools?id=peertube-import-videosjs

[fflorent]

@emansom If these videos were imported through the Import with URL feature, you should have nothing to do, these videos should already be recognized as imported.

They were imported using the peertube-import-videos.js CLI and don't show up in the imported tab. https://docs.joinpeertube.org/maintain-tools?id=peertube-import-videosjs

I see.

That being said, if you create a new sync without importing the existing videos, only videos newer than the creation date are eligible for importation.

[emansom]

I see.

That being said, if you create a new sync without importing the existing videos, only videos newer than the creation date are eligible for importation.

Which database table and which column determines the "is imported" state? Or is it another table with video ID references that I need to add entries into?

[fflorent]

I meant that you should have nothing to do, except creating a synchronization with the option "only watch for new publications". Only the videos uploaded on Youtube after the synchronizations will be imported.

If you are still willing to play with the database, the table is named videoImport and place the state value to 2. Ensure to use the sequence (videoImport_id_seq) for the id column. Note that's ok for this table if you know what you are doing, but avoid manipulating PeerTube DB tables in general (otherwise that can lead to confusion on ActivityPub).

[tio-trom]

I just want to say a huge THANK YOU! Amazing.

[Seyferto]

I meant that you should have nothing to do, except creating a synchronization with the option "only watch for new publications". Only the videos uploaded on Youtube after the synchronizations will be imported.

There's no option to import all prior ones?

[emansom]

There's no option to import all prior ones?

There is, manually. The way I did it was:

1. Change the state as described above by @fflorent

For previous videos that weren't imported within the UI itself, but rather using the CLI tool, the following steps are required:

2. Setting up pgAdmin and connecting to the PeerTube database
3. Export the videos table as CSV, opening it in LibreOffice Calc and transforming it into a structure compatible with the schema of the videoImports table
4. Importing the transformed CSV into the videoImports table using pgAdmin

Ofcourse this could be dealt with better in the form of a SQL migration script, I lack the knowledge on Postgres specific SQL to implement this.

[Seyferto]

This could also be dealt better with just an interface click that automates it... but perhaps I should get used to how mirroring and syncing will always be complex tasks, if not impossible elsewhere...