Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Channel sync #5135

Merged
merged 71 commits into from
Aug 10, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
dafcc07
Add external channel URL for channel update / creation (#754)
fflorent Mar 28, 2022
3d9cdf2
Disallow synchronisation if user has no video quota (#754)
fflorent Mar 28, 2022
5295073
More constraints serverside (#754)
fflorent Mar 28, 2022
2899d4a
Disable sync if server configuration does not allow HTTP import (#754)
fflorent Apr 6, 2022
d78fcc2
Working version synchronizing videos with a job (#754)
fflorent Apr 8, 2022
bd0a4b3
More logs and try/catch (#754)
fflorent Apr 11, 2022
1b7448b
Fix eslint error (#754)
fflorent Apr 11, 2022
6ac7dfd
WIP: support synchronization time change (#754)
fflorent Apr 15, 2022
1cd8538
New frontend #754
fflorent May 15, 2022
60dd68a
WIP: Create sync front (#754)
fflorent Jul 5, 2022
38b568f
Enhance UI, sync creation form (#754)
fflorent Jul 6, 2022
18c5316
Warning message when HTTP upload is disallowed
fflorent Jul 6, 2022
9c5d8a3
More consistent names (#754)
fflorent Jul 6, 2022
7d1f63c
Binding Front with API (#754)
fflorent Jul 13, 2022
beb0bb1
Add a /me API (#754)
fflorent Jul 13, 2022
1a25f15
Improve list UI (#754)
fflorent Jul 13, 2022
7c07eea
Implement creation and deletion routes (#754)
fflorent Jul 14, 2022
f470a38
Lint (#754)
fflorent Jul 14, 2022
9fa21bf
Lint again (#754)
fflorent Jul 14, 2022
0b6d378
WIP: UI for triggering import existing videos (#754)
fflorent Jul 15, 2022
a65364f
Implement jobs for syncing and importing channels
fflorent Jul 17, 2022
54a8995
Don't sync videos before sync creation + avoid concurrency issue (#754)
fflorent Jul 17, 2022
22864f2
Cleanup (#754)
fflorent Jul 19, 2022
9d53de6
Cleanup: OpenAPI + API rework (#754)
fflorent Jul 20, 2022
1e29d34
Remove dead code (#754)
fflorent Jul 20, 2022
4c9ebbc
Eslint (#754)
fflorent Jul 20, 2022
df220b5
Revert the mess with whitespaces in constants.ts (#754)
fflorent Jul 25, 2022
80d7a88
Some fixes after rebase (#754)
fflorent Jul 25, 2022
3e11cf5
Several fixes after PR remarks (#754)
fflorent Jul 26, 2022
11e7b7b
Front + API: Rename video-channels-sync to video-channel-syncs (#754)
fflorent Jul 26, 2022
e250276
Allow enabling channel sync through UI (#754)
fflorent Jul 27, 2022
518f4e0
getChannelInfo (#754)
fflorent Jul 28, 2022
1d148d5
Minor fixes: openapi + model + sql (#754)
fflorent Jul 28, 2022
6c7a269
Simplified API validators (#754)
fflorent Jul 28, 2022
242f827
Rename MChannelSync to MChannelSyncChannel (#754)
fflorent Jul 29, 2022
1970660
Add command for VideoChannelSync (#754)
fflorent Jul 29, 2022
c89c40c
Use synchronization.enabled config (#754)
fflorent Aug 1, 2022
ec21cc3
Check parameters test + some fixes (#754)
fflorent Aug 1, 2022
a1cc50c
Fix conflict mistake (#754)
fflorent Aug 1, 2022
63664bf
Restrict access to video channel sync list API (#754)
fflorent Aug 1, 2022
f1e7276
Start adding unit test for synchronization (#754)
fflorent Aug 1, 2022
a14e312
Continue testing (#754)
fflorent Aug 2, 2022
720f865
Tests finished + convertion of job to scheduler (#754)
fflorent Aug 2, 2022
d5806ae
Add lastSyncAt field (#754)
fflorent Aug 2, 2022
c79ceb9
Fix externalRemoteUrl sort + creation date not well formatted (#754)
fflorent Aug 3, 2022
9e44753
Small fix (#754)
fflorent Aug 3, 2022
203c729
Factorize addYoutubeDLImport and buildVideo (#754)
fflorent Aug 3, 2022
1b5cb11
Check duplicates on channel not on users (#754)
fflorent Aug 3, 2022
b2f35a6
factorize thumbnail generation (#754)
fflorent Aug 4, 2022
0728c5d
Fetch error should return status 400 (#754)
fflorent Aug 4, 2022
4e1bff0
Separate video-channel-import and video-channel-sync-latest (#754)
fflorent Aug 4, 2022
bc405b0
Bump DB migration version after rebase (#754)
fflorent Aug 4, 2022
a9a3848
Prettier states in UI table (#754)
fflorent Aug 4, 2022
9bd1226
Add DefaultScope in VideoChannelSyncModel (#754)
fflorent Aug 4, 2022
7e9c769
Fix audit logs (#754)
fflorent Aug 4, 2022
926379d
Ensure user can upload when importing channel + minor fixes (#754)
fflorent Aug 4, 2022
4f023e6
Mark synchronization as failed on exception + typos (#754)
fflorent Aug 5, 2022
9dc5088
Change REST API for importing videos into channel (#754)
fflorent Aug 5, 2022
c8ab2bc
Add option for fully synchronize a chnanel (#754)
fflorent Aug 5, 2022
81e8e66
Return a whole sync object on creation to avoid tricks in Front (#754)
fflorent Aug 5, 2022
f76b3b1
Various remarks (#754)
fflorent Aug 6, 2022
019fdac
Single quotes by default (#754)
fflorent Aug 6, 2022
84ec5e6
Rename synchronization to video_channel_synchronization
fflorent Aug 6, 2022
b30054a
Add check.latest_videos_count and max_per_user options (#754)
fflorent Aug 6, 2022
29254d2
Better channel rendering in list #754
fflorent Aug 6, 2022
78cef0e
Allow sorting with channel name and state (#754)
fflorent Aug 8, 2022
47f4722
Add missing tests for channel imports (#754)
fflorent Aug 8, 2022
c203b31
Merge branch 'develop' into channel-sync
Chocobozzz Aug 9, 2022
555695a
Prefer using a parent job for channel sync
Chocobozzz Aug 9, 2022
75671b5
Styling
Chocobozzz Aug 9, 2022
770cf55
Client styling
Chocobozzz Aug 10, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Allow enabling channel sync through UI (#754)
  • Loading branch information
fflorent committed Aug 8, 2022
commit e2502762c0c2ba6cb7abd547f7512c00f2a4f541
Original file line number Diff line number Diff line change
Expand Up @@ -267,10 +267,10 @@
inputName="importVideosHttpEnabled" formControlName="enabled"
i18n-labelText labelText="Allow import with HTTP URL (e.g. YouTube)"
>
<ng-container ngProjectAs="description">
<span i18n>⚠️ If enabled, we recommend to use <a class="link-orange" href="https://docs.joinpeertube.org/maintain-configuration?id=security">a HTTP proxy</a> to prevent private URL access from your PeerTube server</span>
</ng-container>
</my-peertube-checkbox>
<ng-container ngProjectAs="description">
<span i18n>⚠️ If enabled, we recommend to use <a class="link-orange" href="https://docs.joinpeertube.org/maintain-configuration?id=security">a HTTP proxy</a> to prevent private URL access from your PeerTube server</span>
</ng-container>
</my-peertube-checkbox>
</div>

<div class="form-group" formGroupName="torrent">
Expand All @@ -285,6 +285,22 @@
</div>

</ng-container>

<ng-container formGroupName="synchronization">
<div class="form-group">
<my-peertube-checkbox
inputName="importSynchronizationEnabled" formControlName="enabled"
i18n-labelText labelText="Allow channel synchronization with channel of other platforms like YouTube (requires allowing import with HTTP URL)"
>
<ng-container ngProjectAs="description">
<span i18n [hidden]="isImportVideosHttpEnabled()">
⛔ You need to allow import with HTTP URL to be able to activate this feature.
</span>
</ng-container>
</my-peertube-checkbox>
</div>
</ng-container>

</ng-container>

<ng-container formGroupName="autoBlacklist">
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ export class EditBasicConfigurationComponent implements OnInit, OnChanges {
ngOnInit () {
this.buildLandingPageOptions()
this.checkSignupField()
this.checkImportSyncField()

this.availableThemes = this.themeService.buildAvailableThemes()
}
Expand Down Expand Up @@ -67,6 +68,14 @@ export class EditBasicConfigurationComponent implements OnInit, OnChanges {
return { 'disabled-checkbox-extra': !this.isSignupEnabled() }
}

isImportVideosHttpEnabled (): boolean {
return this.form.value['import']['videos']['http']['enabled'] === true
}

importSynchronizationChecked () {
return this.isImportVideosHttpEnabled() && this.form.value['import']['synchronization']['enabled']
}

hasUnlimitedSignup () {
return this.form.value['signup']['limit'] === -1
}
Expand Down Expand Up @@ -97,6 +106,21 @@ export class EditBasicConfigurationComponent implements OnInit, OnChanges {
return this.themeService.getDefaultThemeLabel()
}

private checkImportSyncField () {
const importSyncControl = this.form.get('import.synchronization.enabled')
const importVideosHttpControl = this.form.get('import.videos.http.enabled')

importVideosHttpControl.valueChanges
.subscribe((httpImportEnabled) => {
importSyncControl.setValue(httpImportEnabled && importSyncControl.value)
if (httpImportEnabled) {
importSyncControl.enable()
} else {
importSyncControl.disable()
}
})
}

private checkSignupField () {
const signupControl = this.form.get('signup.enabled')

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,9 @@ export class EditCustomConfigComponent extends FormReactive implements OnInit {
torrent: {
enabled: null
}
},
synchronization: {
enabled: null
}
},
trending: {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,13 @@
</td>
</tr>

<tr>
<th i18n class="sub-label" scope="row">Channel synchronization with other platforms (YouTube, Vimeo, ...)</th>
<td>
<my-feature-boolean [value]="serverConfig.import.synchronization.enabled"></my-feature-boolean>
</td>
</tr>

<tr>
<th i18n class="label" colspan="2">Search</th>
</tr>
Expand Down
4 changes: 3 additions & 1 deletion config/default.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,6 @@ user:

video_channels:
max_per_user: 20 # Allows each user to create up to 20 video channels.
synchronization_interval: 15 minutes

# If enabled, the video will be transcoded to mp4 (x264) with `faststart` flag
# In addition, if some resolutions are enabled the mp4 video file will be transcoded to these new resolutions
Expand Down Expand Up @@ -540,6 +539,9 @@ import:
# We recommend to only enable magnet URI/torrent import if you trust your users
# See https://docs.joinpeertube.org/maintain-configuration?id=security for more information
enabled: false
synchronization:
fflorent marked this conversation as resolved.
Show resolved Hide resolved
enabled: false
check_interval: 1 hour

auto_blacklist:
# New videos automatically blacklisted so moderators can review before publishing
Expand Down
3 changes: 3 additions & 0 deletions server/controllers/api/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,9 @@ function customConfig (): CustomConfig {
torrent: {
enabled: CONFIG.IMPORT.VIDEOS.TORRENT.ENABLED
}
},
synchronization: {
enabled: CONFIG.IMPORT.SYNCHRONIZATION.ENABLED
}
},
trending: {
Expand Down
13 changes: 13 additions & 0 deletions server/initializers/checker-after-init.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ function checkConfig () {
checkRemoteRedundancyConfig()
checkStorageConfig()
checkTranscodingConfig()
checkImportConfig()
checkBroadcastMessageConfig()
checkSearchConfig()
checkLiveConfig()
Expand Down Expand Up @@ -198,6 +199,18 @@ function checkTranscodingConfig () {
throw new Error('Video import concurrency should be > 0')
}
}

if (CONFIG.IMPORT.VIDEOS.HTTP.ENABLED || CONFIG.IMPORT.VIDEOS.TORRENT.ENABLED) {
if (CONFIG.IMPORT.VIDEOS.CONCURRENCY <= 0) {
throw new Error('Video import concurrency should be > 0')
}
}
}

function checkImportConfig () {
if (CONFIG.IMPORT.SYNCHRONIZATION.ENABLED && !CONFIG.IMPORT.VIDEOS.HTTP) {
throw new Error("You need to enable HTTP import to allow synchronization")
fflorent marked this conversation as resolved.
Show resolved Hide resolved
}
}

function checkBroadcastMessageConfig () {
Expand Down
8 changes: 6 additions & 2 deletions server/initializers/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -300,8 +300,7 @@ const CONFIG = {
get VIDEO_QUOTA_DAILY () { return parseBytes(config.get<number>('user.video_quota_daily')) }
},
VIDEO_CHANNELS: {
get MAX_PER_USER () { return config.get<number>('video_channels.max_per_user') },
get SYNCHRONIZATION_INTERVAL () { return parseDurationToMs(config.get<string>('video_channels.synchronization_interval')) }
get MAX_PER_USER () { return config.get<number>('video_channels.max_per_user') }
},
TRANSCODING: {
get ENABLED () { return config.get<boolean>('transcoding.enabled') },
Expand Down Expand Up @@ -396,6 +395,10 @@ const CONFIG = {
TORRENT: {
get ENABLED () { return config.get<boolean>('import.videos.torrent.enabled') }
}
},
SYNCHRONIZATION: {
get ENABLED () { return config.get<boolean>('import.synchronization.enabled') },
get CHECK_INTERVAL () { return parseDurationToMs(config.get<string>('import.synchronization.check_interval')) }
}
},
AUTO_BLACKLIST: {
Expand Down Expand Up @@ -497,6 +500,7 @@ const CONFIG = {
get IS_DEFAULT_SEARCH () { return config.get<boolean>('search.search_index.is_default_search') }
}
}

}

function registerConfigChangedHandler (fun: Function) {
Expand Down
3 changes: 3 additions & 0 deletions server/lib/server-config-manager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,9 @@ class ServerConfigManager {
torrent: {
enabled: CONFIG.IMPORT.VIDEOS.TORRENT.ENABLED
}
},
synchronization: {
enabled: CONFIG.IMPORT.SYNCHRONIZATION.ENABLED
}
},
autoBlacklist: {
Expand Down
11 changes: 11 additions & 0 deletions server/middlewares/validators/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ const customConfigUpdateValidator = [
body('import.videos.http.enabled').isBoolean().withMessage('Should have a valid import video http enabled boolean'),
body('import.videos.torrent.enabled').isBoolean().withMessage('Should have a valid import video torrent enabled boolean'),

body('import.synchronization.enabled').isBoolean().withMessage('Should have a valid synchronization enabled boolean'),

body('trending.videos.algorithms.default').exists().withMessage('Should have a valid default trending algorithm'),
body('trending.videos.algorithms.enabled').exists().withMessage('Should have a valid array of enabled trending algorithms'),

Expand Down Expand Up @@ -105,6 +107,7 @@ const customConfigUpdateValidator = [
if (areValidationErrors(req, res)) return
if (!checkInvalidConfigIfEmailDisabled(req.body, res)) return
if (!checkInvalidTranscodingConfig(req.body, res)) return
if (!checkInvalidSynchronizationConfig(req.body, res)) return
if (!checkInvalidLiveConfig(req.body, res)) return
if (!checkInvalidVideoStudioConfig(req.body, res)) return

Expand Down Expand Up @@ -152,6 +155,14 @@ function checkInvalidTranscodingConfig (customConfig: CustomConfig, res: express
return true
}

function checkInvalidSynchronizationConfig (customConfig: CustomConfig, res: express.Response) {
if (customConfig.import.synchronization.enabled && !customConfig.import.videos.http.enabled) {
res.fail({ message: 'You need to enable HTTP video import in order to enable channel synchronization' })
return false
}
return true
}

function checkInvalidLiveConfig (customConfig: CustomConfig, res: express.Response) {
if (customConfig.live.enabled === false) return true

Expand Down
3 changes: 3 additions & 0 deletions server/tests/api/check-params/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,9 @@ describe('Test config API validators', function () {
torrent: {
enabled: false
}
},
synchronization: {
enabled: false
}
},
trending: {
Expand Down
3 changes: 3 additions & 0 deletions server/tests/api/server/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -362,6 +362,9 @@ const newCustomConfig: CustomConfig = {
torrent: {
enabled: false
}
},
synchronization: {
enabled: false
}
},
trending: {
Expand Down
3 changes: 3 additions & 0 deletions shared/models/server/custom-config.model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,9 @@ export interface CustomConfig {
enabled: boolean
}
}
synchronization: {
enabled: boolean
}
}

trending: {
Expand Down
3 changes: 3 additions & 0 deletions shared/models/server/server-config.model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,9 @@ export interface ServerConfig {
enabled: boolean
}
}
synchronization: {
enabled: boolean
}
}

autoBlacklist: {
Expand Down
3 changes: 3 additions & 0 deletions shared/server-commands/server/config-command.ts
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,9 @@ export class ConfigCommand extends AbstractCommand {
torrent: {
enabled: false
}
},
synchronization: {
enabled: false
}
},
trending: {
Expand Down
10 changes: 10 additions & 0 deletions support/doc/api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6582,6 +6582,11 @@ components:
properties:
enabled:
type: boolean
synchronization:
type: object
properties:
enabled:
type: boolean
autoBlacklist:
type: object
properties:
Expand Down Expand Up @@ -6946,6 +6951,11 @@ components:
properties:
enabled:
type: boolean
synchronization:
type: object
properties:
enabled:
type: boolean
autoBlacklist:
type: object
properties:
Expand Down