Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Channel sync #5135

Merged
merged 71 commits into from
Aug 10, 2022
Merged
Changes from 1 commit
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
dafcc07
Add external channel URL for channel update / creation (#754)
fflorent Mar 28, 2022
3d9cdf2
Disallow synchronisation if user has no video quota (#754)
fflorent Mar 28, 2022
5295073
More constraints serverside (#754)
fflorent Mar 28, 2022
2899d4a
Disable sync if server configuration does not allow HTTP import (#754)
fflorent Apr 6, 2022
d78fcc2
Working version synchronizing videos with a job (#754)
fflorent Apr 8, 2022
bd0a4b3
More logs and try/catch (#754)
fflorent Apr 11, 2022
1b7448b
Fix eslint error (#754)
fflorent Apr 11, 2022
6ac7dfd
WIP: support synchronization time change (#754)
fflorent Apr 15, 2022
1cd8538
New frontend #754
fflorent May 15, 2022
60dd68a
WIP: Create sync front (#754)
fflorent Jul 5, 2022
38b568f
Enhance UI, sync creation form (#754)
fflorent Jul 6, 2022
18c5316
Warning message when HTTP upload is disallowed
fflorent Jul 6, 2022
9c5d8a3
More consistent names (#754)
fflorent Jul 6, 2022
7d1f63c
Binding Front with API (#754)
fflorent Jul 13, 2022
beb0bb1
Add a /me API (#754)
fflorent Jul 13, 2022
1a25f15
Improve list UI (#754)
fflorent Jul 13, 2022
7c07eea
Implement creation and deletion routes (#754)
fflorent Jul 14, 2022
f470a38
Lint (#754)
fflorent Jul 14, 2022
9fa21bf
Lint again (#754)
fflorent Jul 14, 2022
0b6d378
WIP: UI for triggering import existing videos (#754)
fflorent Jul 15, 2022
a65364f
Implement jobs for syncing and importing channels
fflorent Jul 17, 2022
54a8995
Don't sync videos before sync creation + avoid concurrency issue (#754)
fflorent Jul 17, 2022
22864f2
Cleanup (#754)
fflorent Jul 19, 2022
9d53de6
Cleanup: OpenAPI + API rework (#754)
fflorent Jul 20, 2022
1e29d34
Remove dead code (#754)
fflorent Jul 20, 2022
4c9ebbc
Eslint (#754)
fflorent Jul 20, 2022
df220b5
Revert the mess with whitespaces in constants.ts (#754)
fflorent Jul 25, 2022
80d7a88
Some fixes after rebase (#754)
fflorent Jul 25, 2022
3e11cf5
Several fixes after PR remarks (#754)
fflorent Jul 26, 2022
11e7b7b
Front + API: Rename video-channels-sync to video-channel-syncs (#754)
fflorent Jul 26, 2022
e250276
Allow enabling channel sync through UI (#754)
fflorent Jul 27, 2022
518f4e0
getChannelInfo (#754)
fflorent Jul 28, 2022
1d148d5
Minor fixes: openapi + model + sql (#754)
fflorent Jul 28, 2022
6c7a269
Simplified API validators (#754)
fflorent Jul 28, 2022
242f827
Rename MChannelSync to MChannelSyncChannel (#754)
fflorent Jul 29, 2022
1970660
Add command for VideoChannelSync (#754)
fflorent Jul 29, 2022
c89c40c
Use synchronization.enabled config (#754)
fflorent Aug 1, 2022
ec21cc3
Check parameters test + some fixes (#754)
fflorent Aug 1, 2022
a1cc50c
Fix conflict mistake (#754)
fflorent Aug 1, 2022
63664bf
Restrict access to video channel sync list API (#754)
fflorent Aug 1, 2022
f1e7276
Start adding unit test for synchronization (#754)
fflorent Aug 1, 2022
a14e312
Continue testing (#754)
fflorent Aug 2, 2022
720f865
Tests finished + convertion of job to scheduler (#754)
fflorent Aug 2, 2022
d5806ae
Add lastSyncAt field (#754)
fflorent Aug 2, 2022
c79ceb9
Fix externalRemoteUrl sort + creation date not well formatted (#754)
fflorent Aug 3, 2022
9e44753
Small fix (#754)
fflorent Aug 3, 2022
203c729
Factorize addYoutubeDLImport and buildVideo (#754)
fflorent Aug 3, 2022
1b5cb11
Check duplicates on channel not on users (#754)
fflorent Aug 3, 2022
b2f35a6
factorize thumbnail generation (#754)
fflorent Aug 4, 2022
0728c5d
Fetch error should return status 400 (#754)
fflorent Aug 4, 2022
4e1bff0
Separate video-channel-import and video-channel-sync-latest (#754)
fflorent Aug 4, 2022
bc405b0
Bump DB migration version after rebase (#754)
fflorent Aug 4, 2022
a9a3848
Prettier states in UI table (#754)
fflorent Aug 4, 2022
9bd1226
Add DefaultScope in VideoChannelSyncModel (#754)
fflorent Aug 4, 2022
7e9c769
Fix audit logs (#754)
fflorent Aug 4, 2022
926379d
Ensure user can upload when importing channel + minor fixes (#754)
fflorent Aug 4, 2022
4f023e6
Mark synchronization as failed on exception + typos (#754)
fflorent Aug 5, 2022
9dc5088
Change REST API for importing videos into channel (#754)
fflorent Aug 5, 2022
c8ab2bc
Add option for fully synchronize a chnanel (#754)
fflorent Aug 5, 2022
81e8e66
Return a whole sync object on creation to avoid tricks in Front (#754)
fflorent Aug 5, 2022
f76b3b1
Various remarks (#754)
fflorent Aug 6, 2022
019fdac
Single quotes by default (#754)
fflorent Aug 6, 2022
84ec5e6
Rename synchronization to video_channel_synchronization
fflorent Aug 6, 2022
b30054a
Add check.latest_videos_count and max_per_user options (#754)
fflorent Aug 6, 2022
29254d2
Better channel rendering in list #754
fflorent Aug 6, 2022
78cef0e
Allow sorting with channel name and state (#754)
fflorent Aug 8, 2022
47f4722
Add missing tests for channel imports (#754)
fflorent Aug 8, 2022
c203b31
Merge branch 'develop' into channel-sync
Chocobozzz Aug 9, 2022
555695a
Prefer using a parent job for channel sync
Chocobozzz Aug 9, 2022
75671b5
Styling
Chocobozzz Aug 9, 2022
770cf55
Client styling
Chocobozzz Aug 10, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert the mess with whitespaces in constants.ts (#754)
  • Loading branch information
fflorent committed Aug 8, 2022
commit df220b59bcc6f03d5ec867a5dab7f3def8aadf9e
61 changes: 26 additions & 35 deletions server/initializers/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ const ACTOR_FOLLOW_SCORE = {
MAX: 10000
}

const FOLLOW_STATES: {[id: string]: FollowState} = {
const FOLLOW_STATES: { [ id: string ]: FollowState } = {
PENDING: 'pending',
ACCEPTED: 'accepted',
REJECTED: 'rejected'
Expand All @@ -140,7 +140,7 @@ const REMOTE_SCHEME = {
WS: 'wss'
}

const JOB_ATTEMPTS: {[id in JobType]: number} = {
const JOB_ATTEMPTS: { [id in JobType]: number } = {
'activitypub-http-broadcast': 1,
'activitypub-http-broadcast-parallel': 1,
'activitypub-http-unicast': 1,
Expand All @@ -158,12 +158,10 @@ const JOB_ATTEMPTS: {[id in JobType]: number} = {
'video-live-ending': 1,
'video-studio-edition': 1,
'manage-video-torrent': 1,
'move-to-object-storage': 3,
'video-channels-sync': 1,
'video-channel-import': 3
'move-to-object-storage': 3
}
// Excluded keys are jobs that can be configured by admins
const JOB_CONCURRENCY: {[id in Exclude<JobType, 'video-transcoding' | 'video-import'>]: number} = {
const JOB_CONCURRENCY: { [id in Exclude<JobType, 'video-transcoding' | 'video-import'>]: number } = {
'activitypub-http-broadcast': 1,
'activitypub-http-broadcast-parallel': 30,
'activitypub-http-unicast': 10,
Expand All @@ -183,7 +181,7 @@ const JOB_CONCURRENCY: {[id in Exclude<JobType, 'video-transcoding' | 'video-imp
'video-channels-sync': 1,
'video-channel-import': 1
}
const JOB_TTL: {[id in JobType]: number} = {
const JOB_TTL: { [id in JobType]: number } = {
'activitypub-http-broadcast': 60000 * 10, // 10 minutes
'activitypub-http-broadcast-parallel': 60000 * 10, // 10 minutes
'activitypub-http-unicast': 60000 * 10, // 10 minutes
Expand All @@ -205,7 +203,7 @@ const JOB_TTL: {[id in JobType]: number} = {
'video-channels-sync': 1000 * 60 * 60 * 4, // 4 hours
'video-channel-import': 1000 * 60 * 60 * 4 // 4 hours
}
const REPEAT_JOBS: {[id in JobType]?: EveryRepeatOptions | CronRepeatOptions} = {
const REPEAT_JOBS: { [ id in JobType ]?: EveryRepeatOptions | CronRepeatOptions } = {
'videos-views-stats': {
cron: randomInt(1, 20) + ' * * * *' // Between 1-20 minutes past the hour
},
Expand Down Expand Up @@ -413,12 +411,12 @@ const VIDEO_TRANSCODING_FPS: VideoTranscodingFPS = {

const DEFAULT_AUDIO_RESOLUTION = VideoResolution.H_480P

const VIDEO_RATE_TYPES: {[id: string]: VideoRateType} = {
const VIDEO_RATE_TYPES: { [ id: string ]: VideoRateType } = {
LIKE: 'like',
DISLIKE: 'dislike'
}

const FFMPEG_NICE: {[id: string]: number} = {
const FFMPEG_NICE: { [ id: string ]: number } = {
// parent process defaults to niceness = 0
// reminder: lower = higher priority, max value is 19, lowest is -20
LIVE: 5, // prioritize over VOD and THUMBNAIL
Expand Down Expand Up @@ -458,9 +456,9 @@ const VIDEO_LICENCES = {
7: 'Public Domain Dedication'
}

const VIDEO_LANGUAGES: {[id: string]: string} = {}
const VIDEO_LANGUAGES: { [id: string]: string } = {}

const VIDEO_PRIVACIES: {[id in VideoPrivacy]: string} = {
const VIDEO_PRIVACIES: { [ id in VideoPrivacy ]: string } = {
[VideoPrivacy.PUBLIC]: 'Public',
[VideoPrivacy.UNLISTED]: 'Unlisted',
[VideoPrivacy.PRIVATE]: 'Private',
Expand All @@ -469,7 +467,7 @@ const VIDEO_PRIVACIES: {[id in VideoPrivacy]: string} = {

const VIDEO_CHANNEL_MAX_SYNC = 3
fflorent marked this conversation as resolved.
Show resolved Hide resolved

const VIDEO_STATES: {[id in VideoState]: string} = {
const VIDEO_STATES: { [ id in VideoState ]: string } = {
[VideoState.PUBLISHED]: 'Published',
[VideoState.TO_TRANSCODE]: 'To transcode',
[VideoState.TO_IMPORT]: 'To import',
Expand All @@ -481,7 +479,7 @@ const VIDEO_STATES: {[id in VideoState]: string} = {
[VideoState.TO_EDIT]: 'To edit*'
}

const VIDEO_IMPORT_STATES: {[id in VideoImportState]: string} = {
const VIDEO_IMPORT_STATES: { [ id in VideoImportState ]: string } = {
[VideoImportState.FAILED]: 'Failed',
[VideoImportState.PENDING]: 'Pending',
[VideoImportState.SUCCESS]: 'Success',
Expand All @@ -490,26 +488,19 @@ const VIDEO_IMPORT_STATES: {[id in VideoImportState]: string} = {
[VideoImportState.PROCESSING]: 'Processing'
}

const VIDEO_CHANNEL_SYNC_STATE: {[id in VideoChannelSyncState]: string} = {
[VideoChannelSyncState.WAITING_FIRST_RUN]: 'Waiting for the first run',
[VideoChannelSyncState.PROCESSING]: 'Processing',
[VideoChannelSyncState.SYNCED]: 'Synchronized',
[VideoChannelSyncState.FAILED]: 'Failed'
}

const ABUSE_STATES: {[id in AbuseState]: string} = {
const ABUSE_STATES: { [ id in AbuseState ]: string } = {
[AbuseState.PENDING]: 'Pending',
[AbuseState.REJECTED]: 'Rejected',
[AbuseState.ACCEPTED]: 'Accepted'
}

const VIDEO_PLAYLIST_PRIVACIES: {[id in VideoPlaylistPrivacy]: string} = {
const VIDEO_PLAYLIST_PRIVACIES: { [ id in VideoPlaylistPrivacy ]: string } = {
[VideoPlaylistPrivacy.PUBLIC]: 'Public',
[VideoPlaylistPrivacy.UNLISTED]: 'Unlisted',
[VideoPlaylistPrivacy.PRIVATE]: 'Private'
}

const VIDEO_PLAYLIST_TYPES: {[id in VideoPlaylistType]: string} = {
const VIDEO_PLAYLIST_TYPES: { [ id in VideoPlaylistType ]: string } = {
[VideoPlaylistType.REGULAR]: 'Regular',
[VideoPlaylistType.WATCH_LATER]: 'Watch later'
}
Expand Down Expand Up @@ -540,12 +531,12 @@ const MIMETYPES = {
'audio/vnd.dolby.dd-raw': '.ac3',
'audio/ac3': '.ac3'
},
EXT_MIMETYPE: null as {[id: string]: string}
EXT_MIMETYPE: null as { [ id: string ]: string }
},
VIDEO: {
MIMETYPE_EXT: null as {[id: string]: string | string[]},
MIMETYPE_EXT: null as { [ id: string ]: string | string[] },
MIMETYPES_REGEX: null as string,
EXT_MIMETYPE: null as {[id: string]: string}
EXT_MIMETYPE: null as { [ id: string ]: string }
},
IMAGE: {
MIMETYPE_EXT: {
Expand All @@ -555,7 +546,7 @@ const MIMETYPES = {
'image/jpg': '.jpg',
'image/jpeg': '.jpg'
},
EXT_MIMETYPE: null as {[id: string]: string}
EXT_MIMETYPE: null as { [ id: string ]: string }
},
VIDEO_CAPTIONS: {
MIMETYPE_EXT: {
Expand Down Expand Up @@ -613,7 +604,7 @@ const ACTIVITY_PUB = {
VIDEO_PLAYLIST_REFRESH_INTERVAL: 3600 * 24 * 1000 * 2 // 2 days
}

const ACTIVITY_PUB_ACTOR_TYPES: {[id: string]: ActivityPubActorType} = {
const ACTIVITY_PUB_ACTOR_TYPES: { [ id: string ]: ActivityPubActorType } = {
GROUP: 'Group',
PERSON: 'Person',
APPLICATION: 'Application',
Expand All @@ -640,7 +631,7 @@ const USER_PASSWORD_CREATE_LIFETIME = 60000 * 60 * 24 * 7 // 7 days

const USER_EMAIL_VERIFY_LIFETIME = 60000 * 60 // 60 minutes

const NSFW_POLICY_TYPES: {[id: string]: NSFWPolicyType} = {
const NSFW_POLICY_TYPES: { [ id: string ]: NSFWPolicyType } = {
DO_NOT_LIST: 'do_not_list',
BLUR: 'blur',
DISPLAY: 'display'
Expand Down Expand Up @@ -688,7 +679,7 @@ const PREVIEWS_SIZE = {
height: 480,
minWidth: 400
}
const ACTOR_IMAGES_SIZE: {[key in ActorImageType]: {width: number, height: number}[]} = {
const ACTOR_IMAGES_SIZE: { [key in ActorImageType]: { width: number, height: number }[]} = {
[ActorImageType.AVATAR]: [
{
width: 120,
Expand Down Expand Up @@ -1134,8 +1125,8 @@ function updateWebserverConfig () {
CONSTRAINTS_FIELDS.VIDEOS.EXTNAME = Object.keys(MIMETYPES.VIDEO.EXT_MIMETYPE)
}

function buildVideoExtMimetype (obj: {[id: string]: string | string[]}) {
const result: {[id: string]: string} = {}
function buildVideoExtMimetype (obj: { [ id: string ]: string | string[] }) {
const result: { [id: string]: string } = {}

for (const mimetype of Object.keys(obj)) {
const value = obj[mimetype]
Expand All @@ -1151,7 +1142,7 @@ function buildVideoExtMimetype (obj: {[id: string]: string | string[]}) {
return result
}

function buildMimetypesRegex (obj: {[id: string]: string | string[]}) {
function buildMimetypesRegex (obj: { [id: string]: string | string[] }) {
return Object.keys(obj)
.map(m => `(${m})`)
.join('|')
Expand All @@ -1164,7 +1155,7 @@ function loadLanguages () {
function buildLanguages () {
const iso639 = require('iso-639-3')

const languages: {[id: string]: string} = {}
const languages: { [id: string]: string } = {}

const additionalLanguages = {
sgn: true, // Sign languages (macro language)
Expand Down