Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Channel sync #5135

Merged
merged 71 commits into from
Aug 10, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
dafcc07
Add external channel URL for channel update / creation (#754)
fflorent Mar 28, 2022
3d9cdf2
Disallow synchronisation if user has no video quota (#754)
fflorent Mar 28, 2022
5295073
More constraints serverside (#754)
fflorent Mar 28, 2022
2899d4a
Disable sync if server configuration does not allow HTTP import (#754)
fflorent Apr 6, 2022
d78fcc2
Working version synchronizing videos with a job (#754)
fflorent Apr 8, 2022
bd0a4b3
More logs and try/catch (#754)
fflorent Apr 11, 2022
1b7448b
Fix eslint error (#754)
fflorent Apr 11, 2022
6ac7dfd
WIP: support synchronization time change (#754)
fflorent Apr 15, 2022
1cd8538
New frontend #754
fflorent May 15, 2022
60dd68a
WIP: Create sync front (#754)
fflorent Jul 5, 2022
38b568f
Enhance UI, sync creation form (#754)
fflorent Jul 6, 2022
18c5316
Warning message when HTTP upload is disallowed
fflorent Jul 6, 2022
9c5d8a3
More consistent names (#754)
fflorent Jul 6, 2022
7d1f63c
Binding Front with API (#754)
fflorent Jul 13, 2022
beb0bb1
Add a /me API (#754)
fflorent Jul 13, 2022
1a25f15
Improve list UI (#754)
fflorent Jul 13, 2022
7c07eea
Implement creation and deletion routes (#754)
fflorent Jul 14, 2022
f470a38
Lint (#754)
fflorent Jul 14, 2022
9fa21bf
Lint again (#754)
fflorent Jul 14, 2022
0b6d378
WIP: UI for triggering import existing videos (#754)
fflorent Jul 15, 2022
a65364f
Implement jobs for syncing and importing channels
fflorent Jul 17, 2022
54a8995
Don't sync videos before sync creation + avoid concurrency issue (#754)
fflorent Jul 17, 2022
22864f2
Cleanup (#754)
fflorent Jul 19, 2022
9d53de6
Cleanup: OpenAPI + API rework (#754)
fflorent Jul 20, 2022
1e29d34
Remove dead code (#754)
fflorent Jul 20, 2022
4c9ebbc
Eslint (#754)
fflorent Jul 20, 2022
df220b5
Revert the mess with whitespaces in constants.ts (#754)
fflorent Jul 25, 2022
80d7a88
Some fixes after rebase (#754)
fflorent Jul 25, 2022
3e11cf5
Several fixes after PR remarks (#754)
fflorent Jul 26, 2022
11e7b7b
Front + API: Rename video-channels-sync to video-channel-syncs (#754)
fflorent Jul 26, 2022
e250276
Allow enabling channel sync through UI (#754)
fflorent Jul 27, 2022
518f4e0
getChannelInfo (#754)
fflorent Jul 28, 2022
1d148d5
Minor fixes: openapi + model + sql (#754)
fflorent Jul 28, 2022
6c7a269
Simplified API validators (#754)
fflorent Jul 28, 2022
242f827
Rename MChannelSync to MChannelSyncChannel (#754)
fflorent Jul 29, 2022
1970660
Add command for VideoChannelSync (#754)
fflorent Jul 29, 2022
c89c40c
Use synchronization.enabled config (#754)
fflorent Aug 1, 2022
ec21cc3
Check parameters test + some fixes (#754)
fflorent Aug 1, 2022
a1cc50c
Fix conflict mistake (#754)
fflorent Aug 1, 2022
63664bf
Restrict access to video channel sync list API (#754)
fflorent Aug 1, 2022
f1e7276
Start adding unit test for synchronization (#754)
fflorent Aug 1, 2022
a14e312
Continue testing (#754)
fflorent Aug 2, 2022
720f865
Tests finished + convertion of job to scheduler (#754)
fflorent Aug 2, 2022
d5806ae
Add lastSyncAt field (#754)
fflorent Aug 2, 2022
c79ceb9
Fix externalRemoteUrl sort + creation date not well formatted (#754)
fflorent Aug 3, 2022
9e44753
Small fix (#754)
fflorent Aug 3, 2022
203c729
Factorize addYoutubeDLImport and buildVideo (#754)
fflorent Aug 3, 2022
1b5cb11
Check duplicates on channel not on users (#754)
fflorent Aug 3, 2022
b2f35a6
factorize thumbnail generation (#754)
fflorent Aug 4, 2022
0728c5d
Fetch error should return status 400 (#754)
fflorent Aug 4, 2022
4e1bff0
Separate video-channel-import and video-channel-sync-latest (#754)
fflorent Aug 4, 2022
bc405b0
Bump DB migration version after rebase (#754)
fflorent Aug 4, 2022
a9a3848
Prettier states in UI table (#754)
fflorent Aug 4, 2022
9bd1226
Add DefaultScope in VideoChannelSyncModel (#754)
fflorent Aug 4, 2022
7e9c769
Fix audit logs (#754)
fflorent Aug 4, 2022
926379d
Ensure user can upload when importing channel + minor fixes (#754)
fflorent Aug 4, 2022
4f023e6
Mark synchronization as failed on exception + typos (#754)
fflorent Aug 5, 2022
9dc5088
Change REST API for importing videos into channel (#754)
fflorent Aug 5, 2022
c8ab2bc
Add option for fully synchronize a chnanel (#754)
fflorent Aug 5, 2022
81e8e66
Return a whole sync object on creation to avoid tricks in Front (#754)
fflorent Aug 5, 2022
f76b3b1
Various remarks (#754)
fflorent Aug 6, 2022
019fdac
Single quotes by default (#754)
fflorent Aug 6, 2022
84ec5e6
Rename synchronization to video_channel_synchronization
fflorent Aug 6, 2022
b30054a
Add check.latest_videos_count and max_per_user options (#754)
fflorent Aug 6, 2022
29254d2
Better channel rendering in list #754
fflorent Aug 6, 2022
78cef0e
Allow sorting with channel name and state (#754)
fflorent Aug 8, 2022
47f4722
Add missing tests for channel imports (#754)
fflorent Aug 8, 2022
c203b31
Merge branch 'develop' into channel-sync
Chocobozzz Aug 9, 2022
555695a
Prefer using a parent job for channel sync
Chocobozzz Aug 9, 2022
75671b5
Styling
Chocobozzz Aug 9, 2022
770cf55
Client styling
Chocobozzz Aug 10, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Front + API: Rename video-channels-sync to video-channel-syncs (#754)
  • Loading branch information
fflorent committed Aug 8, 2022
commit 11e7b7b0be4880b5c6cfce6c7d75f30ed7334462
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ <h1>
</span>

<div>
<a routerLink="/my-library/video-channels-sync" class="button-link">
<a routerLink="/my-library/video-channel-syncs" class="button-link">
<my-global-icon iconName="repeat" aria-hidden="true"></my-global-icon>
<ng-container i18n>My synchronizations</ng-container>
</a>
Expand Down
10 changes: 5 additions & 5 deletions client/src/app/+my-library/my-library-routing.module.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import { MySubscriptionsComponent } from './my-follows/my-subscriptions.componen
import { MyHistoryComponent } from './my-history/my-history.component'
import { MyLibraryComponent } from './my-library.component'
import { MyOwnershipComponent } from './my-ownership/my-ownership.component'
import { MyVideoChannelsSyncComponent } from './my-video-channels-sync/my-video-channels-sync.component'
import { VideoChannelSyncEditComponent } from './my-video-channels-sync/video-channel-sync-edit/video-channel-sync-edit.component'
import { MyVideoChannelSyncsComponent } from './my-video-channel-syncs/my-video-channel-syncs.component'
import { VideoChannelSyncEditComponent } from './my-video-channel-syncs/video-channel-sync-edit/video-channel-sync-edit.component'
import { MyVideoImportsComponent } from './my-video-imports/my-video-imports.component'
import { MyVideoPlaylistCreateComponent } from './my-video-playlists/my-video-playlist-create.component'
import { MyVideoPlaylistElementsComponent } from './my-video-playlists/my-video-playlist-elements.component'
Expand Down Expand Up @@ -136,8 +136,8 @@ const myLibraryRoutes: Routes = [
},

{
path: 'video-channels-sync',
component: MyVideoChannelsSyncComponent,
path: 'video-channel-syncs',
component: MyVideoChannelSyncsComponent,
data: {
meta: {
title: $localize`My synchronizations`
Expand All @@ -146,7 +146,7 @@ const myLibraryRoutes: Routes = [
},

{
path: 'video-channels-sync/create',
path: 'video-channel-syncs/create',
component: VideoChannelSyncEditComponent,
data: {
meta: {
Expand Down
6 changes: 3 additions & 3 deletions client/src/app/+my-library/my-library.module.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ import { MyVideoPlaylistUpdateComponent } from './my-video-playlists/my-video-pl
import { MyVideoPlaylistsComponent } from './my-video-playlists/my-video-playlists.component'
import { VideoChangeOwnershipComponent } from './my-videos/modals/video-change-ownership.component'
import { MyVideosComponent } from './my-videos/my-videos.component'
import { MyVideoChannelsSyncComponent } from './my-video-channels-sync/my-video-channels-sync.component'
import { VideoChannelSyncEditComponent } from './my-video-channels-sync/video-channel-sync-edit/video-channel-sync-edit.component'
import { MyVideoChannelSyncsComponent } from './my-video-channel-syncs/my-video-channel-syncs.component'
import { VideoChannelSyncEditComponent } from './my-video-channel-syncs/video-channel-sync-edit/video-channel-sync-edit.component'

@NgModule({
imports: [
Expand Down Expand Up @@ -65,7 +65,7 @@ import { VideoChannelSyncEditComponent } from './my-video-channels-sync/video-ch
MyOwnershipComponent,
MyAcceptOwnershipComponent,
MyVideoImportsComponent,
MyVideoChannelsSyncComponent,
MyVideoChannelSyncsComponent,
VideoChannelSyncEditComponent,
MySubscriptionsComponent,
MyFollowersComponent,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ import { SortMeta } from 'primeng/api'
import { mergeMap } from 'rxjs'

@Component({
templateUrl: './my-video-channels-sync.component.html',
styleUrls: [ './my-video-channels-sync.component.scss' ]
templateUrl: './my-video-channel-syncs.component.html',
styleUrls: [ './my-video-channel-syncs.component.scss' ]
})
export class MyVideoChannelsSyncComponent extends RestTable implements OnInit {
export class MyVideoChannelSyncsComponent extends RestTable implements OnInit {

error: string = undefined
channelsSync: VideoChannelSync[]
fflorent marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -83,7 +83,7 @@ export class MyVideoChannelsSyncComponent extends RestTable implements OnInit {
}

getVideoChannelCreateLink () {
fflorent marked this conversation as resolved.
Show resolved Hide resolved
return '/my-library/video-channels-sync/create'
return '/my-library/video-channel-syncs/create'
}

getVideoChannelSyncStateClass (syncId: number) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ export class VideoChannelSyncEditComponent extends FormReactive implements OnIni
.subscribe({
next: () => {
this.notifier.success($localize`Synchronization created successfully.`)
this.router.navigate([ '/my-library', 'video-channels-sync' ])
this.router.navigate([ '/my-library', 'video-channel-syncs' ])
},

error: err => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import { Account, AccountService } from '../account'
providedIn: 'root'
})
export class VideoChannelSyncService {
static BASE_VIDEO_CHANNEL_URL = environment.apiUrl + '/api/v1/video-channels-syncs'
static BASE_VIDEO_CHANNEL_URL = environment.apiUrl + '/api/v1/video-channel-syncs'

constructor (
Chocobozzz marked this conversation as resolved.
Show resolved Hide resolved
private authHttp: HttpClient,
Expand All @@ -28,7 +28,7 @@ export class VideoChannelSyncService {
const { pagination, sort, account } = parameters
let params = new HttpParams()
params = this.restService.addRestGetParams(params, pagination, sort)
const url = AccountService.BASE_ACCOUNT_URL + account.nameWithHost + '/video-channels-syncs'
const url = AccountService.BASE_ACCOUNT_URL + account.nameWithHost + '/video-channel-syncs'
return this.authHttp.get<ResultList<VideoChannelSync>>(url, { params })
.pipe(catchError(err => this.restExtractor.handleError(err)))
}
Expand Down
2 changes: 1 addition & 1 deletion server/controllers/api/accounts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ accountsRouter.get('/:accountName/video-channels',
asyncMiddleware(listAccountChannels)
)

accountsRouter.get('/:accountName/video-channels-syncs',
accountsRouter.get('/:accountName/video-channel-syncs',
asyncMiddleware(accountNameWithHostGetValidator),
paginationValidator,
videoChannelSyncsSortValidator,
Expand Down
2 changes: 1 addition & 1 deletion server/controllers/api/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ apiRouter.use('/config', configRouter)
apiRouter.use('/users', usersRouter)
apiRouter.use('/accounts', accountsRouter)
apiRouter.use('/video-channels', videoChannelRouter)
apiRouter.use('/video-channels-syncs', videoChannelSyncRouter)
apiRouter.use('/video-channel-syncs', videoChannelSyncRouter)
apiRouter.use('/video-playlists', videoPlaylistRouter)
apiRouter.use('/videos', videosRouter)
apiRouter.use('/jobs', jobsRouter)
Expand Down
10 changes: 1 addition & 9 deletions server/helpers/custom-validators/video-channels.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,11 @@ function isVideoChannelSupportValid (value: string) {
return value === null || (exists(value) && validator.isLength(value, VIDEO_CHANNELS_CONSTRAINTS_FIELDS.SUPPORT))
}

function isVideoChannelExternalChannelUrlValid (value: string) {
return value === null || (exists(value) &&
validator.matches(value, /^https?:\/\//) &&
validator.isLength(value, VIDEO_CHANNELS_CONSTRAINTS_FIELDS.EXTERNAL_CHANNEL_URL
))
}

// ---------------------------------------------------------------------------

export {
isVideoChannelUsernameValid,
isVideoChannelDescriptionValid,
isVideoChannelDisplayNameValid,
isVideoChannelSupportValid,
isVideoChannelExternalChannelUrlValid
isVideoChannelSupportValid
}
6 changes: 3 additions & 3 deletions server/models/video/video-channel-sync.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import { isVideoChannelExternalChannelUrlValid } from "@server/helpers/custom-validators/video-channels"
import { isVideoChannelSyncStateValid } from "@server/helpers/custom-validators/video-channels-sync"
import { isVideoChannelSyncStateValid } from "@server/helpers/custom-validators/video-channel-syncs"
import { CONSTRAINTS_FIELDS, VIDEO_CHANNEL_SYNC_STATE } from "@server/initializers/constants"
import { VideoChannelSync, VideoChannelSyncState } from "@shared/models"
import { MChannelSyncFormattable } from "@server/types/models/video/video-channel-sync"
Expand All @@ -23,6 +22,7 @@ import { VideoChannelModel } from "./video-channel"
import { ActorModel } from "../actor/actor"
import { Op } from "sequelize"
import { UserModel } from "../user/user"
import { isVideoImportTargetUrlValid } from "@server/helpers/custom-validators/video-imports"

type AvailableForListOptions = {
fflorent marked this conversation as resolved.
Show resolved Hide resolved
accountId: number
Expand All @@ -39,7 +39,7 @@ export class VideoChannelSyncModel extends Model<Partial<AttributesOnly<VideoCha

@AllowNull(false)
@Default(null)
@Is('VideoChannelExternalChannelUrl', value => throwIfNotValid(value, isVideoChannelExternalChannelUrlValid, 'externalChannelUrl', true))
@Is('VideoChannelExternalChannelUrl', value => throwIfNotValid(value, isVideoImportTargetUrlValid, 'externalChannelUrl', true))
fflorent marked this conversation as resolved.
Show resolved Hide resolved
@Column(DataType.STRING(CONSTRAINTS_FIELDS.VIDEO_CHANNEL_SYNCS.EXTERNAL_CHANNEL_URL.max))
externalChannelUrl: string

Expand Down
8 changes: 4 additions & 4 deletions support/doc/api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3288,7 +3288,7 @@ paths:
'204':
description: successful operation

'/video-channels-syncs/':
'/video-channel-syncs/':
fflorent marked this conversation as resolved.
Show resolved Hide resolved
post:
summary: Create a synchronization for a video channels
fflorent marked this conversation as resolved.
Show resolved Hide resolved
operationId: addVideoChannelSync
Expand All @@ -3315,7 +3315,7 @@ paths:
id:
$ref: '#/components/schemas/id'

'/video-channels-syncs/{channelSyncId}':
'/video-channel-syncs/{channelSyncId}':
delete:
summary: Delete a video channel synchronization
operationId: delVideoChannelSync
Expand All @@ -3329,7 +3329,7 @@ paths:
'204':
description: successful operation

'/video-channels-syncs/syncChannel/{channelSyncId}':
'/video-channel-syncs/syncChannel/{channelSyncId}':
fflorent marked this conversation as resolved.
Show resolved Hide resolved
post:
summary: Triggers the channel synchronization job, fetching all the videos from the remote channel
operationId: triggerVideoChannelSync
Expand Down Expand Up @@ -3715,7 +3715,7 @@ paths:
schema:
$ref: '#/components/schemas/VideoChannelList'

'/accounts/{name}/video-channels-syncs':
'/accounts/{name}/video-channel-syncs':
get:
summary: List the synchronizations of video channels of an account
tags:
Expand Down