Commit Graph

1300 Commits

Author SHA1 Message Date
Florent 2a491182e4
Channel sync (#5135)
* Add external channel URL for channel update / creation (#754)

* Disallow synchronisation if user has no video quota (#754)

* More constraints serverside (#754)

* Disable sync if server configuration does not allow HTTP import (#754)

* Working version synchronizing videos with a job (#754)

TODO: refactoring, too much code duplication

* More logs and try/catch (#754)

* Fix eslint error (#754)

* WIP: support synchronization time change (#754)

* New frontend #754

* WIP: Create sync front (#754)

* Enhance UI, sync creation form (#754)

* Warning message when HTTP upload is disallowed

* More consistent names (#754)

* Binding Front with API (#754)

* Add a /me API (#754)

* Improve list UI (#754)

* Implement creation and deletion routes (#754)

* Lint (#754)

* Lint again (#754)

* WIP: UI for triggering import existing videos (#754)

* Implement jobs for syncing and importing channels

* Don't sync videos before sync creation + avoid concurrency issue (#754)

* Cleanup (#754)

* Cleanup: OpenAPI + API rework (#754)

* Remove dead code (#754)

* Eslint (#754)

* Revert the mess with whitespaces in constants.ts (#754)

* Some fixes after rebase (#754)

* Several fixes after PR remarks (#754)

* Front + API: Rename video-channels-sync to video-channel-syncs (#754)

* Allow enabling channel sync through UI (#754)

* getChannelInfo (#754)

* Minor fixes: openapi + model + sql (#754)

* Simplified API validators (#754)

* Rename MChannelSync to MChannelSyncChannel (#754)

* Add command for VideoChannelSync (#754)

* Use synchronization.enabled config (#754)

* Check parameters test + some fixes (#754)

* Fix conflict mistake (#754)

* Restrict access to video channel sync list API (#754)

* Start adding unit test for synchronization (#754)

* Continue testing (#754)

* Tests finished + convertion of job to scheduler (#754)

* Add lastSyncAt field (#754)

* Fix externalRemoteUrl sort + creation date not well formatted (#754)

* Small fix (#754)

* Factorize addYoutubeDLImport and buildVideo (#754)

* Check duplicates on channel not on users (#754)

* factorize thumbnail generation (#754)

* Fetch error should return status 400 (#754)

* Separate video-channel-import and video-channel-sync-latest (#754)

* Bump DB migration version after rebase (#754)

* Prettier states in UI table (#754)

* Add DefaultScope in VideoChannelSyncModel (#754)

* Fix audit logs (#754)

* Ensure user can upload when importing channel + minor fixes (#754)

* Mark synchronization as failed on exception + typos (#754)

* Change REST API for importing videos into channel (#754)

* Add option for fully synchronize a chnanel (#754)

* Return a whole sync object on creation to avoid tricks in Front (#754)

* Various remarks (#754)

* Single quotes by default (#754)

* Rename synchronization to video_channel_synchronization

* Add check.latest_videos_count and max_per_user options (#754)

* Better channel rendering in list #754

* Allow sorting with channel name and state (#754)

* Add missing tests for channel imports (#754)

* Prefer using a parent job for channel sync

* Styling

* Client styling

Co-authored-by: Chocobozzz <me@florianbigard.com>
2022-08-10 09:53:39 +02:00
Chocobozzz 7f529402d1
Fix live max rate 2022-08-09 13:59:51 +02:00
Chocobozzz 51335c72cf
Prevent job failure on concurrent HLS transcoding 2022-08-09 13:21:18 +02:00
Chocobozzz e2b2c726b1
Fix job queue tests 2022-08-09 11:39:31 +02:00
Chocobozzz b42c2c7e89
Avoid concurrency issue on transcoding 2022-08-09 09:32:17 +02:00
Chocobozzz bd911b54b5
Use bullmq job dependency 2022-08-09 09:18:07 +02:00
Chocobozzz 5a921e7b74
Move to bullmq 2022-08-09 09:18:07 +02:00
Chocobozzz 5e2afe4290
Limit import depending on transcoding resolutions 2022-08-09 09:18:07 +02:00
Chocobozzz 64fd6158fd
Fix resolution to transcode hook name 2022-08-09 09:18:07 +02:00
Chocobozzz 84cae54e7a
Add option to not transcode original resolution 2022-08-09 09:18:07 +02:00
Florent 7e0f50d6e0 Use single quotes in server/ and update eslint rules 2022-08-08 08:25:36 +02:00
Florent b37d80e3ce Fix lint on hls.ts 2022-08-04 09:49:26 +02:00
Chocobozzz c795e19663
Automatically rebuild native modules on ABI change 2022-08-03 15:08:36 +02:00
Chocobozzz fd59208e8c
Fix lint 2022-08-03 11:54:08 +02:00
Chocobozzz 785f1897a4
Add notification plugin hook 2022-08-03 11:33:43 +02:00
Chocobozzz 0260dc8aca
Add channel server hooks 2022-08-03 11:24:42 +02:00
Chocobozzz 0b6f531653
Suffix external auth username on conflict 2022-08-03 10:10:26 +02:00
Chocobozzz ebb9e53ada
Add plugin hook on transcoding resolutions building 2022-08-02 16:05:44 +02:00
Chocobozzz 22df69fdec
Add job queue hooks 2022-08-02 15:50:05 +02:00
Chocobozzz 7a9e420a02
Remove uneeded async 2022-08-02 14:41:44 +02:00
Chocobozzz 15eb9e5b8f
Fix typescript error 2022-08-01 15:27:49 +02:00
Chocobozzz 1bb4c9ab2e
Add ability to delete a specific video file 2022-08-01 14:55:10 +02:00
Chocobozzz ce6b3765a2
Add more tracing 2022-07-28 10:56:05 +02:00
Chocobozzz adc94cf09c
Add live and viewers otel metrics 2022-07-27 16:19:25 +02:00
Chocobozzz 50cc1ee48a
Fix process follow 2022-07-27 15:45:04 +02:00
Chocobozzz 073deef886
Handle rejected follows in client
Also add quick filters so it's easier to find pending follows
2022-07-27 13:52:13 +02:00
Chocobozzz 927fa4b11f
Add rejected state to follows
Prevent reprocessing already rejected follows
2022-07-27 13:52:13 +02:00
Chocobozzz 0f58b11f5c
Reduce joins need to generate AP url 2022-07-27 13:52:13 +02:00
Chocobozzz 9e2a4af3a0
Use unique AP id for accept/reject 2022-07-25 11:24:05 +02:00
Chocobozzz 7b6b445d91
Regenerate video filenames on transcoding
In particular when using manual transcoding, to invalidate potential
HTTP caches in front of peertube
2022-07-25 10:57:16 +02:00
Chocobozzz c8fa571f32
Clearer live session
Get the save replay setting when the session started to prevent
inconsistent behaviour when the setting changed before the session was
processed by the live ending job

Display more information about the potential session replay in live
modal information
2022-07-22 15:22:21 +02:00
Chocobozzz 7fd28c99b9
Fix lint 2022-07-19 15:34:44 +02:00
Chocobozzz cd1b8e9a40
Add node process metrics 2022-07-19 11:58:19 +02:00
Chocobozzz 0755cb89e7
Prevent logging error when saving delete live 2022-07-18 15:49:15 +02:00
Chocobozzz 1cc9774668
Fix incorrect error logs 2022-07-18 11:55:13 +02:00
Chocobozzz 654d4ede7f
Correctly handle actors without follow counters 2022-07-15 11:18:29 +02:00
Chocobozzz 2939925639
Fix live tests 2022-07-13 15:37:36 +02:00
Chocobozzz ba2684cedd
Fix lint 2022-07-13 12:14:26 +02:00
Chocobozzz c5cadb2859
Reduce unknown undo logging level 2022-07-13 11:07:01 +02:00
Chocobozzz 39d117a47d
Prevent error logs when aborting a stream 2022-07-13 10:57:04 +02:00
Chocobozzz 14f7ad399f
Add more error logs 2022-07-13 10:17:47 +02:00
Chocobozzz 3b0525106d
Prevent duplicated HLS playlist on transcoding 2022-07-13 10:15:41 +02:00
Chocobozzz b0f4204266
Relax log level
401 is expected when the token has expired
federation issues are not very important
2022-07-13 10:04:22 +02:00
Chocobozzz eaaf316fe1
Don't display errors when SMTP is not configured
info is enough
2022-07-12 16:32:05 +02:00
Chocobozzz 46f7cd6837
Fix handling broken lives
We want the latest session that may already be finished
2022-07-12 16:30:59 +02:00
Chocobozzz 4ddb53f65d
Auto retry video state db query on failure 2022-07-12 16:25:21 +02:00
Chocobozzz a49407524f
Prevent invalid watch sections 2022-07-12 09:51:21 +02:00
Chocobozzz 9452d4fd33
/!\ Use a dedicated config file for development
It means you have to replace NODE_ENV=test to NODE_ENV=dev if you use it
npm run dev:* commands are already updated
2022-07-06 16:12:17 +02:00
Chocobozzz 630d0a1bf5
Introduce experimental telemetry 2022-07-06 15:13:55 +02:00
Chocobozzz cd25344f74
Reduce lazy static error logs 2022-06-30 09:25:17 +02:00
Chocobozzz 031bbcd25d
Upgrade oauth2-server 2022-06-28 15:32:57 +02:00
Chocobozzz 4fae2b1f30
Rename video full loading 2022-06-28 15:06:15 +02:00
Chocobozzz daa0226b0a
Lazy load piscina 2022-06-28 09:09:03 +02:00
Chocobozzz ccd62a4502
Fix workers constants 2022-06-27 15:56:57 +02:00
Chocobozzz 3a54605d4e
Process images in a dedicated worker 2022-06-27 11:53:12 +02:00
Chocobozzz c53853ca1b
Introduce worker threads to process remote images 2022-06-24 16:31:48 +02:00
Chocobozzz 92083e4228
Fix removed sha segments on fast restream 2022-06-24 09:57:55 +02:00
Chocobozzz 64324ac646
Merge branch 'release/4.2.0' into develop 2022-06-23 10:53:16 +02:00
Chocobozzz 53023be33a
Fix fast restream in saved permanent live 2022-06-23 10:30:48 +02:00
Chocobozzz a91e9beb3f
Update server dependencies 2022-06-21 11:23:32 +02:00
Chocobozzz 714e33a742
Merge branch 'release/4.2.0' into develop 2022-06-20 10:23:19 +02:00
Chocobozzz f53e1fcd99
Relax views federation 2022-06-20 10:04:52 +02:00
Chocobozzz aa2ce188d1
Optimize view endpoint 2022-06-17 14:56:16 +02:00
Chocobozzz fba911e2c8
Merge branch 'release/4.2.0' into develop 2022-06-17 14:17:06 +02:00
Chocobozzz f27b7a750f
Send views in a dedicated queue 2022-06-17 14:08:13 +02:00
Chocobozzz 3396e65345
Optimize broadcast job creation 2022-06-17 11:55:03 +02:00
Chocobozzz 68d6e57870
Fix transaction when processing local viewer 2022-06-17 09:03:28 +02:00
Chocobozzz 3256771725
More robust move to object storage job failure 2022-06-16 14:29:53 +02:00
Chocobozzz 0c9a835466
Merge branch 'release/4.2.0' into develop 2022-06-16 14:11:31 +02:00
Chocobozzz cdd838168d
Fix live ending job that breaks new live session 2022-06-16 13:39:57 +02:00
Chocobozzz b89b0bfce9
Clearer video creation from API regarding rates 2022-06-08 15:42:09 +02:00
luz paz 7a4fd56ccd Fix various typos
Found via `codespell -q 3 -S ./CREDITS.md,./CHANGELOG.md,./client/src/locale,./yarn.lock,./client/yarn.lock -L doubleclick,followings,nd,ot,ro,serie,splitted,tread,truthy`
2022-06-07 16:41:21 +02:00
Chocobozzz 6d9ed0b6cb
Merge branch 'release/4.2.0' into develop 2022-06-07 09:59:15 +02:00
Chocobozzz 4d52690134
Fix schema.org context 2022-06-07 08:47:41 +02:00
Chocobozzz e815f566f3
Remove unused function 2022-06-03 16:28:20 +02:00
Chocobozzz d3f4689bde
Add live autostart/messages in embed 2022-05-31 14:24:07 +02:00
Chocobozzz ab623c0e0b
Merge branch 'release/4.2.0' into develop 2022-05-25 15:22:28 +02:00
Chocobozzz c8fdfab0e3
More robust live ending job 2022-05-25 15:18:29 +02:00
Chocobozzz 5333788c08
Fix saving permanent live replay on quick restream 2022-05-25 14:54:16 +02:00
Chocobozzz b34ee7fa5f
Cleanup muxing session method options 2022-05-25 11:08:12 +02:00
Chocobozzz 3eba7ab815
Add ability to set a custom video import timeout 2022-05-24 14:55:07 +02:00
Chocobozzz 3329325a64
Fix email action button label for reports 2022-05-24 09:19:03 +02:00
Chocobozzz 941d28cc7f
Reduce error logs 2022-05-18 12:01:02 +02:00
Chocobozzz ca3d5912e1
Add use proxy for s3 (#4973)
* Fix object storage to be accessible via proxy

* fix lint

* Use hpagent

* Fix lint

* Fix PR

Co-authored-by: noellabo <noel.yoshiba@gmail.com>
2022-05-09 11:49:25 +02:00
Chocobozzz f82ea670d8
Fix collection type 2022-05-09 09:51:11 +02:00
Chocobozzz f40712abbb
Add ability to filter overall video stats by date 2022-05-06 14:23:02 +02:00
Chocobozzz bae616273d
Convert followers/following in raw SQL queries
Prevent weird bug in SQL generation
2022-05-05 10:29:35 +02:00
Chocobozzz 9a82ce2455
Fix quota inconstistencies with lives 2022-05-04 10:32:49 +02:00
Chocobozzz 26e3e98ff0
Support live session in server 2022-05-03 14:49:15 +02:00
Chocobozzz cfbe6be516
Add ability to provide rtmp listening hostname 2022-05-02 14:32:12 +02:00
Chocobozzz 4ec52d04dc
Add ability to save replay of permanent lives 2022-04-21 11:47:57 +02:00
kontrollanten 156cdbac22
object-storage: @aws-sdk/lib-storage for multipart (#4903)
* object-storage: @aws-sdk/lib-storage for multipart

* gitignore: add .DS_Store

* test(object-storage): remove only

* test(object-storage/multipart): generate video

* fix lint issue

* test(obj-storage/video): ensure file size

* Styling

Co-authored-by: Chocobozzz <me@florianbigard.com>
2022-04-19 15:22:18 +02:00
Chocobozzz 77d0ae7b20
Fix 2 hours limit on uploads 2022-04-19 11:44:03 +02:00
Chocobozzz e85a36cb96
Fix email text containing links 2022-04-15 15:17:32 +02:00
Chocobozzz 3c7ddd7daa
Fix email links displayed twice as text 2022-04-15 14:19:07 +02:00
Chocobozzz 3eda9b775a Support interactive video stats graph 2022-04-15 09:49:35 +02:00
Chocobozzz 901bcf5c18 Add ability to set start/end date to timeserie 2022-04-15 09:49:35 +02:00
Chocobozzz ac907dc7c1 Improve viewer counter
More precise, avoid weird decrease, reuse an id to federate viewers
2022-04-15 09:49:35 +02:00
Chocobozzz dfbcefc20d Improve views/viewers documentation 2022-04-15 09:49:35 +02:00
Chocobozzz b211106695 Support video views/viewers stats in server
* Add "currentTime" and "event" body params to view endpoint
 * Merge watching and view endpoints
 * Introduce WatchAction AP activity
 * Add tables to store viewer information of local videos
 * Add endpoints to fetch video views/viewers stats of local videos
 * Refactor views/viewers handlers
 * Support "views" and "viewers" counters for both VOD and live videos
2022-04-15 09:49:35 +02:00
Chocobozzz 69d48ee30c
Don't clean mastodon rates
Workaround of https://github.com/mastodon/mastodon/issues/13571
2022-04-15 09:17:16 +02:00
Chocobozzz c6d20c84a7
Disallow unlisted video indexation 2022-04-08 16:28:14 +02:00
Chocobozzz a219c9100b
Refactor AP context builder 2022-03-24 09:40:46 +01:00
Chocobozzz 7e98a7df7d
Remove activitypub helper
Put functions in lib/activitypub instead
2022-03-23 14:24:50 +01:00
Chocobozzz 92e66e04f7
Rename studio to editor 2022-03-22 17:24:32 +01:00
Chocobozzz 1808a1f8e4
Add video edition finished notification 2022-03-22 16:25:14 +01:00
Chocobozzz f4110e0a72
Update server dependencies 2022-03-21 14:20:47 +01:00
Chocobozzz 57e4e1c1a9
Don't store remote rates of remote videos
In the future we'll stop to expose all available rates to improve users
privacy
2022-03-18 11:21:50 +01:00
Chocobozzz 2769876fb2
Fix client html cache on theme update 2022-03-17 09:09:06 +01:00
Chocobozzz 52fe4b671a
Fix torrent creation 2022-03-16 18:35:05 +01:00
Chocobozzz f012319a64
Process video torrents in order
Prevent update before video torrent generation for example
2022-03-16 15:45:58 +01:00
Doug Luce f9915efa5e Make object storage ACL configurable
Override this value to allow uploads to non-public S3
buckets. Otherwise "AccessDenied: Access Denied" errors will end up in
the log.

Fixes #4850
2022-03-16 10:58:01 +01:00
kontrollanten 60233e90d2
add client.videos.upload.maxChunkSize config (#4857)
* add client.videos.upload.maxChunkSize config

* updates after review

* remove maxChunkSize from custom-config
2022-03-16 10:57:36 +01:00
Chocobozzz f443a74649 Add latency setting support 2022-03-09 09:23:10 +01:00
kontrollanten 2c102aac13
Display avatar in video miniatures (#4823)
* show avatar in video miniatures

closes #4040

* fix: videos for api, include avatars

* add config for display video miniature avatar

* Revert "fix: videos for api, include avatars"

This reverts commit 5520a3eda6.

* video-min: set display avatar to false as default

* remove empty line

* rename display_avatar > display_author_avatar

* fix renaming displayAvatar > displayAuthorAvatar
2022-03-08 13:39:27 +01:00
Chocobozzz cc45cc9ac1
Remove uneeded memoize 2022-03-08 11:39:56 +01:00
Chocobozzz d3d3deaa7a
Refactor user build and express file middlewares 2022-03-04 10:57:36 +01:00
Chocobozzz f7ac03ee94
Fix client html performance regression 2022-02-28 15:13:56 +01:00
Chocobozzz c729caf6cc Add basic video editor support 2022-02-28 10:42:19 +01:00
kontrollanten d0800f7661
Implement avatar miniatures (#4639)
* client: remove unused file

* refactor(client/my-actor-avatar): size from input

Read size from component input instead of scss, to make it possible to
use smaller avatar images when implemented.

* implement avatar miniatures

close #4560

* fix(test): max file size

* fix(search-index): normalize res acc to avatarMini

* refactor avatars to an array

* client/search: resize channel avatar to 120

* refactor(client/videos): remove unused function

* client(actor-avatar): set default size

* fix tests and avatars full result

When findOne is used only an array containting one avatar is returned.

* update migration version and version notations

* server/search: harmonize normalizing

* Cleanup avatar miniature PR

Co-authored-by: Chocobozzz <me@florianbigard.com>
2022-02-28 08:34:43 +01:00
Chocobozzz 9d8ef212ff
Fix broken dep 2022-02-09 17:48:15 +01:00
Chocobozzz 400ed2ab41
Add log on user plugin auth conflict 2022-02-09 11:54:18 +01:00
Chocobozzz 7630e1c893
Fix import timeout inconsistency 2022-02-09 11:54:18 +01:00
Chocobozzz c68e2b2d22
Fix plaintext markdown converter 2022-02-04 10:38:32 +01:00
Chocobozzz b25fdc73fd
Add fixme info 2022-02-02 09:20:40 +01:00
Chocobozzz a2caee9f51
Fix HLS re transcoding with object storage enabled 2022-02-01 14:19:44 +01:00
Chocobozzz 0f11ec8dd3
Prevent HLS transcoding after webtorrent transcoding 2022-02-01 11:27:38 +01:00
lutangar d2d4a5a999 Fix plugin settings manager definition
This mainly fix the `onSettingsChange` argument typing.
I'm not 100% sure the setting `value` can be a boolean though.
But this is how it was typed before so I just made it consistent.

Feel free to change - or suggest change - to the type names / location.
2022-01-20 13:43:20 +01:00
Chocobozzz d511df2890
Add ability to filter my imports by target URL 2022-01-19 14:58:16 +01:00
Chocobozzz 419b520ca4
Add ability to cancel & delete video imports 2022-01-19 14:31:05 +01:00
Chocobozzz 2f0a0ae217
Upgrade uploadx dependency 2022-01-18 10:47:26 +01:00
Chocobozzz ff4d2c73fd
Fix bull redis options 2022-01-14 15:06:33 +01:00
Chocobozzz c0d2eac332
Update server dependencies 2022-01-14 14:15:23 +01:00
Chocobozzz cbe2f36d93
Fix audio transcoding with video only file 2022-01-06 17:55:37 +01:00
Chocobozzz 75b7117f07
Remove unnecessary NPM_RUN_BUILD_OPTS docker arg 2022-01-06 10:37:55 +01:00
Chocobozzz f8360396ff
Stop using tsconfig register
Prefer to replace paths at compile time
2022-01-06 08:26:14 +01:00
Chocobozzz 8f5a1f36b5
Try to fix tests 2022-01-04 09:16:43 +01:00
Chocobozzz 0628157fe9
Move uuid stuff in extra utils
Since it requires an external dependency
2021-12-29 14:44:58 +01:00
Chocobozzz fb3c9e2bf5
Translate plugin settings 2021-12-29 14:08:07 +01:00
Chocobozzz 10a72a7e61
Fix tests 2021-12-28 14:12:57 +01:00
Chocobozzz f1569117f9
Cleanup unavailable remote AP resource 2021-12-28 11:37:05 +01:00
Chocobozzz ae4e417ab0
Update CREDITS 2021-12-24 14:56:33 +01:00
Chocobozzz e5d91a9b9c
Upgrade redis dep 2021-12-24 14:52:24 +01:00
Chocobozzz c7cdac4409
Fix type conflict 2021-12-24 10:40:51 +01:00
Chocobozzz d17c7b4e8c
Fix shared imports 2021-12-24 10:14:47 +01:00
Chocobozzz 5354af75b3
Fix video import transcoding 2021-12-23 13:40:47 +01:00
Chocobozzz dbd9fb44dd
Don't stuck state when move transcoding job failed 2021-12-23 11:13:06 +01:00
Chocobozzz 482b26231b
Fix audio only transcoding 2021-12-23 10:57:55 +01:00
Chocobozzz 293074ae79
Fix video upload with big preview 2021-12-17 15:50:02 +01:00
Chocobozzz f304a1580b
Fix build 2021-12-17 13:58:07 +01:00