Commit Graph

989 Commits

Author SHA1 Message Date
Chocobozzz 7d9ba5c089
Cleanup models directory organization 2021-05-11 11:32:31 +02:00
Chocobozzz c07902b908
Move transcoding files in their own directory 2021-05-11 11:32:31 +02:00
Chocobozzz 1bcb03a100
Use a class for youtube-dl 2021-05-11 11:32:31 +02:00
kontrollanten f6d6e7f861
Resumable video uploads (#3933)
* WIP: resumable video uploads

relates to #324

* fix review comments

* video upload: error handling

* fix audio upload

* fixes after self review

* Update server/controllers/api/videos/index.ts

Co-authored-by: Rigel Kent <par@rigelk.eu>

* Update server/middlewares/validators/videos/videos.ts

Co-authored-by: Rigel Kent <par@rigelk.eu>

* Update server/controllers/api/videos/index.ts

Co-authored-by: Rigel Kent <par@rigelk.eu>

* update after code review

* refactor upload route

- restore multipart upload route
- move resumable to dedicated upload-resumable route
- move checks to middleware
- do not leak internal fs structure in response

* fix yarn.lock upon rebase

* factorize addVideo for reuse in both endpoints

* add resumable upload API to openapi spec

* add initial test and test helper for resumable upload

* typings for videoAddResumable middleware

* avoid including aws and google packages via node-uploadx, by only including uploadx/core

* rename ex-isAudioBg to more explicit name mentioning it is a preview file for audio

* add video-upload-tmp-folder-cleaner job

* stronger typing of video upload middleware

* reduce dependency to @uploadx/core

* add audio upload test

* refactor resumable uploads cleanup from job to scheduler

* refactor resumable uploads scheduler to compare to last execution time

* make resumable upload validator to always cleanup on failure

* move legacy upload request building outside of uploadVideo test helper

* filter upload-resumable middlewares down to POST, PUT, DELETE

also begin to type metadata

* merge add duration functions

* stronger typings and documentation for uploadx behaviour, move init validator up

* refactor(client/video-edit): options > uploadxOptions

* refactor(client/video-edit): remove obsolete else

* scheduler/remove-dangling-resum: rename tag

* refactor(server/video): add UploadVideoFiles type

* refactor(mw/validators): restructure eslint disable

* refactor(mw/validators/videos): rename import

* refactor(client/vid-upload): rename html elem id

* refactor(sched/remove-dangl): move fn to method

* refactor(mw/async): add method typing

* refactor(mw/vali/video): double quote > single

* refactor(server/upload-resum): express use > all

* proper http methud enum server/middlewares/async.ts

* properly type http methods

* factorize common video upload validation steps

* add check for maximum partially uploaded file size

* fix audioBg use

* fix extname(filename) in addVideo

* document parameters for uploadx's resumable protocol

* clear META files in scheduler

* last audio refactor before cramming preview in the initial POST form data

* refactor as mulitpart/form-data initial post request

this allows preview/thumbnail uploads alongside the initial request,
and cleans up the upload form

* Add more tests for resumable uploads

* Refactor remove dangling resumable uploads

* Prepare changelog

* Add more resumable upload tests

* Remove user quota check for resumable uploads

* Fix upload error handler

* Update nginx template for upload-resumable

* Cleanup comment

* Remove unused express methods

* Prefer to use got instead of raw http

* Don't retry on error 500

Co-authored-by: Rigel Kent <par@rigelk.eu>
Co-authored-by: Rigel Kent <sendmemail@rigelk.eu>
Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-05-10 11:13:41 +02:00
Chocobozzz e024fd6a74
Update channel updatedAt when uploading a video 2021-05-10 09:38:11 +02:00
Chocobozzz a66c2e3252
Fix remote actor creation date 2021-05-07 09:00:09 +02:00
kontrollanten 66f77f6343 server: fix HLS playlist format
Remove prefixed comma when there's no video codec.
2021-05-06 08:47:43 +02:00
Chocobozzz b31d72625d
Plugin user.getAuthUser is now async
So we can load the full user
2021-05-05 15:26:28 +02:00
Kimsible 9a911038d9 Add shorter URLs for accounts and channels server-side 2021-05-05 11:47:03 +02:00
Chocobozzz 36217ac537
Fix lint 2021-05-05 09:55:01 +02:00
Chocobozzz a6e37eebfb
Fix transcoding job priority
New resolution jobs are also important if waiting for transcoding is
enabled since we publish the video after the first resolution generation
2021-05-05 09:32:20 +02:00
Chocobozzz 302eba0d89
Add data directory for plugins and some helpers 2021-04-22 10:55:28 +02:00
Chocobozzz b7a27f2860
Add sendmail log in server 2021-04-21 15:18:22 +02:00
Chocobozzz 67baf6478a
Allow plugins to set short translation locale 2021-04-20 16:14:09 +02:00
Chocobozzz 24a792404c
Support async onSettingsChange 2021-04-20 16:02:15 +02:00
Rigel Kent a073c91270 modify tests to support current behaviour regarding plaintext description 2021-04-13 15:45:09 +02:00
Rigel Kent 84bced652c render markdown and plainify descriptions on previews 2021-04-13 15:45:09 +02:00
Chocobozzz 1a578165f9
Fix lint 2021-04-12 17:00:21 +02:00
Chocobozzz c24822a8fd
Don't save other attributes when set as refreshed 2021-04-12 13:44:58 +02:00
Rigel Kent fe19f600da
add channel and playlist stats to server stats endpoint (#3747)
* add channel and playlist stats to nodeinfo

* add tests for active video channels stats

* fix tests for active channel stats
2021-04-12 11:19:07 +02:00
Chocobozzz 8280d0c227
Ensure to install supported plugins 2021-04-12 10:10:48 +02:00
Chocobozzz 78ec174cca
Don't retry failed transaction for refresh
It's just a refresh and could lead to some issues in our tests. If there
is a transaction conflict, it means that the same video is already
beeing updated so aborting should be fine
2021-04-09 16:29:38 +02:00
Chocobozzz a60696ab18 Add scale filter to documentation 2021-04-09 15:53:18 +02:00
Théo Le Calvar 5fb7cfbac5 add support for inputOptions in trancode plugins 2021-04-09 15:53:18 +02:00
Chocobozzz 22820226e5
Add server plugin helpers 2021-04-09 15:17:43 +02:00
Chocobozzz bae9303726
Also decache config 2021-04-09 09:37:46 +02:00
John Livingston 09d535ef98
Fix #3940: unload all children from the plugin module on updates. 2021-04-09 09:32:16 +02:00
Chocobozzz 84531547bc
Add size info in db for actor images 2021-04-08 13:38:04 +02:00
Chocobozzz a0eeb45f14
Update data in DB when regenerate thumbnails 2021-04-08 13:38:04 +02:00
Chocobozzz 213e30ef90 Add banner tests 2021-04-08 10:07:53 +02:00
Chocobozzz 2cb03dc1f4 Add banners support 2021-04-08 10:07:53 +02:00
Chocobozzz f479685678 Agnostic actor image storage 2021-04-08 10:07:53 +02:00
Thavarasa Prasanth 4097c6d66c
fix missing title attribute on <iframe> tag suggested for embedding (#3901)
* title attribute is missing on <iframe> tag suggested for embedding #3861

* fix #3901

* fix: escapeHTML #3901

* fix: playlist title instead of video title #3901

* fix #3901

* assign title directly #3901
2021-03-31 08:32:05 +02:00
Chocobozzz 4bc45da342
Add hooks support for video download 2021-03-24 18:18:41 +01:00
Chocobozzz 97aeb3cc46
Fix external on logout hook 2021-03-24 18:18:41 +01:00
Chocobozzz f43db2f46e
Refactor auth flow
Reimplement some node-oauth2-server methods to remove hacky code needed by our external
login workflow
2021-03-24 18:18:41 +01:00
Chocobozzz cae2df6bdc
Stricter email options typings 2021-03-24 18:18:41 +01:00
Chocobozzz 32a18cbf33
Add new plugin/peertube version notifs 2021-03-24 18:18:41 +01:00
Chocobozzz e7053b1d9d
Fix AP security tests 2021-03-24 18:18:40 +01:00
Chocobozzz 266131e0ca
Don't skip all threads on AP fetcher error 2021-03-24 18:18:40 +01:00
Chocobozzz b5c361089f
Fix 404 AP status codes 2021-03-24 18:18:40 +01:00
Chocobozzz db4b15f21f
Use got instead of request 2021-03-24 18:18:40 +01:00
Chocobozzz 67f87b66fe
Refactor a little bit AP validators 2021-03-24 18:18:40 +01:00
Chocobozzz 452b3bea08
Introduce tags to the logger
That could help to filter unneeded logs and/or select particular
components
2021-03-24 18:18:40 +01:00
Chocobozzz a2eba50b8e
Fix moderation notification 2021-03-04 16:16:37 +01:00
Chocobozzz fbd51e69f2
Upgrade server dependencies 2021-03-03 16:14:32 +01:00
Chocobozzz 6c9c3b7b14
Fix emptying video tags 2021-03-03 11:03:30 +01:00
Chocobozzz 1ef447bd83
Don't create another uneeded preview placeholder 2021-03-03 11:02:34 +01:00
Chocobozzz dcd75f786c
Rename request timeout 2021-03-03 10:10:55 +01:00
Chocobozzz 74d249bc13
Add ability to cleanup remote AP interactions 2021-03-03 10:08:08 +01:00
Chocobozzz 8795d6f254
Fix broken local actors
Some channels can't federate because they don't have public/private
keys, maybe because the generation failed for various reasons
2021-02-26 14:22:25 +01:00
Chocobozzz 92315d979c
More robust channel change federation 2021-02-26 11:50:18 +01:00
Chocobozzz 543442a3be
Add more AP stats to stats endpoint
It will help to understand if the federation correctly works or not
2021-02-26 10:28:11 +01:00
Chocobozzz 88cfa3e8fa
Fix preview upload with capitalized ext 2021-02-26 09:16:31 +01:00
Chocobozzz a77c7327a2
Fix process abuse transactions 2021-02-25 15:55:31 +01:00
Chocobozzz e7812bf091
Add refresh_token_expires_in to oauth res 2021-02-25 15:25:48 +01:00
Chocobozzz 70243d7a35
Correctly wait transcoding before federating 2021-02-22 10:35:37 +01:00
Chocobozzz 4cbdcf4409
Fix sending accept to old follows 2021-02-19 13:55:00 +01:00
Chocobozzz fae6e4da8f Try to speed up server tests 2021-02-19 09:36:50 +01:00
Chocobozzz 53c06121a4
Fix import tests 2021-02-18 14:08:16 +01:00
Chocobozzz 8efc27bf14 Cleanup 2021-02-18 13:38:09 +01:00
Chocobozzz b3d5cb92b1 Add ability to update torrents cache in client 2021-02-18 13:38:09 +01:00
Chocobozzz d9a2a03196 Don't guess remote tracker URL 2021-02-18 13:38:09 +01:00
Chocobozzz 2451916e45 Add video files migration 2021-02-18 13:38:09 +01:00
Chocobozzz 90a8bd305d Dissociate video file names and video uuid 2021-02-18 13:38:09 +01:00
Chocobozzz 1664bc60eb Optimize remote image processing 2021-02-16 10:36:44 +01:00
Chocobozzz 374b725df5 Optimize remote thumbnail processing 2021-02-16 10:36:44 +01:00
Chocobozzz a35a22797c Remove previous thumbnail if needed 2021-02-16 10:36:44 +01:00
Chocobozzz 6302d599cd Generate a name for caption files 2021-02-16 10:36:44 +01:00
Chocobozzz a8b1b40485 Generate a name for thumbnails
Allows aggressive caching
2021-02-16 10:36:44 +01:00
Chocobozzz c15aff666d
Remove old code
We consider Dislike as a classic activity now
2021-02-12 14:53:07 +01:00
Chocobozzz e3b4c084cd
Guess if we need to generate the thumbnail for imports 2021-02-12 09:37:01 +01:00
Chocobozzz 6559da2821
Allow plugins to load a video by id 2021-02-11 10:23:52 +01:00
Chocobozzz 44d1f7f2e8
Painfully debug concurrent import jobs 2021-02-09 11:46:30 +01:00
Chocobozzz 9129b7694d Allow to specify transcoding and import jobs concurrency 2021-02-08 15:38:45 +01:00
Chocobozzz ba5a8d89bb
Update server dependencies 2021-02-03 09:45:08 +01:00
Chocobozzz 69eddafb17
Fix HLS generation after import script 2021-02-02 14:00:46 +01:00
Chocobozzz aaedadd538
Transcode HLS playlists in a tmp directory 2021-02-02 11:50:29 +01:00
Chocobozzz 40930fda86
Transcode audio uploads to lower resolutions
Better consistency
2021-02-02 09:45:42 +01:00
Chocobozzz 89613cb444
Purge entire video from redundancy 2021-02-02 08:50:30 +01:00
Chocobozzz 7e3592d70b
Correctly remove torrents with HLS only 2021-02-01 11:23:12 +01:00
Chocobozzz 33c7131be5
Check banned status for external auths 2021-02-01 11:23:11 +01:00
Chocobozzz a8537c622e
Use veryfast preset for default transcoding profile 2021-01-29 15:31:31 +01:00
Chocobozzz 2498aaead1
Add plugin transcoding profile guide 2021-01-29 14:05:41 +01:00
Chocobozzz 1896bca09e
Support transcoding options/encoders by plugins 2021-01-28 15:55:39 +01:00
Chocobozzz 529b37527c
Use a profile manager for transcoding 2021-01-28 15:55:39 +01:00
Chocobozzz 00b87c5791
Fix live RAM usage when ffmpeg is too slow 2021-01-27 12:05:10 +01:00
Chocobozzz e08ff02a9f
Allow webp image uploads 2021-01-26 10:29:57 +01:00
Chocobozzz 448487a602
Fix sendmail emailer 2021-01-26 09:28:49 +01:00
Chocobozzz 6939cbac48 Safely remove webtorrent files
Only remove them on max quality HLS playlist generation
2021-01-25 14:38:52 +01:00
Chocobozzz 77d7e851dc Add priority to transcoding jobs
(1 = highest priority)

100 for new resolutions
10 for original file optimization

Add a malus for transcoding jobs depending on how many uploads the user did in the
last 7 days
2021-01-25 14:38:52 +01:00
Chocobozzz 24516aa26a
Refactor transcoding job handlers 2021-01-21 15:58:17 +01:00
Chocobozzz 3b01f4c0ac
Support progress for ffmpeg tasks 2021-01-21 14:42:43 +01:00
Chocobozzz 471251fa14
Fix error log 2021-01-20 09:50:22 +01:00
Chocobozzz 805b8619c1
Fix (again) youtube import 2021-01-19 16:36:21 +01:00
Rigel Kent 454c20fa7c
stricter youtubedl format selectors (#3516)
* stricter youtubedl format selectors

make sure selectors avoid av1, and otherwise match as closely to the
maximum resolution enabled for transcoding

* add support for merge formats in youtubedl

* avoid vp9.2 in youtubedl to avoid any HDR

* move getEnabledResolutions, safer replace of imported extension

* add test for youtube-dl selectors
2021-01-15 15:56:56 +01:00
Rigel Kent 1ea7da819e
add ability to remove one's avatar for account and channels (#3467)
* add ability to remove one's avatar for account and channels

* add ability to remove one's avatar for account and channels

* only display avatar edition options after input change
2021-01-13 09:12:55 +01:00
Chocobozzz c2b823826d
Log RTMP server error (address already in use) 2021-01-11 09:33:14 +01:00
Chocobozzz f0484f7a16
Fix autoblacklist notification issue 2021-01-04 10:50:47 +01:00
Chocobozzz cc49be3e35
Fix views processing hour 0 2020-12-30 14:29:17 +01:00
Chocobozzz f88453e233
Fix build 2020-12-22 16:49:17 +01:00
Chocobozzz 236841a1d7
Force HLS transcoding
Fix weird behaviour with some web browsers. Don't really know if it's a
ffmpeg bug, a safari bug or a peertube bug, but forcing transcoding
seems to fix this playback bug
2020-12-22 16:04:42 +01:00
Chocobozzz 9cfeb3cf98
Fix redundancy federation in some cases 2020-12-17 09:23:57 +01:00
Chocobozzz 99afa081bc
Add AP stats 2020-12-15 13:34:58 +01:00
Chocobozzz c9bc850e93
Use -1 for max live duration unlimited 2020-12-15 09:26:37 +01:00
kontrollanten 40c52969f7 emails: remove hardcoded PeerTube names 2020-12-14 15:37:29 +01:00
kontrollanten 2e4b8ae485 emailer: use instance name instead of hostname 2020-12-14 15:37:29 +01:00
kontrollanten 915e2bbb97 video embed: use instance name as html title 2020-12-14 15:25:51 +01:00
kontrollanten 2564d97e0d video-embed: inject custom CSS
closes #3420
2020-12-14 15:25:51 +01:00
Chocobozzz 402145b863
Refactor jobs state 2020-12-14 12:00:35 +01:00
Chocobozzz f98c395295 Keep scope in notifier 2020-12-14 09:34:47 +01:00
kontrollanten b1771455af fix(notifier): notifyModeratorsOfNewAbuse
closes #3445
2020-12-14 09:34:47 +01:00
Rigel Kent 040d6896a3
add display of logs matching any state 2020-12-13 19:27:25 +01:00
Girish Ramakrishnan 75594f474a do not crash if SMTP server is down
just log a warning if the SMTP server is down on startup time

fixes #3457
2020-12-12 08:50:59 +01:00
Chocobozzz a800dbf345
Live views update 2020-12-09 16:31:42 +01:00
Chocobozzz 0d8de2756f
Fix live tests 2020-12-09 14:45:47 +01:00
Chocobozzz b49f22d8f9 Upgrade sequelize to v6 2020-12-09 11:41:22 +01:00
Rigel Kent f2eb23cd87
emit more specific status codes on video upload (#3423)
- reduce http status codes list to potentially useful codes
- convert more codes to typed ones
- factorize html generator for error responses
2020-12-08 21:16:10 +01:00
Chocobozzz a59f210ff3 Refactor a little bit client canonical URL 2020-12-08 11:15:11 +01:00
Thomas Citharel 106fa2249d Set canonical link to original video/playlist url
The local URL was used instead of the original url

Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2020-12-08 11:15:11 +01:00
Chocobozzz 5b9b403a20
Fix replay last segments loop 2020-12-07 16:07:00 +01:00
Rigel Kent 2d53be0267
replace numbers with typed http status codes (#3409) 2020-12-07 14:32:36 +01:00
Chocobozzz aa5ee5017a
Fix lint 2020-12-04 15:58:29 +01:00
Chocobozzz e772bdf14c
Load video in permanent live after last one ended 2020-12-04 15:38:32 +01:00
Chocobozzz 3851e732c4
Fix audio issues with live replay 2020-12-04 15:10:13 +01:00
Chocobozzz e26dc0cdbd
Strict actor url comparison 2020-12-03 16:12:58 +01:00
Chocobozzz 5147a6d945
Try to fix non HTTPS remote accounts 2020-12-03 15:21:16 +01:00
Chocobozzz bb4ba6d94c
Add permanent live support 2020-12-03 15:21:16 +01:00
Chocobozzz 4a54a93941
Fix live invalid save replay duration 2020-12-03 10:05:10 +01:00
Chocobozzz 2650d6d489
Fix live replay duration glitch 2020-12-02 10:18:15 +01:00
Chocobozzz 8c666c44ab
Fix live saving after a server restart 2020-11-30 17:03:13 +01:00
Chocobozzz 6cddd97d1d
Fix lint 2020-11-30 16:02:34 +01:00
Chocobozzz 937581b8f6
Fix high CPU with long live when save replay is true 2020-11-30 15:59:22 +01:00
Chocobozzz 884d2c39ae
Fix live FPS limit 2020-11-26 11:30:18 +01:00
Chocobozzz 6b67897e2e Add transcoding module comments 2020-11-25 10:07:51 +01:00
Chocobozzz 33ff70baa6 Fix audio encoding params 2020-11-25 10:07:51 +01:00
Chocobozzz ca5c612bfd Add live transcoding bit rate tests 2020-11-25 10:07:51 +01:00
Chocobozzz 5a547f69d5 Support encoding profiles 2020-11-25 10:07:51 +01:00
Chocobozzz 9252a33d11 Export encoders options in a dedicated struct 2020-11-25 10:07:51 +01:00
Chocobozzz daf6e48010 Split ffmpeg utils with ffprobe utils 2020-11-25 10:07:51 +01:00
Chocobozzz 74fd2643b4 Provide express request to onLogout call
+ pluginInfo related changes
2020-11-20 15:36:43 +01:00
Chocobozzz 55d5fbc583
Fix lint 2020-11-20 11:35:33 +01:00
Chocobozzz de94ac86a2
Fix incorrect IDs in AP federation 2020-11-20 11:21:08 +01:00
Chocobozzz fce7fe04ee
Update server dependencies 2020-11-19 09:07:20 +01:00
Chocobozzz 9ff36c2d70
Refactor markdown/sanitize html code 2020-11-17 14:34:09 +01:00
Chocobozzz 6bff8ce23a
Process remaining segment hashes on live ending 2020-11-17 14:05:15 +01:00
Chocobozzz 5c0904fc66
Cleanup lives on server restart 2020-11-13 14:36:30 +01:00
Chocobozzz 786b855af7
Fix live sha segments with multiple resolutions 2020-11-13 12:02:29 +01:00
Chocobozzz 17119e4a54 Reorganize left menu and account menu
Add my-settings and my-library in left menu
Move administration below my-library
Split account menu: my-setting and my library
2020-11-13 12:02:21 +01:00
Chocobozzz 501af82d99
Delay notification when waiting for a live 2020-11-12 17:13:32 +01:00