02b286f890
More robust webtorrent redundancy download
...
Avoid issues with inconsistencies between magnet infohash and torrent
infohash, blocking webtorrent upload that will timeout
2021-09-06 16:19:49 +02:00
41fb13c330
esModuleInterop to true
2021-08-27 15:12:22 +02:00
5a298a5a3d
Better logs for transcoding
2021-08-26 10:01:42 +02:00
1f6125be8b
Optimize torrent URL update
2021-08-18 09:14:51 +02:00
0305db28c9
Add support for saving video files to object storage ( #4290 )
...
* Add support for saving video files to object storage
* Add support for custom url generation on s3 stored files
Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.
* Upload files to s3 concurrently and delete originals afterwards
* Only publish after move to object storage is complete
* Use base url instead of url template
* Fix mistyped config field
* Add rudenmentary way to download before transcode
* Implement Chocobozzz suggestions
https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478
The remarks in question:
Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19 uses a string instead of an integer
I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs
* Import correct function
* Support multipart upload
* Remove import of node 15.0 module stream/promises
* Extend maximum upload job length
Using the same value as for redundancy downloading seems logical
* Use dynamic part size for really large uploads
Also adds very small part size for local testing
* Fix decreasePendingMove query
* Resolve various PR comments
* Move to object storage after optimize
* Make upload size configurable and increase default
* Prune webtorrent files that are stored in object storage
* Move files after transcoding jobs
* Fix federation
* Add video path manager
* Support move to external storage job in client
* Fix live object storage tests
Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-08-17 08:26:20 +02:00
679c12e69c
Improve target bitrate calculation
2021-08-06 14:13:26 +02:00
e83d06a702
Remove deprecated transcoding job names
2021-08-05 14:29:44 +02:00
d26836cd95
Refactor notifier
2021-08-02 10:39:51 +02:00
764b1a14fc
Use random names for VOD HLS playlists
2021-07-26 11:29:31 +02:00
83903cb65d
Generate random uuid for video files
2021-07-26 11:29:31 +02:00
c0e8b12e7f
Refactor requests
2021-07-20 15:27:18 +02:00
37a44fc915
Add ability to search playlists
2021-06-25 14:44:01 +02:00
8ebf2a5d5d
Refactor live manager
2021-06-17 08:47:03 +02:00
4c3e4c3d93
Fix lint
2021-06-15 14:19:30 +02:00
eae0365b5c
Fix missing transactions
2021-06-15 10:40:06 +02:00
7500d6c900
Use a global request timeout
2021-06-15 08:21:31 +02:00
4ead40e776
Cache refresh actor promise
2021-06-09 13:34:40 +02:00
765b150245
Fix lint
2021-06-08 10:19:04 +02:00
ea54cd04c1
Fix video upload with a capitalized ext
2021-06-08 10:01:50 +02:00
868fce62f8
fetch -> load
...
avoid confusion with AP fetch functions
2021-06-03 18:10:56 +02:00
10363c74c1
Move middleware utils in middlewares
...
helpers modules should not import models
2021-06-03 18:03:36 +02:00
136d7efde7
Refactor AP actors
2021-06-03 16:40:32 +02:00
49af5ac8c2
Refactor AP playlists
2021-06-03 16:40:32 +02:00
69290ab37b
Refactor AP video update
2021-06-02 16:57:53 +02:00
2539932e16
Instance homepage support ( #4007 )
...
* Prepare homepage parsers
* Add ability to update instance hompage
* Add ability to set homepage as landing page
* Add homepage preview in admin
* Dynamically update left menu for homepage
* Inject home content in homepage
* Add videos list and channel miniature custom markup
* Remove unused elements in markup service
2021-05-27 15:59:55 +02:00
16c016e8b1
Stricter models typing
2021-05-12 14:12:14 +02:00
7d9ba5c089
Cleanup models directory organization
2021-05-11 11:32:31 +02:00
c07902b908
Move transcoding files in their own directory
2021-05-11 11:32:31 +02:00
1bcb03a100
Use a class for youtube-dl
2021-05-11 11:32:31 +02:00
36217ac537
Fix lint
2021-05-05 09:55:01 +02:00
a6e37eebfb
Fix transcoding job priority
...
New resolution jobs are also important if waiting for transcoding is
enabled since we publish the video after the first resolution generation
2021-05-05 09:32:20 +02:00
e7053b1d9d
Fix AP security tests
2021-03-24 18:18:40 +01:00
b5c361089f
Fix 404 AP status codes
2021-03-24 18:18:40 +01:00
db4b15f21f
Use got instead of request
2021-03-24 18:18:40 +01:00
67f87b66fe
Refactor a little bit AP validators
2021-03-24 18:18:40 +01:00
dcd75f786c
Rename request timeout
2021-03-03 10:10:55 +01:00
74d249bc13
Add ability to cleanup remote AP interactions
2021-03-03 10:08:08 +01:00
8795d6f254
Fix broken local actors
...
Some channels can't federate because they don't have public/private
keys, maybe because the generation failed for various reasons
2021-02-26 14:22:25 +01:00
70243d7a35
Correctly wait transcoding before federating
2021-02-22 10:35:37 +01:00
8efc27bf14
Cleanup
2021-02-18 13:38:09 +01:00
2451916e45
Add video files migration
2021-02-18 13:38:09 +01:00
90a8bd305d
Dissociate video file names and video uuid
2021-02-18 13:38:09 +01:00
a35a22797c
Remove previous thumbnail if needed
2021-02-16 10:36:44 +01:00
e3b4c084cd
Guess if we need to generate the thumbnail for imports
2021-02-12 09:37:01 +01:00
44d1f7f2e8
Painfully debug concurrent import jobs
2021-02-09 11:46:30 +01:00
9129b7694d
Allow to specify transcoding and import jobs concurrency
2021-02-08 15:38:45 +01:00
69eddafb17
Fix HLS generation after import script
2021-02-02 14:00:46 +01:00
40930fda86
Transcode audio uploads to lower resolutions
...
Better consistency
2021-02-02 09:45:42 +01:00
7e3592d70b
Correctly remove torrents with HLS only
2021-02-01 11:23:12 +01:00
6939cbac48
Safely remove webtorrent files
...
Only remove them on max quality HLS playlist generation
2021-01-25 14:38:52 +01:00
77d7e851dc
Add priority to transcoding jobs
...
(1 = highest priority)
100 for new resolutions
10 for original file optimization
Add a malus for transcoding jobs depending on how many uploads the user did in the
last 7 days
2021-01-25 14:38:52 +01:00
24516aa26a
Refactor transcoding job handlers
2021-01-21 15:58:17 +01:00
3b01f4c0ac
Support progress for ffmpeg tasks
2021-01-21 14:42:43 +01:00
805b8619c1
Fix (again) youtube import
2021-01-19 16:36:21 +01:00
454c20fa7c
stricter youtubedl format selectors ( #3516 )
...
* stricter youtubedl format selectors
make sure selectors avoid av1, and otherwise match as closely to the
maximum resolution enabled for transcoding
* add support for merge formats in youtubedl
* avoid vp9.2 in youtubedl to avoid any HDR
* move getEnabledResolutions, safer replace of imported extension
* add test for youtube-dl selectors
2021-01-15 15:56:56 +01:00
f88453e233
Fix build
2020-12-22 16:49:17 +01:00
236841a1d7
Force HLS transcoding
...
Fix weird behaviour with some web browsers. Don't really know if it's a
ffmpeg bug, a safari bug or a peertube bug, but forcing transcoding
seems to fix this playback bug
2020-12-22 16:04:42 +01:00
402145b863
Refactor jobs state
2020-12-14 12:00:35 +01:00
040d6896a3
add display of logs matching any state
2020-12-13 19:27:25 +01:00
a800dbf345
Live views update
2020-12-09 16:31:42 +01:00
b49f22d8f9
Upgrade sequelize to v6
2020-12-09 11:41:22 +01:00
5b9b403a20
Fix replay last segments loop
2020-12-07 16:07:00 +01:00
aa5ee5017a
Fix lint
2020-12-04 15:58:29 +01:00
e772bdf14c
Load video in permanent live after last one ended
2020-12-04 15:38:32 +01:00
3851e732c4
Fix audio issues with live replay
2020-12-04 15:10:13 +01:00
bb4ba6d94c
Add permanent live support
2020-12-03 15:21:16 +01:00
4a54a93941
Fix live invalid save replay duration
2020-12-03 10:05:10 +01:00
2650d6d489
Fix live replay duration glitch
2020-12-02 10:18:15 +01:00
8c666c44ab
Fix live saving after a server restart
2020-11-30 17:03:13 +01:00
937581b8f6
Fix high CPU with long live when save replay is true
2020-11-30 15:59:22 +01:00
daf6e48010
Split ffmpeg utils with ffprobe utils
2020-11-25 10:07:51 +01:00
de94ac86a2
Fix incorrect IDs in AP federation
2020-11-20 11:21:08 +01:00
e4bf785617
Handle views for live videos
2020-11-09 15:33:04 +01:00
9e2b2e76ba
Don't add live videos in redundancy
2020-11-09 15:33:04 +01:00
053aed43fb
Regenerate miniature on live save
2020-11-09 15:33:04 +01:00
3bc68dfd61
Fix audio sync after saving replay
...
hls.js seems to not correctly handle audio gaps with fragmented mp4
(but can with a ts playlist)
2020-11-09 15:33:04 +01:00
68e70a745b
Add save replay live tests
2020-11-09 15:33:04 +01:00
97969c4edf
Add check constraints live tests
2020-11-09 15:33:04 +01:00
d846d99c6c
Add modal to display live information
2020-11-09 15:33:04 +01:00
31c82cd914
Fix replay saving
2020-11-09 15:33:04 +01:00
b5b687550d
Add ability to save live replay
2020-11-09 15:33:04 +01:00
fb7194043d
Check live duration and size
2020-11-09 15:33:04 +01:00
a5cf76afa3
Add watch messages if live has not started
2020-11-09 15:33:04 +01:00
c6c0fa6cd8
Live streaming implementation first step
2020-11-09 15:33:04 +01:00
dca0fe12ec
Fix resolution for portrait videos
2020-08-03 16:03:52 +02:00
26d6bf6533
Split types and typings
2020-06-18 10:46:27 +02:00
fd261a8de9
Fix video view filename
2020-06-15 15:18:54 +02:00
2158ac9034
Add server plugin filter hooks for import with torrent and url ( #2621 )
...
* Add server plugin filter hooks for import with torrent and url
* WIP: pre and post-import filter hooks
* Rebased
* Cleanup filters to accept imports
Co-authored-by: Chocobozzz <me@florianbigard.com>
2020-05-14 11:10:26 +02:00
80fdaf0645
Add moderation helpers to plugins
2020-05-07 15:07:01 +02:00
1c32067367
Fix HLS audio only transcoding
2020-05-05 16:27:46 +02:00
e307e4fce3
Add ability for auth plugins to hook tokens validity
2020-05-04 16:21:39 +02:00
e1c5503114
Support logout and add id and pass tests
2020-05-04 16:21:39 +02:00
8dc8a34ee8
Avoir some circular dependencies
2020-05-04 16:21:39 +02:00
b1770a0af4
Add thumbnail / preview generation from url on the fly ( #2646 )
...
* Add thumbnails generation on the fly to URL import
* Display generated preview to import first edit
* Use ternary to get type inference
* Move preview/thumbnail test just after import
Co-authored-by: kimsible <kimsible@users.noreply.github.com>
2020-04-20 10:28:38 +02:00
0dd57e4d43
Fix tests
2020-04-20 09:04:38 +02:00
2c0ccd4b3f
Do not support subscriptions to accounts
2020-04-16 16:37:25 +02:00
d57d1d83c6
Support audio files import
2020-04-03 15:41:39 +02:00
84ebcf345c
Fix federation with pleroma
2020-03-12 14:14:11 +01:00
598edb8af1
Reduce AP context size on specific activities
2020-02-03 11:31:23 +01:00
a15871560f
Move to eslint
2020-02-03 08:31:02 +01:00
ca6d36227a
Add url field in caption and use it for thumbnails
2020-01-30 11:53:38 +01:00
0374b6b5cd
Cleanup server fixme
2020-01-28 14:45:17 +01:00
b764380ac2
Add ability to list redundancies
2020-01-28 11:35:26 +01:00
99849607a6
Remove debug messages
2020-01-24 11:38:35 +01:00
e92269053e
Update http signature
2020-01-07 15:56:09 +01:00
1061c73fde
Add ability to filter per job type
2019-12-04 14:49:59 +01:00
d7a25329f9
Add ability to disable webtorrent
...
In favour of HLS
2019-11-25 10:59:43 +01:00
32d7f2b754
fix a few typos ( #2141 )
...
* fix a few typos
* apply changes to original files instead
* additional correction
2019-09-23 08:17:42 +02:00
8424c4026a
Add auto follow back support for instances
2019-09-04 16:24:58 +02:00
0283eaac2a
Cleanup model types
2019-08-20 14:53:42 +02:00
453e83ea5d
Stronger model typings
2019-08-19 17:26:35 +02:00
6b9c966f64
Automatically remove bad followings
2019-08-06 17:26:51 +02:00
1198edf4bb
Fix user notifications on new follow
2019-08-02 10:53:36 +02:00
2284f20207
Add gitlab ci support
2019-07-29 13:40:39 +02:00
5b77537ce5
Correctly notify on auto blacklist
2019-07-24 10:58:16 +02:00
21d141c296
Merge branch 'release/v1.3.0' into develop
2019-05-21 13:32:00 +02:00
63dc589865
Fix video views
2019-05-21 13:14:27 +02:00
536598cfaf
Add audio support in upload
2019-05-16 17:16:47 +02:00
c28bcdd10a
Fix pleroma follow
2019-04-25 15:20:30 +02:00
47f6409bb8
Use apicache instead of our broken implementation
2019-04-25 11:27:13 +02:00
3acc508440
Upgrade sequelize
2019-04-24 16:26:21 +02:00
e8bafea35b
Create a dedicated table to track video thumbnails
2019-04-24 16:25:52 +02:00
74dc3bca2b
Don't expose constants directly in initializers/
2019-04-11 14:26:41 +02:00
6dd9de95df
Move config in its own file
2019-04-11 13:45:39 +02:00
883993c81e
Add notification on new instance follower (server side)
2019-04-08 17:30:48 +02:00
7ccddd7b52
add quarantine videos feature ( #1637 )
...
* add quarantine videos feature
* increase Notification settings test timeout
to 20000ms. was completing 7000 locally but timing out
after 10000 on travis
* fix quarantine video test issues
-propagate misspelling
-remove skip from server/tests/client.ts
* WIP use blacklist for moderator video approval
instead of video.quarantine boolean
* finish auto-blacklist feature
2019-04-02 11:26:47 +02:00
308421283a
Move video file import in its own file
2019-03-19 17:10:53 +01:00
a0327eedb0
Rename video-file job to video-transcoding
2019-03-19 17:00:08 +01:00
2ba9287131
Cleanup invalid rates/comments/shares
2019-03-19 16:23:02 +01:00
d74d29ad9e
Limit user tokens cache
2019-03-19 14:30:43 +01:00
9f79ade627
Refresh playlists
2019-03-19 14:13:53 +01:00
418d092afa
Playlist server API
2019-03-18 11:17:59 +01:00
374c1db98c
Upgrade server dependencies
2019-02-21 14:22:39 +01:00
dee77e767f
Create send email type
...
Instead of using email job handler payload type
2019-02-20 15:54:32 +01:00
4759fedc61
Fix from header in contact form
2019-02-15 09:43:38 +01:00
597a9266d4
Add player mode in watch/embed urls
2019-02-11 09:13:02 +01:00
0920929696
Add hls support on server
2019-02-11 09:13:02 +01:00
f7effe8dc7
don't notify prior to scheduled update
...
also increase timeouts on user-notification test
2019-02-07 14:49:28 +01:00
744d0eca19
Refresh remote actors on GET enpoints
2019-01-14 11:30:15 +01:00
a4101923e6
Implement contact form on server side
2019-01-10 11:32:37 +01:00
f7cc67b455
Add new follow, mention and user registered notifs
2019-01-09 11:15:15 +01:00
dc13348070
Add import finished and video published notifs
2019-01-09 11:15:15 +01:00
e8d246d526
Add notification settings migration
2019-01-09 11:15:15 +01:00
cef534ed53
Add user notification base code
2019-01-09 11:15:15 +01:00
2f5c6b2fc6
Optimize actor follow scores modifications
2018-12-20 14:31:11 +01:00
56b13bd193
Fix federation of some videos
...
If we don't transcode additional resolutions, and user decided to wait
transcoding before publishing the video
2018-12-19 13:41:47 +01:00
f481c4f9f3
Use move instead rename
...
To avoid EXDEV errors
2018-12-11 15:12:38 +01:00
9f8ca79284
Don't quit on queue error
2018-12-11 09:18:01 +01:00
6040f87d14
Add tmp and redundancy directories
2018-12-04 16:04:15 +01:00
745778256c
Fix thumbnail processing
2018-12-04 16:04:14 +01:00
Chocobozzz
Jelle Besseling
Chocobozzz
Rigel Kent
Kim
Lukas Winkler
Josh Morel