0305db28c9
Add support for saving video files to object storage ( #4290 )
...
* Add support for saving video files to object storage
* Add support for custom url generation on s3 stored files
Uses two config keys to support url generation that doesn't directly go
to (compatible s3). Can be used to generate urls to any cache server or
CDN.
* Upload files to s3 concurrently and delete originals afterwards
* Only publish after move to object storage is complete
* Use base url instead of url template
* Fix mistyped config field
* Add rudenmentary way to download before transcode
* Implement Chocobozzz suggestions
https://github.com/Chocobozzz/PeerTube/pull/4290#issuecomment-891670478
The remarks in question:
Try to use objectStorage prefix instead of s3 prefix for your function/variables/config names
Prefer to use a tree for the config: s3.streaming_playlists_bucket -> object_storage.streaming_playlists.bucket
Use uppercase for config: S3.STREAMING_PLAYLISTS_BUCKETINFO.bucket -> OBJECT_STORAGE.STREAMING_PLAYLISTS.BUCKET (maybe BUCKET_NAME instead of BUCKET)
I suggest to rename moveJobsRunning to pendingMovingJobs (or better, create a dedicated videoJobInfo table with a pendingMove & videoId columns so we could also use this table to track pending transcoding jobs)
https://github.com/Chocobozzz/PeerTube/pull/4290/files#diff-3e26d41ca4bda1de8e1747af70ca2af642abcc1e9e0bfb94239ff2165acfbde5R19 uses a string instead of an integer
I think we should store the origin object storage URL in fileUrl, without base_url injection. Instead, inject the base_url at "runtime" so admins can easily change this configuration without running a script to update DB URLs
* Import correct function
* Support multipart upload
* Remove import of node 15.0 module stream/promises
* Extend maximum upload job length
Using the same value as for redundancy downloading seems logical
* Use dynamic part size for really large uploads
Also adds very small part size for local testing
* Fix decreasePendingMove query
* Resolve various PR comments
* Move to object storage after optimize
* Make upload size configurable and increase default
* Prune webtorrent files that are stored in object storage
* Move files after transcoding jobs
* Fix federation
* Add video path manager
* Support move to external storage job in client
* Fix live object storage tests
Co-authored-by: Chocobozzz <me@florianbigard.com>
2021-08-17 08:26:20 +02:00
27db78400c
Fix backend channel name validator consistency
2021-08-05 13:54:35 +02:00
9e87894973
Merge branch 'release/3.3.0' into develop
2021-07-29 17:18:28 +02:00
cd162f25d2
Fix video format utils
2021-07-29 17:18:09 +02:00
9c9a236b54
Refactor options in models
2021-07-29 14:17:03 +02:00
d688602710
Refactor search query options
2021-07-29 11:54:38 +02:00
b033851fb5
Search channels against handles and not names
2021-07-29 10:27:24 +02:00
fbd67e7f38
Add ability to search by uuids/actor names
2021-07-28 16:40:21 +02:00
fa47956ecf
Filter host for channels and playlists search
2021-07-28 10:32:40 +02:00
29837f8885
Add ability to search by host in server
2021-07-27 17:10:52 +02:00
15a7eafb89
Refactor video links builders
2021-07-26 15:04:37 +02:00
764b1a14fc
Use random names for VOD HLS playlists
2021-07-26 11:29:31 +02:00
4d029ef8ec
Add ability for instances to follow any actor
2021-07-21 13:35:31 +02:00
2a408c40a9
change audio label, 0p > Audio
...
Changes the audio label in the Download modal.
2021-07-12 15:34:32 +02:00
06c2759338
Fix hooks definition
2021-07-01 16:47:14 +02:00
693c6586cb
Fix missing delete cascade video -> channel
2021-07-01 14:53:37 +02:00
c17ee0903e
Don't check enum for video model
...
Licence, language and category could be extended by a remote plugin
2021-06-29 16:21:32 +02:00
d4a8e7a65f
Support short uuid for GET video/playlist
2021-06-29 14:56:35 +02:00
37a44fc915
Add ability to search playlists
2021-06-25 14:44:01 +02:00
8ebf2a5d5d
Refactor live manager
2021-06-17 08:47:03 +02:00
eae0365b5c
Fix missing transactions
2021-06-15 10:40:06 +02:00
520bf885c5
Remove unnecessary log
2021-06-11 16:44:33 +02:00
31d5d916c3
Fix redundancy row grabbing
2021-06-11 16:31:41 +02:00
9f7657b6b6
Include transaction in video hooks
2021-06-11 16:22:54 +02:00
adddb12b04
Fix video with all files query
2021-06-11 16:05:32 +02:00
668f864f5a
Fix live files include
2021-06-11 16:02:26 +02:00
b8afe6f0cd
Fix video rights fetching
2021-06-11 15:53:52 +02:00
69322042c4
Cleanup video model scopes
2021-06-11 14:45:58 +02:00
7c3a6636fd
Move get* video methods to formatter
2021-06-11 14:36:07 +02:00
fd6a74a835
Refactor schedule update
2021-06-11 14:33:34 +02:00
20a206c3d1
Refactor include checks
2021-06-11 14:26:37 +02:00
71d4af1efc
Use raw SQL for most of video queries
2021-06-11 14:09:52 +02:00
3c79c2ce86
Optimize join build
2021-06-11 11:27:45 +02:00
17bb45388e
Optimize rows parsing
2021-06-11 11:15:44 +02:00
ca4b4b2e55
Fetch directly all video attributes for get API
2021-06-11 09:57:19 +02:00
1d43c3a613
Use separate queries for video files
2021-06-11 09:31:59 +02:00
d9bf974f5d
Use raw SQL for video get request
2021-06-10 15:26:18 +02:00
e5dbd5084e
Refactor video query builder
2021-06-10 09:22:58 +02:00
ff0ea0cd8e
Transaction is optional
2021-06-09 17:06:56 +02:00
57a0a9cde4
Optimize AP video captions update
2021-06-09 16:22:01 +02:00
28dfb44b14
Try to speed up AP update transaction
2021-06-08 17:41:48 +02:00
99cb53fd5a
Don't count deleted comment for replies
2021-06-08 13:48:58 +02:00
49af5ac8c2
Refactor AP playlists
2021-06-03 16:40:32 +02:00
9e2e51dc0b
Add tags to live logger
2021-05-28 13:05:59 +02:00
a1eda903a4
Support '/w/' and '/w/p/' for watch page
...
And use them as default in client
2021-05-28 11:38:08 +02:00
8ee37c5f38
Merge branch 'release/3.2.0' into develop
2021-05-26 13:41:15 +02:00
f49b3231d0
Fix broken lives handling
2021-05-26 12:03:44 +02:00
c4d125527a
Avoid error when file has no torrent file
2021-05-25 11:35:29 +02:00
be89e66895
Avoid error when file has no torrent file
2021-05-25 11:34:43 +02:00
16c016e8b1
Stricter models typing
2021-05-12 14:12:14 +02:00
2b02c520e6
Cleanup shared models
2021-05-11 11:32:31 +02:00
7d9ba5c089
Cleanup models directory organization
2021-05-11 11:32:31 +02:00
e024fd6a74
Update channel updatedAt when uploading a video
2021-05-10 09:38:11 +02:00
a66c2e3252
Fix remote actor creation date
2021-05-07 09:00:09 +02:00
1fd61899ea
Add ability to filter my videos by live
2021-05-03 16:42:15 +02:00
c24822a8fd
Don't save other attributes when set as refreshed
2021-04-12 13:44:58 +02:00
fe19f600da
add channel and playlist stats to server stats endpoint ( #3747 )
...
* add channel and playlist stats to nodeinfo
* add tests for active video channels stats
* fix tests for active channel stats
2021-04-12 11:19:07 +02:00
dc48fdbe68
Merge branch 'release/3.1.0' into develop
2021-04-08 15:16:01 +02:00
1afb3c474a
Fix outbox fetch with subtitled videos
2021-04-08 15:04:14 +02:00
a0eeb45f14
Update data in DB when regenerate thumbnails
2021-04-08 13:38:04 +02:00
213e30ef90
Add banner tests
2021-04-08 10:07:53 +02:00
2cb03dc1f4
Add banners support
2021-04-08 10:07:53 +02:00
f479685678
Agnostic actor image storage
2021-04-08 10:07:53 +02:00
42ec411bf8
Fix abuse tests
2021-03-09 16:10:52 +01:00
5a22296bdb
Fix broken hot algorithm
2021-03-03 13:52:58 +01:00
534556052e
Optimize views sort sql query
2021-03-03 11:24:16 +01:00
74d249bc13
Add ability to cleanup remote AP interactions
2021-03-03 10:08:08 +01:00
d61893f723
Async torrent creation
2021-02-25 15:01:07 +01:00
d7df188f23
Fix separate SQL query for video get
2021-02-25 11:18:11 +01:00
2a021e6cb6
Fix thread replies API response
2021-02-19 10:52:31 +01:00
9d6b9d10ef
Fix video comments display with deleted comments
2021-02-19 10:06:52 +01:00
8efc27bf14
Cleanup
2021-02-18 13:38:09 +01:00
f66db4d5c8
Fix feeds
2021-02-18 13:38:09 +01:00
d9a2a03196
Don't guess remote tracker URL
2021-02-18 13:38:09 +01:00
90a8bd305d
Dissociate video file names and video uuid
2021-02-18 13:38:09 +01:00
374b725df5
Optimize remote thumbnail processing
2021-02-16 10:36:44 +01:00
a35a22797c
Remove previous thumbnail if needed
2021-02-16 10:36:44 +01:00
6302d599cd
Generate a name for caption files
2021-02-16 10:36:44 +01:00
a8b1b40485
Generate a name for thumbnails
...
Allows aggressive caching
2021-02-16 10:36:44 +01:00
44d1f7f2e8
Painfully debug concurrent import jobs
2021-02-09 11:46:30 +01:00
9129b7694d
Allow to specify transcoding and import jobs concurrency
2021-02-08 15:38:45 +01:00
81b46cbc34
Optimize videos list API endpoint
2021-02-08 15:38:45 +01:00
3d4e112d16
add best trending strategy based on Reddit's best
...
inspired from https://www.reddit.com/r/changelog/comments/7spgg0/best_is_the_new_hotness/
this implementation only adds freshness, and doesn't personalize based
on subscribed communities yet.
2021-02-04 09:04:47 +01:00
ba5a8d89bb
Update server dependencies
2021-02-03 09:45:08 +01:00
1896bca09e
Support transcoding options/encoders by plugins
2021-01-28 15:55:39 +01:00
923d3d5ad5
add test and openapi for hot sort parameter
2021-01-28 15:55:34 +01:00
3da68f0a78
add default trending page choice, revert comments count for hot strategy
2021-01-28 15:55:34 +01:00
28eeb811c4
only count comments from people other than the video author
2021-01-28 15:55:34 +01:00
5bcbcbe338
modularize abstract video list header and implement video hotness recommendation variant
2021-01-28 15:55:34 +01:00
7a4994873c
Fix rss feed with HLS videos
2021-01-27 16:42:13 +01:00
6939cbac48
Safely remove webtorrent files
...
Only remove them on max quality HLS playlist generation
2021-01-25 14:38:52 +01:00
77d7e851dc
Add priority to transcoding jobs
...
(1 = highest priority)
100 for new resolutions
10 for original file optimization
Add a malus for transcoding jobs depending on how many uploads the user did in the
last 7 days
2021-01-25 14:38:52 +01:00
24516aa26a
Refactor transcoding job handlers
2021-01-21 15:58:17 +01:00
a4d2ca0715
Add user video list hooks
2021-01-20 15:28:34 +01:00
d8b34ee55b
Allow user to search through their watch history ( #3576 )
...
* allow user to search through their watch history
* add tests for search in watch history
* Update client/src/app/shared/shared-main/users/user-history.service.ts
2021-01-13 09:16:15 +01:00
1ab6024345
Fix live badge in videos list
2020-12-16 14:19:43 +01:00
b49f22d8f9
Upgrade sequelize to v6
2020-12-09 11:41:22 +01:00
a59f210ff3
Refactor a little bit client canonical URL
2020-12-08 11:15:11 +01:00
bb4ba6d94c
Add permanent live support
2020-12-03 15:21:16 +01:00
2650d6d489
Fix live replay duration glitch
2020-12-02 10:18:15 +01:00
Jelle Besseling
Chocobozzz
kontrollanten
Rigel Kent