Commit Graph

291 Commits

Author SHA1 Message Date
Chocobozzz 62bf86c186
Introduce bufferStalled playback metric 2024-08-08 14:41:34 +02:00
Chocobozzz 658241d8c6
Support cover when downloading audio 2024-08-08 10:42:15 +02:00
Chocobozzz 84e9b0bfea
Fix import tests 2024-08-08 07:32:11 +02:00
Chocobozzz 56d48fc41b
Notify local users with internal video privacy 2024-08-07 09:26:07 +02:00
Chocobozzz da992cbcab
Fix tests 2024-08-07 08:17:40 +02:00
Chocobozzz 21215122a8
Better resolution label for custom video aspect 2024-08-06 16:01:38 +02:00
Chocobozzz 546265e9ae
Support fowarding query params to oembed
For now only `start` for videos and `playlistPosition` for playlists are
supported
2024-08-06 11:52:21 +02:00
Chocobozzz f00d656ba1
Fix peertube runner tests 2024-08-06 11:15:05 +02:00
Chocobozzz 0a87497a11
Merge branch 'release/6.2.0' into develop 2024-08-05 16:34:58 +02:00
Chocobozzz e7b9311e92
Don't send m3u8 containing non existing chunks 2024-08-05 16:32:55 +02:00
Chocobozzz 5e82c1932e
Correctly set tags to replays of permanent lives 2024-08-05 16:23:44 +02:00
Chocobozzz 816f346a60 Separate HLS audio and video streams
Allows:
  * The HLS player to propose an "Audio only" resolution
  * The live to output an "Audio only" resolution
  * The live to ingest and output an "Audio only" stream

 This feature is under a config for VOD videos and is enabled by default for lives

 In the future we can imagine:
  * To propose multiple audio streams for a specific video
  * To ingest an audio only VOD and just output an audio only "video"
    (the player would play the audio file and PeerTube would not
    generate additional resolutions)

This commit introduce a new way to download videos:
 * Add "/download/videos/generate/:videoId" endpoint where PeerTube can
   mux an audio only and a video only file to a mp4 container
 * The download client modal introduces a new default panel where the
   user can choose resolutions it wants to download
2024-08-05 08:31:59 +02:00
Chocobozzz 5276f236f0
Fix tracker test 2024-07-31 16:00:31 +02:00
Chocobozzz 22e9d9a1fe
Fix updating hls infohash on privacy update 2024-07-31 09:26:54 +02:00
Chocobozzz 44919a8d57
Fix hidden runner jobs tab
If remote runners feature is only enabled on transcription
2024-07-31 08:56:33 +02:00
Chocobozzz a9d08d2646
Correctly handle invalid current password 2024-07-31 08:35:24 +02:00
Chocobozzz 300676f62b
Don't replace caption on transcription 2024-07-12 16:20:26 +02:00
Chocobozzz 6379d42d64
Fix tests build 2024-07-11 16:21:57 +02:00
Chocobozzz e99e4aa93c
Run transcription after studio 2024-07-11 11:29:46 +02:00
Chocobozzz c5de5ef1d0
Fix registering unamed settings 2024-07-10 12:12:07 +02:00
Chocobozzz 07058e17c3
Better whisper-ctranslate vad filter option 2024-07-03 17:08:32 +02:00
Chocobozzz efa58350eb
Fix transcription tests 2024-07-03 16:14:54 +02:00
Chocobozzz 8ab6f23a00
Don't block video update on storyboard generation 2024-07-03 15:40:10 +02:00
Chocobozzz c289c86741
Add vad_filter to ctranslate transcriber
Helps us to correctly detect the language if there is no voice in the
first 30 seconds

Also helps to lower hallucinations
2024-07-03 15:23:26 +02:00
Chocobozzz 2d26eff129
Don't create pendingTranscription twice 2024-07-03 08:52:05 +02:00
Chocobozzz 0870a3fe51
Fix check logs params tests 2024-07-02 13:16:59 +02:00
Chocobozzz 9ee467b9cd
Better whisper logging 2024-07-02 09:01:54 +02:00
Chocobozzz 3a71086e35
Correctly handle transcription conflicts 2024-06-28 14:42:19 +02:00
Chocobozzz 0b30e58f6f
Remove verbose option from transcription
Can be specified on-demand using NODE_DEBUG=execa env variable
2024-06-28 08:51:35 +02:00
Chocobozzz b66963fe6f
Runner can choose job type 2024-06-28 08:44:59 +02:00
Chocobozzz fd4831e502
Metadata to know if the caption is auto generated 2024-06-28 08:44:59 +02:00
Chocobozzz 1bfb791e05
Integrate transcription in PeerTube 2024-06-28 08:44:58 +02:00
lutangar ef14cf4a5c
feat(transcription): groundwork
chore: fiddling around some more

chore: add ctranslate2 and timestamped

chore: add performance markers

chore: refactor test

chore: change worflow name

chore: ensure Python3

chore(duration): convert to chai/mocha syntahx

chore(transcription): add individual tests for others transcribers

chore(transcription): implement formats test of all implementations

Also compare result of other implementation to the reference implementation

chore(transcription): add more test case with other language and models size and local model

chore(test): wip ctranslate 2 adapat

chore(transcription): wip transcript file and benchmark

chore(test): clean a bit

chore(test): clean a bit

chore(test): refacto timestamed spec

chore(test): update workflow

chore(test): fix glob expansion with sh

chore(test): extract some hw info

chore(test): fix async tests

chore(benchmark): add model info

feat(transcription): allow use of a local mode in timestamped-whisper

feat(transcription): extract run and profiling info in own value object

feat(transcription): extract run concept in own class an run more bench

chore(transcription): somplify run object only a uuid is now needed and add more benchmark scenario

docs(transcription): creates own package readme

docs(transcription): add local model usage

docs(transcription): update README

fix(transcription): use fr video for better comparison

chore(transcription): make openai comparison passed

docs(timestamped): clea

chore(transcription): change transcribers transcribe method signature

Introduce whisper builtin model.

fix(transcription): activate language detection

Forbid transcript creation without a language.
Add `languageDetection` flag to an engine and some assertions.

Fix an issue in `whisper-ctranslate2` :
https://github.com/Softcatala/whisper-ctranslate2/pull/93

chore(transcription): use PeerTube time helpers instead of custom ones

Update existing time function to output an integer number of seconds and add a ms human-readable time formatter with hints of tests.

chore(transcription): use PeerTube UUID helpers

chore(transcription): enable CER evaluation

Thanks to this recent fix in Jiwer <3
https://github.com/jitsi/jiwer/issues/873

chore(jiwer): creates JiWer package

I'm not very happy with the TranscriptFileEvaluator constructor... suggestions ?

chore(JiWer): add usage in README

docs(jiwer): update JiWer readme

chore(transcription): use FunMOOC video in fixtures

chore(transcription): add proper english video fixture

chore(transcription): use os tmp directory where relevant

chore(transcription): fix jiwer cli test reference.txt

chore(transcription): move benchmark out of tests

chore(transcription): remove transcription workflow

docs(transcription): add benchmark info

fix(transcription): use ms precision in other transcribers

chore(transcription): simplify most of the tests

chore(transcription): remove slashes when building path with join

chore(transcription): make fromPath method async

chore(transcription): assert path to model is a directory for CTranslate2 transcriber

chore(transcription): ctranslate2 assertion

chore(transcription): ctranslate2 assertion

chore(transcription): add preinstall script for Python dependencies

chore(transcription): add download and unzip utils functions

chore(transcription): add download and unzip utils functions

chore(transcription): download & unzip models fixtures

chore(transcription): zip

chore(transcription): raise download file test timeout

chore(transcription): simplify download file test

chore(transcription): add transcriptions test to CI

chore(transcription): raise test preconditions timeout

chore(transcription): run preinstall scripts before running ci

chore(transcription): create dedicated tmp folder for transcriber tests

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): use short video for local model test

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): raise timeout some more

chore(transcription): setup verbosity based on NODE_ENV value
2024-06-28 08:43:40 +02:00
Chocobozzz ef0a6b267b
Add server restart test 2024-06-27 11:56:48 +02:00
Chocobozzz 985e79f61e
Fix lint and tests 2024-06-21 15:28:09 +02:00
Chocobozzz 990266f149
Owner must not be able to approve its own comment 2024-06-21 10:27:03 +02:00
Chocobozzz 0adbc73eb9
Fix HLS audio desync on some videos 2024-06-12 10:58:07 +02:00
Chocobozzz 585050821a
Also prune storyboards and actor images 2024-06-05 15:47:37 +02:00
Chocobozzz 2b189131fa
Add house-keeping script 2024-06-05 15:35:58 +02:00
Chocobozzz 9b483bcb78
Fix tests build 2024-06-05 09:12:43 +02:00
Chocobozzz 3427330611
Support update object storage urls 2024-06-05 09:01:40 +02:00
Chocobozzz 96b9748585
Support CLI move of original video file 2024-06-04 09:08:24 +02:00
Chocobozzz 54c140c800
Support object storage in prune script
Also prune original files and user exports
2024-06-03 16:37:44 +02:00
Chocobozzz 4a67994775 Use my-embed component 2024-05-30 16:31:16 +02:00
Chocobozzz 29329d6c45 Implement auto tag on comments and videos
* Comments and videos can be automatically tagged using core rules or
   watched word lists
 * These tags can be used to automatically filter videos and comments
 * Introduce a new video comment policy where comments must be approved
   first
 * Comments may have to be approved if the user auto block them using
   core rules or watched word lists
 * Implement FEP-5624 to federate reply control policies
2024-05-29 15:03:14 +02:00
Chocobozzz 4ea659d569
Handle videos with FPS < 1 2024-05-29 08:56:53 +02:00
Chocobozzz 9b5c7a8dc6
Merge branch 'release/6.1.0' into develop 2024-05-17 11:36:43 +02:00
Chocobozzz cb88f25ffa
Don't count deleted comments for stats 2024-05-17 11:15:37 +02:00
kontrollanten 1eb8fc2c06 fix(server/plugins): avoid duplicate settings
Filter settings so that the name property is unique.

closes #6356
2024-05-17 10:43:47 +02:00
Chocobozzz ad5f82495f
Add privacy remover plugin test 2024-05-16 09:41:39 +02:00
Chocobozzz 8d22b9100b
Fix import tests 2024-04-29 08:01:57 +02:00
Chocobozzz afb28272f5
Check video rights before providing AP information 2024-04-26 10:09:38 +02:00
Chocobozzz b8635c2606
Compact json-ld AP objects 2024-04-25 11:21:55 +02:00
Chocobozzz bce0f2f11b
Fix total video file size 2024-04-22 11:48:05 +02:00
Chocobozzz 6f3deaee2e
Fix import tests 2024-04-11 08:13:35 +02:00
Chocobozzz 3bfecf4890
Update runner version 2024-04-04 16:33:06 +02:00
Chocobozzz 5cb3e6a0b8
Use sessionId instead of IP to identify viewer
Breaking: YAML config `ip_view_expiration` is renamed `view_expiration`
Breaking: Views are taken into account after 10 seconds instead of 30
seconds (can be changed in YAML config)

Purpose of this commit is to get closer to other video platforms where
some platforms count views on play (mux, vimeo) or others use a very low
delay (instagram, tiktok)

We also want to improve the viewer identification, where we no longer
use the IP but the `sessionId` generated by the web browser. Multiple
viewers behind a NAT can now be able to be identified as independent
viewers (this method is also used by vimeo or mux)
2024-04-04 16:27:40 +02:00
kontrollanten 6f6abcabfb
feat(plugins): add peertubeHelpers.loadByIdOrUUIDWithFiles (#6302) 2024-04-04 08:20:09 +02:00
kontrollanten cd42491cf0
feat: add support for sub routes under /my-account (#6218)
* feat: add support for sub routes under /my-account

closes #6217

* feat(plugins/client-routes): page titles

Add support for adding custom page titles in client routes.

* fix(client/PluginPages): reload component upon URL change

* Styling

* docs(plugins): update registerClientRoute

---------

Co-authored-by: Chocobozzz <me@florianbigard.com>
2024-04-04 08:17:59 +02:00
Chocobozzz 4b18b0880d
Fix tests 2024-04-03 16:40:43 +02:00
Chocobozzz 7816fa4d48
Fix lint 2024-04-03 16:16:06 +02:00
Chocobozzz 61fec4e4ef
Better seconds to time formatting 2024-04-03 14:50:30 +02:00
Chocobozzz 9b70c8e7e8
Do not take into account empty view sections 2024-04-03 14:25:21 +02:00
kontrollanten 1b323f4f65 feat(plugins): add req.rawBody for routes
Stripe webhooks endpoints requires to read the raw request body.
https://docs.stripe.com/webhooks#verify-webhook-signatures-with-official-libraries
2024-04-02 11:06:34 +02:00
Chocobozzz 107e6e73a6
Fix import tests 2024-04-02 11:05:00 +02:00
Chocobozzz 7eb0189b73
Fix config test with parallel tests 2024-03-28 10:03:47 +01:00
Chocobozzz 81963e0c6e
Simplify config test 2024-03-28 09:20:23 +01:00
Chocobozzz 817f754c22
Remove difficult to maintain config update method 2024-03-28 09:10:36 +01:00
Chocobozzz 11521f231f
Generate small versions of banners too 2024-03-27 15:08:09 +01:00
kontrollanten aaa5acbb0c feat(plugins): add filter:api.user.me.get.result
relates to #6219
2024-03-27 13:32:53 +01:00
Chocobozzz ca889dbbb8
Ensure time to int returns an integer 2024-03-27 09:04:34 +01:00
Chocobozzz a246c44504
Support tr locale 2024-03-26 14:25:12 +01:00
Chocobozzz a159b8b517
Add ability to list and delete original file
In admin
2024-03-26 14:10:02 +01:00
Chocobozzz 058ef6912c
Remove deprecated transcoding type 2024-03-25 15:17:02 +01:00
Chocobozzz da63a030f3
Also add "allow-forms" to oembed 2024-03-19 15:29:32 +01:00
Chocobozzz 302fd93c49
Disable http metrics by default
Can lead to performance issues on prometheus side and peertube side if
many different URLs have been called on peertube side (google indexation
for example)
2024-03-19 14:52:41 +01:00
Chocobozzz c09e27d77a
Optimize transcoding profile building 2024-03-19 09:53:59 +01:00
Chocobozzz 4e98d843da
Success on update "not in processing state" error
Or the job is never "ended"
2024-03-19 09:26:40 +01:00
Chocobozzz 798d8d37ee
Use more specific name 2024-03-18 11:28:43 +01:00
soonsouth 01af40a55f chore: fix some comments
Signed-off-by: soonsouth <cuibuwei@163.com>
2024-03-18 08:12:12 +01:00
chagai95 e57c3024f4
Add config option to keep original video file (basic first version) (#6157)
* testing not removing old file and adding columb to db

* implement feature

* remove unnecessary config changes

* use only keptOriginalFileName, change keptOriginalFileName to keptOriginalFilename for consistency with with videoFile table, slight refactor with basename()

* save original video files to dedicated directory original-video-files

* begin implementing object storage (bucket) support

---------

Co-authored-by: chagai.friedlander <chagai.friedlander@fairkom.eu>
Co-authored-by: Ian <ian.kraft@hotmail.com>
Co-authored-by: Chocobozzz <me@florianbigard.com>
2024-03-15 15:47:18 +01:00
Chocobozzz 0f67adf98a
Fix live state on user archive import 2024-03-12 11:11:53 +01:00
Chocobozzz 46950fbcc8
Fix encoder after transcoding profile deletion 2024-03-12 10:44:20 +01:00
Chocobozzz 4e29a6f7ba
Improve storyboard grid algorithm
Try to reduce missing sprites at the end of the video
2024-03-12 06:29:56 +01:00
Chocobozzz 888b142e2e
Fix storyboard generation with some videos
Where `t` variable is never an integer due to our setpts formula
2024-03-11 15:40:14 +01:00
Chocobozzz 4596ec2a9f
Add ability to sort videos by file size 2024-03-08 15:15:03 +01:00
Chocobozzz 10e78bb778
Remove twitter whitelisted option
It doesn't seem to be required
2024-03-08 10:54:29 +01:00
Chocobozzz 98781f353d
Add user history in import/export 2024-02-28 16:48:59 +01:00
Chocobozzz 7be401ac76
Fix playlist elements merge on import 2024-02-28 15:55:37 +01:00
Chocobozzz f6fb7cd3e9
Ensure server actor AP object has avatars 2024-02-28 14:47:39 +01:00
Chocobozzz 4434e78c56
Fix tests 2024-02-28 10:00:01 +01:00
Chocobozzz a4b497c2a9
Add postingRestrictedToMods to channels AP objects
See https://github.com/LemmyNet/lemmy/issues/3837
2024-02-27 15:46:38 +01:00
Chocobozzz b6b1aaa56f
Add video aspect ratio in server 2024-02-27 15:24:34 +01:00
Chocobozzz 68d006b6fc
Add config to disable storing lives in s3 2024-02-26 16:14:40 +01:00
Chocobozzz 109e93c139
Generate 600x600 and 1500x1500 avatars 2024-02-26 15:00:22 +01:00
Chocobozzz fb2dc40858
Improve generated image quality 2024-02-26 11:33:17 +01:00
Chocobozzz 1abf74e464
Try to re-fix tests 2024-02-26 09:36:11 +01:00
Chocobozzz e2b2c46339
Fix tests 2024-02-23 16:29:52 +01:00
Chocobozzz 094accd873
Fix CI 2024-02-23 15:07:09 +01:00
Chocobozzz 0165786f65
Add total moderators/admins in stats 2024-02-23 15:04:18 +01:00