From ed75f06fb2c6c1397886a2618f7808e652f5541d Mon Sep 17 00:00:00 2001 From: Paul Schneider Date: Sat, 24 Oct 2026 23:09:37 +0100 Subject: [PATCH] fix(android): correct camera frame rotation and recorder timestamps Rotate camera frames before encoding while accounting for Camera2 texture transforms. Align camera timestamps with MediaRecorder and cover capture duration, tracks and rotation with regression tests. Include LibVLC ProGuard rules and update documentation and changelog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 15 +- INSTALL.md | 9 + doc/testing.md | 53 +++ .../PostIt.Android/PostIt.Android.csproj | 1 + .../Services/AndroidCameraFrameRotator.cs | 308 ++++++++++++++++++ .../Services/AndroidPodcastCapture.cs | 49 ++- .../Services/CameraFrameClock.cs | 24 ++ .../Services/CameraFrameOrientation.cs | 55 ++++ src/PostIt/PostIt.Android/proguard-rules.pro | 9 + .../AndroidCameraFrameRotationTestActivity.cs | 147 +++++++++ .../AndroidPodcastCaptureTestActivity.cs | 27 +- .../PostIt.Tests/CameraFrameClockTests.cs | 37 +++ .../CameraFrameOrientationTests.cs | 61 ++++ src/PostIt/PostIt.Tests/PostIt.Tests.csproj | 5 +- 14 files changed, 792 insertions(+), 8 deletions(-) create mode 100644 src/PostIt/PostIt.Android/Services/AndroidCameraFrameRotator.cs create mode 100644 src/PostIt/PostIt.Android/Services/CameraFrameClock.cs create mode 100644 src/PostIt/PostIt.Android/Services/CameraFrameOrientation.cs create mode 100644 src/PostIt/PostIt.Android/proguard-rules.pro create mode 100644 src/PostIt/PostIt.Tests/AndroidCameraFrameRotationTestActivity.cs create mode 100644 src/PostIt/PostIt.Tests/CameraFrameClockTests.cs create mode 100644 src/PostIt/PostIt.Tests/CameraFrameOrientationTests.cs diff --git a/CHANGELOG.md b/CHANGELOG.md index 950759449..224008f7e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,19 @@ # Changelog +## [1.0.8-rc33] - unstable + +### Added + +### Changed + +### Fixed + +* [PostIt.Android] Avoid applying camera sensor rotation twice: account for the rotation already included in Camera2's SurfaceTexture matrix before rotating encoder input, while retaining portrait output dimensions. +* [PostIt.Android] Fix zero-second camera captures after adding OpenGL rotation: align Camera2 timestamps with the recorder's monotonic clock, preserve frame intervals, ignore duplicate frames and start camera requests only after the encoder is ready. Add a real Camera2/microphone regression check for segment duration and H.264/AAC tracks. +* [PostIt.Android] Keep the `VideoLAN.LibVLC.Android` JNI/native bindings alive in Release builds with a ProGuard rule set so the bundled `libvlc`/`libc++_shared` libraries are not stripped. +* [PostIt.Android] Rotate camera frames before H.264 encoding using an OpenGL surface, based on sensor orientation, lens facing and display rotation at capture start. Portrait captures swap the encoded dimensions; live MPEG-TS/HLS and replays no longer depend on an ineffective MP4-only orientation hint. Restore compilation by reading display orientation from the activity rather than the capture service. + ## [1.0.8-rc32] - unstable ### Added @@ -9,8 +22,8 @@ ### Fixed -* [PostIt.Android] Upgrade `VideoLAN.LibVLC.Android` to 3.7.0-beta to resolve the runtime `failed to load libc++_shared.so` problem on 16 KiB-page Android devices and document the native-library compatibility requirement. * Performer notification audience +* [PostIt.Android] Upgrade `VideoLAN.LibVLC.Android` to 3.7.0-beta to resolve the runtime `failed to load libc++_shared.so` problem on 16 KiB-page Android devices and document the native-library compatibility requirement. ## [1.0.8-rc31] - unstable diff --git a/INSTALL.md b/INSTALL.md index 6d92d63f8..5cea2b6f0 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -166,6 +166,15 @@ at IDR frames, with timestamps used for segment durations. Closed segments enter the same ordered upload/retry/quota/replay pipeline as desktop. Devices without a supported camera/encoder report a startup error. +Camera frames are rotated through an OpenGL surface before encoding, so both +live playback and replays have upright pixels without relying on rotation +metadata (not supported by the MPEG-TS recorder). The sensor, front/rear lens +and display orientation are sampled when capture starts; portrait output swaps +width and height. Camera2 may already include sensor rotation in the +SurfaceTexture transform; only the remaining rotation is applied by OpenGL. +Keep that device orientation during a broadcast, or stop and +start a new broadcast after turning the device. + A camera/microphone foreground service and a partial wake lock maintain the broadcast when PostIt is backgrounded or the screen locks. Its ongoing notification returns to the app and offers **Arreter et publier**. Stopping diff --git a/doc/testing.md b/doc/testing.md index 8b5e8eff1..7b5de174f 100644 --- a/doc/testing.md +++ b/doc/testing.md @@ -97,6 +97,59 @@ decodage authentifie du replay. Aucun FFmpeg n'est requis par ces tests ni par l'application (le fichier d'entree peut provenir de n'importe quelle source). Les tests materiels camera/microphone Windows/Linux restent manuels. +### Rotation de la capture Android + +Le calcul capteur/ecran/camera avant et les quatre transformations de texture +sont couverts sans appareil : + +```bash +dotnet test --project src/PostIt/PostIt.Tests/PostIt.Tests.csproj \ + --filter-class '*CameraFrameOrientationTests' --filter-class '*CameraFrameClockTests' +``` + +Un test Android supplementaire verifie le chemin OpenGL et l'encodeur reels, +sans ouvrir la camera, sans microphone et sans contacter l'API : + +```bash +dotnet build src/PostIt/PostIt.Android/PostIt.Android.csproj -c Debug \ + -p:EmbedAssembliesIntoApk=true +adb install -r -t src/PostIt/PostIt.Android/bin/Debug/net10.0-android/fr.pschneider.postit-Signed.apk +adb shell am start -n fr.pschneider.postit/fr.pschneider.postit.CameraFrameRotationTestActivity +adb shell run-as fr.pschneider.postit cat files/camera-rotation-test-result.txt +``` + +Attendre la fin du test : `RUNNING` doit devenir `PASS`. Il encode quatre +quadrants colores en MPEG-TS pour 0, 90, 180 et 270 degres, decode une image et +verifie les dimensions (640x480 ou 480x640) et les couleurs aux quatre positions. +La tolerance de chaque composante RGB est de 40 pour tenir compte de H.264. +Une erreur de rendu, d'encodage, de decodage ou de comparaison produit `FAIL` +avec l'exception. Cette activite est exclue des builds Release. + +Le test synthetique ne couvre pas l'horloge Camera2 ni sa synchronisation avec +le microphone. Avec l'accord de la personne utilisant le telephone, lancer +aussi le test local reel (camera et microphone actifs, aucun envoi a l'API) : + +```bash +adb shell am start -n fr.pschneider.postit/fr.pschneider.postit.PodcastCaptureTestActivity --es microphone true +adb shell run-as fr.pschneider.postit cat files/podcast-capture-test-result.txt +``` + +Attendre `PASS` : au moins trois segments, 8 a 16 secondes capturees, +pistes H.264 et AAC presentes, et progression des segments en arriere-plan. +Ce test detecte notamment les captures de zero seconde dues au decalage entre +l'horloge Camera2 et celle de MediaRecorder. Les fichiers restent dans le +repertoire prive indique par le resultat pour verification locale. +Le resultat inclut les angles capteur/ecran et la premiere matrice +SurfaceTexture recue. Les tests unitaires couvrent notamment la matrice +reelle `[0,-1,0,0,-1,0,0,0,0,0,1,0,1,1,0,1]` : pour un capteur a 90 degres +et un ecran a 0, elle porte deja la rotation et ne doit pas recevoir 90 degres +supplementaires. Le test synthetique seul ne detecte pas cette double rotation. + +Verifier aussi manuellement les cameras avant/arriere en portrait et paysage, +le son, le direct et le replay, et la poursuite de l'emission en arriere-plan. +L'orientation est fixee au demarrage d'une emission ; tourner le telephone +pendant l'emission ne reconfigure pas le format encode. + ## Quand ne PAS écrire de test Un test qui ne détecte rien n'est pas un test. Si l'invariant diff --git a/src/PostIt/PostIt.Android/PostIt.Android.csproj b/src/PostIt/PostIt.Android/PostIt.Android.csproj index 0923e8a12..5b7f4f968 100644 --- a/src/PostIt/PostIt.Android/PostIt.Android.csproj +++ b/src/PostIt/PostIt.Android/PostIt.Android.csproj @@ -31,6 +31,7 @@ +