Compare commits

..
Author SHA1 Message Date
ed75f06fb2
fix(android): correct camera frame rotation and recorder timestamps
All checks were successful
Dotnet build and test / build (push) Successful in 9m59s
Forgejo Release / release (push) Successful in 9m41s
Rotate camera frames before encoding while accounting for Camera2 texture transforms. Align camera timestamps with MediaRecorder and cover capture duration, tracks and rotation with regression tests. Include LibVLC ProGuard rules and update documentation and changelog.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-10-24 23:09:37 +01:00
14 changed files with 792 additions and 8 deletions

View file

@ -1,6 +1,19 @@
# Changelog
## [1.0.8-rc33] - unstable
### Added
### Changed
### Fixed
* [PostIt.Android] Avoid applying camera sensor rotation twice: account for the rotation already included in Camera2's SurfaceTexture matrix before rotating encoder input, while retaining portrait output dimensions.
* [PostIt.Android] Fix zero-second camera captures after adding OpenGL rotation: align Camera2 timestamps with the recorder's monotonic clock, preserve frame intervals, ignore duplicate frames and start camera requests only after the encoder is ready. Add a real Camera2/microphone regression check for segment duration and H.264/AAC tracks.
* [PostIt.Android] Keep the `VideoLAN.LibVLC.Android` JNI/native bindings alive in Release builds with a ProGuard rule set so the bundled `libvlc`/`libc++_shared` libraries are not stripped.
* [PostIt.Android] Rotate camera frames before H.264 encoding using an OpenGL surface, based on sensor orientation, lens facing and display rotation at capture start. Portrait captures swap the encoded dimensions; live MPEG-TS/HLS and replays no longer depend on an ineffective MP4-only orientation hint. Restore compilation by reading display orientation from the activity rather than the capture service.
## [1.0.8-rc32] - unstable
### Added
@ -9,8 +22,8 @@
### Fixed
* [PostIt.Android] Upgrade `VideoLAN.LibVLC.Android` to 3.7.0-beta to resolve the runtime `failed to load libc++_shared.so` problem on 16 KiB-page Android devices and document the native-library compatibility requirement.
* Performer notification audience
* [PostIt.Android] Upgrade `VideoLAN.LibVLC.Android` to 3.7.0-beta to resolve the runtime `failed to load libc++_shared.so` problem on 16 KiB-page Android devices and document the native-library compatibility requirement.
## [1.0.8-rc31] - unstable

View file

@ -166,6 +166,15 @@ at IDR frames, with timestamps used for segment durations. Closed segments
enter the same ordered upload/retry/quota/replay pipeline as desktop.
Devices without a supported camera/encoder report a startup error.
Camera frames are rotated through an OpenGL surface before encoding, so both
live playback and replays have upright pixels without relying on rotation
metadata (not supported by the MPEG-TS recorder). The sensor, front/rear lens
and display orientation are sampled when capture starts; portrait output swaps
width and height. Camera2 may already include sensor rotation in the
SurfaceTexture transform; only the remaining rotation is applied by OpenGL.
Keep that device orientation during a broadcast, or stop and
start a new broadcast after turning the device.
A camera/microphone foreground service and a partial wake lock maintain the
broadcast when PostIt is backgrounded or the screen locks. Its ongoing
notification returns to the app and offers **Arreter et publier**. Stopping

View file

@ -97,6 +97,59 @@ decodage authentifie du replay. Aucun FFmpeg n'est requis par ces tests ni
par l'application (le fichier d'entree peut provenir de n'importe quelle source).
Les tests materiels camera/microphone Windows/Linux restent manuels.
### Rotation de la capture Android
Le calcul capteur/ecran/camera avant et les quatre transformations de texture
sont couverts sans appareil :
```bash
dotnet test --project src/PostIt/PostIt.Tests/PostIt.Tests.csproj \
--filter-class '*CameraFrameOrientationTests' --filter-class '*CameraFrameClockTests'
```
Un test Android supplementaire verifie le chemin OpenGL et l'encodeur reels,
sans ouvrir la camera, sans microphone et sans contacter l'API :
```bash
dotnet build src/PostIt/PostIt.Android/PostIt.Android.csproj -c Debug \
-p:EmbedAssembliesIntoApk=true
adb install -r -t src/PostIt/PostIt.Android/bin/Debug/net10.0-android/fr.pschneider.postit-Signed.apk
adb shell am start -n fr.pschneider.postit/fr.pschneider.postit.CameraFrameRotationTestActivity
adb shell run-as fr.pschneider.postit cat files/camera-rotation-test-result.txt
```
Attendre la fin du test : `RUNNING` doit devenir `PASS`. Il encode quatre
quadrants colores en MPEG-TS pour 0, 90, 180 et 270 degres, decode une image et
verifie les dimensions (640x480 ou 480x640) et les couleurs aux quatre positions.
La tolerance de chaque composante RGB est de 40 pour tenir compte de H.264.
Une erreur de rendu, d'encodage, de decodage ou de comparaison produit `FAIL`
avec l'exception. Cette activite est exclue des builds Release.
Le test synthetique ne couvre pas l'horloge Camera2 ni sa synchronisation avec
le microphone. Avec l'accord de la personne utilisant le telephone, lancer
aussi le test local reel (camera et microphone actifs, aucun envoi a l'API) :
```bash
adb shell am start -n fr.pschneider.postit/fr.pschneider.postit.PodcastCaptureTestActivity --es microphone true
adb shell run-as fr.pschneider.postit cat files/podcast-capture-test-result.txt
```
Attendre `PASS` : au moins trois segments, 8 a 16 secondes capturees,
pistes H.264 et AAC presentes, et progression des segments en arriere-plan.
Ce test detecte notamment les captures de zero seconde dues au decalage entre
l'horloge Camera2 et celle de MediaRecorder. Les fichiers restent dans le
repertoire prive indique par le resultat pour verification locale.
Le resultat inclut les angles capteur/ecran et la premiere matrice
SurfaceTexture recue. Les tests unitaires couvrent notamment la matrice
reelle `[0,-1,0,0,-1,0,0,0,0,0,1,0,1,1,0,1]` : pour un capteur a 90 degres
et un ecran a 0, elle porte deja la rotation et ne doit pas recevoir 90 degres
supplementaires. Le test synthetique seul ne detecte pas cette double rotation.
Verifier aussi manuellement les cameras avant/arriere en portrait et paysage,
le son, le direct et le replay, et la poursuite de l'emission en arriere-plan.
L'orientation est fixee au demarrage d'une emission ; tourner le telephone
pendant l'emission ne reconfigure pas le format encode.
## Quand ne PAS écrire de test
Un test qui ne détecte rien n'est pas un test. Si l'invariant

View file

@ -31,6 +31,7 @@
</ItemGroup>
<ItemGroup Condition="'$(Configuration)' == 'Debug'">
<Compile Include="..\PostIt.Tests\AndroidPodcastCaptureTestActivity.cs" Link="Tests\AndroidPodcastCaptureTestActivity.cs" />
<Compile Include="..\PostIt.Tests\AndroidCameraFrameRotationTestActivity.cs" Link="Tests\AndroidCameraFrameRotationTestActivity.cs" />
</ItemGroup>
<!-- This is an Avalonia app, not a MAUI app: exclude the .NET MAUI VS Code
extension's injected HotReload agent (references Microsoft.Maui.*).

View file

@ -0,0 +1,308 @@
using System;
using System.Threading.Tasks;
using Android.Graphics;
using Android.Opengl;
using Android.OS;
using Android.Views;
using Java.Nio;
namespace PostIt.Android.Services;
internal sealed class AndroidCameraFrameRotator : IDisposable
{
private const int ExternalTexture = 0x8D65;
private const int RecordableAndroid = 0x3142;
private const string VertexShader = """
attribute vec2 position;
attribute vec2 textureCoordinate;
uniform mat4 textureMatrix;
varying vec2 uv;
void main() {
gl_Position = vec4(position, 0.0, 1.0);
uv = (textureMatrix * vec4(textureCoordinate, 0.0, 1.0)).xy;
}
""";
private const string FragmentShader = """
#extension GL_OES_EGL_image_external : require
precision mediump float;
uniform samplerExternalOES cameraTexture;
varying vec2 uv;
void main() {
gl_FragColor = texture2D(cameraTexture, uv);
}
""";
private readonly HandlerThread _thread = new("PostIt.CameraRotation");
private readonly Handler _handler;
private readonly Action<Exception> _failed;
private readonly float[] _textureMatrix = new float[16];
private readonly CameraFrameClock _clock = new();
private EGLDisplay? _display;
private EGLContext? _context;
private EGLSurface? _window;
private SurfaceTexture? _texture;
private FrameListener? _listener;
private Surface? _cameraSurface;
private FloatBuffer? _vertices;
private FloatBuffer? _coordinates;
private int _program;
private int _textureId;
private int _matrixLocation;
private bool _rendering;
private bool _disposed;
private int _rotation;
private int? _remainingRotation;
internal string? FirstTextureTransform { get; private set; }
internal Surface CameraSurface => _cameraSurface
?? throw new InvalidOperationException("Surface de rotation camera indisponible.");
private AndroidCameraFrameRotator(Action<Exception> failed)
{
_failed = failed;
_thread.Start();
_handler = new Handler(_thread.Looper!);
}
internal static async Task<AndroidCameraFrameRotator> CreateAsync(
Surface encoderSurface, int width, int height, int rotation, Action<Exception> failed)
{
var rotator = new AndroidCameraFrameRotator(failed);
try
{
await rotator.InvokeAsync(() => rotator.Initialize(encoderSurface, width, height, rotation));
return rotator;
}
catch
{
rotator.Dispose();
throw;
}
}
private void Initialize(Surface encoderSurface, int width, int height, int rotation)
{
_rotation = rotation;
_display = EGL14.EglGetDisplay(EGL14.EglDefaultDisplay);
if (_display is null || _display.Equals(EGL14.EglNoDisplay))
throw EglError("Obtention de l'affichage EGL");
if (!EGL14.EglInitialize(_display, new int[2], 0, new int[2], 0))
throw EglError("Initialisation EGL");
int[] attributes =
[
EGL14.EglRedSize, 8, EGL14.EglGreenSize, 8, EGL14.EglBlueSize, 8,
EGL14.EglRenderableType, EGL14.EglOpenglEs2Bit,
EGL14.EglSurfaceType, EGL14.EglWindowBit, RecordableAndroid, 1, EGL14.EglNone
];
var configs = new EGLConfig[1];
var count = new int[1];
if (!EGL14.EglChooseConfig(_display, attributes, 0, configs, 0, 1, count, 0) || count[0] == 0)
throw EglError("Configuration EGL pour l'encodeur");
_context = EGL14.EglCreateContext(_display, configs[0], EGL14.EglNoContext,
[EGL14.EglContextClientVersion, 2, EGL14.EglNone], 0);
if (_context is null || _context.Equals(EGL14.EglNoContext))
throw EglError("Creation du contexte EGL");
_window = EGL14.EglCreateWindowSurface(_display, configs[0], encoderSurface, [EGL14.EglNone], 0);
if (_window is null || _window.Equals(EGL14.EglNoSurface))
throw EglError("Creation de la surface EGL");
if (!EGL14.EglMakeCurrent(_display, _window, _window, _context))
throw EglError("Activation du contexte EGL");
_program = CreateProgram();
var ids = new int[1];
GLES20.GlGenTextures(1, ids, 0);
_textureId = ids[0];
GLES20.GlBindTexture(ExternalTexture, _textureId);
GLES20.GlTexParameteri(ExternalTexture, GLES20.GlTextureMinFilter, GLES20.GlLinear);
GLES20.GlTexParameteri(ExternalTexture, GLES20.GlTextureMagFilter, GLES20.GlLinear);
GLES20.GlTexParameteri(ExternalTexture, GLES20.GlTextureWrapS, GLES20.GlClampToEdge);
GLES20.GlTexParameteri(ExternalTexture, GLES20.GlTextureWrapT, GLES20.GlClampToEdge);
_vertices = CreateBuffer([-1, -1, 1, -1, -1, 1, 1, 1]);
_coordinates = CreateBuffer(CameraFrameOrientation.GetTextureCoordinates(0));
GLES20.GlUseProgram(_program);
BindAttribute("position", _vertices);
BindAttribute("textureCoordinate", _coordinates);
_matrixLocation = GLES20.GlGetUniformLocation(_program, "textureMatrix");
GLES20.GlUniform1i(GLES20.GlGetUniformLocation(_program, "cameraTexture"), 0);
GLES20.GlViewport(0, 0, rotation is 90 or 270 ? height : width, rotation is 90 or 270 ? width : height);
CheckGl();
_texture = new SurfaceTexture(_textureId);
_texture.SetDefaultBufferSize(width, height);
_listener = new FrameListener(this);
_texture.SetOnFrameAvailableListener(_listener, _handler);
_cameraSurface = new Surface(_texture);
}
internal Task StartAsync() => InvokeAsync(() => _rendering = true);
private void DrawFrame(SurfaceTexture texture)
{
if (!_rendering) return;
try
{
texture.UpdateTexImage();
var presentationTime = _clock.GetPresentationTime(texture.Timestamp, Java.Lang.JavaSystem.NanoTime());
if (presentationTime is null) return;
texture.GetTransformMatrix(_textureMatrix);
var remainingRotation = CameraFrameOrientation.GetRemainingRotation(_rotation, _textureMatrix);
if (_remainingRotation != remainingRotation)
{
_remainingRotation = remainingRotation;
FirstTextureTransform = string.Join(", ", _textureMatrix);
_coordinates!.Position(0);
_coordinates.Put(CameraFrameOrientation.GetTextureCoordinates(remainingRotation));
_coordinates.Position(0);
global::Android.Util.Log.Info("PostIt.Podcast",
$"Camera texture transform: [{FirstTextureTransform}], remaining rotation={remainingRotation}");
}
GLES20.GlUniformMatrix4fv(_matrixLocation, 1, false, _textureMatrix, 0);
GLES20.GlDrawArrays(GLES20.GlTriangleStrip, 0, 4);
CheckGl();
if (!EGLExt.EglPresentationTimeANDROID(_display, _window, presentationTime.Value))
throw EglError("Horodatage de l'image camera");
if (!EGL14.EglSwapBuffers(_display, _window))
throw EglError("Envoi de l'image a l'encodeur");
}
catch (Exception ex)
{
_rendering = false;
_failed(new InvalidOperationException("Rotation de la capture Android impossible.", ex));
}
}
private Task InvokeAsync(Action action)
{
var completion = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
if (!_handler.Post(() =>
{
try { action(); completion.SetResult(); }
catch (Exception ex) { completion.SetException(ex); }
}))
completion.SetException(new InvalidOperationException("Le thread de rotation camera est arrete."));
return completion.Task;
}
private static FloatBuffer CreateBuffer(float[] values)
{
using var bytes = ByteBuffer.AllocateDirect(values.Length * sizeof(float))
?? throw new InvalidOperationException("Allocation du tampon OpenGL impossible.");
bytes.Order(ByteOrder.NativeOrder()!);
var buffer = bytes.AsFloatBuffer()
?? throw new InvalidOperationException("Creation du tampon OpenGL impossible.");
buffer.Put(values);
buffer.Position(0);
return buffer;
}
private void BindAttribute(string name, FloatBuffer buffer)
{
var location = GLES20.GlGetAttribLocation(_program, name);
GLES20.GlEnableVertexAttribArray(location);
GLES20.GlVertexAttribPointer(location, 2, GLES20.GlFloat, false, 0, buffer);
}
private static int CreateShader(int type, string source)
{
var shader = GLES20.GlCreateShader(type);
GLES20.GlShaderSource(shader, source);
GLES20.GlCompileShader(shader);
var status = new int[1];
GLES20.GlGetShaderiv(shader, GLES20.GlCompileStatus, status, 0);
if (status[0] != 0) return shader;
var error = GLES20.GlGetShaderInfoLog(shader);
GLES20.GlDeleteShader(shader);
throw new InvalidOperationException($"Compilation du shader camera impossible : {error}");
}
private static int CreateProgram()
{
var vertex = CreateShader(GLES20.GlVertexShader, VertexShader);
var fragment = 0;
var program = 0;
try
{
fragment = CreateShader(GLES20.GlFragmentShader, FragmentShader);
program = GLES20.GlCreateProgram();
GLES20.GlAttachShader(program, vertex);
GLES20.GlAttachShader(program, fragment);
GLES20.GlLinkProgram(program);
var status = new int[1];
GLES20.GlGetProgramiv(program, GLES20.GlLinkStatus, status, 0);
if (status[0] == 0)
throw new InvalidOperationException($"Edition du programme camera impossible : {GLES20.GlGetProgramInfoLog(program)}");
return program;
}
catch
{
if (program != 0) GLES20.GlDeleteProgram(program);
throw;
}
finally
{
GLES20.GlDeleteShader(vertex);
if (fragment != 0) GLES20.GlDeleteShader(fragment);
}
}
private static void CheckGl()
{
var error = GLES20.GlGetError();
if (error != GLES20.GlNoError)
throw new InvalidOperationException($"Erreur OpenGL de capture : 0x{error:X}.");
}
private static InvalidOperationException EglError(string operation)
=> new($"{operation} impossible : EGL 0x{EGL14.EglGetError():X}.");
public void Dispose()
{
if (_disposed) return;
_disposed = true;
try { InvokeAsync(Release).GetAwaiter().GetResult(); }
finally
{
_thread.QuitSafely();
_thread.Join();
_handler.Dispose();
_thread.Dispose();
}
}
private void Release()
{
_rendering = false;
_texture?.SetOnFrameAvailableListener(null);
_cameraSurface?.Release();
_cameraSurface?.Dispose();
_texture?.Release();
_texture?.Dispose();
_listener?.Dispose();
if (_program != 0) GLES20.GlDeleteProgram(_program);
if (_textureId != 0) GLES20.GlDeleteTextures(1, [_textureId], 0);
_vertices?.Dispose();
_coordinates?.Dispose();
if (_display is not null && !_display.Equals(EGL14.EglNoDisplay))
{
EGL14.EglMakeCurrent(_display, EGL14.EglNoSurface, EGL14.EglNoSurface, EGL14.EglNoContext);
if (_window is not null && !_window.Equals(EGL14.EglNoSurface))
EGL14.EglDestroySurface(_display, _window);
if (_context is not null && !_context.Equals(EGL14.EglNoContext))
EGL14.EglDestroyContext(_display, _context);
EGL14.EglReleaseThread();
EGL14.EglTerminate(_display);
}
_window?.Dispose();
_context?.Dispose();
_display?.Dispose();
}
private sealed class FrameListener(AndroidCameraFrameRotator owner)
: Java.Lang.Object, SurfaceTexture.IOnFrameAvailableListener
{
public void OnFrameAvailable(SurfaceTexture? texture)
{
if (texture is not null) owner.DrawFrame(texture);
}
}
}

View file

@ -26,6 +26,7 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
private CameraDevice? _camera;
private CameraCaptureSession? _session;
private MediaRecorder? _recorder;
private AndroidCameraFrameRotator? _rotator;
private Surface? _surface;
private ParcelFileDescriptor? _readPipe;
private ParcelFileDescriptor? _writePipe;
@ -36,6 +37,8 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
private bool _started;
private bool _disposed;
public Task Completion => _completion.Task;
internal string OrientationDiagnostics { get; private set; } = "";
internal string? TextureTransform => _rotator?.FirstTextureTransform;
private AndroidPodcastCapture(PodcastCaptureService service) => _service = service;
@ -103,6 +106,30 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
var size = sizes.Where(s => s.Width <= 640 && s.Height <= 480)
.OrderByDescending(s => s.Width * s.Height).FirstOrDefault()
?? throw new InvalidOperationException("La camera ne propose aucun format jusqu'a 480p.");
using var sensor = characteristics.Get(CameraCharacteristics.SensorOrientation) as Java.Lang.Integer
?? throw new InvalidOperationException("Orientation du capteur camera indisponible.");
using var facing = characteristics.Get(CameraCharacteristics.LensFacing) as Java.Lang.Integer
?? throw new InvalidOperationException("Position du capteur camera indisponible.");
var activity = MainActivity.Current ?? throw new InvalidOperationException("Activite Android indisponible.");
#pragma warning disable CS0618, CA1422
var display = OperatingSystem.IsAndroidVersionAtLeast(30)
? activity.Display : activity.WindowManager?.DefaultDisplay;
#pragma warning restore CS0618, CA1422
var displayRotation = display?.Rotation
?? throw new InvalidOperationException("Orientation de l'ecran Android indisponible.");
var displayDegrees = displayRotation switch
{
SurfaceOrientation.Rotation0 => 0,
SurfaceOrientation.Rotation90 => 90,
SurfaceOrientation.Rotation180 => 180,
SurfaceOrientation.Rotation270 => 270,
_ => throw new InvalidOperationException($"Rotation d'ecran Android inconnue : {displayRotation}.")
};
var rotation = CameraFrameOrientation.GetRotation(sensor.IntValue(), displayDegrees,
facing.IntValue() == (int)LensFacing.Front);
global::Android.Util.Log.Info("PostIt.Podcast",
$"Camera orientation: sensor={sensor.IntValue()}, display={displayDegrees}, facing={facing.IntValue()}, rotation={rotation}");
OrientationDiagnostics = $"sensor={sensor.IntValue()}, display={displayDegrees}, facing={facing.IntValue()}, rotation={rotation}";
#pragma warning disable CS0618
_recorder = OperatingSystem.IsAndroidVersionAtLeast(31)
? new MediaRecorder(_service) : new MediaRecorder();
@ -111,7 +138,8 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
if (microphone) _recorder.SetAudioSource(AudioSource.Mic);
_recorder.SetVideoSource(VideoSource.Surface);
_recorder.SetOutputFormat(OutputFormat.Mpeg2Ts);
_recorder.SetVideoSize(size.Width, size.Height);
_recorder.SetVideoSize(rotation is 90 or 270 ? size.Height : size.Width,
rotation is 90 or 270 ? size.Width : size.Height);
_recorder.SetVideoFrameRate(30);
_recorder.SetVideoEncodingBitRate(1200000);
_recorder.SetVideoEncoder(VideoEncoder.H264);
@ -131,20 +159,23 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
ct.ThrowIfCancellationRequested();
if (_stopRequested) throw new System.OperationCanceledException("Demarrage de la capture annule.");
_surface = _recorder.Surface ?? throw new InvalidOperationException("Surface encodeur indisponible.");
_rotator = await AndroidCameraFrameRotator.CreateAsync(_surface, size.Width, size.Height, rotation, Fail);
ct.ThrowIfCancellationRequested();
if (_stopRequested) throw new System.OperationCanceledException("Demarrage de la capture annule.");
var cameraSurface = _rotator.CameraSurface;
var opened = new TaskCompletionSource<CameraDevice>(TaskCreationOptions.RunContinuationsAsynchronously);
manager.OpenCamera(cameraId, new DeviceCallback(this, opened), _handler);
_camera = await opened.Task.WaitAsync(TimeSpan.FromSeconds(10), ct);
var configured = new TaskCompletionSource<CameraCaptureSession>(TaskCreationOptions.RunContinuationsAsynchronously);
#pragma warning disable CS0618, CA1422
_camera.CreateCaptureSession(new List<Surface> { _surface }, new SessionCallback(this, configured), _handler);
_camera.CreateCaptureSession(new List<Surface> { cameraSurface }, new SessionCallback(this, configured), _handler);
#pragma warning restore CS0618, CA1422
_session = await configured.Task.WaitAsync(TimeSpan.FromSeconds(10), ct);
ct.ThrowIfCancellationRequested();
if (_stopRequested) throw new System.OperationCanceledException("Demarrage de la capture annule.");
using var request = _camera.CreateCaptureRequest(CameraTemplate.Record);
request.AddTarget(_surface);
request.AddTarget(cameraSurface);
using var built = request.Build();
_session.SetRepeatingRequest(built, null, _handler);
var handle = new SafeFileHandle((IntPtr)_readPipe.Fd, ownsHandle: false);
var stream = new FileStream(handle, FileAccess.Read);
_pump = Task.Run(async () =>
@ -169,6 +200,8 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
});
await Task.Run(_recorder.Start, ct);
_started = true;
await _rotator.StartAsync();
_session.SetRepeatingRequest(built, null, _handler);
_writePipe.Close();
_writePipe.Dispose();
_writePipe = null;
@ -261,7 +294,11 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
private void StopNative()
{
try { if (_started) _recorder?.Stop(); }
try
{
// Keep frames flowing until MediaRecorder has drained its encoder.
if (_started) _recorder?.Stop();
}
catch (Java.Lang.RuntimeException ex)
{
_failure = ex;
@ -276,6 +313,8 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
_camera?.Close();
_camera?.Dispose();
_camera = null;
_rotator?.Dispose();
_rotator = null;
if (_recorder is not null)
{
_recorder.Error -= OnRecorderError;

View file

@ -0,0 +1,24 @@
using System;
namespace PostIt.Android.Services;
internal sealed class CameraFrameClock
{
private long? _offset;
private long? _previousTimestamp;
internal long? GetPresentationTime(long cameraTimestamp, long monotonicNow)
{
if (_previousTimestamp is { } previous)
{
if (cameraTimestamp == previous) return null;
if (cameraTimestamp < previous)
throw new InvalidOperationException("Les horodatages de la camera Android ont recule.");
}
// Camera2 may use elapsed realtime (including suspend), whereas MediaRecorder
// and its microphone use monotonic time. Preserve frame intervals, not that epoch.
_offset ??= monotonicNow - cameraTimestamp;
_previousTimestamp = cameraTimestamp;
return checked(cameraTimestamp + _offset.Value);
}
}

View file

@ -0,0 +1,55 @@
using System;
namespace PostIt.Android.Services;
internal static class CameraFrameOrientation
{
internal static int GetRotation(int sensorDegrees, int displayDegrees, bool frontFacing)
{
Validate(sensorDegrees);
Validate(displayDegrees);
return (sensorDegrees + (frontFacing ? displayDegrees : -displayDegrees) + 360) % 360;
}
internal static float[] GetTextureCoordinates(int clockwiseDegrees)
{
Validate(clockwiseDegrees);
float[] corners = [0, 0, 1, 0, 0, 1, 1, 1];
for (var i = 0; i < corners.Length; i += 2)
{
var x = corners[i];
var y = corners[i + 1];
(corners[i], corners[i + 1]) = clockwiseDegrees switch
{
90 => (1 - y, x),
180 => (1 - x, 1 - y),
270 => (y, 1 - x),
_ => (x, y)
};
}
return corners;
}
internal static int GetRemainingRotation(int clockwiseDegrees, float[] textureMatrix)
{
Validate(clockwiseDegrees);
if (textureMatrix.Length != 16)
throw new ArgumentException("A SurfaceTexture transform must contain 16 elements.", nameof(textureMatrix));
// SurfaceTexture's usual Y flip changes the sign of the first column's Y.
// Camera2 can also put the sensor rotation in this matrix; don't apply it twice.
var angle = Math.Atan2(-textureMatrix[1], textureMatrix[0]) * 180 / Math.PI;
if (!double.IsFinite(angle) || (textureMatrix[0] == 0 && textureMatrix[1] == 0))
throw new ArgumentException("Invalid SurfaceTexture orientation.", nameof(textureMatrix));
var quarterTurns = (int)Math.Round(angle / 90);
if (Math.Abs(angle - quarterTurns * 90) > 0.01)
throw new ArgumentException("SurfaceTexture orientation is not a quarter turn.", nameof(textureMatrix));
var producerRotation = (quarterTurns * 90 + 360) % 360;
return (clockwiseDegrees - producerRotation + 360) % 360;
}
private static void Validate(int degrees)
{
if (degrees is not (0 or 90 or 180 or 270))
throw new ArgumentOutOfRangeException(nameof(degrees), degrees, "Rotation must be 0, 90, 180 or 270 degrees.");
}
}

View file

@ -0,0 +1,9 @@
# Keep the LibVLC Android classes and JNI/native bindings intact in Release builds.
# R8/ProGuard otherwise can strip or rename the Java-side entry points that load
# the native libvlc and libc++_shared libraries bundled by VideoLAN.LibVLC.Android.
-keep class org.videolan.libvlc.** { *; }
-keep class org.videolan.libvlc.** { native *; }
-keep class org.videolan.medialibrary.** { *; }
-keepattributes *Annotation*
-dontwarn org.videolan.libvlc.**
-dontwarn org.videolan.medialibrary.**

View file

@ -0,0 +1,147 @@
#if ANDROID
using System;
using System.IO;
using System.Runtime.Versioning;
using System.Threading;
using System.Threading.Tasks;
using Android.App;
using Android.Graphics;
using Android.Media;
using Android.OS;
using PostIt.Android.Services;
using Path = System.IO.Path;
namespace PostIt.Tests;
/// <summary>Debug-only synthetic encoder test; never opens the camera or contacts the API.</summary>
[Activity(Name = "fr.pschneider.postit.CameraFrameRotationTestActivity", Exported = true)]
[SupportedOSPlatform("android26.0")]
public sealed class AndroidCameraFrameRotationTestActivity : Activity
{
protected override async void OnCreate(Bundle? state)
{
base.OnCreate(state);
var resultPath = Path.Combine(FilesDir!.AbsolutePath, "camera-rotation-test-result.txt");
File.WriteAllText(resultPath, "RUNNING");
try
{
foreach (var rotation in new[] { 0, 90, 180, 270 })
await VerifyRotationAsync(rotation);
File.WriteAllText(resultPath, "PASS\nrotations=0,90,180,270\nformat=MPEG-TS\nsource=synthetic");
}
catch (Exception ex)
{
File.WriteAllText(resultPath, $"FAIL\n{ex}");
}
finally { Finish(); }
}
private async Task VerifyRotationAsync(int rotation)
{
const int width = 640;
const int height = 480;
var outputWidth = rotation is 90 or 270 ? height : width;
var outputHeight = rotation is 90 or 270 ? width : height;
var path = Path.Combine(FilesDir!.AbsolutePath, $"camera-rotation-{rotation}.ts");
void Stage(string stage) => File.WriteAllText(
Path.Combine(FilesDir.AbsolutePath, "camera-rotation-test-result.txt"), $"RUNNING\nrotation={rotation}\nstage={stage}");
Stage("prepare");
#pragma warning disable CS0618
using var recorder = OperatingSystem.IsAndroidVersionAtLeast(31)
? new MediaRecorder(this) : new MediaRecorder();
#pragma warning restore CS0618
recorder.SetVideoSource(VideoSource.Surface);
recorder.SetOutputFormat(OutputFormat.Mpeg2Ts);
recorder.SetVideoSize(outputWidth, outputHeight);
recorder.SetVideoFrameRate(30);
recorder.SetVideoEncodingBitRate(1200000);
recorder.SetVideoEncoder(VideoEncoder.H264);
recorder.SetOutputFile(path);
recorder.Prepare();
using var encoderSurface = recorder.Surface
?? throw new InvalidOperationException("Encoder surface unavailable.");
Exception? failure = null;
var stopping = 0;
Stage("initialize EGL");
using (var rotator = await AndroidCameraFrameRotator.CreateAsync(
encoderSurface, width, height, rotation, error =>
{
if (Volatile.Read(ref stopping) != 0) return;
failure = error;
File.WriteAllText(Path.Combine(FilesDir.AbsolutePath, "camera-rotation-test-result.txt"), $"FAIL\n{error}");
}))
{
Stage("start recorder");
recorder.Start();
Task? stop = null;
try
{
await rotator.StartAsync();
using var paint = new Paint();
Color[] colors = [Color.Red, Color.Green, Color.Blue, Color.Yellow];
Stage("draw frames");
for (var frame = 0; frame < 90; frame++)
{
if (failure is not null) throw failure;
if (frame == 45)
{
Volatile.Write(ref stopping, 1);
stop = Task.Run(recorder.Stop);
}
if (stop?.IsCompleted == true) break;
var canvas = rotator.CameraSurface.LockCanvas(null)
?? throw new InvalidOperationException("Synthetic camera canvas unavailable.");
try
{
for (var i = 0; i < colors.Length; i++)
{
paint.Color = colors[i];
var x = i % 2 * width / 2;
var y = i / 2 * height / 2;
canvas.DrawRect(x, y, x + width / 2, y + height / 2, paint);
}
}
finally
{
rotator.CameraSurface.UnlockCanvasAndPost(canvas);
canvas.Dispose();
}
await Task.Delay(34);
}
if (failure is not null) throw failure;
Stage("wait for recorder stop");
await (stop ?? throw new InvalidOperationException("Recorder stop was not requested."))
.WaitAsync(TimeSpan.FromSeconds(10));
}
finally
{
Stage("dispose EGL");
rotator.Dispose();
}
}
Stage("decode");
using var retriever = new MediaMetadataRetriever();
retriever.SetDataSource(path);
using var bitmap = retriever.GetFrameAtTime(0, Option.ClosestSync)
?? throw new InvalidOperationException($"Cannot decode rotation {rotation}.");
if (bitmap.Width != outputWidth || bitmap.Height != outputHeight)
throw new InvalidOperationException($"Rotation {rotation}: dimensions {bitmap.Width}x{bitmap.Height}, expected {outputWidth}x{outputHeight}.");
Color[] expected = rotation switch
{
90 => [Color.Blue, Color.Red, Color.Yellow, Color.Green],
180 => [Color.Yellow, Color.Blue, Color.Green, Color.Red],
270 => [Color.Green, Color.Yellow, Color.Red, Color.Blue],
_ => [Color.Red, Color.Green, Color.Blue, Color.Yellow]
};
for (var i = 0; i < expected.Length; i++)
{
var actual = new Color(bitmap.GetPixel((i % 2 * 2 + 1) * outputWidth / 4,
(i / 2 * 2 + 1) * outputHeight / 4));
var target = expected[i];
if (Math.Abs(actual.R - target.R) > 40 || Math.Abs(actual.G - target.G) > 40 || Math.Abs(actual.B - target.B) > 40)
throw new InvalidOperationException($"Rotation {rotation}, quadrant {i}: {actual}, expected {target}.");
}
File.Delete(path);
}
}
#endif

View file

@ -1,11 +1,13 @@
#if ANDROID
using System;
using System.IO;
using System.Linq;
using System.Runtime.Versioning;
using System.Threading;
using System.Threading.Tasks;
using Android.App;
using Android.Content;
using Android.Media;
using Android.OS;
using Avalonia.Threading;
using PostIt.Android;
@ -34,7 +36,7 @@ public sealed class AndroidPodcastCaptureTestActivity : Activity
try
{
await RunAsync(result);
File.WriteAllText(resultPath, $"PASS\nsegments={result.GetInt("segments")}\ndirectory={result.GetString("directory")}\nbackground={result.GetString("background")}");
File.WriteAllText(resultPath, $"PASS\nsegments={result.GetInt("segments")}\ndurationSeconds={result.GetDouble("durationSeconds")}\naudio={result.GetBoolean("audio")}\ndirectory={result.GetString("directory")}\nbackground={result.GetString("background")}\norientation={result.GetString("orientation")}\ntextureTransform={result.GetString("textureTransform")}");
}
catch (Exception ex)
{
@ -68,6 +70,11 @@ public sealed class AndroidPodcastCaptureTestActivity : Activity
await Task.Delay(9000);
var after = SegmentCount(directory);
if (capture.Completion.IsFaulted) await capture.Completion;
if (capture is AndroidPodcastCapture androidCapture)
{
result.PutString("orientation", androidCapture.OrientationDiagnostics);
result.PutString("textureTransform", androidCapture.TextureTransform);
}
if (after <= before) throw new InvalidOperationException($"Capture did not continue in background: {before} -> {after} segments.");
if (_notificationStop)
{
@ -81,7 +88,25 @@ public sealed class AndroidPodcastCaptureTestActivity : Activity
await capture.Completion.WaitAsync(TimeSpan.FromSeconds(15));
var segments = HlsCapturePlaylist.Parse(await File.ReadAllTextAsync(Path.Combine(directory, "index.m3u8")));
if (segments.Count < 3) throw new InvalidOperationException("Insufficient capture segments.");
var duration = segments.Sum(segment => segment.DurationSeconds);
if (duration < 8 || duration > 16)
throw new InvalidOperationException($"Expected 8-16 seconds of capture, got {duration}.");
using var extractor = new MediaExtractor();
extractor.SetDataSource(Path.Combine(directory, segments[0].FileName));
var video = false;
var audio = false;
for (var track = 0; track < extractor.TrackCount; track++)
{
using var format = extractor.GetTrackFormat(track);
var mime = format.GetString(MediaFormat.KeyMime);
video |= mime == "video/avc";
audio |= mime == "audio/mp4a-latm";
}
if (!video || (_microphone && !audio))
throw new InvalidOperationException($"Missing capture tracks: H.264={video}, AAC={audio}.");
result.PutInt("segments", segments.Count);
result.PutDouble("durationSeconds", duration);
result.PutBoolean("audio", audio);
result.PutString("directory", directory);
result.PutString("background", $"segments {before} -> {after}");
}

View file

@ -0,0 +1,37 @@
using PostIt.Android.Services;
namespace PostIt.Tests;
public sealed class CameraFrameClockTests
{
[Theory]
[InlineData(4_213_000_000_000L)]
[InlineData(0L)]
[InlineData(-1_000_000_000L)]
public void Aligns_camera_epoch_to_encoder_and_preserves_frame_intervals(long cameraOffset)
{
var clock = new CameraFrameClock();
const long now = 20_000_000_000_000L;
var camera = now + cameraOffset;
Assert.Equal(now, clock.GetPresentationTime(camera, now));
Assert.Equal(now + 33_333_333, clock.GetPresentationTime(camera + 33_333_333, now + 40_000_000));
Assert.Equal(now + 100_000_000, clock.GetPresentationTime(camera + 100_000_000, now + 110_000_000));
}
[Fact]
public void Does_not_submit_the_same_camera_image_twice()
{
var clock = new CameraFrameClock();
Assert.Equal(100, clock.GetPresentationTime(1000, 100));
Assert.Null(clock.GetPresentationTime(1000, 110));
Assert.Equal(120, clock.GetPresentationTime(1020, 130));
}
[Fact]
public void Reports_a_camera_clock_that_goes_backwards()
{
var clock = new CameraFrameClock();
clock.GetPresentationTime(1000, 100);
Assert.Throws<InvalidOperationException>(() => clock.GetPresentationTime(999, 110));
}
}

View file

@ -0,0 +1,61 @@
using PostIt.Android.Services;
namespace PostIt.Tests;
public sealed class CameraFrameOrientationTests
{
[Theory]
[InlineData(90, 0, false, 90)]
[InlineData(90, 90, false, 0)]
[InlineData(90, 180, false, 270)]
[InlineData(90, 270, false, 180)]
[InlineData(270, 0, true, 270)]
[InlineData(270, 90, true, 0)]
[InlineData(270, 180, true, 90)]
[InlineData(270, 270, true, 180)]
[InlineData(0, 90, false, 270)]
[InlineData(180, 270, true, 90)]
public void Uses_sensor_display_and_lens_facing(int sensor, int display, bool front, int expected)
=> Assert.Equal(expected, CameraFrameOrientation.GetRotation(sensor, display, front));
[Theory]
[InlineData(0, new float[] { 0, 0, 1, 0, 0, 1, 1, 1 })]
[InlineData(90, new float[] { 1, 0, 1, 1, 0, 0, 0, 1 })]
[InlineData(180, new float[] { 1, 1, 0, 1, 1, 0, 0, 0 })]
[InlineData(270, new float[] { 0, 1, 0, 0, 1, 1, 1, 0 })]
public void Maps_output_corners_to_input_for_clockwise_rotation(int rotation, float[] expected)
=> Assert.Equal(expected, CameraFrameOrientation.GetTextureCoordinates(rotation));
[Theory]
[InlineData(90, 0, -1, 0)]
[InlineData(270, 0, 1, 0)]
[InlineData(180, -1, 0, 0)]
[InlineData(90, 1, 0, 90)]
[InlineData(270, 1, 0, 270)]
[InlineData(0, 0, -1, 270)]
[InlineData(180, 0, -1, 90)]
public void Does_not_repeat_rotation_already_in_surface_texture(
int desired, float x, float y, int expected)
{
float[] matrix = [x, y, 0, 0, y, -x, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1];
Assert.Equal(expected, CameraFrameOrientation.GetRemainingRotation(desired, matrix));
}
[Fact]
public void Recognizes_the_measured_rear_camera_transform_even_with_cropping()
{
float[] matrix = [0, -0.9f, 0, 0, -0.8f, 0, 0, 0, 0, 0, 1, 0, 0.9f, 0.95f, 0, 1];
Assert.Equal(0, CameraFrameOrientation.GetRemainingRotation(90, matrix));
}
[Theory]
[InlineData(-90)]
[InlineData(45)]
[InlineData(360)]
public void Rejects_invalid_angles(int rotation)
{
Assert.Throws<ArgumentOutOfRangeException>(() => CameraFrameOrientation.GetRotation(rotation, 0, false));
Assert.Throws<ArgumentOutOfRangeException>(() => CameraFrameOrientation.GetRotation(90, rotation, false));
Assert.Throws<ArgumentOutOfRangeException>(() => CameraFrameOrientation.GetTextureCoordinates(rotation));
}
}

View file

@ -25,5 +25,8 @@
<ItemGroup>
<Using Include="Xunit" />
</ItemGroup>
<ItemGroup></ItemGroup>
<ItemGroup>
<Compile Include="..\PostIt.Android\Services\CameraFrameOrientation.cs" Link="Android\CameraFrameOrientation.cs" />
<Compile Include="..\PostIt.Android\Services\CameraFrameClock.cs" Link="Android\CameraFrameClock.cs" />
</ItemGroup>
</Project>