fix(android): correct camera frame rotation and recorder timestamps
Rotate camera frames before encoding while accounting for Camera2 texture transforms. Align camera timestamps with MediaRecorder and cover capture duration, tracks and rotation with regression tests. Include LibVLC ProGuard rules and update documentation and changelog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
parent
dfef7652a4
commit
ed75f06fb2
14 changed files with 792 additions and 8 deletions
15
CHANGELOG.md
15
CHANGELOG.md
|
|
@ -1,6 +1,19 @@
|
||||||
|
|
||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## [1.0.8-rc33] - unstable
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
* [PostIt.Android] Avoid applying camera sensor rotation twice: account for the rotation already included in Camera2's SurfaceTexture matrix before rotating encoder input, while retaining portrait output dimensions.
|
||||||
|
* [PostIt.Android] Fix zero-second camera captures after adding OpenGL rotation: align Camera2 timestamps with the recorder's monotonic clock, preserve frame intervals, ignore duplicate frames and start camera requests only after the encoder is ready. Add a real Camera2/microphone regression check for segment duration and H.264/AAC tracks.
|
||||||
|
* [PostIt.Android] Keep the `VideoLAN.LibVLC.Android` JNI/native bindings alive in Release builds with a ProGuard rule set so the bundled `libvlc`/`libc++_shared` libraries are not stripped.
|
||||||
|
* [PostIt.Android] Rotate camera frames before H.264 encoding using an OpenGL surface, based on sensor orientation, lens facing and display rotation at capture start. Portrait captures swap the encoded dimensions; live MPEG-TS/HLS and replays no longer depend on an ineffective MP4-only orientation hint. Restore compilation by reading display orientation from the activity rather than the capture service.
|
||||||
|
|
||||||
## [1.0.8-rc32] - unstable
|
## [1.0.8-rc32] - unstable
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
@ -9,8 +22,8 @@
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
* [PostIt.Android] Upgrade `VideoLAN.LibVLC.Android` to 3.7.0-beta to resolve the runtime `failed to load libc++_shared.so` problem on 16 KiB-page Android devices and document the native-library compatibility requirement.
|
|
||||||
* Performer notification audience
|
* Performer notification audience
|
||||||
|
* [PostIt.Android] Upgrade `VideoLAN.LibVLC.Android` to 3.7.0-beta to resolve the runtime `failed to load libc++_shared.so` problem on 16 KiB-page Android devices and document the native-library compatibility requirement.
|
||||||
|
|
||||||
## [1.0.8-rc31] - unstable
|
## [1.0.8-rc31] - unstable
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -166,6 +166,15 @@ at IDR frames, with timestamps used for segment durations. Closed segments
|
||||||
enter the same ordered upload/retry/quota/replay pipeline as desktop.
|
enter the same ordered upload/retry/quota/replay pipeline as desktop.
|
||||||
Devices without a supported camera/encoder report a startup error.
|
Devices without a supported camera/encoder report a startup error.
|
||||||
|
|
||||||
|
Camera frames are rotated through an OpenGL surface before encoding, so both
|
||||||
|
live playback and replays have upright pixels without relying on rotation
|
||||||
|
metadata (not supported by the MPEG-TS recorder). The sensor, front/rear lens
|
||||||
|
and display orientation are sampled when capture starts; portrait output swaps
|
||||||
|
width and height. Camera2 may already include sensor rotation in the
|
||||||
|
SurfaceTexture transform; only the remaining rotation is applied by OpenGL.
|
||||||
|
Keep that device orientation during a broadcast, or stop and
|
||||||
|
start a new broadcast after turning the device.
|
||||||
|
|
||||||
A camera/microphone foreground service and a partial wake lock maintain the
|
A camera/microphone foreground service and a partial wake lock maintain the
|
||||||
broadcast when PostIt is backgrounded or the screen locks. Its ongoing
|
broadcast when PostIt is backgrounded or the screen locks. Its ongoing
|
||||||
notification returns to the app and offers **Arreter et publier**. Stopping
|
notification returns to the app and offers **Arreter et publier**. Stopping
|
||||||
|
|
|
||||||
|
|
@ -97,6 +97,59 @@ decodage authentifie du replay. Aucun FFmpeg n'est requis par ces tests ni
|
||||||
par l'application (le fichier d'entree peut provenir de n'importe quelle source).
|
par l'application (le fichier d'entree peut provenir de n'importe quelle source).
|
||||||
Les tests materiels camera/microphone Windows/Linux restent manuels.
|
Les tests materiels camera/microphone Windows/Linux restent manuels.
|
||||||
|
|
||||||
|
### Rotation de la capture Android
|
||||||
|
|
||||||
|
Le calcul capteur/ecran/camera avant et les quatre transformations de texture
|
||||||
|
sont couverts sans appareil :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dotnet test --project src/PostIt/PostIt.Tests/PostIt.Tests.csproj \
|
||||||
|
--filter-class '*CameraFrameOrientationTests' --filter-class '*CameraFrameClockTests'
|
||||||
|
```
|
||||||
|
|
||||||
|
Un test Android supplementaire verifie le chemin OpenGL et l'encodeur reels,
|
||||||
|
sans ouvrir la camera, sans microphone et sans contacter l'API :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dotnet build src/PostIt/PostIt.Android/PostIt.Android.csproj -c Debug \
|
||||||
|
-p:EmbedAssembliesIntoApk=true
|
||||||
|
adb install -r -t src/PostIt/PostIt.Android/bin/Debug/net10.0-android/fr.pschneider.postit-Signed.apk
|
||||||
|
adb shell am start -n fr.pschneider.postit/fr.pschneider.postit.CameraFrameRotationTestActivity
|
||||||
|
adb shell run-as fr.pschneider.postit cat files/camera-rotation-test-result.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Attendre la fin du test : `RUNNING` doit devenir `PASS`. Il encode quatre
|
||||||
|
quadrants colores en MPEG-TS pour 0, 90, 180 et 270 degres, decode une image et
|
||||||
|
verifie les dimensions (640x480 ou 480x640) et les couleurs aux quatre positions.
|
||||||
|
La tolerance de chaque composante RGB est de 40 pour tenir compte de H.264.
|
||||||
|
Une erreur de rendu, d'encodage, de decodage ou de comparaison produit `FAIL`
|
||||||
|
avec l'exception. Cette activite est exclue des builds Release.
|
||||||
|
|
||||||
|
Le test synthetique ne couvre pas l'horloge Camera2 ni sa synchronisation avec
|
||||||
|
le microphone. Avec l'accord de la personne utilisant le telephone, lancer
|
||||||
|
aussi le test local reel (camera et microphone actifs, aucun envoi a l'API) :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
adb shell am start -n fr.pschneider.postit/fr.pschneider.postit.PodcastCaptureTestActivity --es microphone true
|
||||||
|
adb shell run-as fr.pschneider.postit cat files/podcast-capture-test-result.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Attendre `PASS` : au moins trois segments, 8 a 16 secondes capturees,
|
||||||
|
pistes H.264 et AAC presentes, et progression des segments en arriere-plan.
|
||||||
|
Ce test detecte notamment les captures de zero seconde dues au decalage entre
|
||||||
|
l'horloge Camera2 et celle de MediaRecorder. Les fichiers restent dans le
|
||||||
|
repertoire prive indique par le resultat pour verification locale.
|
||||||
|
Le resultat inclut les angles capteur/ecran et la premiere matrice
|
||||||
|
SurfaceTexture recue. Les tests unitaires couvrent notamment la matrice
|
||||||
|
reelle `[0,-1,0,0,-1,0,0,0,0,0,1,0,1,1,0,1]` : pour un capteur a 90 degres
|
||||||
|
et un ecran a 0, elle porte deja la rotation et ne doit pas recevoir 90 degres
|
||||||
|
supplementaires. Le test synthetique seul ne detecte pas cette double rotation.
|
||||||
|
|
||||||
|
Verifier aussi manuellement les cameras avant/arriere en portrait et paysage,
|
||||||
|
le son, le direct et le replay, et la poursuite de l'emission en arriere-plan.
|
||||||
|
L'orientation est fixee au demarrage d'une emission ; tourner le telephone
|
||||||
|
pendant l'emission ne reconfigure pas le format encode.
|
||||||
|
|
||||||
## Quand ne PAS écrire de test
|
## Quand ne PAS écrire de test
|
||||||
|
|
||||||
Un test qui ne détecte rien n'est pas un test. Si l'invariant
|
Un test qui ne détecte rien n'est pas un test. Si l'invariant
|
||||||
|
|
|
||||||
|
|
@ -31,6 +31,7 @@
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
<ItemGroup Condition="'$(Configuration)' == 'Debug'">
|
<ItemGroup Condition="'$(Configuration)' == 'Debug'">
|
||||||
<Compile Include="..\PostIt.Tests\AndroidPodcastCaptureTestActivity.cs" Link="Tests\AndroidPodcastCaptureTestActivity.cs" />
|
<Compile Include="..\PostIt.Tests\AndroidPodcastCaptureTestActivity.cs" Link="Tests\AndroidPodcastCaptureTestActivity.cs" />
|
||||||
|
<Compile Include="..\PostIt.Tests\AndroidCameraFrameRotationTestActivity.cs" Link="Tests\AndroidCameraFrameRotationTestActivity.cs" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
<!-- This is an Avalonia app, not a MAUI app: exclude the .NET MAUI VS Code
|
<!-- This is an Avalonia app, not a MAUI app: exclude the .NET MAUI VS Code
|
||||||
extension's injected HotReload agent (references Microsoft.Maui.*).
|
extension's injected HotReload agent (references Microsoft.Maui.*).
|
||||||
|
|
|
||||||
308
src/PostIt/PostIt.Android/Services/AndroidCameraFrameRotator.cs
Normal file
308
src/PostIt/PostIt.Android/Services/AndroidCameraFrameRotator.cs
Normal file
|
|
@ -0,0 +1,308 @@
|
||||||
|
using System;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using Android.Graphics;
|
||||||
|
using Android.Opengl;
|
||||||
|
using Android.OS;
|
||||||
|
using Android.Views;
|
||||||
|
using Java.Nio;
|
||||||
|
|
||||||
|
namespace PostIt.Android.Services;
|
||||||
|
|
||||||
|
internal sealed class AndroidCameraFrameRotator : IDisposable
|
||||||
|
{
|
||||||
|
private const int ExternalTexture = 0x8D65;
|
||||||
|
private const int RecordableAndroid = 0x3142;
|
||||||
|
private const string VertexShader = """
|
||||||
|
attribute vec2 position;
|
||||||
|
attribute vec2 textureCoordinate;
|
||||||
|
uniform mat4 textureMatrix;
|
||||||
|
varying vec2 uv;
|
||||||
|
void main() {
|
||||||
|
gl_Position = vec4(position, 0.0, 1.0);
|
||||||
|
uv = (textureMatrix * vec4(textureCoordinate, 0.0, 1.0)).xy;
|
||||||
|
}
|
||||||
|
""";
|
||||||
|
private const string FragmentShader = """
|
||||||
|
#extension GL_OES_EGL_image_external : require
|
||||||
|
precision mediump float;
|
||||||
|
uniform samplerExternalOES cameraTexture;
|
||||||
|
varying vec2 uv;
|
||||||
|
void main() {
|
||||||
|
gl_FragColor = texture2D(cameraTexture, uv);
|
||||||
|
}
|
||||||
|
""";
|
||||||
|
|
||||||
|
private readonly HandlerThread _thread = new("PostIt.CameraRotation");
|
||||||
|
private readonly Handler _handler;
|
||||||
|
private readonly Action<Exception> _failed;
|
||||||
|
private readonly float[] _textureMatrix = new float[16];
|
||||||
|
private readonly CameraFrameClock _clock = new();
|
||||||
|
private EGLDisplay? _display;
|
||||||
|
private EGLContext? _context;
|
||||||
|
private EGLSurface? _window;
|
||||||
|
private SurfaceTexture? _texture;
|
||||||
|
private FrameListener? _listener;
|
||||||
|
private Surface? _cameraSurface;
|
||||||
|
private FloatBuffer? _vertices;
|
||||||
|
private FloatBuffer? _coordinates;
|
||||||
|
private int _program;
|
||||||
|
private int _textureId;
|
||||||
|
private int _matrixLocation;
|
||||||
|
private bool _rendering;
|
||||||
|
private bool _disposed;
|
||||||
|
private int _rotation;
|
||||||
|
private int? _remainingRotation;
|
||||||
|
internal string? FirstTextureTransform { get; private set; }
|
||||||
|
|
||||||
|
internal Surface CameraSurface => _cameraSurface
|
||||||
|
?? throw new InvalidOperationException("Surface de rotation camera indisponible.");
|
||||||
|
|
||||||
|
private AndroidCameraFrameRotator(Action<Exception> failed)
|
||||||
|
{
|
||||||
|
_failed = failed;
|
||||||
|
_thread.Start();
|
||||||
|
_handler = new Handler(_thread.Looper!);
|
||||||
|
}
|
||||||
|
|
||||||
|
internal static async Task<AndroidCameraFrameRotator> CreateAsync(
|
||||||
|
Surface encoderSurface, int width, int height, int rotation, Action<Exception> failed)
|
||||||
|
{
|
||||||
|
var rotator = new AndroidCameraFrameRotator(failed);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await rotator.InvokeAsync(() => rotator.Initialize(encoderSurface, width, height, rotation));
|
||||||
|
return rotator;
|
||||||
|
}
|
||||||
|
catch
|
||||||
|
{
|
||||||
|
rotator.Dispose();
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void Initialize(Surface encoderSurface, int width, int height, int rotation)
|
||||||
|
{
|
||||||
|
_rotation = rotation;
|
||||||
|
_display = EGL14.EglGetDisplay(EGL14.EglDefaultDisplay);
|
||||||
|
if (_display is null || _display.Equals(EGL14.EglNoDisplay))
|
||||||
|
throw EglError("Obtention de l'affichage EGL");
|
||||||
|
if (!EGL14.EglInitialize(_display, new int[2], 0, new int[2], 0))
|
||||||
|
throw EglError("Initialisation EGL");
|
||||||
|
int[] attributes =
|
||||||
|
[
|
||||||
|
EGL14.EglRedSize, 8, EGL14.EglGreenSize, 8, EGL14.EglBlueSize, 8,
|
||||||
|
EGL14.EglRenderableType, EGL14.EglOpenglEs2Bit,
|
||||||
|
EGL14.EglSurfaceType, EGL14.EglWindowBit, RecordableAndroid, 1, EGL14.EglNone
|
||||||
|
];
|
||||||
|
var configs = new EGLConfig[1];
|
||||||
|
var count = new int[1];
|
||||||
|
if (!EGL14.EglChooseConfig(_display, attributes, 0, configs, 0, 1, count, 0) || count[0] == 0)
|
||||||
|
throw EglError("Configuration EGL pour l'encodeur");
|
||||||
|
_context = EGL14.EglCreateContext(_display, configs[0], EGL14.EglNoContext,
|
||||||
|
[EGL14.EglContextClientVersion, 2, EGL14.EglNone], 0);
|
||||||
|
if (_context is null || _context.Equals(EGL14.EglNoContext))
|
||||||
|
throw EglError("Creation du contexte EGL");
|
||||||
|
_window = EGL14.EglCreateWindowSurface(_display, configs[0], encoderSurface, [EGL14.EglNone], 0);
|
||||||
|
if (_window is null || _window.Equals(EGL14.EglNoSurface))
|
||||||
|
throw EglError("Creation de la surface EGL");
|
||||||
|
if (!EGL14.EglMakeCurrent(_display, _window, _window, _context))
|
||||||
|
throw EglError("Activation du contexte EGL");
|
||||||
|
|
||||||
|
_program = CreateProgram();
|
||||||
|
var ids = new int[1];
|
||||||
|
GLES20.GlGenTextures(1, ids, 0);
|
||||||
|
_textureId = ids[0];
|
||||||
|
GLES20.GlBindTexture(ExternalTexture, _textureId);
|
||||||
|
GLES20.GlTexParameteri(ExternalTexture, GLES20.GlTextureMinFilter, GLES20.GlLinear);
|
||||||
|
GLES20.GlTexParameteri(ExternalTexture, GLES20.GlTextureMagFilter, GLES20.GlLinear);
|
||||||
|
GLES20.GlTexParameteri(ExternalTexture, GLES20.GlTextureWrapS, GLES20.GlClampToEdge);
|
||||||
|
GLES20.GlTexParameteri(ExternalTexture, GLES20.GlTextureWrapT, GLES20.GlClampToEdge);
|
||||||
|
_vertices = CreateBuffer([-1, -1, 1, -1, -1, 1, 1, 1]);
|
||||||
|
_coordinates = CreateBuffer(CameraFrameOrientation.GetTextureCoordinates(0));
|
||||||
|
GLES20.GlUseProgram(_program);
|
||||||
|
BindAttribute("position", _vertices);
|
||||||
|
BindAttribute("textureCoordinate", _coordinates);
|
||||||
|
_matrixLocation = GLES20.GlGetUniformLocation(_program, "textureMatrix");
|
||||||
|
GLES20.GlUniform1i(GLES20.GlGetUniformLocation(_program, "cameraTexture"), 0);
|
||||||
|
GLES20.GlViewport(0, 0, rotation is 90 or 270 ? height : width, rotation is 90 or 270 ? width : height);
|
||||||
|
CheckGl();
|
||||||
|
|
||||||
|
_texture = new SurfaceTexture(_textureId);
|
||||||
|
_texture.SetDefaultBufferSize(width, height);
|
||||||
|
_listener = new FrameListener(this);
|
||||||
|
_texture.SetOnFrameAvailableListener(_listener, _handler);
|
||||||
|
_cameraSurface = new Surface(_texture);
|
||||||
|
}
|
||||||
|
|
||||||
|
internal Task StartAsync() => InvokeAsync(() => _rendering = true);
|
||||||
|
|
||||||
|
private void DrawFrame(SurfaceTexture texture)
|
||||||
|
{
|
||||||
|
if (!_rendering) return;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
texture.UpdateTexImage();
|
||||||
|
var presentationTime = _clock.GetPresentationTime(texture.Timestamp, Java.Lang.JavaSystem.NanoTime());
|
||||||
|
if (presentationTime is null) return;
|
||||||
|
texture.GetTransformMatrix(_textureMatrix);
|
||||||
|
var remainingRotation = CameraFrameOrientation.GetRemainingRotation(_rotation, _textureMatrix);
|
||||||
|
if (_remainingRotation != remainingRotation)
|
||||||
|
{
|
||||||
|
_remainingRotation = remainingRotation;
|
||||||
|
FirstTextureTransform = string.Join(", ", _textureMatrix);
|
||||||
|
_coordinates!.Position(0);
|
||||||
|
_coordinates.Put(CameraFrameOrientation.GetTextureCoordinates(remainingRotation));
|
||||||
|
_coordinates.Position(0);
|
||||||
|
global::Android.Util.Log.Info("PostIt.Podcast",
|
||||||
|
$"Camera texture transform: [{FirstTextureTransform}], remaining rotation={remainingRotation}");
|
||||||
|
}
|
||||||
|
GLES20.GlUniformMatrix4fv(_matrixLocation, 1, false, _textureMatrix, 0);
|
||||||
|
GLES20.GlDrawArrays(GLES20.GlTriangleStrip, 0, 4);
|
||||||
|
CheckGl();
|
||||||
|
if (!EGLExt.EglPresentationTimeANDROID(_display, _window, presentationTime.Value))
|
||||||
|
throw EglError("Horodatage de l'image camera");
|
||||||
|
if (!EGL14.EglSwapBuffers(_display, _window))
|
||||||
|
throw EglError("Envoi de l'image a l'encodeur");
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_rendering = false;
|
||||||
|
_failed(new InvalidOperationException("Rotation de la capture Android impossible.", ex));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private Task InvokeAsync(Action action)
|
||||||
|
{
|
||||||
|
var completion = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
|
if (!_handler.Post(() =>
|
||||||
|
{
|
||||||
|
try { action(); completion.SetResult(); }
|
||||||
|
catch (Exception ex) { completion.SetException(ex); }
|
||||||
|
}))
|
||||||
|
completion.SetException(new InvalidOperationException("Le thread de rotation camera est arrete."));
|
||||||
|
return completion.Task;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static FloatBuffer CreateBuffer(float[] values)
|
||||||
|
{
|
||||||
|
using var bytes = ByteBuffer.AllocateDirect(values.Length * sizeof(float))
|
||||||
|
?? throw new InvalidOperationException("Allocation du tampon OpenGL impossible.");
|
||||||
|
bytes.Order(ByteOrder.NativeOrder()!);
|
||||||
|
var buffer = bytes.AsFloatBuffer()
|
||||||
|
?? throw new InvalidOperationException("Creation du tampon OpenGL impossible.");
|
||||||
|
buffer.Put(values);
|
||||||
|
buffer.Position(0);
|
||||||
|
return buffer;
|
||||||
|
}
|
||||||
|
|
||||||
|
private void BindAttribute(string name, FloatBuffer buffer)
|
||||||
|
{
|
||||||
|
var location = GLES20.GlGetAttribLocation(_program, name);
|
||||||
|
GLES20.GlEnableVertexAttribArray(location);
|
||||||
|
GLES20.GlVertexAttribPointer(location, 2, GLES20.GlFloat, false, 0, buffer);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static int CreateShader(int type, string source)
|
||||||
|
{
|
||||||
|
var shader = GLES20.GlCreateShader(type);
|
||||||
|
GLES20.GlShaderSource(shader, source);
|
||||||
|
GLES20.GlCompileShader(shader);
|
||||||
|
var status = new int[1];
|
||||||
|
GLES20.GlGetShaderiv(shader, GLES20.GlCompileStatus, status, 0);
|
||||||
|
if (status[0] != 0) return shader;
|
||||||
|
var error = GLES20.GlGetShaderInfoLog(shader);
|
||||||
|
GLES20.GlDeleteShader(shader);
|
||||||
|
throw new InvalidOperationException($"Compilation du shader camera impossible : {error}");
|
||||||
|
}
|
||||||
|
|
||||||
|
private static int CreateProgram()
|
||||||
|
{
|
||||||
|
var vertex = CreateShader(GLES20.GlVertexShader, VertexShader);
|
||||||
|
var fragment = 0;
|
||||||
|
var program = 0;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
fragment = CreateShader(GLES20.GlFragmentShader, FragmentShader);
|
||||||
|
program = GLES20.GlCreateProgram();
|
||||||
|
GLES20.GlAttachShader(program, vertex);
|
||||||
|
GLES20.GlAttachShader(program, fragment);
|
||||||
|
GLES20.GlLinkProgram(program);
|
||||||
|
var status = new int[1];
|
||||||
|
GLES20.GlGetProgramiv(program, GLES20.GlLinkStatus, status, 0);
|
||||||
|
if (status[0] == 0)
|
||||||
|
throw new InvalidOperationException($"Edition du programme camera impossible : {GLES20.GlGetProgramInfoLog(program)}");
|
||||||
|
return program;
|
||||||
|
}
|
||||||
|
catch
|
||||||
|
{
|
||||||
|
if (program != 0) GLES20.GlDeleteProgram(program);
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
GLES20.GlDeleteShader(vertex);
|
||||||
|
if (fragment != 0) GLES20.GlDeleteShader(fragment);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void CheckGl()
|
||||||
|
{
|
||||||
|
var error = GLES20.GlGetError();
|
||||||
|
if (error != GLES20.GlNoError)
|
||||||
|
throw new InvalidOperationException($"Erreur OpenGL de capture : 0x{error:X}.");
|
||||||
|
}
|
||||||
|
|
||||||
|
private static InvalidOperationException EglError(string operation)
|
||||||
|
=> new($"{operation} impossible : EGL 0x{EGL14.EglGetError():X}.");
|
||||||
|
|
||||||
|
public void Dispose()
|
||||||
|
{
|
||||||
|
if (_disposed) return;
|
||||||
|
_disposed = true;
|
||||||
|
try { InvokeAsync(Release).GetAwaiter().GetResult(); }
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
_thread.QuitSafely();
|
||||||
|
_thread.Join();
|
||||||
|
_handler.Dispose();
|
||||||
|
_thread.Dispose();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void Release()
|
||||||
|
{
|
||||||
|
_rendering = false;
|
||||||
|
_texture?.SetOnFrameAvailableListener(null);
|
||||||
|
_cameraSurface?.Release();
|
||||||
|
_cameraSurface?.Dispose();
|
||||||
|
_texture?.Release();
|
||||||
|
_texture?.Dispose();
|
||||||
|
_listener?.Dispose();
|
||||||
|
if (_program != 0) GLES20.GlDeleteProgram(_program);
|
||||||
|
if (_textureId != 0) GLES20.GlDeleteTextures(1, [_textureId], 0);
|
||||||
|
_vertices?.Dispose();
|
||||||
|
_coordinates?.Dispose();
|
||||||
|
if (_display is not null && !_display.Equals(EGL14.EglNoDisplay))
|
||||||
|
{
|
||||||
|
EGL14.EglMakeCurrent(_display, EGL14.EglNoSurface, EGL14.EglNoSurface, EGL14.EglNoContext);
|
||||||
|
if (_window is not null && !_window.Equals(EGL14.EglNoSurface))
|
||||||
|
EGL14.EglDestroySurface(_display, _window);
|
||||||
|
if (_context is not null && !_context.Equals(EGL14.EglNoContext))
|
||||||
|
EGL14.EglDestroyContext(_display, _context);
|
||||||
|
EGL14.EglReleaseThread();
|
||||||
|
EGL14.EglTerminate(_display);
|
||||||
|
}
|
||||||
|
_window?.Dispose();
|
||||||
|
_context?.Dispose();
|
||||||
|
_display?.Dispose();
|
||||||
|
}
|
||||||
|
|
||||||
|
private sealed class FrameListener(AndroidCameraFrameRotator owner)
|
||||||
|
: Java.Lang.Object, SurfaceTexture.IOnFrameAvailableListener
|
||||||
|
{
|
||||||
|
public void OnFrameAvailable(SurfaceTexture? texture)
|
||||||
|
{
|
||||||
|
if (texture is not null) owner.DrawFrame(texture);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -26,6 +26,7 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
|
||||||
private CameraDevice? _camera;
|
private CameraDevice? _camera;
|
||||||
private CameraCaptureSession? _session;
|
private CameraCaptureSession? _session;
|
||||||
private MediaRecorder? _recorder;
|
private MediaRecorder? _recorder;
|
||||||
|
private AndroidCameraFrameRotator? _rotator;
|
||||||
private Surface? _surface;
|
private Surface? _surface;
|
||||||
private ParcelFileDescriptor? _readPipe;
|
private ParcelFileDescriptor? _readPipe;
|
||||||
private ParcelFileDescriptor? _writePipe;
|
private ParcelFileDescriptor? _writePipe;
|
||||||
|
|
@ -36,6 +37,8 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
|
||||||
private bool _started;
|
private bool _started;
|
||||||
private bool _disposed;
|
private bool _disposed;
|
||||||
public Task Completion => _completion.Task;
|
public Task Completion => _completion.Task;
|
||||||
|
internal string OrientationDiagnostics { get; private set; } = "";
|
||||||
|
internal string? TextureTransform => _rotator?.FirstTextureTransform;
|
||||||
|
|
||||||
private AndroidPodcastCapture(PodcastCaptureService service) => _service = service;
|
private AndroidPodcastCapture(PodcastCaptureService service) => _service = service;
|
||||||
|
|
||||||
|
|
@ -103,6 +106,30 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
|
||||||
var size = sizes.Where(s => s.Width <= 640 && s.Height <= 480)
|
var size = sizes.Where(s => s.Width <= 640 && s.Height <= 480)
|
||||||
.OrderByDescending(s => s.Width * s.Height).FirstOrDefault()
|
.OrderByDescending(s => s.Width * s.Height).FirstOrDefault()
|
||||||
?? throw new InvalidOperationException("La camera ne propose aucun format jusqu'a 480p.");
|
?? throw new InvalidOperationException("La camera ne propose aucun format jusqu'a 480p.");
|
||||||
|
using var sensor = characteristics.Get(CameraCharacteristics.SensorOrientation) as Java.Lang.Integer
|
||||||
|
?? throw new InvalidOperationException("Orientation du capteur camera indisponible.");
|
||||||
|
using var facing = characteristics.Get(CameraCharacteristics.LensFacing) as Java.Lang.Integer
|
||||||
|
?? throw new InvalidOperationException("Position du capteur camera indisponible.");
|
||||||
|
var activity = MainActivity.Current ?? throw new InvalidOperationException("Activite Android indisponible.");
|
||||||
|
#pragma warning disable CS0618, CA1422
|
||||||
|
var display = OperatingSystem.IsAndroidVersionAtLeast(30)
|
||||||
|
? activity.Display : activity.WindowManager?.DefaultDisplay;
|
||||||
|
#pragma warning restore CS0618, CA1422
|
||||||
|
var displayRotation = display?.Rotation
|
||||||
|
?? throw new InvalidOperationException("Orientation de l'ecran Android indisponible.");
|
||||||
|
var displayDegrees = displayRotation switch
|
||||||
|
{
|
||||||
|
SurfaceOrientation.Rotation0 => 0,
|
||||||
|
SurfaceOrientation.Rotation90 => 90,
|
||||||
|
SurfaceOrientation.Rotation180 => 180,
|
||||||
|
SurfaceOrientation.Rotation270 => 270,
|
||||||
|
_ => throw new InvalidOperationException($"Rotation d'ecran Android inconnue : {displayRotation}.")
|
||||||
|
};
|
||||||
|
var rotation = CameraFrameOrientation.GetRotation(sensor.IntValue(), displayDegrees,
|
||||||
|
facing.IntValue() == (int)LensFacing.Front);
|
||||||
|
global::Android.Util.Log.Info("PostIt.Podcast",
|
||||||
|
$"Camera orientation: sensor={sensor.IntValue()}, display={displayDegrees}, facing={facing.IntValue()}, rotation={rotation}");
|
||||||
|
OrientationDiagnostics = $"sensor={sensor.IntValue()}, display={displayDegrees}, facing={facing.IntValue()}, rotation={rotation}";
|
||||||
#pragma warning disable CS0618
|
#pragma warning disable CS0618
|
||||||
_recorder = OperatingSystem.IsAndroidVersionAtLeast(31)
|
_recorder = OperatingSystem.IsAndroidVersionAtLeast(31)
|
||||||
? new MediaRecorder(_service) : new MediaRecorder();
|
? new MediaRecorder(_service) : new MediaRecorder();
|
||||||
|
|
@ -111,7 +138,8 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
|
||||||
if (microphone) _recorder.SetAudioSource(AudioSource.Mic);
|
if (microphone) _recorder.SetAudioSource(AudioSource.Mic);
|
||||||
_recorder.SetVideoSource(VideoSource.Surface);
|
_recorder.SetVideoSource(VideoSource.Surface);
|
||||||
_recorder.SetOutputFormat(OutputFormat.Mpeg2Ts);
|
_recorder.SetOutputFormat(OutputFormat.Mpeg2Ts);
|
||||||
_recorder.SetVideoSize(size.Width, size.Height);
|
_recorder.SetVideoSize(rotation is 90 or 270 ? size.Height : size.Width,
|
||||||
|
rotation is 90 or 270 ? size.Width : size.Height);
|
||||||
_recorder.SetVideoFrameRate(30);
|
_recorder.SetVideoFrameRate(30);
|
||||||
_recorder.SetVideoEncodingBitRate(1200000);
|
_recorder.SetVideoEncodingBitRate(1200000);
|
||||||
_recorder.SetVideoEncoder(VideoEncoder.H264);
|
_recorder.SetVideoEncoder(VideoEncoder.H264);
|
||||||
|
|
@ -131,20 +159,23 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
|
||||||
ct.ThrowIfCancellationRequested();
|
ct.ThrowIfCancellationRequested();
|
||||||
if (_stopRequested) throw new System.OperationCanceledException("Demarrage de la capture annule.");
|
if (_stopRequested) throw new System.OperationCanceledException("Demarrage de la capture annule.");
|
||||||
_surface = _recorder.Surface ?? throw new InvalidOperationException("Surface encodeur indisponible.");
|
_surface = _recorder.Surface ?? throw new InvalidOperationException("Surface encodeur indisponible.");
|
||||||
|
_rotator = await AndroidCameraFrameRotator.CreateAsync(_surface, size.Width, size.Height, rotation, Fail);
|
||||||
|
ct.ThrowIfCancellationRequested();
|
||||||
|
if (_stopRequested) throw new System.OperationCanceledException("Demarrage de la capture annule.");
|
||||||
|
var cameraSurface = _rotator.CameraSurface;
|
||||||
var opened = new TaskCompletionSource<CameraDevice>(TaskCreationOptions.RunContinuationsAsynchronously);
|
var opened = new TaskCompletionSource<CameraDevice>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
manager.OpenCamera(cameraId, new DeviceCallback(this, opened), _handler);
|
manager.OpenCamera(cameraId, new DeviceCallback(this, opened), _handler);
|
||||||
_camera = await opened.Task.WaitAsync(TimeSpan.FromSeconds(10), ct);
|
_camera = await opened.Task.WaitAsync(TimeSpan.FromSeconds(10), ct);
|
||||||
var configured = new TaskCompletionSource<CameraCaptureSession>(TaskCreationOptions.RunContinuationsAsynchronously);
|
var configured = new TaskCompletionSource<CameraCaptureSession>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||||
#pragma warning disable CS0618, CA1422
|
#pragma warning disable CS0618, CA1422
|
||||||
_camera.CreateCaptureSession(new List<Surface> { _surface }, new SessionCallback(this, configured), _handler);
|
_camera.CreateCaptureSession(new List<Surface> { cameraSurface }, new SessionCallback(this, configured), _handler);
|
||||||
#pragma warning restore CS0618, CA1422
|
#pragma warning restore CS0618, CA1422
|
||||||
_session = await configured.Task.WaitAsync(TimeSpan.FromSeconds(10), ct);
|
_session = await configured.Task.WaitAsync(TimeSpan.FromSeconds(10), ct);
|
||||||
ct.ThrowIfCancellationRequested();
|
ct.ThrowIfCancellationRequested();
|
||||||
if (_stopRequested) throw new System.OperationCanceledException("Demarrage de la capture annule.");
|
if (_stopRequested) throw new System.OperationCanceledException("Demarrage de la capture annule.");
|
||||||
using var request = _camera.CreateCaptureRequest(CameraTemplate.Record);
|
using var request = _camera.CreateCaptureRequest(CameraTemplate.Record);
|
||||||
request.AddTarget(_surface);
|
request.AddTarget(cameraSurface);
|
||||||
using var built = request.Build();
|
using var built = request.Build();
|
||||||
_session.SetRepeatingRequest(built, null, _handler);
|
|
||||||
var handle = new SafeFileHandle((IntPtr)_readPipe.Fd, ownsHandle: false);
|
var handle = new SafeFileHandle((IntPtr)_readPipe.Fd, ownsHandle: false);
|
||||||
var stream = new FileStream(handle, FileAccess.Read);
|
var stream = new FileStream(handle, FileAccess.Read);
|
||||||
_pump = Task.Run(async () =>
|
_pump = Task.Run(async () =>
|
||||||
|
|
@ -169,6 +200,8 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
|
||||||
});
|
});
|
||||||
await Task.Run(_recorder.Start, ct);
|
await Task.Run(_recorder.Start, ct);
|
||||||
_started = true;
|
_started = true;
|
||||||
|
await _rotator.StartAsync();
|
||||||
|
_session.SetRepeatingRequest(built, null, _handler);
|
||||||
_writePipe.Close();
|
_writePipe.Close();
|
||||||
_writePipe.Dispose();
|
_writePipe.Dispose();
|
||||||
_writePipe = null;
|
_writePipe = null;
|
||||||
|
|
@ -261,7 +294,11 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
|
||||||
|
|
||||||
private void StopNative()
|
private void StopNative()
|
||||||
{
|
{
|
||||||
try { if (_started) _recorder?.Stop(); }
|
try
|
||||||
|
{
|
||||||
|
// Keep frames flowing until MediaRecorder has drained its encoder.
|
||||||
|
if (_started) _recorder?.Stop();
|
||||||
|
}
|
||||||
catch (Java.Lang.RuntimeException ex)
|
catch (Java.Lang.RuntimeException ex)
|
||||||
{
|
{
|
||||||
_failure = ex;
|
_failure = ex;
|
||||||
|
|
@ -276,6 +313,8 @@ internal sealed class AndroidPodcastCapture : IPodcastCapture
|
||||||
_camera?.Close();
|
_camera?.Close();
|
||||||
_camera?.Dispose();
|
_camera?.Dispose();
|
||||||
_camera = null;
|
_camera = null;
|
||||||
|
_rotator?.Dispose();
|
||||||
|
_rotator = null;
|
||||||
if (_recorder is not null)
|
if (_recorder is not null)
|
||||||
{
|
{
|
||||||
_recorder.Error -= OnRecorderError;
|
_recorder.Error -= OnRecorderError;
|
||||||
|
|
|
||||||
24
src/PostIt/PostIt.Android/Services/CameraFrameClock.cs
Normal file
24
src/PostIt/PostIt.Android/Services/CameraFrameClock.cs
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
using System;
|
||||||
|
|
||||||
|
namespace PostIt.Android.Services;
|
||||||
|
|
||||||
|
internal sealed class CameraFrameClock
|
||||||
|
{
|
||||||
|
private long? _offset;
|
||||||
|
private long? _previousTimestamp;
|
||||||
|
|
||||||
|
internal long? GetPresentationTime(long cameraTimestamp, long monotonicNow)
|
||||||
|
{
|
||||||
|
if (_previousTimestamp is { } previous)
|
||||||
|
{
|
||||||
|
if (cameraTimestamp == previous) return null;
|
||||||
|
if (cameraTimestamp < previous)
|
||||||
|
throw new InvalidOperationException("Les horodatages de la camera Android ont recule.");
|
||||||
|
}
|
||||||
|
// Camera2 may use elapsed realtime (including suspend), whereas MediaRecorder
|
||||||
|
// and its microphone use monotonic time. Preserve frame intervals, not that epoch.
|
||||||
|
_offset ??= monotonicNow - cameraTimestamp;
|
||||||
|
_previousTimestamp = cameraTimestamp;
|
||||||
|
return checked(cameraTimestamp + _offset.Value);
|
||||||
|
}
|
||||||
|
}
|
||||||
55
src/PostIt/PostIt.Android/Services/CameraFrameOrientation.cs
Normal file
55
src/PostIt/PostIt.Android/Services/CameraFrameOrientation.cs
Normal file
|
|
@ -0,0 +1,55 @@
|
||||||
|
using System;
|
||||||
|
|
||||||
|
namespace PostIt.Android.Services;
|
||||||
|
|
||||||
|
internal static class CameraFrameOrientation
|
||||||
|
{
|
||||||
|
internal static int GetRotation(int sensorDegrees, int displayDegrees, bool frontFacing)
|
||||||
|
{
|
||||||
|
Validate(sensorDegrees);
|
||||||
|
Validate(displayDegrees);
|
||||||
|
return (sensorDegrees + (frontFacing ? displayDegrees : -displayDegrees) + 360) % 360;
|
||||||
|
}
|
||||||
|
|
||||||
|
internal static float[] GetTextureCoordinates(int clockwiseDegrees)
|
||||||
|
{
|
||||||
|
Validate(clockwiseDegrees);
|
||||||
|
float[] corners = [0, 0, 1, 0, 0, 1, 1, 1];
|
||||||
|
for (var i = 0; i < corners.Length; i += 2)
|
||||||
|
{
|
||||||
|
var x = corners[i];
|
||||||
|
var y = corners[i + 1];
|
||||||
|
(corners[i], corners[i + 1]) = clockwiseDegrees switch
|
||||||
|
{
|
||||||
|
90 => (1 - y, x),
|
||||||
|
180 => (1 - x, 1 - y),
|
||||||
|
270 => (y, 1 - x),
|
||||||
|
_ => (x, y)
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return corners;
|
||||||
|
}
|
||||||
|
|
||||||
|
internal static int GetRemainingRotation(int clockwiseDegrees, float[] textureMatrix)
|
||||||
|
{
|
||||||
|
Validate(clockwiseDegrees);
|
||||||
|
if (textureMatrix.Length != 16)
|
||||||
|
throw new ArgumentException("A SurfaceTexture transform must contain 16 elements.", nameof(textureMatrix));
|
||||||
|
// SurfaceTexture's usual Y flip changes the sign of the first column's Y.
|
||||||
|
// Camera2 can also put the sensor rotation in this matrix; don't apply it twice.
|
||||||
|
var angle = Math.Atan2(-textureMatrix[1], textureMatrix[0]) * 180 / Math.PI;
|
||||||
|
if (!double.IsFinite(angle) || (textureMatrix[0] == 0 && textureMatrix[1] == 0))
|
||||||
|
throw new ArgumentException("Invalid SurfaceTexture orientation.", nameof(textureMatrix));
|
||||||
|
var quarterTurns = (int)Math.Round(angle / 90);
|
||||||
|
if (Math.Abs(angle - quarterTurns * 90) > 0.01)
|
||||||
|
throw new ArgumentException("SurfaceTexture orientation is not a quarter turn.", nameof(textureMatrix));
|
||||||
|
var producerRotation = (quarterTurns * 90 + 360) % 360;
|
||||||
|
return (clockwiseDegrees - producerRotation + 360) % 360;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void Validate(int degrees)
|
||||||
|
{
|
||||||
|
if (degrees is not (0 or 90 or 180 or 270))
|
||||||
|
throw new ArgumentOutOfRangeException(nameof(degrees), degrees, "Rotation must be 0, 90, 180 or 270 degrees.");
|
||||||
|
}
|
||||||
|
}
|
||||||
9
src/PostIt/PostIt.Android/proguard-rules.pro
vendored
Normal file
9
src/PostIt/PostIt.Android/proguard-rules.pro
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
# Keep the LibVLC Android classes and JNI/native bindings intact in Release builds.
|
||||||
|
# R8/ProGuard otherwise can strip or rename the Java-side entry points that load
|
||||||
|
# the native libvlc and libc++_shared libraries bundled by VideoLAN.LibVLC.Android.
|
||||||
|
-keep class org.videolan.libvlc.** { *; }
|
||||||
|
-keep class org.videolan.libvlc.** { native *; }
|
||||||
|
-keep class org.videolan.medialibrary.** { *; }
|
||||||
|
-keepattributes *Annotation*
|
||||||
|
-dontwarn org.videolan.libvlc.**
|
||||||
|
-dontwarn org.videolan.medialibrary.**
|
||||||
|
|
@ -0,0 +1,147 @@
|
||||||
|
#if ANDROID
|
||||||
|
using System;
|
||||||
|
using System.IO;
|
||||||
|
using System.Runtime.Versioning;
|
||||||
|
using System.Threading;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using Android.App;
|
||||||
|
using Android.Graphics;
|
||||||
|
using Android.Media;
|
||||||
|
using Android.OS;
|
||||||
|
using PostIt.Android.Services;
|
||||||
|
using Path = System.IO.Path;
|
||||||
|
|
||||||
|
namespace PostIt.Tests;
|
||||||
|
|
||||||
|
/// <summary>Debug-only synthetic encoder test; never opens the camera or contacts the API.</summary>
|
||||||
|
[Activity(Name = "fr.pschneider.postit.CameraFrameRotationTestActivity", Exported = true)]
|
||||||
|
[SupportedOSPlatform("android26.0")]
|
||||||
|
public sealed class AndroidCameraFrameRotationTestActivity : Activity
|
||||||
|
{
|
||||||
|
protected override async void OnCreate(Bundle? state)
|
||||||
|
{
|
||||||
|
base.OnCreate(state);
|
||||||
|
var resultPath = Path.Combine(FilesDir!.AbsolutePath, "camera-rotation-test-result.txt");
|
||||||
|
File.WriteAllText(resultPath, "RUNNING");
|
||||||
|
try
|
||||||
|
{
|
||||||
|
foreach (var rotation in new[] { 0, 90, 180, 270 })
|
||||||
|
await VerifyRotationAsync(rotation);
|
||||||
|
File.WriteAllText(resultPath, "PASS\nrotations=0,90,180,270\nformat=MPEG-TS\nsource=synthetic");
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
File.WriteAllText(resultPath, $"FAIL\n{ex}");
|
||||||
|
}
|
||||||
|
finally { Finish(); }
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task VerifyRotationAsync(int rotation)
|
||||||
|
{
|
||||||
|
const int width = 640;
|
||||||
|
const int height = 480;
|
||||||
|
var outputWidth = rotation is 90 or 270 ? height : width;
|
||||||
|
var outputHeight = rotation is 90 or 270 ? width : height;
|
||||||
|
var path = Path.Combine(FilesDir!.AbsolutePath, $"camera-rotation-{rotation}.ts");
|
||||||
|
void Stage(string stage) => File.WriteAllText(
|
||||||
|
Path.Combine(FilesDir.AbsolutePath, "camera-rotation-test-result.txt"), $"RUNNING\nrotation={rotation}\nstage={stage}");
|
||||||
|
Stage("prepare");
|
||||||
|
#pragma warning disable CS0618
|
||||||
|
using var recorder = OperatingSystem.IsAndroidVersionAtLeast(31)
|
||||||
|
? new MediaRecorder(this) : new MediaRecorder();
|
||||||
|
#pragma warning restore CS0618
|
||||||
|
recorder.SetVideoSource(VideoSource.Surface);
|
||||||
|
recorder.SetOutputFormat(OutputFormat.Mpeg2Ts);
|
||||||
|
recorder.SetVideoSize(outputWidth, outputHeight);
|
||||||
|
recorder.SetVideoFrameRate(30);
|
||||||
|
recorder.SetVideoEncodingBitRate(1200000);
|
||||||
|
recorder.SetVideoEncoder(VideoEncoder.H264);
|
||||||
|
recorder.SetOutputFile(path);
|
||||||
|
recorder.Prepare();
|
||||||
|
using var encoderSurface = recorder.Surface
|
||||||
|
?? throw new InvalidOperationException("Encoder surface unavailable.");
|
||||||
|
Exception? failure = null;
|
||||||
|
var stopping = 0;
|
||||||
|
Stage("initialize EGL");
|
||||||
|
using (var rotator = await AndroidCameraFrameRotator.CreateAsync(
|
||||||
|
encoderSurface, width, height, rotation, error =>
|
||||||
|
{
|
||||||
|
if (Volatile.Read(ref stopping) != 0) return;
|
||||||
|
failure = error;
|
||||||
|
File.WriteAllText(Path.Combine(FilesDir.AbsolutePath, "camera-rotation-test-result.txt"), $"FAIL\n{error}");
|
||||||
|
}))
|
||||||
|
{
|
||||||
|
Stage("start recorder");
|
||||||
|
recorder.Start();
|
||||||
|
Task? stop = null;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await rotator.StartAsync();
|
||||||
|
using var paint = new Paint();
|
||||||
|
Color[] colors = [Color.Red, Color.Green, Color.Blue, Color.Yellow];
|
||||||
|
Stage("draw frames");
|
||||||
|
for (var frame = 0; frame < 90; frame++)
|
||||||
|
{
|
||||||
|
if (failure is not null) throw failure;
|
||||||
|
if (frame == 45)
|
||||||
|
{
|
||||||
|
Volatile.Write(ref stopping, 1);
|
||||||
|
stop = Task.Run(recorder.Stop);
|
||||||
|
}
|
||||||
|
if (stop?.IsCompleted == true) break;
|
||||||
|
var canvas = rotator.CameraSurface.LockCanvas(null)
|
||||||
|
?? throw new InvalidOperationException("Synthetic camera canvas unavailable.");
|
||||||
|
try
|
||||||
|
{
|
||||||
|
for (var i = 0; i < colors.Length; i++)
|
||||||
|
{
|
||||||
|
paint.Color = colors[i];
|
||||||
|
var x = i % 2 * width / 2;
|
||||||
|
var y = i / 2 * height / 2;
|
||||||
|
canvas.DrawRect(x, y, x + width / 2, y + height / 2, paint);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
rotator.CameraSurface.UnlockCanvasAndPost(canvas);
|
||||||
|
canvas.Dispose();
|
||||||
|
}
|
||||||
|
await Task.Delay(34);
|
||||||
|
}
|
||||||
|
if (failure is not null) throw failure;
|
||||||
|
Stage("wait for recorder stop");
|
||||||
|
await (stop ?? throw new InvalidOperationException("Recorder stop was not requested."))
|
||||||
|
.WaitAsync(TimeSpan.FromSeconds(10));
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
Stage("dispose EGL");
|
||||||
|
rotator.Dispose();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Stage("decode");
|
||||||
|
using var retriever = new MediaMetadataRetriever();
|
||||||
|
retriever.SetDataSource(path);
|
||||||
|
using var bitmap = retriever.GetFrameAtTime(0, Option.ClosestSync)
|
||||||
|
?? throw new InvalidOperationException($"Cannot decode rotation {rotation}.");
|
||||||
|
if (bitmap.Width != outputWidth || bitmap.Height != outputHeight)
|
||||||
|
throw new InvalidOperationException($"Rotation {rotation}: dimensions {bitmap.Width}x{bitmap.Height}, expected {outputWidth}x{outputHeight}.");
|
||||||
|
Color[] expected = rotation switch
|
||||||
|
{
|
||||||
|
90 => [Color.Blue, Color.Red, Color.Yellow, Color.Green],
|
||||||
|
180 => [Color.Yellow, Color.Blue, Color.Green, Color.Red],
|
||||||
|
270 => [Color.Green, Color.Yellow, Color.Red, Color.Blue],
|
||||||
|
_ => [Color.Red, Color.Green, Color.Blue, Color.Yellow]
|
||||||
|
};
|
||||||
|
for (var i = 0; i < expected.Length; i++)
|
||||||
|
{
|
||||||
|
var actual = new Color(bitmap.GetPixel((i % 2 * 2 + 1) * outputWidth / 4,
|
||||||
|
(i / 2 * 2 + 1) * outputHeight / 4));
|
||||||
|
var target = expected[i];
|
||||||
|
if (Math.Abs(actual.R - target.R) > 40 || Math.Abs(actual.G - target.G) > 40 || Math.Abs(actual.B - target.B) > 40)
|
||||||
|
throw new InvalidOperationException($"Rotation {rotation}, quadrant {i}: {actual}, expected {target}.");
|
||||||
|
}
|
||||||
|
File.Delete(path);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
@ -1,11 +1,13 @@
|
||||||
#if ANDROID
|
#if ANDROID
|
||||||
using System;
|
using System;
|
||||||
using System.IO;
|
using System.IO;
|
||||||
|
using System.Linq;
|
||||||
using System.Runtime.Versioning;
|
using System.Runtime.Versioning;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using System.Threading.Tasks;
|
using System.Threading.Tasks;
|
||||||
using Android.App;
|
using Android.App;
|
||||||
using Android.Content;
|
using Android.Content;
|
||||||
|
using Android.Media;
|
||||||
using Android.OS;
|
using Android.OS;
|
||||||
using Avalonia.Threading;
|
using Avalonia.Threading;
|
||||||
using PostIt.Android;
|
using PostIt.Android;
|
||||||
|
|
@ -34,7 +36,7 @@ public sealed class AndroidPodcastCaptureTestActivity : Activity
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
await RunAsync(result);
|
await RunAsync(result);
|
||||||
File.WriteAllText(resultPath, $"PASS\nsegments={result.GetInt("segments")}\ndirectory={result.GetString("directory")}\nbackground={result.GetString("background")}");
|
File.WriteAllText(resultPath, $"PASS\nsegments={result.GetInt("segments")}\ndurationSeconds={result.GetDouble("durationSeconds")}\naudio={result.GetBoolean("audio")}\ndirectory={result.GetString("directory")}\nbackground={result.GetString("background")}\norientation={result.GetString("orientation")}\ntextureTransform={result.GetString("textureTransform")}");
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
|
|
@ -68,6 +70,11 @@ public sealed class AndroidPodcastCaptureTestActivity : Activity
|
||||||
await Task.Delay(9000);
|
await Task.Delay(9000);
|
||||||
var after = SegmentCount(directory);
|
var after = SegmentCount(directory);
|
||||||
if (capture.Completion.IsFaulted) await capture.Completion;
|
if (capture.Completion.IsFaulted) await capture.Completion;
|
||||||
|
if (capture is AndroidPodcastCapture androidCapture)
|
||||||
|
{
|
||||||
|
result.PutString("orientation", androidCapture.OrientationDiagnostics);
|
||||||
|
result.PutString("textureTransform", androidCapture.TextureTransform);
|
||||||
|
}
|
||||||
if (after <= before) throw new InvalidOperationException($"Capture did not continue in background: {before} -> {after} segments.");
|
if (after <= before) throw new InvalidOperationException($"Capture did not continue in background: {before} -> {after} segments.");
|
||||||
if (_notificationStop)
|
if (_notificationStop)
|
||||||
{
|
{
|
||||||
|
|
@ -81,7 +88,25 @@ public sealed class AndroidPodcastCaptureTestActivity : Activity
|
||||||
await capture.Completion.WaitAsync(TimeSpan.FromSeconds(15));
|
await capture.Completion.WaitAsync(TimeSpan.FromSeconds(15));
|
||||||
var segments = HlsCapturePlaylist.Parse(await File.ReadAllTextAsync(Path.Combine(directory, "index.m3u8")));
|
var segments = HlsCapturePlaylist.Parse(await File.ReadAllTextAsync(Path.Combine(directory, "index.m3u8")));
|
||||||
if (segments.Count < 3) throw new InvalidOperationException("Insufficient capture segments.");
|
if (segments.Count < 3) throw new InvalidOperationException("Insufficient capture segments.");
|
||||||
|
var duration = segments.Sum(segment => segment.DurationSeconds);
|
||||||
|
if (duration < 8 || duration > 16)
|
||||||
|
throw new InvalidOperationException($"Expected 8-16 seconds of capture, got {duration}.");
|
||||||
|
using var extractor = new MediaExtractor();
|
||||||
|
extractor.SetDataSource(Path.Combine(directory, segments[0].FileName));
|
||||||
|
var video = false;
|
||||||
|
var audio = false;
|
||||||
|
for (var track = 0; track < extractor.TrackCount; track++)
|
||||||
|
{
|
||||||
|
using var format = extractor.GetTrackFormat(track);
|
||||||
|
var mime = format.GetString(MediaFormat.KeyMime);
|
||||||
|
video |= mime == "video/avc";
|
||||||
|
audio |= mime == "audio/mp4a-latm";
|
||||||
|
}
|
||||||
|
if (!video || (_microphone && !audio))
|
||||||
|
throw new InvalidOperationException($"Missing capture tracks: H.264={video}, AAC={audio}.");
|
||||||
result.PutInt("segments", segments.Count);
|
result.PutInt("segments", segments.Count);
|
||||||
|
result.PutDouble("durationSeconds", duration);
|
||||||
|
result.PutBoolean("audio", audio);
|
||||||
result.PutString("directory", directory);
|
result.PutString("directory", directory);
|
||||||
result.PutString("background", $"segments {before} -> {after}");
|
result.PutString("background", $"segments {before} -> {after}");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
37
src/PostIt/PostIt.Tests/CameraFrameClockTests.cs
Normal file
37
src/PostIt/PostIt.Tests/CameraFrameClockTests.cs
Normal file
|
|
@ -0,0 +1,37 @@
|
||||||
|
using PostIt.Android.Services;
|
||||||
|
|
||||||
|
namespace PostIt.Tests;
|
||||||
|
|
||||||
|
public sealed class CameraFrameClockTests
|
||||||
|
{
|
||||||
|
[Theory]
|
||||||
|
[InlineData(4_213_000_000_000L)]
|
||||||
|
[InlineData(0L)]
|
||||||
|
[InlineData(-1_000_000_000L)]
|
||||||
|
public void Aligns_camera_epoch_to_encoder_and_preserves_frame_intervals(long cameraOffset)
|
||||||
|
{
|
||||||
|
var clock = new CameraFrameClock();
|
||||||
|
const long now = 20_000_000_000_000L;
|
||||||
|
var camera = now + cameraOffset;
|
||||||
|
Assert.Equal(now, clock.GetPresentationTime(camera, now));
|
||||||
|
Assert.Equal(now + 33_333_333, clock.GetPresentationTime(camera + 33_333_333, now + 40_000_000));
|
||||||
|
Assert.Equal(now + 100_000_000, clock.GetPresentationTime(camera + 100_000_000, now + 110_000_000));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Does_not_submit_the_same_camera_image_twice()
|
||||||
|
{
|
||||||
|
var clock = new CameraFrameClock();
|
||||||
|
Assert.Equal(100, clock.GetPresentationTime(1000, 100));
|
||||||
|
Assert.Null(clock.GetPresentationTime(1000, 110));
|
||||||
|
Assert.Equal(120, clock.GetPresentationTime(1020, 130));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Reports_a_camera_clock_that_goes_backwards()
|
||||||
|
{
|
||||||
|
var clock = new CameraFrameClock();
|
||||||
|
clock.GetPresentationTime(1000, 100);
|
||||||
|
Assert.Throws<InvalidOperationException>(() => clock.GetPresentationTime(999, 110));
|
||||||
|
}
|
||||||
|
}
|
||||||
61
src/PostIt/PostIt.Tests/CameraFrameOrientationTests.cs
Normal file
61
src/PostIt/PostIt.Tests/CameraFrameOrientationTests.cs
Normal file
|
|
@ -0,0 +1,61 @@
|
||||||
|
using PostIt.Android.Services;
|
||||||
|
|
||||||
|
namespace PostIt.Tests;
|
||||||
|
|
||||||
|
public sealed class CameraFrameOrientationTests
|
||||||
|
{
|
||||||
|
[Theory]
|
||||||
|
[InlineData(90, 0, false, 90)]
|
||||||
|
[InlineData(90, 90, false, 0)]
|
||||||
|
[InlineData(90, 180, false, 270)]
|
||||||
|
[InlineData(90, 270, false, 180)]
|
||||||
|
[InlineData(270, 0, true, 270)]
|
||||||
|
[InlineData(270, 90, true, 0)]
|
||||||
|
[InlineData(270, 180, true, 90)]
|
||||||
|
[InlineData(270, 270, true, 180)]
|
||||||
|
[InlineData(0, 90, false, 270)]
|
||||||
|
[InlineData(180, 270, true, 90)]
|
||||||
|
public void Uses_sensor_display_and_lens_facing(int sensor, int display, bool front, int expected)
|
||||||
|
=> Assert.Equal(expected, CameraFrameOrientation.GetRotation(sensor, display, front));
|
||||||
|
|
||||||
|
[Theory]
|
||||||
|
[InlineData(0, new float[] { 0, 0, 1, 0, 0, 1, 1, 1 })]
|
||||||
|
[InlineData(90, new float[] { 1, 0, 1, 1, 0, 0, 0, 1 })]
|
||||||
|
[InlineData(180, new float[] { 1, 1, 0, 1, 1, 0, 0, 0 })]
|
||||||
|
[InlineData(270, new float[] { 0, 1, 0, 0, 1, 1, 1, 0 })]
|
||||||
|
public void Maps_output_corners_to_input_for_clockwise_rotation(int rotation, float[] expected)
|
||||||
|
=> Assert.Equal(expected, CameraFrameOrientation.GetTextureCoordinates(rotation));
|
||||||
|
|
||||||
|
[Theory]
|
||||||
|
[InlineData(90, 0, -1, 0)]
|
||||||
|
[InlineData(270, 0, 1, 0)]
|
||||||
|
[InlineData(180, -1, 0, 0)]
|
||||||
|
[InlineData(90, 1, 0, 90)]
|
||||||
|
[InlineData(270, 1, 0, 270)]
|
||||||
|
[InlineData(0, 0, -1, 270)]
|
||||||
|
[InlineData(180, 0, -1, 90)]
|
||||||
|
public void Does_not_repeat_rotation_already_in_surface_texture(
|
||||||
|
int desired, float x, float y, int expected)
|
||||||
|
{
|
||||||
|
float[] matrix = [x, y, 0, 0, y, -x, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1];
|
||||||
|
Assert.Equal(expected, CameraFrameOrientation.GetRemainingRotation(desired, matrix));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Recognizes_the_measured_rear_camera_transform_even_with_cropping()
|
||||||
|
{
|
||||||
|
float[] matrix = [0, -0.9f, 0, 0, -0.8f, 0, 0, 0, 0, 0, 1, 0, 0.9f, 0.95f, 0, 1];
|
||||||
|
Assert.Equal(0, CameraFrameOrientation.GetRemainingRotation(90, matrix));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Theory]
|
||||||
|
[InlineData(-90)]
|
||||||
|
[InlineData(45)]
|
||||||
|
[InlineData(360)]
|
||||||
|
public void Rejects_invalid_angles(int rotation)
|
||||||
|
{
|
||||||
|
Assert.Throws<ArgumentOutOfRangeException>(() => CameraFrameOrientation.GetRotation(rotation, 0, false));
|
||||||
|
Assert.Throws<ArgumentOutOfRangeException>(() => CameraFrameOrientation.GetRotation(90, rotation, false));
|
||||||
|
Assert.Throws<ArgumentOutOfRangeException>(() => CameraFrameOrientation.GetTextureCoordinates(rotation));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -25,5 +25,8 @@
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<Using Include="Xunit" />
|
<Using Include="Xunit" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
<ItemGroup></ItemGroup>
|
<ItemGroup>
|
||||||
|
<Compile Include="..\PostIt.Android\Services\CameraFrameOrientation.cs" Link="Android\CameraFrameOrientation.cs" />
|
||||||
|
<Compile Include="..\PostIt.Android\Services\CameraFrameClock.cs" Link="Android\CameraFrameClock.cs" />
|
||||||
|
</ItemGroup>
|
||||||
</Project>
|
</Project>
|
||||||
Loading…
Reference in a new issue