ClientControllerCollectionTests: IndexOutOfRangeException from EF InMemory on multi-Include #3

New issue

Closed

opened 2026-07-11 21:03:38 +01:00 by notazof · 1 comment

notazof commented 2026-07-11 21:03:38 +01:00

Owner

Copy link

Status

Bug, pre-existing on main and fix/blog-detail. Not introduced by any recent commit. The four tests in src/Yavsc.Org.Tests/Controllers/ClientControllerCollectionTests.cs have been red since their creation in commit 6aaff740 (the ClientController overhaul that introduced per-collection pages).

Observed

Running dotnet test src/Yavsc.Org.Tests/Yavsc.Org.Tests.csproj fails 4 tests with the same stack:

System.IndexOutOfRangeException : Index was outside the bounds of the array.
   at lambda_method(Closure, ValueBuffer)
   at System.Linq.Enumerable.ListWhereSelectIterator`2.MoveNext()
   at InMemoryShapedQueryCompilingExpressionVisitor.ShaperExpressionProcessingExpressionVisitor.IncludeCollection[...]
   at InMemoryShapedQueryCompilingExpressionVisitor.QueryingEnumerable`1.Enumerator.MoveNextHelper()
   at InMemoryShapedQueryCompilingExpressionVisitor.QueryingEnumerable`1.Enumerator.MoveNextAsync()
   at ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[...]
   at Yavsc.Controllers.ClientController.LoadClientAsync(Int32 id) in ClientController.Collections.cs:line 289

The 4 failing tests are all in ClientControllerCollectionTests:

• Edit_GET_returns_200_for_admin
• EditRedirectUris_GET_returns_200_and_lists_seeded_uri
• AddRedirectUri_POST_appends_to_database
• RemoveRedirectUri_POST_with_foreign_rowId_returns_NotFound

Root cause

ClientController.LoadClientAsync (in src/Yavsc.Org/Controllers/Administration/ClientController.Collections.cs) issues a SingleOrDefaultAsync with 9 Includes on collections of the Client entity:

.Include(c => c.RedirectUris)
.Include(c => c.PostLogoutRedirectUris)
.Include(c => c.AllowedScopes)
.Include(c => c.AllowedGrantTypes)
.Include(c => c.AllowedCorsOrigins)
.Include(c => c.IdentityProviderRestrictions)
.Include(c => c.Claims)
.Include(c => c.Properties)
.Include(c => c.ClientSecrets)

The EF Core in-memory provider (used per the test policy in doc/testing.md) cannot materialise this query: its InMemoryShapedQueryCompilingExpressionVisitor reads a value buffer indexed by a slot counter that goes out of bounds when the same shape expression fans out across many collections. This is a long-standing limitation of the in-memory provider, documented in dotnet/efcore issues since 2019, and explicitly not fixed (the in-memory provider is "best effort", not a faithful mock).

The test seeds a Client with three populated collections (AllowedGrantTypes, AllowedScopes, RedirectUris) and six empty ones. The provider crashes on the include walk before any of the assertions can run.

Reproduction

dotnet test src/Yavsc.Org.Tests/Yavsc.Org.Tests.csproj \
  --filter "FullyQualifiedName~ClientControllerCollectionTests.Edit_GET_returns_200_for_admin"

Result: 1 failed, 0 passed, 0 skipped. Same crash in isolation (no test-order dependence).

Options to fix

1. Switch the in-memory provider to SQLite in-memory (Data Source=:memory:) for this fixture. Faithful materialisation, no code change to the controller. Blocked by the test policy in doc/testing.md (UseInMemoryDatabase is the only EF driver for unit tests; no SQLite, no real DB, no Docker).
2. Refactor LoadClientAsync to load collections in separate queries (one per Include, or Entry(c).Collection(...).LoadAsync() for each). Trades one round-trip for nine in production, but the controller is admin-only and the row count is small. Would unblock the in-memory provider while keeping the test policy.
3. Use AsSplitQuery() (EF Core 5+): changes the in-memory provider's expression tree to a sequence of per-collection queries. Worth a try as a one-line fix.

Recommendation

Try option 3 first (cheapest). If AsSplitQuery() doesn't satisfy the in-memory provider, refactor to option 2 (separate queries) — this is an admin endpoint, performance is not on the critical path.

## Status **Bug, pre-existing on `main` and `fix/blog-detail`.** Not introduced by any recent commit. The four tests in `src/Yavsc.Org.Tests/Controllers/ClientControllerCollectionTests.cs` have been red since their creation in commit `6aaff740` (the ClientController overhaul that introduced per-collection pages). ## Observed Running `dotnet test src/Yavsc.Org.Tests/Yavsc.Org.Tests.csproj` fails 4 tests with the same stack: ``` System.IndexOutOfRangeException : Index was outside the bounds of the array. at lambda_method(Closure, ValueBuffer) at System.Linq.Enumerable.ListWhereSelectIterator`2.MoveNext() at InMemoryShapedQueryCompilingExpressionVisitor.ShaperExpressionProcessingExpressionVisitor.IncludeCollection[...] at InMemoryShapedQueryCompilingExpressionVisitor.QueryingEnumerable`1.Enumerator.MoveNextHelper() at InMemoryShapedQueryCompilingExpressionVisitor.QueryingEnumerable`1.Enumerator.MoveNextAsync() at ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync[...] at Yavsc.Controllers.ClientController.LoadClientAsync(Int32 id) in ClientController.Collections.cs:line 289 ``` The 4 failing tests are all in `ClientControllerCollectionTests`: - `Edit_GET_returns_200_for_admin` - `EditRedirectUris_GET_returns_200_and_lists_seeded_uri` - `AddRedirectUri_POST_appends_to_database` - `RemoveRedirectUri_POST_with_foreign_rowId_returns_NotFound` ## Root cause `ClientController.LoadClientAsync` (in `src/Yavsc.Org/Controllers/Administration/ClientController.Collections.cs`) issues a `SingleOrDefaultAsync` with **9 `Include`s** on collections of the `Client` entity: ```csharp .Include(c => c.RedirectUris) .Include(c => c.PostLogoutRedirectUris) .Include(c => c.AllowedScopes) .Include(c => c.AllowedGrantTypes) .Include(c => c.AllowedCorsOrigins) .Include(c => c.IdentityProviderRestrictions) .Include(c => c.Claims) .Include(c => c.Properties) .Include(c => c.ClientSecrets) ``` The EF Core in-memory provider (used per the test policy in `doc/testing.md`) cannot materialise this query: its `InMemoryShapedQueryCompilingExpressionVisitor` reads a value buffer indexed by a slot counter that goes out of bounds when the same shape expression fans out across many collections. This is a long-standing limitation of the in-memory provider, documented in dotnet/efcore issues since 2019, and explicitly **not** fixed (the in-memory provider is "best effort", not a faithful mock). The test seeds a `Client` with three populated collections (`AllowedGrantTypes`, `AllowedScopes`, `RedirectUris`) and six empty ones. The provider crashes on the include walk before any of the assertions can run. ## Reproduction ``` dotnet test src/Yavsc.Org.Tests/Yavsc.Org.Tests.csproj \ --filter "FullyQualifiedName~ClientControllerCollectionTests.Edit_GET_returns_200_for_admin" ``` Result: 1 failed, 0 passed, 0 skipped. Same crash in isolation (no test-order dependence). ## Options to fix 1. **Switch the in-memory provider to SQLite in-memory (`Data Source=:memory:`) for this fixture.** Faithful materialisation, no code change to the controller. *Blocked by* the test policy in `doc/testing.md` (`UseInMemoryDatabase` is the only EF driver for unit tests; no SQLite, no real DB, no Docker). 2. **Refactor `LoadClientAsync` to load collections in separate queries** (one per `Include`, or `Entry(c).Collection(...).LoadAsync()` for each). Trades one round-trip for nine in production, but the controller is admin-only and the row count is small. Would unblock the in-memory provider while keeping the test policy. 3. **Use `AsSplitQuery()`** (EF Core 5+): changes the in-memory provider's expression tree to a sequence of per-collection queries. Worth a try as a one-line fix. ## Recommendation Try option 3 first (cheapest). If `AsSplitQuery()` doesn't satisfy the in-memory provider, refactor to option 2 (separate queries) — this is an admin endpoint, performance is not on the critical path.

notazof commented 2026-07-11 21:37:42 +01:00

Author

Owner

Copy link

Investigation 2026-07-11: AsSplitQuery and Entry.LoadAsync both fail

Tried both recommendations from the issue body on a local branch
fix/issue-3-splitquery. Neither worked against the EF Core 10
in-memory provider.

Attempt 1: .AsSplitQuery() before SingleOrDefaultAsync

.AsSplitQuery()
.SingleOrDefaultAsync(c => c.Id == id)

Result: still 4 failures. Stack trace identical to the original:

System.IndexOutOfRangeException : Index was outside the bounds of the array.
   at InMemoryShapedQueryCompilingExpressionVisitor.ShaperExpressionProcessingExpressionVisitor.IncludeCollection[...]

Reason: AsSplitQuery is a no-op on the in-memory provider. There
is no SQL LEFT JOIN to split — the provider still walks the same
multi-Include expression tree, with the same IncludeCollection slot
counter that overflows.

Attempt 2: Entry().Collection().LoadAsync() per navigation

Refactored LoadClientAsync to first SingleOrDefaultAsync the
Client and then call dbContext.Entry(client).Collection(...).LoadAsync()
once per navigation (9 calls). All Includes removed.

Result: still 4 failures. The crash moved:

System.IndexOutOfRangeException : Index was outside the bounds of the array.
   at InMemoryShapedQueryCompilingExpressionVisitor.QueryingEnumerable`1.Enumerator.MoveNextAsync()
   at EntityFrameworkQueryableExtensions.LoadAsync[TSource](...)

Reason: the in-memory provider cannot materialise a query that
returns a navigation-property entity of IdentityServer8.EntityFramework.Entities.
The shape expression it builds references a slot that does not exist
in the value buffer, regardless of how the query is composed. The
issue is not the multi-Include shape — it is the entity type.

Next steps

• Migrate the test driver to SQLite in-memory
  (Data Source=:memory:) for this fixture. This is the only
  path that materialises the IdentityServer8 entity model
  faithfully. Currently blocked by the test policy in
  doc/testing.md ("No SQLite, no real DB, no Docker" for unit
  tests).
• Or relax the test policy for the ClientController tests
  specifically, since IdentityServer8-shaped entities are out of
  scope for a generic in-memory round-trip.
• Or skip the four tests with a clear marker — the production
  code is fine, the unit tests are not exercisable on InMemory.

No code change proposed yet. Awaiting policy decision.

## Investigation 2026-07-11: AsSplitQuery and Entry.LoadAsync both fail Tried both recommendations from the issue body on a local branch `fix/issue-3-splitquery`. Neither worked against the EF Core 10 in-memory provider. ### Attempt 1: `.AsSplitQuery()` before `SingleOrDefaultAsync` ```csharp .AsSplitQuery() .SingleOrDefaultAsync(c => c.Id == id) ``` Result: still 4 failures. Stack trace identical to the original: System.IndexOutOfRangeException : Index was outside the bounds of the array. at InMemoryShapedQueryCompilingExpressionVisitor.ShaperExpressionProcessingExpressionVisitor.IncludeCollection[...] **Reason:** `AsSplitQuery` is a no-op on the in-memory provider. There is no SQL LEFT JOIN to split — the provider still walks the same multi-Include expression tree, with the same `IncludeCollection` slot counter that overflows. ### Attempt 2: `Entry().Collection().LoadAsync()` per navigation Refactored `LoadClientAsync` to first `SingleOrDefaultAsync` the `Client` and then call `dbContext.Entry(client).Collection(...).LoadAsync()` once per navigation (9 calls). All `Include`s removed. Result: still 4 failures. The crash moved: System.IndexOutOfRangeException : Index was outside the bounds of the array. at InMemoryShapedQueryCompilingExpressionVisitor.QueryingEnumerable`1.Enumerator.MoveNextAsync() at EntityFrameworkQueryableExtensions.LoadAsync[TSource](...) **Reason:** the in-memory provider cannot materialise a query that returns a navigation-property entity of `IdentityServer8.EntityFramework.Entities`. The shape expression it builds references a slot that does not exist in the value buffer, regardless of how the query is composed. The issue is not the multi-Include shape — it is the entity type. ### Next steps - **Migrate the test driver to SQLite in-memory (`Data Source=:memory:`)** for this fixture. This is the only path that materialises the `IdentityServer8` entity model faithfully. *Currently blocked by* the test policy in `doc/testing.md` ("No SQLite, no real DB, no Docker" for unit tests). - **Or relax the test policy for the `ClientController` tests** specifically, since `IdentityServer8`-shaped entities are out of scope for a generic in-memory round-trip. - **Or skip the four tests** with a clear marker — the production code is fine, the unit tests are not exercisable on InMemory. No code change proposed yet. Awaiting policy decision.

notazof closed this issue 2026-08-02 23:56:11 +01:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

release/1.0.8-rc3

feat/ui-testing

1.0.8-rc2

1.0.8-rc1

1.0.7

1.0.6

1.0.5

1.0.4

1.0.2

1.0.1

1.0.0

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

notazof/yavsc#3

No description provided.