Add Device Tests

[ne0rrmatrix]

[PR] Add Device Testing Infrastructure (DeviceRunners)

Description of Change

This branch introduces a comprehensive device testing infrastructure for the .NET MAUI Community Toolkit, powered by DeviceRunners — the same test-runner infrastructure recommended by the .NET MAUI team. Unlike standard xunit unit tests that run in a headless .NET process, device tests run inside a real MAUI application on actual devices and emulators (iOS, Android, Mac Catalyst, Windows). This enables testing of platform-specific behavior — handler creation, layout measurement, control rendering, and platform service interactions — that cannot be verified through unit tests alone.

Update (2026-08-03): All reflection-based tests that accessed internal types have been rewritten to use direct type references via InternalsVisibleTo. This eliminates TypeInitializationException failures on macOS Catalyst where Assembly.GetType() cannot resolve internal types. See Reflection Removal below for details.

Update (2026-10-01): Addressed the Copilot review feedback, and changed how the device tests are invoked. After merging main, global.json opts dotnet test into Microsoft.Testing.Platform, which rejects this VSTest-based project, so CI and local runs now call the DeviceRunners VSTest target through dotnet build -t:VSTest. See CI Pipeline and Running Tests.

---

Key Components

Component	Type	Purpose
CommunityToolkit.Maui.DeviceTests	MAUI App (multi-targeted)	Hosts and executes tests on-device
DeviceRunners.VisualRunners.Maui	NuGet (0.1.0-preview.12)	MAUI visual runner UI (pages, view models, diagnostics)
DeviceRunners.VisualRunners.Xunit	NuGet (0.1.0-preview.12)	xUnit v2 test discovery and execution adapter
DeviceRunners.Testing.Targets	NuGet (0.1.0-preview.12)	MSBuild targets (a replacement VSTest target) that build, deploy, run and produce TRX for device projects
xunit v2.9.3	NuGet	xUnit v2 testing framework
MauiProgram.cs	Static class	Configures UseVisualTestRunner
GlobalUsings.cs	Assembly config	CollectionBehavior (sequential, no parallelization)

---

CI Pipeline

Device tests run in the dotnet-build.yml workflow via two dedicated jobs.

Both jobs invoke the DeviceRunners VSTest target with dotnet build -t:VSTest rather than dotnet test. global.json sets test.runner to Microsoft.Testing.Platform, and in that mode dotnet test refuses any project that is not an MTP application (The following test projects are using VSTest test runner). DeviceRunners is VSTest-based, so the target is called directly; the filter, logger and results directory that dotnet test would pass are supplied as the VSTestTestCaseFilter, VSTestLogger and VSTestResultsDirectory MSBuild properties.

device_test_windows (windows-latest)

device_test_windows:
    name: Run Device Tests (Windows)
    runs-on: windows-latest
    steps:
        - checkout
        - setup-dotnet (TOOLKIT_NET_VERSION)
        - dotnet workload install maui
        - Enable Windows Developer Mode (registry)
        - dotnet restore
        - Install Windows App Runtime framework (MSIX from NuGet cache)
        - dotnet build -t:VSTest
            -f net10.0-windows10.0.19041.0
            -c Release
            -p:IsCI=true
            -p:VSTestTestCaseFilter=Category!=ExpectedFailure
            -p:VSTestLogger=trx%3BLogFileName=test-results.trx
            -p:VSTestResultsDirectory=<workspace>/artifacts/test-results
        - Upload test results artifact

Key details:

• Enables Developer Mode via registry before running (required for sideloaded MAUI apps)
• Installs Microsoft.WindowsAppSDK MSIX runtime from the NuGet package cache ($env:USERPROFILE/.nuget/packages)
• Filters out ExpectedFailure tests so known-flaky tests don't block CI
• Produces TRX test results and MSBuild binlog as artifacts

device_test_maccatalyst (macos-26)

device_test_maccatalyst:
    name: Run Device Tests (macOS Catalyst)
    runs-on: macos-26
    env:
        CommunityToolkitLibrary_Xcode_Version: '26.6'
    steps:
        - checkout
        - setup-xcode (26.6)
        - setup-dotnet (TOOLKIT_NET_VERSION)
        - dotnet workload install maui
        - dotnet build -t:VSTest
            -f net10.0-maccatalyst
            -r maccatalyst-x64
            -c Release
            -p:IsCI=true
            -p:VSTestTestCaseFilter=Category!=ExpectedFailure
            -p:VSTestLogger=trx%3BLogFileName=test-results.trx
            -p:VSTestResultsDirectory=<workspace>/artifacts/test-results
        - Upload test results artifact

Key details:

• Uses -r maccatalyst-x64 to target the x64 Mac Catalyst runtime
• Selects Xcode via maxim-lobanov/setup-xcode using the job-level CommunityToolkitLibrary_Xcode_Version variable (26.6)
• Same Category!=ExpectedFailure filter and TRX output as Windows

Category!=ExpectedFailure filter

Tests marked with [Trait("Category", "ExpectedFailure")] are excluded from CI runs but still execute locally. This prevents known platform-specific issues from blocking PRs while keeping the tests visible for investigation. Use [Trait("Category", "ExpectedFailure")] instead of [Fact(Skip = "...")] — Skip hides the test entirely and may cause silent regressions.

---

Reflection Removal

Problem

The original device tests used Assembly.GetType() and System.Reflection to access internal types (e.g., AlertDefaults, CameraViewDefaults, MediaElementDefaults, Bounds, WeakReferenceExtensions). On macOS Catalyst, Assembly.GetType() fails to resolve internal types, causing TypeInitializationException in the static reflection helper classes and 128 test failures.

Solution

Two-part fix:

1. InternalsVisibleTo — Added <InternalsVisibleTo Include="CommunityToolkit.Maui.DeviceTests" /> to 5 projects so their internal types are accessible from device tests:

Project	Internal types exposed
CommunityToolkit.Maui.Core	20 *Defaults static classes (AlertDefaults, AvatarViewDefaults, DrawingViewDefaults, etc.), NullableExtensions
CommunityToolkit.Maui	WeakReferenceExtensions, SafeFireAndForgetExtensions, PropertyChangedEventArgsExtensions, CryptographyExtensions, Options (internal ctor + static props)
CommunityToolkit.Maui.Camera	CameraViewDefaults
CommunityToolkit.Maui.MediaElement	MediaElementDefaults, MediaElementOptions (internal ctor + static props), StreamExtensions
CommunityToolkit.Maui.Maps	Bounds, Center, EventIdentifier, EventMessage, InfoWindow, SerializerContext

2. Test rewrites — 7 test files rewritten to use direct type references instead of reflection:

File	Changes
Tests/Core/DefaultsTests.cs	Removed DefaultsReflectionHelper class (~25 lines). All ~68 tests now use direct access (e.g., AlertDefaults.FontSize, RatingViewDefaults.FillColor).
Tests/Camera/CameraTests.cs	CameraViewDefaultsTests uses CameraViewDefaults.CameraFlashMode, .IsTorchOn, etc. directly. Removed using System.Reflection.
Tests/MediaElement/MediaElementTests.cs	Three classes rewritten: MediaElementDefaultsTests → MediaElementDefaults.Speed, etc.; MediaElementOptionsTests → new MediaElementOptions() + direct property access; StreamExtensionsTests → stream.GetMimeType(). Added using CommunityToolkit.Maui.Core.Extensions.
Tests/Maps/MapsTests.cs	Removed MapsReflectionHelper (~35 lines). All 6 test classes use new Bounds(), new Center(), typeof(EventIdentifier), SerializerContext.Default.EventMessage, etc. Added using CommunityToolkit.Maui.Maps.Handlers.
Tests/Extensions/InternalExtensionsTests.cs	Removed MauiExtensionsReflectionHelper (~30 lines). CryptographyExtensionsTests → "string".GetMd5Hash(). WeakReferenceExtensionsTests → weakRef.GetTargetOrDefault(). SafeFireAndForgetExtensionsTests → task.SafeFireAndForget().
Tests/Additional/ExtensionsAndEventArgsTests.cs	PropertyChangedEventArgsExtensionsTests → "propertyName".IsOneOf(...) directly. NullableExtensionsTests → typeof(x).IsNullable() directly.
Tests/Additional/PlatformBehaviorsAndServicesTests.cs	OptionsTests → new Options() and Options.ShouldSuppressExceptionsIn* directly. Removed the reflective CreateOptions / GetInternalStaticProperty helpers.

xUnit Analyzer Fixes

During the rewrite, several xUnit analyzer violations were also fixed:

• xUnit2004: Assert.Equal(true/false, ...) → Assert.True(...) / Assert.False(...) wherever the value is statically a bool. Converter tests whose result is object? keep Assert.Equal(true, result); the analyzer does not flag those, and the assertion also checks the boxed type.
• xUnit2000: Swapped expected/actual arguments so constants are the expected argument. Assert.Equal(StateViewDefaults.StateKey, string.Empty) is intentionally in this order: StateKey is a const and string.Empty is not.
• xUnit1031: Replaced tcs.Task.Result with await tcs.Task
• xUnit1004: No longer suppressed in the project; the four CameraInfo tests use the ExpectedFailure trait instead of Skip.

xUnit1051 (TestContext.Current.CancellationToken) does not apply: the project is on xunit v2, where TestContext does not exist.

---

Architecture

flowchart TB
    subgraph Boot["MAUI App Boot"]
        MP["MauiProgram.cs"]
        VTR["UseVisualTestRunner(conf => conf
        .AddCliConfiguration()
        .AddConsoleResultChannel()
        .AddTestAssembly(...)
        .AddXunit())"]
        MP --> VTR
    end

    subgraph Runner["DeviceRunners Visual Runner"]
        D1["1. Discover tests via DeviceRunners.VisualRunners.Xunit"]
        D2["2. Execute tests on-device"]
        D3["3. Display pass/fail UI with per-test details"]
        D4["4. Stream results via TCP (VSTest target) or console"]
        D1 --> D2 --> D3 --> D4
    end

    subgraph CI["dotnet build -t:VSTest (CI)"]
        C1["1. Build"]
        C2["2. Deploy to device"]
        C3["3. Launch app"]
        C4["4. App auto-starts, connects via TCP port 16384"]
        C5["5. NDJSON events streamed back to CLI"]
        C6["6. TRX file generated"]
        C1 --> C2 --> C3 --> C4 --> C5 --> C6
    end

    Boot --> Runner
    Runner --> CI

Loading

---

Test Categories Implemented

• Smoke Tests — Verifies the MAUI application boots correctly (app, window, dispatcher, services all available)
• Platform Detection — Confirms DeviceInfo.Platform, DeviceInfo.Idiom, and OS version reporting using #if ANDROID / #elif IOS / #elif MACCATALYST / #elif WINDOWS conditional compilation
• Handler Tests — Validates that MAUI controls (Label, Button, Entry, StackLayout) create platform handlers via element.ToHandler(context) on the main thread
• Behaviors — TextValidationBehavior tests (valid/invalid lengths, regex patterns, null values)
• Converters — InvertedBoolConverter, IntToBoolConverter, IsNullConverter, color converters, and more
• Views — Popup and PopupOptions property defaults, event wiring, and configuration
• Core — DockLayoutManager measurement, FolderPickerResult, FileSaverResult, DrawingLine, math operators, primitives, and extension methods
• Additional — Color/theme tests, platform behaviors, service interactions, and extension method tests
• Defaults — All *Defaults static classes in CommunityToolkit.Maui.Core (20 classes, ~68 tests for const fields and static properties)
• Extensions — Internal extension methods (CryptographyExtensions.GetMd5Hash, WeakReferenceExtensions.GetTargetOrDefault, SafeFireAndForgetExtensions.SafeFireAndForget, PropertyChangedEventArgsExtensions.IsOneOf)
• Camera / Maps / MediaElement — Package-specific integration tests including handler types, serialization, and options configuration

---

Design Decisions & Conventions

• DeviceRunners, not custom runners — Uses the community-standard DeviceRunners packages. Do not build custom XunitFrontController wrappers or DeviceRunner classes — DeviceRunners handles discovery, execution, visual runner UI, result collection, and the headless VSTest target.
• No parallelization — [assembly: CollectionBehavior(DisableTestParallelization = true, MaxParallelThreads = 1)] prevents UI threading issues
• Handler creation via element.ToHandler(context) — Uses Application.Current?.Handler?.MauiContext to create handlers directly without replacing window.Page. Must run on MainThread.InvokeOnMainThreadAsync.
• No builder.UseMauiApp<App>() — DeviceRunners registers its own VisualRunnerApp via UseVisualTestRunner
• GenerateTestingPlatformEntryPoint>false — DeviceRunners provides Program.Main
• All packages from nuget.org — No custom feeds; XHarness is a transitive dependency
• dotnet build -t:VSTest, not dotnet test — The root global.json sets test.runner to Microsoft.Testing.Platform for the unit test projects. The DeviceTests project is xunit v2/VSTest (DeviceRunners.Testing.Targets sets <IsTestingPlatformApplication>false</IsTestingPlatformApplication>), so dotnet test rejects it in that mode. Invoking the VSTest target directly lets both coexist without changing global.json.
• InternalsVisibleTo for internal type access — Device tests reference internal types via <InternalsVisibleTo> instead of Assembly.GetType() reflection. This avoids TypeInitializationException on macOS Catalyst and makes tests faster and more maintainable.
• ExpectedFailure trait for known issues — Use [Trait("Category", "ExpectedFailure")] instead of [Fact(Skip = "...")]. CI filters with Category!=ExpectedFailure so known failures don't block PRs, but tests still run locally for investigation.

---

How to Use and Integrate Device Testing

Prerequisites

• .NET 10 SDK with MAUI workload: dotnet workload install maui
• Android: Android emulator or physical device
• iOS / Mac Catalyst: macOS with Xcode installed
• Windows: Windows 10/11 with Windows App SDK

Running Tests

VSTest target (CI / Headless, Recommended)

dotnet build src/CommunityToolkit.Maui.DeviceTests/CommunityToolkit.Maui.DeviceTests.csproj -t:VSTest -f net10.0-android
dotnet build src/CommunityToolkit.Maui.DeviceTests/CommunityToolkit.Maui.DeviceTests.csproj -t:VSTest -f net10.0-ios
dotnet build src/CommunityToolkit.Maui.DeviceTests/CommunityToolkit.Maui.DeviceTests.csproj -t:VSTest -f net10.0-maccatalyst
dotnet build src/CommunityToolkit.Maui.DeviceTests/CommunityToolkit.Maui.DeviceTests.csproj -t:VSTest -f net10.0-windows10.0.19041.0

# With filter
dotnet build ... -t:VSTest -f net10.0-android "-p:VSTestTestCaseFilter=FullyQualifiedName~StatusBarBehavior"

# Exclude known-failing tests (CI pattern)
dotnet build ... -t:VSTest -f net10.0-maccatalyst "-p:VSTestTestCaseFilter=Category!=ExpectedFailure"

DeviceRunners.Testing.Targets handles build, deploy, run, and TRX automatically. No extra tooling needed.

dotnet test does not work for this project while global.json selects Microsoft.Testing.Platform; it stops with The following test projects are using VSTest test runner.

Visual Runner (IDE / Interactive)

dotnet build src/CommunityToolkit.Maui.DeviceTests/CommunityToolkit.Maui.DeviceTests.csproj -f net10.0-android -t:Run

Or press F5 in VS Code / Visual Studio. The DeviceRunners visual runner displays test results in-app.

Adding New Device Tests

1. Create a test class in src/CommunityToolkit.Maui.DeviceTests/Tests/<Category>/:

using Xunit;

namespace CommunityToolkit.Maui.DeviceTests.Tests.Behaviors;

public class MyNewBehaviorTests
{
    [Fact]
    public void MyBehavior_DefaultValue_IsCorrect()
    {
        var behavior = new MyBehavior();
        Assert.Equal(expectedValue, behavior.SomeProperty);
    }
}

2. For handler-dependent tests, use MainThread.InvokeOnMainThreadAsync + element.ToHandler(context):

[Fact]
public async Task MyControlHandlerIsCreated()
{
    var control = new MyControl();
    var handler = await MainThread.InvokeOnMainThreadAsync(() =>
    {
        var context = Application.Current?.Handler?.MauiContext;
        return control.ToHandler(context);
    });
    Assert.NotNull(handler.PlatformView);
}

3. For platform-specific tests, use conditional compilation:

#if ANDROID
[Fact]
public void AndroidSpecificBehavior() { ... }
#elif IOS
[Fact]
public void IOSSpecificBehavior() { ... }
#endif

4. For tests accessing internal types, ensure InternalsVisibleTo is in the target project's .csproj:

<ItemGroup>
    <InternalsVisibleTo Include="CommunityToolkit.Maui.DeviceTests" />
</ItemGroup>

Then reference the types directly — no reflection needed:

using CommunityToolkit.Maui.Core;

[Fact]
public void AlertDefaults_FontSize_Is14()
{
    Assert.Equal(14d, AlertDefaults.FontSize);
}

---

Project Structure

src/
└── CommunityToolkit.Maui.DeviceTests/
    ├── MauiProgram.cs                  # UseVisualTestRunner configuration
    ├── GlobalUsings.cs                 # xunit CollectionBehavior (sequential)
    ├── SmokeTests.cs                   # App boot verification
    ├── PlatformDetectionTests.cs       # Platform identity tests
    ├── HandlerTests.cs                 # Control handler creation tests
    ├── Tests/
    │   ├── Additional/                 # Cross-cutting toolkit tests
    │   ├── Behaviors/                  # Behavior tests
    │   ├── Camera/                     # Camera package tests
    │   ├── Converters/                 # Converter tests
    │   ├── Core/                       # Core library tests (Defaults tests)
    │   ├── Extensions/                 # Internal extension tests
    │   ├── Maps/                       # Maps package tests
    │   ├── MediaElement/               # MediaElement package tests
    │   └── Views/                      # View/control tests
    ├── Platforms/                      # Platform entry points
    ├── Resources/                      # App icon, splash
    └── Properties/
        └── launchSettings.json

Platforms Tested

Status as of 2026-10-01 (commit 5e98149):

Platform	TFM	Result
Windows	net10.0-windows10.0.19041.0	✅ 877 passed, 0 failed (local run, Category!=ExpectedFailure). The 11 ExpectedFailure tests are excluded; 10 of them fail on Windows.
Android	net10.0-android	Builds with no new warnings. Previously reported passing; tests not re-run after the October changes.
iOS Simulator	net10.0-ios	Previously reported passing; not re-built or re-run after the October changes.
Mac Catalyst	net10.0-maccatalyst	❌ CI job currently fails before the tests run: the runner installs the MacCatalyst 27.0 SDK, which requires Xcode 27, while the job selects Xcode 26.6. The same mismatch breaks the macOS sample app build and is not caused by the test code. Tests last passed in CI in August.

[Copilot]

Pull request overview

This PR introduces a new on-device testing harness for the .NET MAUI Community Toolkit by adding a MAUI “DeviceTests” app that discovers and runs xUnit tests via reflection, plus a companion unit test project that validates the runner behavior. It also updates solution definitions and test-runner configuration to support the new test projects.

Changes:

• Added CommunityToolkit.Maui.DeviceTests MAUI app project (platform entrypoints, resources, smoke/platform/handler tests, plus many toolkit API tests).
• Added CommunityToolkit.Maui.DeviceTests.UnitTests project to validate InAppTestRunner behavior (pass/fail/skip/theory/disposal/cancellation).
• Updated .slnx files and repo test configuration (global.json, Directory.Build.props) to include/enable the new testing setup.

Reviewed changes

Copilot reviewed 50 out of 53 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/CommunityToolkit.Maui.slnx	Adds build configuration mappings and includes the new device test projects in the main solution.
samples/CommunityToolkit.Maui.Sample.slnx	Mirrors .slnx configuration/mappings and includes the device test project in the sample solution.
global.json	Enables Microsoft.Testing.Platform mode via the test section.
Directory.Build.props	Updates commentary around MTP enablement; removes obsolete TestingPlatformDotnetTestSupport usage.
src/CommunityToolkit.Maui.DeviceTests/CommunityToolkit.Maui.DeviceTests.csproj	Adds the new MAUI device test app project and its dependencies.
src/CommunityToolkit.Maui.DeviceTests/App.cs	App UI that runs tests on startup and displays live output + summary.
src/CommunityToolkit.Maui.DeviceTests/MauiProgram.cs	MAUI app builder setup for the device test host.
src/CommunityToolkit.Maui.DeviceTests/InAppTestRunner.cs	Reflection-based xUnit test discovery/execution and summary/output capture.
src/CommunityToolkit.Maui.DeviceTests/GlobalUsings.cs	Disables xUnit parallelization at assembly level.
src/CommunityToolkit.Maui.DeviceTests/SmokeTests.cs	Basic boot/DI/dispatcher smoke tests.
src/CommunityToolkit.Maui.DeviceTests/PlatformDetectionTests.cs	Device platform/idiom/version checks with per-platform assertions.
src/CommunityToolkit.Maui.DeviceTests/HandlerTests.cs	Handler creation tests for common controls with polling for handler attachment.
src/CommunityToolkit.Maui.DeviceTests/README.md	Local run instructions and structure notes for the device test app.
src/CommunityToolkit.Maui.DeviceTests/Properties/launchSettings.json	Windows launch profile for local debugging.
src/CommunityToolkit.Maui.DeviceTests/Resources/Splash/splash.svg	Splash screen asset for the device test app.
src/CommunityToolkit.Maui.DeviceTests/Resources/AppIcon/appicon.svg	App icon asset for the device test app.
src/CommunityToolkit.Maui.DeviceTests/Resources/AppIcon/appiconfg.svg	App icon foreground asset for the device test app.
src/CommunityToolkit.Maui.DeviceTests/Platforms/Android/AndroidManifest.xml	Android app manifest for the device test host.
src/CommunityToolkit.Maui.DeviceTests/Platforms/Android/MainActivity.cs	Android MAUI activity entry point.
src/CommunityToolkit.Maui.DeviceTests/Platforms/Android/MainApplication.cs	Android MAUI application entry point.
src/CommunityToolkit.Maui.DeviceTests/Platforms/iOS/Info.plist	iOS app metadata for the device test host.
src/CommunityToolkit.Maui.DeviceTests/Platforms/iOS/Program.cs	iOS main entry point.
src/CommunityToolkit.Maui.DeviceTests/Platforms/iOS/AppDelegate.cs	iOS MAUI app delegate.
src/CommunityToolkit.Maui.DeviceTests/Platforms/MacCatalyst/Info.plist	Mac Catalyst app metadata for the device test host.
src/CommunityToolkit.Maui.DeviceTests/Platforms/MacCatalyst/Program.cs	Mac Catalyst main entry point.
src/CommunityToolkit.Maui.DeviceTests/Platforms/MacCatalyst/AppDelegate.cs	Mac Catalyst MAUI app delegate.
src/CommunityToolkit.Maui.DeviceTests/Platforms/Windows/App.xaml	Windows app XAML entry point.
src/CommunityToolkit.Maui.DeviceTests/Platforms/Windows/App.xaml.cs	Windows MAUI app class implementation.
src/CommunityToolkit.Maui.DeviceTests/Platforms/Windows/Package.appxmanifest	Windows packaging manifest (even though project uses unpackaged mode).
src/CommunityToolkit.Maui.DeviceTests/Platforms/Windows/app.manifest	Windows app manifest (DPI awareness etc.).
src/CommunityToolkit.Maui.DeviceTests/Tests/Maps/MapsTests.cs	Reflection-based tests for internal Maps handler types and serializer context.
src/CommunityToolkit.Maui.DeviceTests/Tests/Extensions/InternalExtensionsTests.cs	Reflection-based tests for internal extension methods (crypto/weakref/safe fire-and-forget).
src/CommunityToolkit.Maui.DeviceTests/Tests/Core/PrimitivesTests.cs	Enum + primitives/event args tests for core primitives.
src/CommunityToolkit.Maui.DeviceTests/Tests/Core/LayoutsTests.cs	Layout manager measure/arrange tests with test layout/view implementations.
src/CommunityToolkit.Maui.DeviceTests/Tests/Core/ExtensionsTests.cs	Tests for core extension methods (observable collections, color conversions, math).
src/CommunityToolkit.Maui.DeviceTests/Tests/Core/EssentialsTests.cs	Tests for result/exception primitives (FolderPicker/FileSaver/SpeechToText).
src/CommunityToolkit.Maui.DeviceTests/Tests/Core/DrawingLineAndMathOperatorTests.cs	Tests for drawing primitives and math operator primitives.
src/CommunityToolkit.Maui.DeviceTests/Tests/Core/DefaultsTests.cs	Reflection-based tests for internal Core Defaults classes/values.
src/CommunityToolkit.Maui.DeviceTests/Tests/Core/AdditionalCoreTests.cs	Additional tests for drawing/rating/toast options + EnsureSuccess behavior.
src/CommunityToolkit.Maui.DeviceTests/Tests/Camera/CameraTests.cs	Camera primitives/defaults tests; some tests explicitly skipped due to hardware needs.
src/CommunityToolkit.Maui.DeviceTests/Tests/Additional/ExtensionsAndEventArgsTests.cs	Tests covering various extensions/event args/markup extensions + reflection for internal helpers.
src/CommunityToolkit.Maui.DeviceTests/Tests/Additional/ConvertersAndBehaviorsTests.cs	Tests for converters and behaviors defaults/basic behavior.
src/CommunityToolkit.Maui.DeviceTests/Tests/Additional/ColorAndThemeTests.cs	Tests for color conversion extensions and theme-related helpers.
src/CommunityToolkit.Maui.DeviceTests/Tests/Additional/AdditionalMauiTests.cs	Additional tests for animations/layouts/state container/etc.
src/CommunityToolkit.Maui.DeviceTests.UnitTests/CommunityToolkit.Maui.DeviceTests.UnitTests.csproj	Adds the new runner unit test project (MTP + coverage) and links InAppTestRunner.cs.
src/CommunityToolkit.Maui.DeviceTests.UnitTests/InAppTestRunnerTests.cs	Unit tests validating runner exit codes, discovery, output, cancellation, disposal, etc.
src/CommunityToolkit.Maui.DeviceTests.UnitTests/Fixtures/TestFixtures.cs	Internal fixtures used to drive runner behavior (pass/fail/skip/theory/etc.).
src/CommunityToolkit.Maui.DeviceTests.UnitTests/xunit.runner.json	Disables xUnit parallelization for the unit test project.

[TheCodeTraveler]

Does this leverage the open Device Tests PR? Does it include its logic/tests? Or does it replace it?

#1444

[ne0rrmatrix]

Does this leverage the open Device Tests PR? Does it include its logic/tests? Or does it replace it?

This PR is an independent implementation that replaces #1444 rather than building on it. It does not include any code or tests from that PR.

The key difference is the approach: #1444 relies on the DeviceRunners/XHarness infrastructure (external packages from private NuGet feeds, UseXHarnessTestRunner/UseVisualTestRunner in MauiProgram, and Microsoft.DotNet.XHarness.CLI for CI). This PR instead uses a custom InAppTestRunner — a lightweight, reflection-based test discovery and execution engine that runs xunit tests directly inside the MAUI app with no external runner dependencies or private feeds.

It also adds a companion CommunityToolkit.Maui.DeviceTests.UnitTests project that validates the runner itself (pass/fail/skip/theory/disposal/cancellation scenarios), and targets net10.0 with xunit.v3 + Microsoft.Testing.Platform.

If we prefer the DeviceRunners/XHarness approach from #1444, this PR could be adapted to use that infrastructure instead. Happy to align with whichever direction the team prefers.

[TheCodeTraveler]

I would recommend following the same approach that the .NET MAUI team uses for their Device Tests. That will be the most-robust and future-proof way of proceeding.

https://github.com/dotnet/maui/tree/main/src/Controls/tests/DeviceTests

[ne0rrmatrix]

I would recommend following the same approach that the .NET MAUI team uses for their Device Tests. That will be the most-robust and future-proof way of proceeding.

https://github.com/dotnet/maui/tree/main/src/Controls/tests/DeviceTests

Updated as suggested

[pictos]

I think you should use the https://github.com/mattleibow/DeviceRunners project from @mattleibow, that is what .net maui uses and can run on CI

[ne0rrmatrix]

I will migrate to @matt-bartholomew project design

[matt-bartholomew]

I will migrate to @matt-bartholomew project design

I think you meant @mattleibow

[ne0rrmatrix]

I have refactored it as suggested

[pictos]

using the package, as nuget a package there's no need for import this tool. I'm using it on my project here https://github.com/pictos/LogViewer/tree/main/LogViewer.DeviceTests. It's an app but you can take the key idea on how to use it on CI

[ne0rrmatrix]

I removed the dotnet tools json file. You are right. It builds and runs fine without it. Good catch.

[pictos]

maybe they will be needed for android on CI, if so, let me know that I can look into my previous research what info is needed

[ne0rrmatrix]

I followed what I saw and only added for windows and macos. I do not know if we have access to emulators for android/ios testing? For local testing android/ios works.

[Copilot]

Copilot review overview

🟡 Changes recommended

The Catalyst CI job is currently guaranteed to fail from an acknowledged workload and Xcode version mismatch.

Review effort: Balanced
Findings: 1 · 1 · 2

Open (4)

• Pin workload SDK to the selected Xcode version · New
• Initialize toolkit packages and platform extensions in the test app
• Use VSTestTestCaseFilter instead of unsupported --filter · New
• Document supported DeviceRunners VSTest integration · New

Resolved since last review (4)

• Align Android minimum version with MediaElement dependency
• Correct the comment describing MSIX packaging
• Use the supported VSTest command in scoped instructions
• Use VSTest commands instead of dotnet test in guidance

Previously missed (1)

In code that hasn't changed since last review

Fix xUnit2032 violations instead of suppressing the analyzer

src/​CommunityToolkit.Maui.DeviceTests/​CommunityToolkit.Maui.DeviceTests.csproj:32

This new test project globally suppresses xUnit2032 even though the repository explicitly promotes that diagnostic to an error in Directory.Build.props:225; none of these tests are pre-existing in this project. Update the affected assertions to the analyzer-supported form rather than bypassing the repository-wide analyzer policy.

[Copilot]

Copilot review overview

🟡 Changes recommended

The Mac Catalyst device-test job remains red due to its Xcode/workload mismatch, and the new documentation contains conflicting runner guidance.

Review effort: Balanced
Findings: 1 · 4

Open (5)

• Initialize toolkit packages and platform extensions in the test app
• Correct comment attributing generated entry point behavior · New
• Clarify MSIX packaging does not enable notification test support · New
• Document supported DeviceRunners VSTest integration
• Use VSTestTestCaseFilter instead of unsupported --filter

Resolved since last review (1)

• Pin workload SDK to the selected Xcode version

[Copilot]

Copilot review overview

🔵 Needs a closer look

Several tests are vacuous, nondeterministic, or locale-sensitive, while the Catalyst CI job also remains incompatible with its selected Xcode version.

Review effort: Balanced
Findings: 1

Open (1)

• Initialize toolkit packages and platform extensions in the test app

Resolved since last review (4)

• Clarify MSIX packaging does not enable notification test support
• Correct comment attributing generated entry point behavior
• Document supported DeviceRunners VSTest integration
• Use VSTestTestCaseFilter instead of unsupported --filter

Previously missed (4)

In code that hasn't changed since last review

Use invariant culture for locale-independent alpha formatting test

src/​CommunityToolkit.Maui.DeviceTests/​Tests/​Core/​ExtensionsTests.cs:77

ToRgbaString() formats alpha with the current culture when no culture is supplied. On devices using a decimal-comma locale this returns an alpha such as 0,5, so the invariant 0.5 assertion fails even though the implementation is correct. Pass an explicit invariant culture to keep this device test locale-independent.

Width clamping test uses a child that contributes no width

src/​CommunityToolkit.Maui.DeviceTests/​Tests/​Core/​LayoutsTests.cs:96

This does not exercise width clamping: a Top child contributes only to the measured height, so result.Width is zero and the inequality passes regardless of the width constraint. Dock the oversized child left/right and assert the constrained width so a regression in Math.Min(width, widthConstraint) is detectable.

Height clamping test uses a child that contributes no height

src/​CommunityToolkit.Maui.DeviceTests/​Tests/​Core/​LayoutsTests.cs:109

This does not exercise height clamping: a Left child contributes only to the measured width, so result.Height is zero and the inequality always passes. Use a top/bottom child whose height exceeds the constraint and assert the constrained height.

Weak-reference test passes without verifying target collection

src/​CommunityToolkit.Maui.DeviceTests/​Tests/​Extensions/​InternalExtensionsTests.cs:120

This conditional lets the test pass without asserting the collected-reference behavior whenever the target survives. Collection also happens while the helper's target local is still in scope, which makes that no-op path more likely. Return the weak reference first, force collection after the no-inline helper has exited, and require the target to be gone before checking GetTargetOrDefault().

[ne0rrmatrix]

Re: Copilot review. All four "previously missed" findings are valid and fixed in 6d38df2, along with the Catalyst/Xcode mismatch called out in the summary.

• Locale-dependent alpha formatting (ExtensionsTests.cs): the test now calls ToRgbaString(CultureInfo.InvariantCulture), so it no longer depends on the device locale.
• Width clamping test (LayoutsTests.cs): the oversized child is now docked Left with HorizontalSpacing = 10 and the test asserts Width == 200. The spacing is needed because TestView clamps its own measure to the constraint, so only the trailing spacing pushes the total past it (210) and reaches the Math.Min(width, widthConstraint) clamp.
• Height clamping test (LayoutsTests.cs): same change on the other axis, a Top child with VerticalSpacing = 10, asserting Height == 200.
• Weak-reference test (InternalExtensionsTests.cs): the conditional is gone. The target is created by a no-inline helper on another thread, collection is forced after that helper has returned (retried up to 10 times), and the test asserts the target is gone before asserting GetTargetOrDefault() returns null.
• Catalyst CI job: the job had its own 26.6 Xcode pin, which the Xcode 27 update on main did not touch, and failed with "requires Xcode 27.0". It now runs on the xcode-27 runner with CommunityToolkitSample_Xcode_Version, mirroring the sample build job that already passes with the same workload update.

The affected test classes pass locally on Mac Catalyst (26 passed, 0 failed). The workflow change is verified by the CI run on this commit.

The remaining open thread on MauiProgram.cs (Maps and Windows Snackbar initialization) is unchanged and still waiting on a maintainer decision.

[Copilot]

Copilot review overview

🟡 Changes recommended

The Catalyst CI job currently fails during Release linking before any device tests execute.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Fix Catalyst test-app build failures from MAUI linker errors · New
• Initialize toolkit packages and platform extensions in the test app

[TheCodeTraveler]

Hey James!

I just opened a new Draft PR (I have another big commit coming tomorrow) that builds on top of the work you did in this PR:

• Adds fixes to the library where the device tests found a bug
• Links Fixed Issues that will close when it’s merged
• Adds additional device tests to increase code coverage

You’re welcome to give it a glance when you get a chance. It’s a massive PR, similar to this one. And it is branched off of the branch for this PR.

I recommend we close this PR and move forward with #3344 since it includes the new bug fixes. Once we merge it we can release v16.0.0 since it does add some subtle breaking changes in how executions are thrown/not thrown for certain classes (and in include those breaking changes in the Release Notes).

[ne0rrmatrix]

Yes I agree. I kind of stalled out not sure of exactly where we should go. Appreciate you taking up the mantle and getting it done. I have been looking forward to being able to use device testing for several years and I am happy to see you continuing it. I am closing this PR in favor of the one you started.