Skip to content

Testing Strategy

New since Warden Supreme 1.0.5/1.1.4

Warden Supreme 1.0.5/1.1.4 add the at.asitplus.warden:generator test utility. It replaces the home-grown certificate chain builders previously needed for generated Android attestations and is exercised by Warden Supreme's own test suite. See Attestation Generator for details.

Attestation tests should exercise the full pipeline without making production policy more permissive. Keep test trust anchors separate, generate realistic evidence for automation, and make non-production environments behave like production. Automated tests can then run offline in CI, while end-to-end tests cover real devices and staged services.

Principles

  • Separate Trust Anchors — Use distinct roots for test identities so production policies stay strict and verifiable. Never reuse production roots or signer digests in tests.
  • Record and Replay — Capture real attestation inputs and verification outcomes (e.g., with Warden Supreme’s WardenDebugAttestationStatement as explained in Debugging) and replay them offline for regression suites. This yields deterministic tests across CI and local runs.
  • Provisioning Realities — On Android, Remote Key Provisioning can exhaust offline key pools during load; pre‑connect devices and warm pools before tests. Prefer emulators running Android 12 or below, and use a separate trust anchor for automation.
  • Stage Fidelity on iOS — App Attest has sandbox and production environments with different AAGUIDs, producing attestation statements with properties set accordingly. Keep configurations consistent with the active stage to avoid cross‑stage contamination.

Test Clients and Staging Pattern

Load tests and monitors must exercise attestation end to end without weakening production policy. Use a test app whose cryptographic identity is distinct from the production app and admit it under a dedicated trust anchor.

  • Configure a second app entry (e.g., AndroidAttestationConfiguration.AppData) with:
    • A different packageName
    • A deliberately obvious, non‑production signing certificate digest
    • An androidVersionOverride that is clearly out of band (e.g., very high) for easy detection
  • Set up a test PKI:
    • A root whose public key is registered as a trustAnchorOverride
    • An intermediate that issues short‑lived leaves (the attestation certificate chain used by Android is always at least three certificates long, and its exact shape must match the claimed security level — see the warning under Automated Attestation Tests)
    • A leaf certificate generated on the fly by the test client so validity is always fresh
  • Admit exactly two identities at the service:
    • The production app under the normal policy and roots
    • The test app under the test root and explicit allowlist
  • Protect the test root private key rigorously; never issue test binding certificates that could be confused with production artefacts or grant production access.

This preserves production policy while enabling full-fidelity tests of binding, boot-state enforcement, and application identity. Android-only fleet testing is often sufficient because iOS hardware behaves consistently.

Staging Environments

Most deployments run one or two non-production stages before production. This section uses T (test), Q (staging), and P (production).

  • P admits only real production apps under production trust anchors. Nothing else validates there except for monitoring and the occasional load tests (see above).
  • T and Q additionally configure faux apps (see Test Clients and Staging Pattern), each with its own custom trust anchors and signer digests.
  • T and Q mirror whatever proxy configuration is needed to reach external resources such as revocation lists.
  • T and Q may point at custom, hardcoded revocation list sources so automated tests can control revocation state (see Flexible Android Revocation Configuration).

Since the custom trust anchors live only on T and Q, any attestation proof minted against them is rejected on P by construction.

Automated Attestation Tests

Recorded evidence catches regressions against devices already seen in the wild. Generated evidence complements it by covering every property enforced by policy, including combinations that may be difficult to obtain from a real device. Each case is signed and placed in a valid chain below the test root. P cannot validate it because P does not trust this root.

This applies the Test Clients and Staging Pattern to individual policy checks: feed each one acceptable values, rejected values, and the combinations worth keeping an eye on.

Generating Attestation Statements

The Attestation Generator, available since Warden Supreme 1.0.5/1.1.4, creates statements and matching certificate chains from Kotlin or the command line. It supports factory-provisioned and remotely provisioned chains, along with valid and malformed statement properties. Point it at the test root and production will remain blissfully unaware of the resulting proofs.

Generated chains must match the claimed security level

Starting with Warden Supreme 1.0.2, the verifier now derives the attestation security level from the certificate chain when asserting SecurityLevel.STRONGBOX and requires it to match the level advertised in the attestation extension (keymasterSecurityLevel). Chain generators must therefore shape their output to match the claimed level:

  • StrongBox (factory-provisioned): ROOT → FACTORY_INTERMEDIATE → ATTESTATION → TARGET (four certificates).
    • The factory intermediate's subject must carry a serial number (OID 2.5.4.5) and the title StrongBox (OID 2.5.4.12).
    • The same is true for TEE Security level, but this is not hard-asserted, since the root will already indicate a proper hardware-backed attestation.
    • RKP chains are checked just as strictly. The Attestation Generator can produce them since Warden Supreme 1.0.5/1.1.4.
  • Software-backed: ROOT → ATTESTATION → TARGET (three certificates), with no such title on the intermediate.
    • This is not enforced by the verifier.

Tagging Apps via Custom Configuration Properties

Warden Supreme lets you attach customProperties — string key/value maps — to an app's attestation configuration (see Attaching Custom Configuration Properties). This makes the attestation configuration a source of application classification for downstream logic. An app tagged test-only, for example, can receive more verbose logging or feed performance monitors without involving production user data. These tags should not alter the core trust decision.

End-to-End Tests on Real Devices

End-to-end testing starts after the unit and integration suites pass. Warden Supreme already tests captured evidence from real devices alongside generated cases. Service integrators should then deploy release-candidate builds to physical phones that can reach the staged service.

Attestation Collector

The Attestation Collector is also useful for quickly exploring what an individual Android device produces. It is a public diagnostic service, not a replacement for testing your own app, signing identity, backend, and production policy.

Early testing benefits from the broadest available device pool: team devices, partner devices, and the devices already used for compatibility testing. Most failures should match documented quirks, but the fleet may still challenge assumptions about the target audience. In particular, expect more devices with discontinued security updates than initial estimates suggest.

Once the service has reached its target audience, the attestation path usually stabilises. Policy changes can then be checked against recorded evidence and a small representative in-house fleet before staged rollout.

Rolling Out an App Update

One practical rollout pattern is a release build with a hidden switch that points the app at Q. Typically, such a switch swaps endpoint URLs, trust anchors, and the like, but it should never change the app's behaviour. It also makes sense that Q is configured to accept additional signing keys that P does not. This way, a local release build is accepted on Q even though it was never distributed through official channels. With this setup, the rollout is:

  1. Deploy a release build to physical devices.
  2. Flip the hidden switch to Q.
  3. Run end-to-end tests with test credentials.
  4. If everything checks out, distribute the updated app through official channels.

Rolling Out a Service Update

Simpler, because nothing has to be rebuilt:

  1. Deploy the updated service to Q.
  2. Point current production apps at Q by using the hidden switch.
  3. Run end-to-end tests.
  4. Stage the updated service on P, ready to go.
  5. Roll the update out on P if everything checked out on Q.

Sharing Recorded Failures

Downstream integrators made many of Warden Supreme's production workarounds possible by sharing batches of recorded, failed attestation attempts.

Sharing Recorded Failures

We offer downstream integrators to share collected failures with us: Send serializeCompact()-ed debug statements (see Debugging) to support.attestation@a-sit.at, and we will use them to analyse potential device quirks. Use the compact format, never the human-readable one. However, we will not accept any logs that contain IP addresses or any other data that could be used to correlate them with real user data, EVER!

Attestation attempts collected downstream and shared with us are failures causing a hard termination of enrolment processes, which means that the users of these devices never gained access. Hence, no user data is generated for those attempts in the first place, so no personally identifiable data can be associated with those attestation attempts. We also never accepted any logs that contained IP addresses or any other data that could be used to correlate them with real user data. We keep the collected failures private for a simple reason: The configuration needed to replay and analyse the failures includes the downstream integrator's package identifier and signer fingerprints. So while there is no user data involved in failed attempts, the whole downstream configuration is included verbatim. Otherwise, we could not analyse the failures meaningfully. For this reason, we run these collected failures in a private CI pipeline, locally, where there is no risk of exposing them.

Tip

See the dedicated Debugging page for practical details on diagnosing attestation errors.