A Payer's Self-Audit Checklist for FHIR API Conformance
A Payer’s Self-Audit Checklist for FHIR API Conformance
This is a checklist a payer engineering or compliance team can run against your own FHIR APIs without buying anything. Twenty-five items, grouped by audit phase, in roughly the order you should run them. Each item is concrete enough to hand to a junior engineer with a single-sentence question: can we answer yes, with evidence?
The checklist assumes you operate one or more of the FHIR APIs governed by CMS-0057-F (Patient Access expanded, Provider Access, Payer-to-Payer, Prior Authorization) or its predecessor CMS-9115-F (Patient Access, Provider Directory, Drug Formulary). CMS-0057-F and its predecessor CMS-9115-F together require 5 FHIR APIs — four under CMS-0057-F plus CMS-9115-F’s Provider Directory.
Run it. Find the gaps. Fix them. Then ask whether running this every month is something a person should be doing manually.
Phase 1 — Discovery and metadata (the audit starts here)
These are the checks any external auditor or third-party app developer can run against your public endpoints in 30 seconds. If you fail one of these, you fail loudly.
1. GET /metadata returns a valid CapabilityStatement. Status 200, content-type application/fhir+json, parses as a v4.0.1 CapabilityStatement resource without errors.
2. CapabilityStatement.fhirVersion matches the version you certified against. A common drift source: a server upgrade silently bumped this to 4.0.1 but your Implementation Guide pins 4.0.0. That is Spec Version Mismatch — see our drift taxonomy for what auditors do with that finding.
3. CapabilityStatement.implementationGuide lists every IG you claim conformance with, version-pinned. “We support US Core” without hl7.fhir.us.core@6.1.0 (or whatever version) is not a conformance claim. It is a vibe.
4. Every advertised resource has a profile reference. A CapabilityStatement.rest.resource entry without a profile is unenforceable; you have not actually committed to any structure.
5. /.well-known/smart-configuration returns 200 and lists the auth methods your IG requires. Da Vinci PAS requires Backend Services with private_key_jwt. CARIN Blue Button supports SMART App Launch with code flow. If your .well-known/smart-configuration advertises client_secret_basic and the IG requires private_key_jwt, that is Auth Deviation drift.
Phase 2 — Structural conformance against the IG
This is the work of comparing your declared structure against the published Implementation Guide. It is doable manually for a single resource and exhausting for a full payer surface.
6. Every mustSupport: true element with min >= 1 in your IG is declared in your CapabilityStatement or profile snapshot. Absent mandatory elements are Mandatory Element Removal drift — the highest-severity category.
7. Every required element’s type matches the IG. Claim.created declared as dateTime in CARIN Blue Button but observed as date? That is Type/Cardinality Change drift.
8. Every required element’s min and max cardinality matches the IG. Patient.identifier declared as 1..* in US Core but observed as 0..1 is the same category, more dangerous: it permits the API to return shapes the spec explicitly forbids.
9. Every profile reference resolves. meta.profile URLs that 404 are an audit finding regardless of what the underlying resource looks like.
10. Every valueSet binding strength is declared and matches the IG. Required bindings to a value set you have not loaded are a runtime failure waiting to happen.
Phase 3 — Authorization conformance
The category regulators notice fastest, because it gates whether a third-party app can legally access your API at all.
11. Token endpoints reject none as an auth method. This is a one-line check. If your token endpoint accepts unauthenticated requests, stop reading this checklist and fix that today.
12. Scope strings match the SMART App Launch v2 grammar exactly. patient/Patient.read is correct; patient.Patient.read is not. A subtle but real source of cross-implementation breakage.
13. PKCE is enforced for public clients. Authorization-code flows without code_challenge/code_verifier parameters are an audit finding under SMART v2.
14. Refresh tokens have a documented lifetime and rotation policy. “We use long-lived tokens” is a finding waiting to be written up.
15. Audit logging captures every authorization decision. Not the same as application logs — see why application logs aren’t compliance evidence.
Phase 4 — Behavioral conformance
The category most easily missed by schema-only validators. Your CapabilityStatement declares a behavior; your runtime delivers a different one.
16. Every advertised search parameter actually filters results. Practitioner?name=Smith and Practitioner?name=Jones should return different result sets. If they return identical results, the search parameter is decorative — Endpoint Behavioral Change drift.
17. Pagination links resolve and return the correct subsequent page. Bundle.link[rel="next"] URLs that 404 or return the same first page are a common bulk-export bug.
18. _include and _revinclude parameters return the resources they claim to. A Patient?_include=Patient:link query that drops the linked resources is a behavioral break that a schema validator will not catch.
By item 18, the answer to “should we be running this monthly by hand” is usually “no.” A pipeline of probes that knows the IG and the runtime is the right shape of solution. Most teams have that thought somewhere around item 18.
Phase 5 — Operational evidence
These items are the bridge from “we are conformant today” to “we can prove we were conformant on a specific historical date.” That distinction is what an auditor cares about.
19. You can produce, on demand, the exact CapabilityStatement your API returned at any timestamp in the last 12 months. Most payers cannot. They can produce the current one and a guess about prior ones.
20. Every conformance check you have run is signed and timestamped. Unsigned reports are not evidence — they are claims. An Ed25519 signature with a published public key is reproducible by any auditor.
21. Conformance reports are linked into a tamper-evident chain. Each report references the hash of the prior one. Editing any historical report breaks every downstream link, which is exactly the property an audit committee wants. See our evidence chain post for one implementation.
22. The chain is replayable by an external party using only public information. If your evidence is “trust our internal audit log,” the auditor will not. If it is “here is the signed verdict and the public key — verify it yourself,” that conversation goes differently.
Phase 6 — Continuous monitoring posture
The hardest items, because they ask whether you have built a process or just done a one-time fix.
23. You have an automated alert for any change in CapabilityStatement. Not “we will notice in QA” — an actual alert with an on-call rotation and a runbook.
24. You have an automated alert for any change in your IG version pinning. A developer bumping hl7.fhir.us.core@6.1.0 to 6.2.0 should generate a ticket, not a silent merge.
25. You can answer the question “what changed last Tuesday at 03:00 UTC” with a signed artifact, not a memory of a Slack thread. This is the question CMS will eventually ask. The team that has the answer already has the evidence chain. The team that does not has homework.
What this checklist is not
This is not a HITRUST control list, a SOC 2 audit framework, or a HIPAA Security Rule walkthrough. Those are different artifacts with different scopes. This is specifically the FHIR API conformance posture under CMS-0057-F and CMS-9115-F — the technical surface CMS will examine when they audit your interoperability obligations.
A clean run of these 25 items does not make you HITRUST-certified. It makes you defensibly conformant on the specific obligations the rule names.
What we built
If by item 18 you concluded that doing this monthly by hand is not the right shape of work, that is the conclusion most teams reach. Tessara is the productized version of this checklist: continuous, automated, cryptographically signed conformance evidence for the FHIR APIs CMS-0057-F and CMS-9115-F mandate.
We probe only public /metadata and declared profile snapshots — the same data the auditor will pull. The difference is we do it every day, on a signed chain, against the IG versions you have pinned. See pricing or contact us.
References: HL7 FHIR R4 CapabilityStatement, SMART App Launch v2, US Core IG, Da Vinci PAS IG, CARIN Blue Button IG.