ONC Certification (g)(10) Standardized API Test Kit v3.0.0

The ONC Certification (g)(10) Standardized API Test Kit v3.0.0 is a major update which adds support for versions of standards approved in the Standard Versions Advancement Process:

• US Core 4.0.0 / USCDI v1
• US Core 5.0.1 / USCDI v2
• SMART App Launch 2.0.0
• Bulk Data 2.0.0

See the wiki for specific details about these test updates.

As a result of these updates, it is necessary to re-run the terminology build process to add support for the terminology used by US Core 4.0.0 and 5.0.1. When starting a test session, users will select which versions of the standards they want to test against.

As a major version release, this update incorporates the following test change that may cause systems that passed v2.3.0 tests to now fail:

• Tests for an EHR Launch with patient scopes have been added when using SMART App Launch 1.0.0 or 2.0.0 (#224).

ONC Certification (g)(10) Standardized API Test Kit v2.3.0

The ONC Certification (g)(10) Standardized API Test Kit v2.3.0 provides several test and user experience improvements based on community feedback.

As a minor version release, this update incorporates the following test changes that may cause systems that passed v2.2.2 tests to now fail:

• Updates 1.5 Unrestricted Resource Type Access test and 2.2 Restricted Resource Types Access test to fix a bug introduced in v2.0 where some searches were not properly formed for systems that require the status search parameter for all searches. (#207)
• Updates 2 Limited App Launch test to enforce the test requirement that the app requests the same scopes as were requested in Standalone Patient App test. Testers no longer enter separate scopes for the Limited App Launch. This ensures that the app user is given granular control over which resources are granted to the app as required by the certification criteria. (#170)

This version also includes a number of other tests updates that allow more flexibility for the systems under test:

• Updates tests that query well-known endpoints to allow servers to redirect clients to alternate URLs instead of serving well-known content directly. Tests were inadvertently updated to deny this behavior beginning in v2.0, and this update reverts that change. (#180)
• Fixes incorrect failure when a resource of a different type is returned from a search that includes a comparator. (#171)

This version includes the following user experience updates that should have no impact on test behavior:

• Provides clarification on Inferno web browser requirements for EHR Practitioner Launch, and documents an approved method for executing this test if the EHR-embedded web browser does not meet Inferno browser requirements.
• Suppresses misleading warning message on Limited App Launch test that states that some scopes were denied, which is the expected behavior of the system under test in this case.
• Adds ability for users running the test kit locally to provide a custom Bulk Data JWKS URL.
• Allows users to represent presets in .erb files, so presets can change based on environment or other settings.

ONC Certification (g)(10) Standardized API Test Kit v2.2.2

The ONC Certification (g)(10) Standardized API Test Kit v2.2.2 provides several test and user experience improvements based on community feedback. As a patch release, systems that pass v2.2.1 tests should continue to pass in v2.2.2.

This version includes a number of tests updates that allow more flexibility for the systems under test:

• Fixes bug where systems that provided a Device Type filter as an input to the Implantable Device tests received an incorrect failure in some cases.
• Removes the SMART Invalid Launch Parameter tests from Additional Tests.
• Updates requests for Well-known configuration endpoints to include an Accept header, per convention.
• Relaxes constraint that prevented systems from returning additional resource types in search responses. Systems may return other resource types if they believe them to be relevant, and tests now provide informational messages in this case.

This version also includes the following user experience updates that should have no impact on test behavior:

• Incorporates UI changes to improve wait, info, warning, and error message readability.
• Includes links to Inferno's Github repository and issues page in testing view.

Please note that in removing the SMART Invalid Launch Parameter test (6.4 in v2.2.1), some test numbers in section 6 Additional Tests have changed. The SMART Invalid Token Request test (6.5 in v2.2.1) is now 6.4 and the Visual Inspection test (6.6 in v2.2.1) is now 6.5. If you maintain an external mapping using the test Ids, please update this mapping to reflect the updated Ids.

ONC Certification (g)(10) Standardized API Test Kit v2.2.1

The ONC Certification (g)(10) Standardized API Test Kit v2.2.1 addresses issues with the Multi-Patient API tests based on community feedback. As a patch release, systems that pass v2.2.0 tests should continue to pass in v2.2.1.

• 5.5.05: Fixes issue where private keys are included in the JWK Set response when Inferno is acting as a Backend Services client. While these private keys are openly published within the GitHub repository to support testing, exposing them in the JWK Set response is forbidden. This issue was introduced in v2.0.0 of the (g)(10) Standardized API tests.
• 5.5.05: Reintroduces Content-Type: application/json header in the JWK Set response. This header was incorrectly removed in the v2.0.0 of the (g)(10) Standardized API Tests.

ONC Certification (g)(10) Standardized API Test Kit v2.2.0

The ONC Certification (g)(10) Standardized API Test Kit v2.2.0 provides several test and user experience improvements based on community feedback.

As a minor version release, this update incorporates the following test changes that may cause systems that passed v2.1.1 tests to now fail:

• 3.1.05: A bug was introduced in v2.0.0 where required SMART core capabilities provided by the 'well-known endpoint' were being incorrectly verified for EHR Launches. Instead of checking that the system stated support for capabilities relevant to EHR launches, they were being checked for capabilities relevant to Standalone launches. This change will not affect systems that use the same FHIR endpoint for both Standalone and EHR launched apps.
• 1.3.06, 3.3.06: OpenID Connect tests were not verifying that the sub claim was provided as required. The tests were updated to ensure that the claim is present and meets stated string length requirements.

This version also includes a number of other tests updates that allow more flexibility for the systems under test:

• 4.13.06: Test was incorrectly providing a 'SKIP' message for systems that do not implement the Medication resource, which is equivalent to 'FAIL', even though this is allowed by US Core in certain cases. This issue was introduced in v2.0.0. This test was updated to properly provide an 'OMIT' result instead in this case, which is interpreted the same as a 'PASS'.
• 4.31.02: Limited test criteria so that Clinical Notes Guidance provided in US Core only apply to DiagnosticReport resources of specific types, instead of all DiagnosticReport resources.

This version includes the following user experience updates that should have no impact to test behavior:

• 5.3.06 - 5.3.23: Multi-Patient API Group Compartment Export validation tests will now validate all returned resources instead of stopping validation after encountering a single invalid resource. The test reports the total number of invalid resources, along with the line number of the invalid resources.
• 5.3.22 - 5.3.23 Multi-Patient API Group Compartment Export tests for Medication and Location resources will now provide a more precise success message when no resources are returned.
• Updated to inferno-core v0.3.2, which provides a number of user interface updates and minor changes to test result reporting (see release notes).
• Update links in test descriptions for Multi-Patient API tests to point to v1.0.1 of the Bulk Data Implementation Guide, rather than v1.0.0.

ONC Certification (g)(10) Standardized API Test Kit v2.2.0 Release Candidate 1

Release 2.2.0.rc1 (#120)

• bump version
• bump us_core_test_kit and smart_app_launch_test_kit gems
• update CHANGELOG.md
• add rc1 to version and bundle
• change version
• generate matrix

ONC Certification (g)(10) Standardized API Test Kit v2.1.1

The ONC Certification (g)(10) Standardized API Test Kit v2.1.1 provides a user experience improvement based on community feedback.

As a patch release, systems that passed (g)(10) Test Kit v2.1.0 should continue to pass in v2.1.1. This update incorporates the following change:

• Fixed bug caused by leaving the 'Additional Patient IDs' input in the Single Patient API group empty.

ONC Certification (g)(10) Standardized API Test Kit v2.1.0

The ONC Certification (g)(10) Standardized API Test Kit v2.1.0 provides several test and user experience improvements based on community feedback.

As a minor version release, this update incorporates the following test change that may cause systems that passed v2.0.0 tests to now fail:

• 4.10.09: A Data Absent Reason can no longer be used within the DocumentReference.content.attachment.url or DocumentReference.content.attachment.data elements to satisfy the 'Must Support' requirement for attachment data.

This version also includes a number of other tests updates that allow more flexibility for the systems under test:

• Modified reference resolution tests so that only references in elements marked as "Must Support" are fetched, and in order to pass the system only needs to demonstrate one successful read for each must support element. Previously, the tests required all references within every resource to be successfully read.
• Removed reference resolution tests for Patient, Organization and Practitioner resource types, because the associated US Core v3.1.1 profile does not have any Must Support elements of type Reference.
• Fixed search tests for reference type search parameters (e.g. search by patient reference). If systems provided resources with references for that parameter as an absolute URL, the search test would fail the system. The tests now properly verifies references in resources that are absolute URLs match the search parameter.
• Systems may now publish their own OperationDefinition for the Bulk Data Group Export operation within their CapabilityStatement. This flexibility may be provided by the FHIR specification, thus systems will no longer fail test 5.2.02 for doing so.
• In some cases, terminology errors for codes within unknown CodeSystems were being provided underlying FHIR validation engine. These are not relevant for (g)(10) certification and will no longer cause test failure.
• Bulk Data file downloads will now follow HTTP redirection without passing an authentication header in the redirected request. Systems are responsible for appropriately securing URLs provided in redirects. Systems should document this expected client behavior if they choose to implement this functionality, as no guidance is provided within relevant standards on how clients must behave in this situation.
• Fixed issue introduced in v2.0.0 where the Bulk Data client did not send an 'Accept' header, as is encouraged for HTTP clients.
• Relaxed restriction introduced in v2.0.0 where SMART access tokens that were refreshed required the same scopes as presented in the original access token. Tokens that are refreshed can now alternatively be a subset of the scopes granted in the original access token.
• Relaxed restriction introduced in v2.0.0 where Bulk Data tests did not look across all files of a given resource type when performing Must Support checks.

This version includes the following user experience updates that should have no impact to test behavior:

• Fixed bugs relating to terminology build process, for users locally installing the (g)(10) test kit.
• Reference resolution tests did not store requests, which hindered users from identifying the source of errors in these tests. These requests are now presented in the user interface, consistent with other requests made by the tests.
• Fixed error message in Read tests when id does not match. Previously, the test error message incorrectly provided the test ID instead of the ID of the resource within the message.
• Updated to inferno-core v0.3.0, which provides a number of user interface updates (see release notes). This includes the ability to copy/paste all inputs for any test in structured JSON format, improved ordering of test inputs, improved report display, and many other bug fixes and visual improvements.

Users with local installations of the (g)(10) Test Kit upgrading from v2.0.0 can upgrade by running setup.sh. New users, or users upgrading from Inferno Program Edition v1.9 or prior must install a new copy of this application as described in the Installation Instructions.

ONC Certification (g)(10) Standardized API Test Kit v2.1.0 Release Candidate 1

Release 2.1.0.rc1 (#80)

* bump smart test kit version

* bump us core test kit version

* update us core class references

* only require us core v311

* bump version

* update changelog

* fix linting errors

* fix spec

ONC Certification (g)(10) Standardized API Test Kit v2.0.0

The ONC Certification (g)(10) Standardized API Test Kit is the successor to Inferno Program Edition. This is a major update to underlying infrastructure of the Inferno-based (g)(10) Standardized API certification tests. Since this release will have a larger impact to users of this testing system that previously released versions, Inferno Program Edition v1.9 will remain available for use in the near term. Users should expect Inferno Program Edition software to be retired over the coming months, and all major test updates to the Inferno-based (g)(10) Standardized API tests to occur here.

Test behavior changes between ONC Certification (g)(10) Standardized API Test Kit v2.0 and Inferno Program Edition v1.9 are intended to be minimal. However, underlying components of Inferno have been substantially modernized to improve flexibility and maintainability of the test system, and to enable better support for testing new or updated Implementation Guides. Given the scope of this effort there may be some unintended changes to test results for systems between v1.9 and v2.0. Please report any discrepancies in GitHub Issues section of the (g)(10) Standardized API Test Kit repository so they can be documented, and if necessary fixed.

Known test updates from Inferno Program Edition v1.9 that may cause previously passing systems under test to fail:

• Scopes granted to refreshed tokens are now expected to match those of the original access token, as described in the SMART App Launch Guide. Previous versions of these tests allowed for the granted scope of the refreshed token to be a strict subset of the scopes granted to the original access token.
• For systems that use the optional Medication resource type, terminology in instances of those resources is now validated in a similar manner to other resources. Previous versions of these tests did not validate terminology for the optional Medication resource type.
• For SMART App Launch discovery, tests now ensure that all SMART related fields match between the .well-known endpoint and the CapabilityStatement. Previously, only required fields were checked for equality. See #53

Test updates from Inferno Program Edition v1.9 that provide additional flexibility for systems under test:

• Allows use of Proof Key for Code Exchange (PKCE) during OAuth 2.0 authorization.
• Allows use of the "requiresAccessToken" field to enable alternate access control schemes besides OAuth 2.0 for file downloads as specified in Bulk Data Implementation Guide.

This release also includes the following updates to the testing software from Inferno Program Edition v1.9 that will impact the tester experience but should not affect testing behavior:

• Renames from "Inferno Program Edition" to "ONC Certification (g)(10) Standardized API Test Kit" to accurately describe the scope of this test kit.
• Reusable components have been extracted into the Inferno Framework, and tests have been rewritten as needed to use this shared testing infrastructure.
• Test identifiers have changed. A mapping between the old test identifiers and the new identifiers are attached to this release.
• User interface has changed substantially from Inferno Program Edition v1.9. However, the test organization and general workflow remains the same.