Vetting Requirements

Profile Requirements tell Vendors and Site Staff what they must do to claim conformance with the Profile.

The point of conformance to a Profile is to achieve the technical performance stated in the Claim.

So, each requirement in a Profile should meet three criteria:

• It discernably impacts the performance of the biomarker (Impact Criteria)
• It is violated often enough in practice to be worth addressing (Violation Criteria)
• The impact exceeds the effort it takes to conform (Effort Criteria)

Every extra requirement makes the profile incrementally longer to read, more work to assess, and harder to conform to. Each increment consumes time and attention at every conforming site and likely discourages one more site from conforming. Dropping requirements is a key mission once the initial brainstorming is done.

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." - Antoine de Saint-Exupéry

Impact Criteria

The requirement discernably impacts the performance of the biomarker as expressed in the Claim.

If it's not impacting the Claim, it's not serving the purpose of the Profile. The Biomarker Committee should be able to state, for each requirement, the way it impacts the Claim.

Consider having a line in the Discussion section for each requirement that explains how it impacts performance, even if only a consensus hypothesis. If there is groundwork or literature you can reference that proves the hypothesis, that's even better.

For each requirement:

• Imagine dropping it and consider the impact on the claim.
• Conversely, would you fail a site that didn't conform to this requirement but conformed to all the rest? If not, it's not critical to meeting the Claim and can be dropped, or moved to the "extra credit" section of non-required recommendations.

Stages:

• During Stage 0 Groundwork: this is a key focus since many experiments are intended to specifically identify sources of variance and their impact.
• During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel may not meet the Impact Criteria.

• Before releasing the Public Comment draft, the BC should review each requirement to make sure that someone can briefly state the actual mechanism by which it impacts the claim.

• During Stage 2 Consensus: this is a chance for the Biomarker Committee members to revisit this question
• During Stage 3 Technical Confirmation: When field test sites indicate they did not conform to a requirement (or would not in practice), review the reason/opinion they provided. They may be challenging whether the requirement has a meaningful impact.

Examples:

• The ability of a staff member to provide a professional certification document likely does not in itself impact biomarker performance. Consider if there is a particular skill or expertise they may acquire during that certification which would affect biomarker performance. You could require they demonstrate having that skill and perhaps allow a certification record as an example method of demonstration. This also avoids narrowly requiring a specific national certification.
• General best practices and safety guidelines for imaging are wonderful, but such recommendations are a job already being done (well) by others. The job of QIBA is to deliver accurate, reproducible biomarkers. Consider briefly referencing such best practice guidelines in the informative text of the Profile. But do not burden the profile by including an (incomplete) set of those guidelines as requirements.
• Confirming assumptions that the claim depends on DOES contribute to achieving the claim if there is a reasonable possibility that the assumptions don't hold at a conforming site. Consider requirements that help sites confirm those assumptions hold locally. E.g. if the Profile performance model has assumed the relationship between the measurements and the ground truth is linear (with a certain slope), and that linearity might vary from site to site, then the profile may need a requirement to confirm that linearity and slope.
• Sources of variance exposed by groundwork are a great basis for requirements with proven impact.
• Since technology and practices change over time, it may be appropriate to periodically review profiles and consider adding or removing requirements as existing sources of variance are resolved and new ones emerge.
• An example of an impactful requirement that often gets overlooked is the need to mandate retention of certain DICOM private tags, such as the b-value information for diffusion MR, that otherwise get removed during the anonymization of data in a clinical trial setting.

Violation Criteria

The requirement should be violated often enough in practice to be worth addressing.

If a requirement is confirming something that is always true, it's not a good use of time and effort. In the extreme, scanning the patient is undeniably a requirement for good measurements and would impact the claim. But realistically, sites don't need us to tell them. It's not a requirement that sites need us to remind them not to violate.

For requirements that are violated but only very rarely, consider "expected value" (probability * value). If the impact of violation is very high, it may still be worth addressing even if the probability is low. If the impact is also low, it might not be worthwhile to burden everyone with the requirement that only rarely provides a small value.

Imagine being called in to investigate poor site performance. Focus the requirements on the things that you would reasonably expect to sometimes be the culprit.

For each requirement, imagine dropping it and estimate how many sites would fail to do it if we didn't tell them.

Stages:

• During Stage 0 Groundwork: it's OK to initially cast a wide net and include things where you are not sure how often they are violated in practice.
• During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel are unlikely violated in practice. Public Comment is our best opportunity to get broad input "from the field". Cast a wide net when circulating the profile.
• During Stage 2 Consensus: this is a chance for the Biomarker Committee members to incorporate the Public Comment feedback.
• During Stage 3 Technical Confirmation:

• When field test sites indicate they did not conform to a requirement (or would not in practice), take comfort that clearly the requirement gets violated in practice, thus passing the Violation criteria, ... but you should still confirm the Impact and Cost criteria.
• Conversely, sites may provide feedback that the requirement addresses something that is never a problem in practice since everyone always conforms. Look into whether it might make sense to drop the requirement, especially if assessment is non-trivial.

Examples:

• Placing the patient roughly centered laterally when doing head, chest or abdominal CT or MR imaging is standard practice at most/all sites. If the patient is significantly off-center there is likely an unavoidable reason for it. If the profile depends on a tight tolerance for that centering, it might be worth a requirement. Otherwise, consider moving it out of requirements into a section on "Assumptions".

Effort Criteria

The requirement's impact exceeds the effort it takes to conform.

The best requirements take minimal effort and yield significant impact. Next best are those that provide a modest impact for a small effort.

Requirements that have minimal impact but require significant effort to conform and assess should be dropped from the profile.

Finally, those where the effort and impact are both significant, call for careful consideration. If the level of performance after dropping the requirement would still be good enough for the driving clinical tasks, then it is a nice-to-have not a need-to have. If the clinical task depends on the requirement being met, then that effort is "the price of admission" but be prepared to concisely document that justification of the effort to implementers.

For each requirement, discuss your estimate of the "cost-benefit", even if you can't perform groundwork to validate.

Stages:

• During Stage 0 Groundwork: this is an opportunity to get estimates of the benefit of key requirements.
• During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel may be burdensome relative to their impact.
• During Stage 2 Consensus: this is a chance for the Biomarker Committee members to incorporate the Public Comment feedback.
• During Stage 3 Technical Confirmation:

• When field test sites actually perform the Profile, you will get concrete values for the cost (effort) of conforming to each requirement. If it exceeds your estimates, re-assess whether the requirement should be kept.
• If sites indicate they did not conform to a requirement (or would not in practice), review the reason/opinion they provided. They may be challenging whether the requirement cost is warranted.

Strategies:

• Remove: If you have enough room in your "performance budget" (between the performance delivered by the Profile and the performance needed to support the clinical tasks), consider dropping the requirement and degrading the claim. Implementers will appreciate the easier profile today and you can consider adding the requirement in a future edition.
• Relax: Consider whether there's a way to split the difference and relax both the requirement and the claim a bit to make it more tolerable. In some cases, meeting 50% of the requirement might provide 80% of the impact.
• Devise: If the assessment procedure is burdensome, see if you can devise a different way to assess conformance that is less "costly".
• Justify: Document clearly in the Profile why the requirement (and associated effort) are unavoidable to achieve adequate performance, so sites see why they HAVE to do it.

Examples:

• Requiring a daily QC procedure would likely represent excessive effort for a minimal gain when compared to a weekly or monthly QC check.

See Also

• Guidance on Expressing Requirements