Vetting Requirements

Profile Requirements tell Vendors and Site Staff what they must do to claim conformance with the Profile.

Vetting Requirements

And the point of conformance to a Profile is to achieve the technical performance stated in the Claim,

So, Each requirement in a Profile should meet three criteria:

• It discernably impacts the performance of the biomarker (Impact Criteria)
• It is violated often enough in practice to be worth addressing (Violation Criteria)
• The impact exceeds the effort it takes to conform (Cost-Benefit Criteria)

Every extra requirement makes the profile incrementally longer to read, more work to assess, and harder to conform to. Each increment consumes time and attention at every conforming site and likely discourages one more site from conforming. Dropping requirements is a key mission once the initial brainstorming is done.

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." - Antoine de Saint-Exupéry

Impact Criteria

The requirement discernably impacts the performance of the biomarker.

If it's not impacting the Claim, it's not serving the purpose of the Profile. The Biomarker Committee should be able to state, for each requirement, the way it impacts the Claim.

Consider documenting that statement by having a line in the Discussion section for each requirement that explains the hypothesis for how it impacts the performance. If there is groundwork or literature that you can reference to prove the hypothesis, that's even better.

For each requirement:

• Imagine dropping it and consider the impact on the claim.
• Conversely, would you be willing to fail a site that didn't conform to this requirement but conformed to all the rest? If not, it can be dropped, or moved to the "extra credit" section of non-required recommendations.

Stages:

• During Stage 0 Groundwork: this is a key focus since many experiments are intended to specifically identify sources of variance and their impact.
• During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel may not meet the Impact Criteria.
• During Stage 2 Consensus: this is a chance for the Biomarker Committee members to revisit this question
• During Stage 3 Technical Confirmation: When field test sites indicate they did not conform to a requirement (or would not in practice), review the reason/opinion they provided. They may be challenging whether the requirement has a meaningful impact.

Examples:

• The ability of a staff member to provide a professional certification likely does not impact biomarker performance. Consider if there is a particular skill or expertise they may acquire during that certification which would affect biomarker performance. You could require they demonstrate having that skill and perhaps allow a certification record as an example method of demonstration. This also avoids narrowly requiring a specific national certification.
• Recommending general best practices and safety guidelines for imaging is a job already being done (well) by others. The job of QIBA is to deliver accurate, reproducible biomarkers. Consider briefly referencing such best practice guidelines in the informative text of the Profile. But do not include an incomplete set of those guidelines as requirements in the profile. After QIBA has succeeded in our job, we can start trying to do the job of others.
• Confirming assumptions that the claim depends on DOES contribute to achieving the claim if there is a reasonable possibility that the assumptions don't hold a conforming site. Understand the assumptions (including the statistical ones) that underlie the claim. Consider requirements that help sites confirm those assumptions hold locally.

Violation Criteria

The requirement should be violated often enough in practice to be worth addressing.

If a requirement is confirming something that is always true, it's not a good use of time and effort. In the extreme, scanning the patient is undeniably a requirement for good measurements and would impact the claim. But realistically, sites don't need us to tell them. it's not a requirement that sites need us to remind them not to violate.

Imagine being called in to investigate poor performance at lots of sites. Focus the requirements on the things that you expect would often or sometimes turn out to be the culprit.

For each requirement, imagine dropping it and estimate how many sites would fail to do it if we didn't tell them.

Stages:

• During Stage 0 Groundwork: it's OK to initially cast a wide net and include things where you are not sure how often they are violated in practice.
• During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel are unlikely violated in practice. Public Comment is our best opportunity to get broad input "from the field". Cast a wide net when circulating the profile.
• During Stage 2 Consensus: this is a chance for the Biomarker Committee members to revisit this question.
• During Stage 3 Technical Confirmation: When field test sites indicate they did not conform to a requirement (or would not in practice), take comfort that clearly the requirement gets violated in practice, thus passing this criteria, ... but confirm the other two criteria.

Examples:

• Placing the patient roughly centered laterally when doing head, chest or abdominal CT or MR imaging is fairly standard practice. If the profile depends on a tight tolerance for that centering, it might be worth a requirement. Otherwise, consider moving it out of requirements into a section on "Assumptions".

Cost-Benefit Criteria

The requirement's impact exceeds the effort it takes to conform.

The best requirements take minimal effort and yield significant impact. Next best are those that provide a modest impact for a small effort.

Requirements that have minimal impact but require significant effort to conform and assess should be dropped from the profile.

Finally, those where the effort and impact are both significant, call for careful consideration. If the level of performance after dropping the requirement would still exceed that needed for the driving clinical tasks, then it is a nice-to-have not a need-to have. If the clinical task depends on meeting the requirement, then it is "the price of admission" but be prepared to communicate that story to implementers to justify the effort.

For each requirement, discuss the estimated cost-benefit, even if you can't perform groundwork to validate.

Stages:

• During Stage 0 Groundwork: this is an opportunity to get estimates of the benefit of key requirements.
• During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel may be burdensome relative to their impact.
• During Stage 2 Consensus: this is a chance for the Biomarker Committee members to revisit this question.
• During Stage 3 Technical Confirmation:

• When field test sites actually perform the Profile, you will get concrete values for the cost (effort) of conforming to each requirement. If it exceeds your estimates, re-assess whether the requirement should be kept.
• If sites indicate they did not conform to a requirement (or would not in practice), review the reason/opinion they provided. They may be challenging whether the requirement cost is warranted.

Strategies:

• Remove: If you have enough room in your "performance budget" (between the performance delivered by the Profile and the performance needed to support the clinical tasks), consider dropping the requirement and degrading the claim. Implementers will appreciate the easier profile now and you can consider adding the requirement in a future edition.
• Relax: Consider whether there's a way to split the difference and relax the requirement (and the claim) a bit to make it more tolerable. In some cases, meeting 50% of the requirement might provide 80% of the impact.
• Devise: If the assessment procedure is burdensome, see if you can devise a different way to assess conformance that is less "costly".

Examples:

• .

Expressing Requirements

What exactly are you asking me to do?

Is it clear whether I've passed?