Vetting Requirements

From QIBA Wiki
Jump to navigation Jump to search

Profile Requirements tell Vendors and Site Staff what they must do to claim conformance with the Profile.


Vetting Requirements

The point of conformance to a Profile is to achieve the technical performance stated in the Claim,

So, Each requirement in a Profile should meet three criteria:

Every extra requirement makes the profile incrementally longer to read, more work to assess, and harder to conform to. Each increment consumes time and attention at every conforming site and likely discourages one more site from conforming. Dropping requirements is a key mission once the initial brainstorming is done.

"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." - Antoine de Saint-Exupéry


Impact Criteria

The requirement discernably impacts the performance of the biomarker as expressed in the Claim.

If it's not impacting the Claim, it's not serving the purpose of the Profile. The Biomarker Committee should be able to state, for each requirement, the way it impacts the Claim.

Consider having a line in the Discussion section for each requirement that explains how it impacts performance, even if only a consensus hypothesis. If there is groundwork or literature you can reference that proves the hypothesis, that's even better.

For each requirement:

  • Imagine dropping it and consider the impact on the claim.
  • Conversely, would you fail a site that didn't conform to this requirement but conformed to all the rest? If not, it's not critical to meeting the Claim and can be dropped, or moved to the "extra credit" section of non-required recommendations.

Stages:

  • During Stage 0 Groundwork: this is a key focus since many experiments are intended to specifically identify sources of variance and their impact.
  • During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel may not meet the Impact Criteria.
  • Before releasing the Public Comment draft, the BC should review all the requirements to make sure that someone can state the mechanism by which each impacts the claim.
  • During Stage 2 Consensus: this is a chance for the Biomarker Committee members to revisit this question
  • During Stage 3 Technical Confirmation: When field test sites indicate they did not conform to a requirement (or would not in practice), review the reason/opinion they provided. They may be challenging whether the requirement has a meaningful impact.

Examples:

  • The ability of a staff member to provide a professional certification document likely does not in itself impact biomarker performance. Consider if there is a particular skill or expertise they may acquire during that certification which would affect biomarker performance. You could require they demonstrate having that skill and perhaps allow a certification record as an example method of demonstration. This also avoids narrowly requiring a specific national certification.
  • General best practices and safety guidelines for imaging are wonderful, but such recommendations are a job already being done (well) by others. The job of QIBA is to deliver accurate, reproducible biomarkers. Consider briefly referencing such best practice guidelines in the informative text of the Profile. But do not burden the profile by including an (incomplete) set of those guidelines as requirements.
  • Confirming assumptions that the claim depends on DOES contribute to achieving the claim if there is a reasonable possibility that the assumptions don't hold at a conforming site. Consider requirements that help sites confirm those assumptions hold locally. E.g. if the Profile performance model has assumed the relationship between the measurements and the ground truth is linear (with a certain slope), and that linearity might vary from site to site, then the profile may need a requirement to confirm that linearity and slope.
  • Sources of variance exposed by groundwork are a great basis for requirements with proven impact.
  • Since technology and practices change over time, it may be appropriate to periodically review profiles and consider adding or removing requirements as existing sources of variance are resolved and new ones emerge.
  • An example of an impactful requirement that often gets overlooked is the need to mandate retention of certain DICOM private tags, such as the b-value information for diffusion MR, that otherwise get removed during the anonymization of data in a clinical trial setting.

Violation Criteria

The requirement should be violated often enough in practice to be worth addressing.

If a requirement is confirming something that is always true, it's not a good use of time and effort. In the extreme, scanning the patient is undeniably a requirement for good measurements and would impact the claim. But realistically, sites don't need us to tell them. It's not a requirement that sites need us to remind them not to violate.

TODO Explain "expected value" - if it is critical impact, then if the probability is non-zero, it's worth addressing even if the probability is low. For lower impact, if it's low probability it might not be worth burdening everyone since it's relatively rare.

Imagine being called in to investigate poor performance. Focus the requirements on the things that you expect would often or sometimes turn out to be the culprit.

For each requirement, imagine dropping it and estimate how many sites would fail to do it if we didn't tell them.

Stages:

  • During Stage 0 Groundwork: it's OK to initially cast a wide net and include things where you are not sure how often they are violated in practice.
  • During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel are unlikely violated in practice. Public Comment is our best opportunity to get broad input "from the field". Cast a wide net when circulating the profile.
  • During Stage 2 Consensus: this is a chance for the Biomarker Committee members to incorporate the Public Comment feedback.
  • During Stage 3 Technical Confirmation: When field test sites indicate they did not conform to a requirement (or would not in practice), take comfort that clearly the requirement gets violated in practice, thus passing the Violation criteria, ... but you should confirm the Impact and Cost criteria. TODO Conversely, sites may provide feedback that this would never be a problem in practice.

Examples:

  • Placing the patient roughly centered laterally when doing head, chest or abdominal CT or MR imaging is standard practice at most/all sites. If the patient is significantly off-center there is likely an unavoidable reason for it. If the profile depends on a tight tolerance for that centering, it might be worth a requirement. Otherwise, consider moving it out of requirements into a section on "Assumptions".

Effort Criteria

The requirement's impact exceeds the effort it takes to conform.

The best requirements take minimal effort and yield significant impact. Next best are those that provide a modest impact for a small effort.

Requirements that have minimal impact but require significant effort to conform and assess should be dropped from the profile.

Finally, those where the effort and impact are both significant, call for careful consideration. If the level of performance after dropping the requirement would still exceed that needed for the driving clinical tasks, then it is a nice-to-have not a need-to have. If the clinical task depends on meeting the requirement, then it is "the price of admission" but be prepared to concisely document that justification of the effort to implementers.

For each requirement, discuss the estimated "cost-benefit", even if you can't perform groundwork to validate.

Stages:

  • During Stage 0 Groundwork: this is an opportunity to get estimates of the benefit of key requirements.
  • During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel may be burdensome relative to their impact.
  • During Stage 2 Consensus: this is a chance for the Biomarker Committee members to incorporate the Public Comment feedback.
  • During Stage 3 Technical Confirmation:
  • When field test sites actually perform the Profile, you will get concrete values for the cost (effort) of conforming to each requirement. If it exceeds your estimates, re-assess whether the requirement should be kept.
  • If sites indicate they did not conform to a requirement (or would not in practice), review the reason/opinion they provided. They may be challenging whether the requirement cost is warranted.

Strategies:

  • Remove: If you have enough room in your "performance budget" (between the performance delivered by the Profile and the performance needed to support the clinical tasks), consider dropping the requirement and degrading the claim. Implementers will appreciate the easier profile today and you can consider adding the requirement in a future edition.
  • Relax: Consider whether there's a way to split the difference and relax both the requirement and the claim a bit to make it more tolerable. In some cases, meeting 50% of the requirement might provide 80% of the impact.
  • Devise: If the assessment procedure is burdensome, see if you can devise a different way to assess conformance that is less "costly".
  • Justify: TODO make a better case you just HAVE to do this.

Examples:

  • TODO

Expressing Requirements

See also the guidance on Expressing Requirements.

Retrieved from "https://qibawiki.rsna.org/index.php?title=Vetting_Requirements&oldid=17447"