Difference between revisions of "Vetting Requirements"
| Line 23: | Line 23: | ||
'''Stages''': | '''Stages''': | ||
| − | * During Stage 0 Groundwork: this is a key focus since many experiments are intended to specifically identify sources of variance and their impact. | + | * During Stage 0 Groundwork: this is a '''key focus''' since many experiments are intended to specifically '''identify sources of variance''' and their impact. |
| − | * During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel may not meet the Impact Criteria. | + | * During Stage 1 Public Comment: specifically '''ask reviewers''' to identify requirements they feel may not meet the Impact Criteria. |
* During Stage 2 Consensus: this is a chance for the Biomarker Committee members to revisit this question | * During Stage 2 Consensus: this is a chance for the Biomarker Committee members to revisit this question | ||
* During Stage 3 Technical Confirmation: When field test sites indicate they did not conform to a requirement (or would not in practice), review the reason/opinion they provided. They may be challenging whether the requirement has a meaningful impact. | * During Stage 3 Technical Confirmation: When field test sites indicate they did not conform to a requirement (or would not in practice), review the reason/opinion they provided. They may be challenging whether the requirement has a meaningful impact. | ||
| Line 31: | Line 31: | ||
* The ability of a staff member to provide a professional certification likely does not impact biomarker performance. If there is a particular skill or expertise they may acquire during that certification which would affect biomarker performance, consider requiring they demonstrate having that skill and perhaps allow a certification record as an example method of demonstration. This also avoids narrowly requiring a specific national certification. | * The ability of a staff member to provide a professional certification likely does not impact biomarker performance. If there is a particular skill or expertise they may acquire during that certification which would affect biomarker performance, consider requiring they demonstrate having that skill and perhaps allow a certification record as an example method of demonstration. This also avoids narrowly requiring a specific national certification. | ||
* Recommending general best practices and safety guidelines for imaging is a job already being done (well) by others. The job of QIBA is to deliver accurate, reproducible biomarkers. Briefly referencing such best practice guidelines in the informative text of the Profile is fair. But do not include an incomplete set of those guidelines as requirements in the profile. After QIBA has succeeded in our job, we can start trying to do the job of others. | * Recommending general best practices and safety guidelines for imaging is a job already being done (well) by others. The job of QIBA is to deliver accurate, reproducible biomarkers. Briefly referencing such best practice guidelines in the informative text of the Profile is fair. But do not include an incomplete set of those guidelines as requirements in the profile. After QIBA has succeeded in our job, we can start trying to do the job of others. | ||
| − | |||
===Violation Criteria=== | ===Violation Criteria=== | ||
Revision as of 21:12, 5 October 2021
Profile Requirements tell Vendors and Site Staff what they must do to claim conformance with the Profile.
Vetting Requirements
And the point of conformance to a Profile is to achieve the technical performance stated in the Claim,
So, Each requirement in a Profile should meet three criteria:
- It discernably impacts the performance of the biomarker (Impact Criteria)
- It is violated often enough in practice to be worth addressing (Violation Criteria)
- The impact exceeds the effort it takes to conform (Cost-Benefit Criteria)
Every requirement that fails one of the above criteria consumes time and attention unnecessarily at every conforming site. Some also consume tangible effort and resources to assess. Having Too Many Requirements is an issue with almost every Profile. Every extra requirement potentially discourages more sites from conforming. Dropping requirements is a key mission once the initial brainstorming is done.
Impact Criteria
The requirement discernably impacts the performance of the biomarker.
The Biomarker Committee should be able to state, for each requirement, the way it impacts the Claim.
Consider documenting that statement by having a line in the Discussion section for each requirement that explains the hypothesis for how it impacts the performance. If there is groundwork or literature that you can reference to prove the hypothesis, that's even better.
Stages:
- During Stage 0 Groundwork: this is a key focus since many experiments are intended to specifically identify sources of variance and their impact.
- During Stage 1 Public Comment: specifically ask reviewers to identify requirements they feel may not meet the Impact Criteria.
- During Stage 2 Consensus: this is a chance for the Biomarker Committee members to revisit this question
- During Stage 3 Technical Confirmation: When field test sites indicate they did not conform to a requirement (or would not in practice), review the reason/opinion they provided. They may be challenging whether the requirement has a meaningful impact.
Examples:
- The ability of a staff member to provide a professional certification likely does not impact biomarker performance. If there is a particular skill or expertise they may acquire during that certification which would affect biomarker performance, consider requiring they demonstrate having that skill and perhaps allow a certification record as an example method of demonstration. This also avoids narrowly requiring a specific national certification.
- Recommending general best practices and safety guidelines for imaging is a job already being done (well) by others. The job of QIBA is to deliver accurate, reproducible biomarkers. Briefly referencing such best practice guidelines in the informative text of the Profile is fair. But do not include an incomplete set of those guidelines as requirements in the profile. After QIBA has succeeded in our job, we can start trying to do the job of others.
Violation Criteria
The requirement should be violated often enough in practice to be worth addressing.
As with the Impact Criteria, requirements that fail the Violation Criteria should be easy to drop.
TODO Link to Stage 1 feedback, Stage 0 brainstorming, etc.
Cost-Benefit Criteria
TODO Link to Stage 3 Tech Conf, Stage 1 feedback, etc.
Expressing Requirements
What exactly are you asking me to do?
Is it clear whether I've passed?