Difference between revisions of "QIBA Profile Stages"

Revision as of 02:42, 8 December 2021

Over time QIBA revises draft Profiles, collects information, and becomes increasingly confident the stated claims can be achieved by sites implementing the Profile. However, QIBA does not want to withhold Profiles until they are "perfect".

Depending on how they intend to use the Profile, some users will find Profiles valuable at each level of QIBA confidence, while others will count on achieving the claims exactly. Profile stages are intended to communicate the stage of maturity or confidence clearly and allow users to make their own choice about what meets their needs.

Briefly:

Stage 1 Public Comment	The Biomarker Committee experts have drafted the profile and believe it is practical and expect it to achieve the claimed performance.
Stage 2 Consensus	The wider community has read the profile and judged it to be practical and expect it to achieve the claimed performance
Stage 3 Technically Confirmed	Several sites have performed the profile and found it to be practical and expect it to achieve the claimed performance
Stage 4 Claim Confirmed	Some sites have performed the profile and found it achieved the claimed performance
Stage 5 Clinically Confirmed	Many sites have performed the profile and demonstrated the claimed performance is widely achievable.

The Criteria are checklist items/metrics/Rules-Of-Thumb the Biomarker Committee considers before publishing a Profile as having reached that stage.

Stage 1: Public Comment

Meaning:

The Profile describes the key factors that affect the claim and the Biomarker Committee (BC) has proposed recommended procedures that address each/most of the factors.
The Profile is believed to be practical.
The Profile is expected to achieve the claimed performance.
The Profile is ready for Public Comment.
The Profile claim and requirements will likely change based on comments.

Criteria:

All open issues have been clearly listed
Most open issues have candidate resolutions drafted into the Profile
Some groundwork projects may be underway
All major solution components and Profile details are complete enough to implement
For each actor in the Profile, it is clear what is required for a system or organization to claim conformance
The Biomarker Committee has done a pre-publication pass to confirm that all requirements meet the Impact Criteria and Violation Criteria

Ideally this is informed by groundwork and literature search references

Each activity in the Profile has a justification based on literature data, phantom studies, or TC consensus
Conformance requirements appear sufficient to accomplish the Claim of the Profile
Profile claims are written in a statistically correct format (see Claim Guidance document).
The statistical assumptions underlying the Profile claims are listed under the claim statements (e.g. for longitudinal claims, the assumption of linearity and the estimate of the wCV are stated; for cross-sectional claims, the estimate of the bias and precision are stated).
The statistical assumptions underlying the Profile claims have been reviewed by a statistician to ensure that they are sufficient for the claims.
An outline of an approach is described for testing the statistical assumptions underlying the Profile Claims (e.g. a phantom study to test linearity and bias of the measurements from image analysis software; a clinical study to test precision of the measurements).

See Also: Process - Getting to Stage 1

Stage 2: Consensus

Meaning:

The Profile represents a consensus of experts within and beyond the Biomarker Committee.
The Profile is widely believed to be practical.
The Profile is expected to achieve the claimed performance.
The Profile is ready for Feasibility testing.
The Profile claim and requirements may change based on Feasibility testing results.

Criteria:

All public comments have been addressed

Supporting Document: Comment Resolution Spreadsheet
Specifically review comments to confirm the Impact Criteria and Violation Criteria are met for each requirement

All open issues necessary for conformant deployment have been resolved
Few, if any, groundwork projects remain active
All recommended procedures have been tested in one or more groundwork project(s) or referenced studies. (Reasonable deviations from Profile details may exist.)
Requirement specifications include requirements and assessment procedures for the statistical assumptions underlying the Profile Claims (e.g. assessment procedure includes sample sizes, details of phantoms and data collection methods, metric computation, requirement specified thresholds).

See Also: Process - Getting to Stage 2

Stage 3: Technically Confirmed

Meaning:

The Profile has been found to be practical. (Reading and conforming to the profile was not burdensome)
The Profile is expected to achieve the claimed performance.
The Profile is ready for Claim testing.
The Profile claim and requirements may change based on Claim testing results.

The BC will generally coordinate with sites/vendors to achieve this stage as it does with groundwork projects

Consider what documentation would be important. Think about how site conformance might be audited.

Criteria:

All Profile procedures and requirements have been performed/checked on at least two vendor platforms and at three or more sites and found to be clear and not burdensome/impractical.

Specifically be sure to collect and review feedback to assess the Effort Criteria
Supporting Document: Feedback Resolution Spreadsheet

At least one of each Profile actor have demonstrated conformance (met all requirements).
The assessment data from the actor conformance results have been reviewed by a statistician and found to validate the statistical assumptions underlying the claims.

See Also: Process - Getting to Stage 3

Stage 4: Claim Confirmed/Achieved/Demonstrated?

Meaning:

The Profile has been found to be practical.
The Profile has been found to achieve the claimed performance. (but your mileage may vary)
The Profile is ready for Clinical testing.
The Profile claim and requirements may change (rarely) based on Clinical testing results.

Criteria:

Groundwork projects are complete
All Profile procedures and requirements have been implemented on at least two vendor platforms and at two or more sites
All Profile procedures and requirements have been tested in the field as described in the Profile
Profile results under experimental conditions are consistent with the Claims.
Biomarker measurements collected from at least a few sites meet all the Profile claims.

Supporting Document: Claim Confirmation Document

A sample size of X Biomarker measurements are compiled and a CI (from CoV) and it is found to be consistent with the Profile claim.
Q. should the results at EACH of the sites also be consistent with the Profile claim?
Some claims (for patient-independent metrics) might be confirmable with phantoms/DROs (and may be part of assessment procedures) but likely measurement of performance in phantoms/DRO is not sufficient by itself to declare patient oriented claims to be confirmed.
Q. how does all this play into conformance testing?
QIBA Registry role? Mountain of data aggregation?

See Also: Process - Getting to Stage 4

(There will be guidance on the level of data/sample size/power, but it will not be specific numbers since there are so many factors) <Want this level to be easy enough to be doable for QIBA, but confident enough to be useful> (Number of scanner models, sites, patients, some variations in data, testing of boundaries) (Difference in Stage 4 and 5 is one of degree/scale not types of activities/process)

Stage 5: Clinically Confirmed

The meaning and criteria of Stage 5 - Clinically Confirmed are still under discussion.

QIBA won't DO this stage but may be a facilitator/enabler to get there.

(More generalizability - spectrum of data characteristics, actor characteristics, ) (

Meaning:

The Profile is proven to be practical.
The Profile is proven to achieve the claimed performance.
The Profile claim and requirements are not longer expected to change.

Criteria:

All recommended procedures and Profile details have been implemented by multiple systems or sites
All recommended procedures and Profile details have been tested in the field as described in the Profile
The Profile Claim has been achieved in clinical conditions
Additional proposed criteria that will be discussed:

The degree of statistical certainty has been documented and exceeds …
Some level of Clinical validation has been demonstrated
The Profile has been implemented in a human research study

See Also: Process - Getting to Stage 5

@@ Line 1: / Line 1: @@
-__NOTOC__
+Over time QIBA revises draft Profiles, collects information, and becomes '''increasingly confident''' the stated claims can be achieved by sites implementing the Profile. However, QIBA does not want to withhold Profiles until they are "perfect".
-[[Media:QIBA_Profile_Stages-20160210.pdf|QIBA Profile Stage Descriptions]] ''February 2016''
+Depending on '''[[How to use QIBA Profiles|how they intend to use the Profile]]''', some users will find Profiles valuable at each level of QIBA confidence, while others will count on achieving the claims exactly. Profile stages are intended to communicate the stage of maturity or confidence clearly and allow users to make their own choice about what meets their needs.
-Over time QIBA revises draft Profiles, collects information, and becomes increasingly confident of its ability to achieve the stated claim. However QIBA does not want to withhold Profiles until they are "perfect".
-Depending on '''[[How to use QIBA Profiles|how they intend to use the Profile]]''', some users will find Profiles valuable even with only modest QIBA confidence, while others will count on achieving the claims exactly. Profile stages are intended to communicate the stage of maturity or confidence clearly and allow users to make their own choice about what meets their needs.
 Briefly:
@@ Line 11: / Line 7: @@
 |style="width:150px" |Stage 1
 '''Public Comment'''
-|The Biomarker Committee experts have drafted the profile and believe it is practical and expect it to achieve the claimed performance.
+|The Biomarker Committee experts have '''drafted the profile''' and believe it is practical and expect it to achieve the claimed performance.
 |-
 |Stage 2
 '''Consensus'''
-|The wider community has read the profile and believe it to be practical and expect it to achieve the claimed performance
+|The wider community has read the profile and '''judged it to be practical''' and expect it to achieve the claimed performance
 |-
 |Stage 3
 '''Technically Confirmed'''
-|Site(s) have performed the profile and found it to be practical and expect it to achieve the claimed performance
+|Several sites have '''performed the profile and found it to be practical''' and expect it to achieve the claimed performance
 |-
 |Stage 4
 '''Claim Confirmed'''
-|Site(s) have performed the profile and found it achieved the claimed performance
+|Some sites have performed the profile and '''found it achieved the claimed performance'''
 |-
 |Stage 5
 '''Clinically Confirmed'''
-|Many sites have performed the profile and demonstrated the claimed performance is widely achievable.
+|Many sites have performed the profile and demonstrated the '''claimed performance is widely achievable'''.
 |}
@@ Line 35: / Line 31: @@
 ==Stage 1: Public Comment==
 '''Meaning:'''
-:* The Profile is ready for public comment.
+:* The Profile describes the key factors that affect the claim and the Biomarker Committee (BC) has proposed recommended procedures that address each/most of the factors.
-:* The Profile claim and requirements will likely change based on comments.
 :* The Profile is believed to be practical.
 :* The Profile is expected to achieve the claimed performance.
-:* The Biomarker Committee (BC) finds that the Profile describes the key factors that affect the claim and has proposed recommended procedures that address each/most of the factors.
+:* The Profile is ready for Public Comment.
+:* The Profile claim and requirements will likely change based on comments.
 '''Criteria:'''
@@ Line 47: / Line 43: @@
 :* All major solution components and Profile details are complete enough to implement
 :* For each actor in the Profile, it is clear what is required for a system or organization to claim conformance
+:* The Biomarker Committee has done a pre-publication pass to confirm that all requirements meet the '''[[Vetting_Requirements#Impact_Criteria|Impact Criteria]]''' and '''[[Vetting_Requirements#Violation_Criteria|Violation Criteria]]'''
+::* Ideally this is informed by groundwork and literature search references
 :* Each activity in the Profile has a justification based on literature data, phantom studies, or TC consensus
 :* Conformance requirements appear sufficient to accomplish the Claim of the Profile
+:* Profile claims are written in a statistically correct format (see Claim Guidance document).
+:* The statistical assumptions underlying the Profile claims are listed under the claim statements (e.g. for longitudinal claims, the assumption of linearity and the estimate of the wCV are stated; for cross-sectional claims, the estimate of the bias and precision are stated).
+:* The statistical assumptions underlying the Profile claims have been reviewed by a statistician to ensure that they are sufficient for the claims.
+:* An outline of an approach is described for testing the statistical assumptions underlying the Profile Claims (e.g. a phantom study to test linearity and bias of the measurements from image analysis software; a clinical study to test precision of the measurements).
-==Stage 2: Consensus (Publicly Reviewed)==
+'''See Also:''' [[Process#Getting_to_Stage_1:_Public_Comment| Process - Getting to Stage 1]]
+==Stage 2: Consensus==
 '''Meaning:'''
+:* '''The Profile represents a consensus of experts within and beyond the Biomarker Committee.'''
+:* The Profile is widely believed to be practical.
+:* The Profile is expected to achieve the claimed performance.
 :* The Profile is ready for Feasibility testing.
 :* The Profile claim and requirements may change based on Feasibility testing results.
-:* The Profile is widely believed to be practical.
-:* The Profile is expected to achieve the claimed performance.
-:*
 '''Criteria:'''
 :* All public comments have been addressed
+::* '''Supporting Document''': [[Public Comment Process|Comment Resolution Spreadsheet]]
+::* Specifically review comments to confirm the '''[[Vetting_Requirements#Impact_Criteria|Impact Criteria]]''' and '''[[Vetting_Requirements#Violation_Criteria|Violation Criteria]]''' are met for each requirement
 :* All open issues necessary for conformant deployment have been resolved
 :* Few, if any, groundwork projects remain active
 :* All recommended procedures have been tested in one or more groundwork project(s) or referenced studies. (Reasonable deviations from Profile details may exist.)
+:* Requirement specifications include requirements and assessment procedures for the statistical assumptions underlying the Profile Claims (e.g. assessment procedure includes sample sizes, details of phantoms and data collection methods, metric computation, requirement specified thresholds).
+'''See Also:''' [[Process#Getting_to_Stage_2:_Consensus| Process - Getting to Stage 2]]
 ==Stage 3: Technically Confirmed==
 '''Meaning:'''
+:* '''The Profile has been found to be practical.''' (Reading and conforming to the profile was not burdensome)
+:* The Profile is expected to achieve the claimed performance.
 :* The Profile is ready for Claim testing.
 :* The Profile claim and requirements may change based on Claim testing results.
-:* The Profile has been found to be practical.
-:* The Profile is expected to achieve the claimed performance.
+The BC will generally coordinate with sites/vendors to achieve this stage as it does with groundwork projects
-:*
+''Consider what documentation would be important. Think about how site conformance might be audited.''
 '''Criteria:'''
-:* All Profile procedures and details have been implemented on at least two vendor platforms and at two or more sites and found to be clear and not burdensome/impractical
+:* All Profile procedures and requirements have been performed/checked on at least two vendor platforms and at three or more sites and found to be clear and not burdensome/impractical.
-:* An example of each Profile actor have demonstrated conformance
+::* Specifically be sure to collect and review feedback to assess the '''[[Vetting_Requirements#Effort_Criteria|Effort Criteria]]'''
-:* The BC will generally coordinate with sites/vendors to achieve this stage as it does with groundwork projects
+::* Supporting Document: [[Technical Confirmation Process|Feedback Resolution Spreadsheet]]
-:* ''Consider what documentation would be important. Think about how site conformance might be audited.''
+:* At least one of each Profile actor have demonstrated conformance (met all requirements).
+:* The assessment data from the actor conformance results have been reviewed by a statistician and found to validate the statistical assumptions underlying the claims.
+'''See Also:''' [[Process#Getting_to_Stage_3:_Technically_Confirmed| Process - Getting to Stage 3]]
-==Stage 4: Claim Confirmed==
+==Stage 4: Claim Confirmed/Achieved/Demonstrated?==
 '''Meaning:'''
+:* The Profile has been found to be practical.
+:* '''The Profile has been found to achieve the claimed performance.''' (but your mileage may vary)
 :* The Profile is ready for Clinical testing.
 :* The Profile claim and requirements may change (rarely) based on Clinical testing results.
-:* The Profile has been found to be practical.
-:* The Profile has been found to achieve the claimed performance.
 '''Criteria:'''
-Groundwork projects are complete
+:* Groundwork projects are complete
-:* All Profile procedures and details have been implemented on at least two vendor platforms and at two or more sites
+:* All Profile procedures and requirements have been implemented on at least two vendor platforms and at two or more sites
-:* All Profile procedures and details have been tested in the field as described in the Profile
+:* All Profile procedures and requirements have been tested in the field as described in the Profile
-:* Profile results under experimental conditions are consistent with the Claim.
+:* Profile results under experimental conditions are consistent with the Claims.
+:* ''Biomarker measurements collected from at least a few sites meet all the Profile claims.''
+::* '''Supporting Document''': [[Claim Confirmation Process|Claim Confirmation Document]]
+:* A sample size of X Biomarker measurements are compiled and a CI (from CoV) and it is found to be consistent with the Profile claim.
+:* Q. should the results at EACH of the sites also be consistent with the Profile claim?
 :* ''Some claims (for patient-independent metrics) might be confirmable with phantoms/DROs (and may be part of assessment procedures) but likely measurement of performance in phantoms/DRO is not sufficient by itself to declare patient oriented claims to be confirmed.''
 :* ''Q. how does all this play into conformance testing?''
 :* ''QIBA Registry role? Mountain of data aggregation?''
+'''See Also:''' [[Process#Getting_to_Stage_4:_Claim_Confirmed| Process - Getting to Stage 4]]
+(There will be guidance on the level of data/sample size/power, but it will not be specific numbers since there are so many factors)
+<Want this level to be easy enough to be doable for QIBA, but confident enough to be useful>
+(Number of scanner models, sites, patients, some variations in data, testing of boundaries)
+(Difference in Stage 4 and 5 is one of degree/scale not types of activities/process)
 ==Stage 5: Clinically Confirmed==
@@ Line 98: / Line 124: @@
 ''QIBA won't DO this stage but may be a facilitator/enabler to get there.''
+(More generalizability - spectrum of data characteristics, actor characteristics, )
+(
 '''Meaning:'''
-:* The Profile claim and requirements are not longer expected to change.
 :* The Profile is proven to be practical.
 :* The Profile is proven to achieve the claimed performance.
+:* The Profile claim and requirements are not longer expected to change.
 '''Criteria:'''
@@ Line 113: / Line 142: @@
 ::* The Profile has been implemented in a human research study
+'''See Also:''' [[Process#Getting_to_Stage_5:_Clinically_Confirmed| Process - Getting to Stage 5]]
 ==See Also==
+:* [[Media:QIBA_Profile_Stages-20160210.pdf|QIBA Profile Stage Descriptions]] ''February 2016''
 :*[[How to use QIBA Profiles]]
+__TOC__

Difference between revisions of "QIBA Profile Stages"

Revision as of 02:42, 8 December 2021

Stage 1: Public Comment

Stage 2: Consensus

Stage 3: Technically Confirmed

Stage 4: Claim Confirmed/Achieved/Demonstrated?

Stage 5: Clinically Confirmed

See Also

Contents

Navigation menu

Search