QIBA Profile Stages

Over time QIBA revises draft Profiles, collects information, and becomes increasingly confident the stated claims can be achieved by sites implementing the Profile. However, QIBA does not want to withhold Profiles until they are "perfect".

Depending on how they intend to use the Profile, some users will find Profiles valuable at each level of QIBA confidence, while others will count on achieving the claims exactly. Profile stages are intended to communicate the stage of maturity or confidence clearly and allow users to make their own choice about what meets their needs.

Briefly:

Stage 1 Public Comment	The Biomarker Committee experts have drafted the profile and believe it is practical and expect it to achieve the claimed performance.
Stage 2 Consensus	The wider community has read the profile and believe it to be practical and expect it to achieve the claimed performance
Stage 3 Technically Confirmed	Several sites have performed the profile and found it to be practical and expect it to achieve the claimed performance
Stage 4 Claim Confirmed	Some sites have performed the profile and found it achieved the claimed performance
Stage 5 Clinically Confirmed	Many sites have performed the profile and demonstrated the claimed performance is widely achievable.

The Criteria are checklist items/metrics/Rules-Of-Thumb the Biomarker Committee considers before publishing a Profile as having reached that stage.

Stage 1: Public Comment

Meaning:

• The Profile describes the key factors that affect the claim and the Biomarker Committee (BC) has proposed recommended procedures that address each/most of the factors.
• The Profile is believed to be practical.
• The Profile is expected to achieve the claimed performance.
• The Profile is ready for Public Comment.
• The Profile claim and requirements will likely change based on comments.

Criteria:

• All open issues have been clearly listed
• Most open issues have candidate resolutions drafted into the Profile
• Some groundwork projects may be underway
• All major solution components and Profile details are complete enough to implement
• For each actor in the Profile, it is clear what is required for a system or organization to claim conformance
• Each activity in the Profile has a justification based on literature data, phantom studies, or TC consensus
• Conformance requirements appear sufficient to accomplish the Claim of the Profile
• Profile claims are written in a statistically correct format (see Claim Guidance document).
• The statistical assumptions underlying the Profile claims are listed under the claim statements (e.g. for longitudinal claims, the assumption of linearity and the estimate of the wCV are stated; for cross-sectional claims, the estimate of the bias and precision are stated).
• The statistical assumptions underlying the Profile claims have been reviewed by a statistician to ensure that they are sufficient for the claims.
• An outline of an approach is described for testing the statistical assumptions underlying the Profile Claims (e.g. a phantom study to test linearity and bias of the measurements from image analysis software; a clinical study to test precision of the measurements).

Stage 2: Consensus

Meaning:

• The Profile represents a consensus of experts within and beyond the Biomarker Committee.
• The Profile is widely believed to be practical.
• The Profile is expected to achieve the claimed performance.
• The Profile is ready for Feasibility testing.
• The Profile claim and requirements may change based on Feasibility testing results.

Criteria:

• All public comments have been addressed
• All open issues necessary for conformant deployment have been resolved
• Few, if any, groundwork projects remain active
• All recommended procedures have been tested in one or more groundwork project(s) or referenced studies. (Reasonable deviations from Profile details may exist.)
• Requirement specifications include requirements and assessment procedures for the statistical assumptions underlying the Profile Claims (e.g. assessment procedure includes sample sizes, details of phantoms and data collection methods, metric computation, requirement specified thresholds).

Stage 3: Technically Confirmed

Meaning:

• The Profile has been found to be practical.
• The Profile is expected to achieve the claimed performance.
• The Profile is ready for Claim testing.
• The Profile claim and requirements may change based on Claim testing results.

The BC will generally coordinate with sites/vendors to achieve this stage as it does with groundwork projects

Consider what documentation would be important. Think about how site conformance might be audited.

Criteria:

• All Profile procedures and requirements have been performed/checked on at least two vendor platforms and at two or more sites and found to be clear and not burdensome/impractical.
• At least one of each Profile actor have demonstrated conformance (met all requirements).
• The assessment data from the actor conformance results have been reviewed by a statistician and found to validate the statistical assumptions underlying the claims.

Stage 4: Claim Confirmed/Achieved/Demonstrated?

Meaning:

• The Profile has been found to be practical.
• The Profile has been found to achieve the claimed performance. (but your mileage may vary)
• The Profile is ready for Clinical testing.
• The Profile claim and requirements may change (rarely) based on Clinical testing results.

Criteria: Groundwork projects are complete

• All Profile procedures and requirements have been implemented on at least two vendor platforms and at two or more sites
• All Profile procedures and requirements have been tested in the field as described in the Profile
• Profile results under experimental conditions are consistent with the Claims.
• Biomarker measurements collected from at least a few sites meet all the Profile claims.
• A sample size of X Biomarker measurements are compiled and a CI (from CoV) and it is found to be consistent with the Profile claim.
• Q. should the results at EACH of the sites also be consistent with the Profile claim?
• Some claims (for patient-independent metrics) might be confirmable with phantoms/DROs (and may be part of assessment procedures) but likely measurement of performance in phantoms/DRO is not sufficient by itself to declare patient oriented claims to be confirmed.
• Q. how does all this play into conformance testing?
• QIBA Registry role? Mountain of data aggregation?

(There will be guidance on the level of data/sample size/power, but it will not be specific numbers since there are so many factors) <Want this level to be easy enough to be doable for QIBA, but confident enough to be useful> (Number of scanner models, sites, patients, some variations in data, testing of boundaries) (Difference in Stage 4 and 5 is one of degree/scale not types of activities/process)

Stage 5: Clinically Confirmed

The meaning and criteria of Stage 5 - Clinically Confirmed are still under discussion.

QIBA won't DO this stage but may be a facilitator/enabler to get there.

(More generalizability - spectrum of data characteristics, actor characteristics, ) (

Meaning:

• The Profile is proven to be practical.
• The Profile is proven to achieve the claimed performance.
• The Profile claim and requirements are not longer expected to change.

Criteria:

• All recommended procedures and Profile details have been implemented by multiple systems or sites
• All recommended procedures and Profile details have been tested in the field as described in the Profile
• The Profile Claim has been achieved in clinical conditions
• Additional proposed criteria that will be discussed:

• The degree of statistical certainty has been documented and exceeds …
• Some level of Clinical validation has been demonstrated
• The Profile has been implemented in a human research study

See Also

• QIBA Profile Stage Descriptions February 2016
• How to use QIBA Profiles