QIBA Profile Stages

From QIBA Wiki
Jump to navigation Jump to search

Over time QIBA revises draft Profiles, collects information, and becomes increasingly confident the stated claims can be achieved by sites implementing the Profile. However, QIBA does not want to withhold Profiles until they are "perfect".

Depending on how they intend to use the Profile, some users will find Profiles valuable at each level of QIBA confidence, while others will count on achieving the claims exactly. Profile stages are intended to communicate the stage of maturity or confidence clearly and allow users to make their own choice about what meets their needs.

Briefly:

Stage 1

Public Comment

The Biomarker Committee experts have drafted the profile and believe it is practical and expect it to achieve the claimed performance.
Stage 2

Consensus

The wider community has read the profile and judged it to be practical and expect it to achieve the claimed performance
Stage 3

Clinically Feasible

Several sites have performed the profile and found it to be practical and expect it to achieve the claimed performance
Stage 4

Claim Confirmed

Some sites have performed the profile and found it achieved the claimed performance


The Criteria are checklist items/metrics/Rules-Of-Thumb the Biomarker Committee considers before publishing a Profile as having reached that stage.

Stage 1: Public Comment

Meaning:

  • The Profile describes the key factors that affect the claim and the Biomarker Committee (BC) has proposed recommended procedures that address each/most of the factors.
  • The Profile is believed to be practical.
  • The Profile is expected to achieve the claimed performance.
  • The Profile is ready for Public Comment.
  • The Profile claim and requirements will likely change based on comments.

Criteria:

  • All open issues have been clearly listed
  • Most open issues have candidate resolutions drafted into the Profile
  • Some groundwork projects may be underway
  • All major solution components and Profile details are complete enough to implement
  • For each actor in the Profile, it is clear what is required for a system or organization to claim conformance
  • The Biomarker Committee has done a pre-publication pass to confirm that all requirements meet the Impact Criteria and Violation Criteria
  • Ideally this is informed by groundwork and literature search references
  • Each activity in the Profile has a justification based on literature data, phantom studies, or TC consensus
  • Conformance requirements appear sufficient to accomplish the Claim of the Profile
  • Profile claims are written in a statistically correct format (see Claim Guidance document).
  • The statistical assumptions underlying the Profile claims are listed under the claim statements (e.g. for longitudinal claims, the assumption of linearity and the estimate of the wCV are stated; for cross-sectional claims, the estimate of the bias and precision are stated).
  • The statistical assumptions underlying the Profile claims have been reviewed by a statistician to ensure that they are sufficient for the claims.
  • An outline of an approach is described for testing the statistical assumptions underlying the Profile Claims (e.g. a phantom study to test linearity and bias of the measurements from image analysis software; a clinical study to test precision of the measurements).

See Also: Process - Getting to Stage 1

Stage 2: Consensus

Meaning:

  • The Profile represents a consensus of experts within and beyond the Biomarker Committee.
  • The Profile is widely believed to be practical.
  • The Profile is expected to achieve the claimed performance.
  • The Profile is ready for Feasibility testing.
  • The Profile claim and requirements may change based on Feasibility testing results.

Criteria:

  • All public comments have been addressed
  • All open issues necessary for conformant deployment have been resolved
  • Few, if any, groundwork projects remain active
  • All recommended procedures have been tested in one or more groundwork project(s) or referenced studies. (Reasonable deviations from Profile details may exist.)
  • Requirement specifications include requirements and assessment procedures for the statistical assumptions underlying the Profile Claims (e.g. assessment procedure includes sample sizes, details of phantoms and data collection methods, metric computation, requirement specified thresholds).

See Also: Process - Getting to Stage 2

Stage 3: Clinically Feasible (formerly Technically Confirmed)

Meaning:

  • The Profile has been found to be practical. (Reading and conforming to the profile was not burdensome)
  • The Profile is expected to achieve the claimed performance.
  • The Profile is ready for Claim testing.
  • The Profile claim and requirements may change based on Claim testing results.

The BC will generally coordinate with sites/vendors to achieve this stage as it does with groundwork projects

Consider what documentation would be important. Think about how site conformance might be audited.

Criteria:

  • All Profile procedures and requirements have been performed/checked on at least two vendor platforms and at three or more sites and found to be clear and not burdensome/impractical.
  • At least one of each Profile actor have demonstrated conformance (met all requirements).
  • The assessment data from the actor conformance results have been reviewed by a statistician and found to validate the statistical assumptions underlying the claims.

See Also: Process - Getting to Stage 3

Stage 4: Claim Confirmed/Achieved/Demonstrated?

Meaning:

  • The Profile has been found to be practical.
  • The Profile has been found to achieve the claimed performance. (but your mileage may vary)
  • The Profile is ready for Clinical testing.
  • The Profile claim and requirements may change (rarely) based on Clinical testing results.
  • Note that Claims are phrased in terms of technical performance, NOT in terms of clinical outcomes. QIBA focuses on metrology.

Criteria:

  • Groundwork projects are complete
  • All Profile procedures and requirements have been implemented on at least two vendor platforms and at two or more sites
  • All Profile procedures and requirements have been tested in the field as described in the Profile
  • Profile results under experimental conditions are consistent with the Claims.
  • Biomarker measurements collected from at least a few sites meet all the Profile claims.
  • A sample size of X Biomarker measurements are compiled and a CI (from CoV) and it is found to be consistent with the Profile claim.
  • Q. should the results at EACH of the sites also be consistent with the Profile claim?
  • Some claims (for patient-independent metrics) might be confirmable with phantoms/DROs (and may be part of assessment procedures) but likely measurement of performance in phantoms/DRO is not sufficient by itself to declare patient oriented claims to be confirmed.
  • Q. how does all this play into conformance testing?
  • QIBA Registry role? Mountain of data aggregation?

See Also: Process - Getting to Stage 4

(There will be guidance on the level of data/sample size/power, but it will not be specific numbers since there are so many factors) <Want this level to be easy enough to be doable for QIBA, but confident enough to be useful> (Number of scanner models, sites, patients, some variations in data, testing of boundaries) (Difference in Stage 4 and 5 is one of degree/scale not types of activities/process)

See Also

  • Clinically Confirmed Stage
  • QIBA has considered a fifth Stage but it is not part of the current model.
  • The fifth stage would involve some external party with sufficient resources performing a trial similar to Stage 4 but at a larger scale to demonstrate that the profile claims are achievable by most sites that adopt the profile requirements.
  • This would involve more generalizability by exploring a wider spectrum of data and actor characteristics
  • QIBA would potentially advise or provide expertise to such activities
Retrieved from "https://qibawiki.rsna.org/index.php?title=QIBA_Profile_Stages&oldid=18716"