Difference between revisions of "Technical Confirmation Process"

From QIBA Wiki
Jump to navigation Jump to search
m
 
(59 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
{{TOCright}}
 +
The '''[[QIBA Profile Stages#Stage 3: Clinically Feasible (formerly Technically Confirmed)|Clinically Feasible (formerly Technical Confirmation) Stage]]''' '''confirms it is practical for sites to perform the technical requirements''' to conform to the Profile.
  
−
==Feedback during Field Test==
+
Users in the field (clinical oncologists, imaging scientists, pharma, CRO’s, imaging vendors, software providers, etc.) attempt to conform to the Profile and provide feedback on the practicality of the requirements (and the usability of the document).
−
The purpose of this process step is to collect feedback on use of the document.  Whereas the review and public comment steps prior to this focused on the document itself, this step now seeks feedback on initial use of it given that it has been placed into trial implementation.  As such, comments are solicited specifically from those that have used it and only those that have used it.
 
  
−
A procedure is outlined below listing what is done (by whom).
+
Whereas the preceding [[QIBA_Profile_Stages|Public Comment Stage]] collected feedback based on people reading the Profile document itself, this stage seeks '''feedback on how well it works in a pilot implementation'''. As such, comments are solicited specifically from those that have used it and only those that have used it. 
  
 +
Considerations:
 +
* Profile authors have written requirements they believe are within the range of current reasonable practice in clinical use.
 +
* Sites are asked to consider, as they generate the biomarker according to the Profile, whether they would find the requirements '''practical in routine work'''.
 +
:* Sites do not need to commit to using the Profile for all routine work going forward (although it would certainly be positive feedback if they chose to).
 +
* Sites are strongly encouraged to try to meet all the requirements
 +
:* If the site is not comfortable using the procedures for '''patient scans''', they can scan a phantom and note in their feedback that they used a phantom, and what details of the procedure made them uncomfortable.
 +
::* If sites are uncomfortable meeting certain requirements with patients, that would make it hard to deploy the Profile so the Biomarker Committee may need to consider revising that requirement or better explaining/justifying it for sites.
 +
:* Try to follow all requirements, even if scanning a phantom.
 +
::* E.g., if the Profile has required breathing instructions, giving them to the phantom may help a site realize that the wording is incompatible with their preferred practice, or the text is poorly written, etc.
 +
:* If the site is '''unable to conform''' to any requirements, comments on the nature of their difficulty or their reason for not conforming are very helpful to the Profile Authoring Group.
  
−
Period: 30 days
 
  
−
* Collect list of people that have participated in the field test (leader of field test)
+
The following Procedures list what is done ('''by whom''').
−
* Finalize comment form that will be used in the survey (secretariat with technical committee)
 
−
* Send announcement to mailing lists (secretariat)
 
−
** reminder copy of document that has been placed into field test
 
−
** provide a link to the [[:Media:QIBA_User_Feedback_Template-2011.01.doc | QIBA Trial Implementation Feedback Template]]
 
−
* Email comments to Field Test leader (Commenters)
 
−
** commenters include full range of users of the document, e.g., clinical oncologists, imaging scientists, pharma, CRO’s, etc.  However, only people who have used document are included.
 
−
** please use the provided comment form
 
−
* Collate all comments into a spreadsheet (secretariat/Field Test leader)
 
−
** Summarize range of feedback to the questions into a report for review by the technical committee provided by the Field Test leader.
 
−
** Technical Committee decides what changes to make (if any) to the document and whether to refer it to the Modality Committee for publication, whether to conduct another round of field test, or to place the document into an inactive status
 
−
* Record resolution in spreadsheet
 
−
** Regardless of recommended status for this document, ensure that the results are made available to related activities
 
  
 +
==Planning & Recruitment==
  
−
==Feedback after Publication==
+
* Identify sites/people that will participate in the field test ('''Field Test Leader''')
−
The purpose of this process step is to actively seek and periodically respond to feedback on use of published documents.  It is important to put this into context with other activities that may seem similar at first: Whereas the review and public comment steps that are part of the Drafting phase focus on the document itself, prior to its use, and the Collect Feedback step as part of the Field Test phase collects feedback on the initial use of the document given that it has been placed into trial implementation, this step is where longer-term, extended use of the document allows for actual assessment of its performance in practice. That is, over the course of time clinical trials may complete that had been designed with this, and/or other results from field usage may become evident either with the user or supplier base. As such, comments are solicited periodically for a wide range of the public that may connect with it in some way.
+
** At least 3 sites are needed to get reasonable confirmation of results
 +
** Often Profile Authoring Group members may have personal contacts they can explore.
 +
*** "External" sites should be recruited to bring "fresh eyes" to better assess the clarity of the profile and bring different assumptions about routine practice for this biomarker.   
 +
*** Sites where Profile Authoring Group members work may be easier to recruit since there is an existing champion, but they should make a point of engaging other fresh eyes (techs, physicists, radiologists, etc.) and possibly stepping back when it comes time to perform the actual field test.
 +
*** It can be helpful to have a mix of large academic centers, large hospitals, and smaller hospitals or imaging centers.
 +
*** International feedback is desirable due to variations in practice and language; and Profiles are intended to be internationally applicable.
 +
**** ''<TODO: Add links to contacts for Europe, Japan, US, etc.>''
 +
*** Core Labs can be effective field testers since they have experience with a variety of site practices and measurement needs
 +
** It may also be useful to call for volunteer sites
 +
** If a profile covers multiple organs/cases, the goal is to test all of them.  
 +
*** If each site tests all organs/cases, test coverage is simplest; however, it is acceptable for sites to test subsets as long as everything gets covered.
  
−
A procedure is outlined below listing what is done (by whom).
+
* Prepare field test materials/documents ('''Profile Authoring Group''')
 +
** Cover Letter
 +
*** Many times, a meeting is arranged between members of the field test site and members of the Profile Authoring Group to describe the purpose of the field test and answer any questions the site may have.
 +
**** Consider recording the questions/answers since they may point to useful clarifications in the Profile
 +
**** If a meeting is not possible, the information may be conveyed in a Cover Letter.
 +
** Site Details
 +
*** To collect basic site information and prompt general feedback, consider the [[Media:QIBA User Feedback Template-2011.01.doc| QIBA Trial Implementation Feedback Template]]
 +
** Profile Requirements
 +
*** Generally, the Profile Checklists are the core document.
 +
*** The full Profile should also be made available as some sites will find it useful to have the context provided in the Overview and the explanations/clarifications in the Discussion sections.
 +
** Feedback Forms
 +
*** One effective approach is to add two columns to the conformance checklists, to indicate for each requirement whether the site conformed to the requirement, and whether they found the requirement practical for routine use.
 +
*** Additional General comments can be collected on a form similar to Public Comment.
  
 +
==Collecting & Processing Feedback==
  
−
Period: 30 days
+
* Provide field test materials (see above) to each site ('''Profile Authoring Group Leadership''')
 +
* Email feedback to Field Test Leader ('''Commenters''')
  
−
* Periodically, collect list of people that may have used the document and/or professional societies that may have a particular relationship to the document (secretariat)
+
''<TODO More cleanup>''
−
* Finalize comment form that will be used in the survey (secretariat with technical committee)
+
 
−
* Send announcement to mailing lists (secretariat)
+
* Collate all Feedback comments into a '''[https://docs.google.com/spreadsheets/d/1JW-_iJ7vTJvCf1-y4wsxZ_coO8Po2zWqMNDNrBVyUXM/edit?usp=sharing Feedback Resolution spreadsheet]''' ('''Field Test Leader''')
−
** reminder copy of document that has been placed into field test
+
** See the Instructions sheet in the spreadsheet for details.
−
** provide a link to the [[:Media:QIBA_Maintenance_Phase_Feedback_Template-2011.01.doc | QIBA User Feedback Template]]
+
* Assess test coverage ('''Profile Authoring Group''')
−
* Email comments to secretariat (Commenters)
+
** Were all requirements tested.  If coverage is incomplete <including organ coverage>, consider dropping the requirement or document the rationale for finding it technically confirmed in the absence of testing. <some sites may have done less than scheduled/intended><might not have needed coils on hand><BC notes lack of coverage and rationale for how to proceed><are there substantive practical issues, if not OK for Stage 3. Are there substantive performance issues, comes up in Stage 4>
−
** commenters include full range of users of the document, e.g., vendors/suppliers, clinical oncologists, imaging scientists, pharma, CRO’s, etc.  However, only people who have used document are included.
+
* Review comments, making changes to the document as needed ('''Profile Authoring Group''')
−
** please use the provided comment form
+
* Record comment resolutions in spreadsheet.
−
* Collate all comments into a spreadsheet (secretariat)
+
 
−
** Summarize range of feedback to the questions into a report for review by the technical committee provided by the secretariat.
+
 
−
** Technical Committee decides what changes to make (if any) to the document and whether to refer it to the Modality Committee for re-publication of an updated version, whether to branch the document leaving the current version in place but spawning a derivative, or to retire the document
+
(Cmte can decide if the full procedure has been "adequately" tested).  Sites not doing requirement X is key feedback.
−
* Record resolution in spreadsheet
+
* (Doing clinical patients exposes nuances,
−
** Regardless of recommended status for this document, ensure that the results are made available to related activities
+
* <Committee may be faced with a response where none of the sites were comfortable executing the profile on patients and responded based on reading.  In this case, the committee has done two public comments and no technical confirmation and has sites saying they are not comfortable with the profile.  Should the committee advance the profile in this case? That would be pretty negative feedback>
 +
* <Profiles are generally written not to be unusual procedures><NOTE Didn't conform isn't a killer, it's the information we are trying to get. Might be better procedures, or might be superfluous>
 +
* <"Not routine, but can" also useful feedback - can do it but we really have to make the case for the value>
 +
 
 +
 
 +
* Vote on next step for the document ('''Profile Authoring Group'''), either: <clarify with new wording>
 +
** refer the document for Technically Confirmed publication
 +
** conduct another round of field test on revised document
 +
** place the document into inactive status
 +
 
 +
 
 +
* Post '''Feedback Resolution spreadsheet''', revised document to the '''[[Comment Resolutions|Wiki]]''' ('''Field Test Leader)
 +
* If profile requirements didn't change significantly, test sites may be able to claim conformance.
 +
 
 +
<When the TC 2020 version comes out, and a site tried the Consensus 2020 but only missed one requirement. Then the BC relaxed that requirement in the TC2020 (or 2021 explain ) draft, it would be nice if there was "change tracking" so they could see that now they would pass. Could have a "change tracked" document between each draft (but just the preceding, not all the combos, you have to step through the changes of history) Practically, we can point them to the newest, and maybe provide one "generation" of change track>
 +
 
 +
==See Also==
 +
:** An earlier alternative writeup - [[QIBA Protocol/Profile Implementation Feedback Process]]

Latest revision as of 17:27, 19 January 2024

The Clinically Feasible (formerly Technical Confirmation) Stage confirms it is practical for sites to perform the technical requirements to conform to the Profile.

Users in the field (clinical oncologists, imaging scientists, pharma, CRO’s, imaging vendors, software providers, etc.) attempt to conform to the Profile and provide feedback on the practicality of the requirements (and the usability of the document).

Whereas the preceding Public Comment Stage collected feedback based on people reading the Profile document itself, this stage seeks feedback on how well it works in a pilot implementation. As such, comments are solicited specifically from those that have used it and only those that have used it.

Considerations:

  • Profile authors have written requirements they believe are within the range of current reasonable practice in clinical use.
  • Sites are asked to consider, as they generate the biomarker according to the Profile, whether they would find the requirements practical in routine work.
  • Sites do not need to commit to using the Profile for all routine work going forward (although it would certainly be positive feedback if they chose to).
  • Sites are strongly encouraged to try to meet all the requirements
  • If the site is not comfortable using the procedures for patient scans, they can scan a phantom and note in their feedback that they used a phantom, and what details of the procedure made them uncomfortable.
  • If sites are uncomfortable meeting certain requirements with patients, that would make it hard to deploy the Profile so the Biomarker Committee may need to consider revising that requirement or better explaining/justifying it for sites.
  • Try to follow all requirements, even if scanning a phantom.
  • E.g., if the Profile has required breathing instructions, giving them to the phantom may help a site realize that the wording is incompatible with their preferred practice, or the text is poorly written, etc.
  • If the site is unable to conform to any requirements, comments on the nature of their difficulty or their reason for not conforming are very helpful to the Profile Authoring Group.


The following Procedures list what is done (by whom).

Planning & Recruitment

  • Identify sites/people that will participate in the field test (Field Test Leader)
    • At least 3 sites are needed to get reasonable confirmation of results
    • Often Profile Authoring Group members may have personal contacts they can explore.
      • "External" sites should be recruited to bring "fresh eyes" to better assess the clarity of the profile and bring different assumptions about routine practice for this biomarker.
      • Sites where Profile Authoring Group members work may be easier to recruit since there is an existing champion, but they should make a point of engaging other fresh eyes (techs, physicists, radiologists, etc.) and possibly stepping back when it comes time to perform the actual field test.
      • It can be helpful to have a mix of large academic centers, large hospitals, and smaller hospitals or imaging centers.
      • International feedback is desirable due to variations in practice and language; and Profiles are intended to be internationally applicable.
        • <TODO: Add links to contacts for Europe, Japan, US, etc.>
      • Core Labs can be effective field testers since they have experience with a variety of site practices and measurement needs
    • It may also be useful to call for volunteer sites
    • If a profile covers multiple organs/cases, the goal is to test all of them.
      • If each site tests all organs/cases, test coverage is simplest; however, it is acceptable for sites to test subsets as long as everything gets covered.
  • Prepare field test materials/documents (Profile Authoring Group)
    • Cover Letter
      • Many times, a meeting is arranged between members of the field test site and members of the Profile Authoring Group to describe the purpose of the field test and answer any questions the site may have.
        • Consider recording the questions/answers since they may point to useful clarifications in the Profile
        • If a meeting is not possible, the information may be conveyed in a Cover Letter.
    • Site Details
    • Profile Requirements
      • Generally, the Profile Checklists are the core document.
      • The full Profile should also be made available as some sites will find it useful to have the context provided in the Overview and the explanations/clarifications in the Discussion sections.
    • Feedback Forms
      • One effective approach is to add two columns to the conformance checklists, to indicate for each requirement whether the site conformed to the requirement, and whether they found the requirement practical for routine use.
      • Additional General comments can be collected on a form similar to Public Comment.

Collecting & Processing Feedback

  • Provide field test materials (see above) to each site (Profile Authoring Group Leadership)
  • Email feedback to Field Test Leader (Commenters)

<TODO More cleanup>

  • Collate all Feedback comments into a Feedback Resolution spreadsheet (Field Test Leader)
    • See the Instructions sheet in the spreadsheet for details.
  • Assess test coverage (Profile Authoring Group)
    • Were all requirements tested. If coverage is incomplete <including organ coverage>, consider dropping the requirement or document the rationale for finding it technically confirmed in the absence of testing. <some sites may have done less than scheduled/intended><might not have needed coils on hand><BC notes lack of coverage and rationale for how to proceed><are there substantive practical issues, if not OK for Stage 3. Are there substantive performance issues, comes up in Stage 4>
  • Review comments, making changes to the document as needed (Profile Authoring Group)
  • Record comment resolutions in spreadsheet.


(Cmte can decide if the full procedure has been "adequately" tested). Sites not doing requirement X is key feedback.

  • (Doing clinical patients exposes nuances,
  • <Committee may be faced with a response where none of the sites were comfortable executing the profile on patients and responded based on reading. In this case, the committee has done two public comments and no technical confirmation and has sites saying they are not comfortable with the profile. Should the committee advance the profile in this case? That would be pretty negative feedback>
  • <Profiles are generally written not to be unusual procedures><NOTE Didn't conform isn't a killer, it's the information we are trying to get. Might be better procedures, or might be superfluous>
  • <"Not routine, but can" also useful feedback - can do it but we really have to make the case for the value>


  • Vote on next step for the document (Profile Authoring Group), either: <clarify with new wording>
    • refer the document for Technically Confirmed publication
    • conduct another round of field test on revised document
    • place the document into inactive status


  • Post Feedback Resolution spreadsheet, revised document to the Wiki (Field Test Leader)
  • If profile requirements didn't change significantly, test sites may be able to claim conformance.

<When the TC 2020 version comes out, and a site tried the Consensus 2020 but only missed one requirement. Then the BC relaxed that requirement in the TC2020 (or 2021 explain ) draft, it would be nice if there was "change tracking" so they could see that now they would pass. Could have a "change tracked" document between each draft (but just the preceding, not all the combos, you have to step through the changes of history) Practically, we can point them to the newest, and maybe provide one "generation" of change track>

See Also

Retrieved from "https://qibawiki.rsna.org/index.php?title=Technical_Confirmation_Process&oldid=19632"