Technical Confirmation Process

This step confirms it is practical for sites to perform the technical requirements to conform to the Profile.

Users in the field, including clinical oncologists, imaging scientists, pharma, CRO’s, imaging vendors, software providers, etc. attempt to conform to the Profile and provide feedback on the practicality of the requirements (and the usability of the document).

Whereas the preceding Public Comment Stage collected feedback based on people reading the Profile document itself, this stage seeks feedback on how well it works in a pilot implementation. As such, comments are solicited specifically from those that have used it and only those that have used it.

The point is to find out if it can work. We are not asking sites to adopt the profile in their routine work (but to think about whether their pilot experience indicates it would be practical to use it in routine work).

• bullets for comprehension and execution
• Describe flexibility that we "strongly encourage" patient scans, but it's up to sites if they scan a phantom instead, but do try to follow all the steps regardless, e.g. give breathing instructions to the phantom. Tell committee if you used a phantom. (Cmte can decide if the full procedure has been "adequately" tested). Sites not doing requirement X is key feedback.
• If a site doesn't follow we want to know why. That's important information.
• (Actual execution helps find places where the profiles are not clearly written)
• (Doing clinical patients exposes nuances,
• <Committee may be faced with a response where none of the sites were comfortable executing the profile on patients and responded based on reading. In this case, the committee has done two public comments and no technical confirmation and has sites saying they are not comfortable with the profile. Should the committee advance the profile in this case? That would be pretty negative feedback>
• <Profiles are generally written not to be unusual procedures><NOTE Didn't conform isn't a killer, it's the information we are trying to get. Might be better procedures, or might be superfluous>
• <"Not routine, but can" also useful feedback - can do it but we really have to make the case for the value>

The following Procedures list what is done (by whom).

Planning & Recuitment

• <Outreach "personal" focused and broad>Announce Call for Feedback to mailing lists (RSNA Staff), including
  • link to the document
  • link to the QIBA Trial Implementation Feedback Template
• Collect list of sites/people that will participate in the field test (Field Test Leader)
  • At least 3 sites are needed to get reasonable confirmation of results
  • If the profile covers multiple organs/cases, it is not required (but is convenient) for each site to test all cases. But each case must get tested at multiple sites (e.g. 5 sites in total, each testing overlapping subsets of cases)
• Prepare the comment form that will be used to collect results (RSNA Staff/Technical Committee)
  • One effective approach is to take the conformance checklists and for each requirement add a column to indicate if the site could do it, and a column to indicate if they found it practical to do. General comments can be collected on a form similar to Public Comment.
  • describe considerations when BC member sites participate - need the interest, want some "fresh eyes" - don't be too strict on excluding since the BC member is only one person and they will be engaging multiple other fresh eyes in the techs, physicists, rads, etc.
    • They work at large academic centers, there are certainly independent reviewers, and what did we learn from EARL
    • International feedback is always desirable due to variations in practice and language. And QIBA is trying to write International Specs.
    • Link to contacts for Europe, Japan, US, etc.

Collecting Feedback

• <Send materials to the site with instructions>
• Email completed Feedback Template to Field Test leader (Commenters)
• Collate all Feedback comments into a Feedback Resolution spreadsheet (RSNA Staff/Field Test Leader)
  • <Currently some flexibility is allowed in the form of the document. Process Cmte is working on a more tuned template>
  • Document test coverage: were all requirements tested. If coverage is incomplete <including organ coverage>, consider dropping the requirement or document the rationale for finding it technically confirmed in the absence of testing. <some sites may have done less than scheduled/intended><might not have needed coils on hand><BC notes lack of coverage and rationale for how to proceed><are there substantive practical issues, if not OK for Stage 3. Are there substantive performance issues, comes up in Stage 4>
• Additionally, summarize feedback into a report (Field Test Leader).
• Review comments, making changes to the document as needed (Technical Committee)
• Record comment resolutions in spreadsheet
• Vote on next step for the document (Technical Committee), either: <clarify with new wording>
  • refer the document to the Modality Committee for Technically Confirmed publication
  • conduct another round of field test on revised document
  • place the document into inactive status
• Post Feedback Resolution spreadsheet, revised document to the Wiki (Field Test Leader/RSNA Staff)
• If profile requirements didn't change significantly, test sites may be able to claim conformance.

<When the TC 2020 version comes out, and a site tried the Consensus 2020 but only missed one requirement. Then the BC relaxed that requirement in the TC2020 draft, it would be nice if there was "change tracking" so they could see that now they would pass. Could have a "change tracked" document between each draft (but just the preceding, not all the combos, you have to step through the changes of history) Practically, we can point them to the newest, and maybe provide one "generation" of change track>

Feedback after Publication

The purpose of this process step is to actively seek and periodically respond to feedback on use of published documents. It is important to put this into context with other activities that may seem similar at first: Whereas the review and public comment steps that are part of the Drafting phase focus on the document itself, prior to its use, and the Collect Feedback step as part of the Field Test phase collects feedback on the initial use of the document given that it has been placed into trial implementation, this step is where longer-term, extended use of the document allows for actual assessment of its performance in practice. That is, over the course of time clinical trials may complete that had been designed with this, and/or other results from field usage may become evident either with the user or supplier base. As such, comments are solicited periodically for a wide range of the public that may connect with it in some way.

• Periodically, collect list of people that may have used the document and/or professional societies that may have a particular relationship to the document (RSNA Staff)
• Finalize comment form that will be used in the survey (RSNA Staff/Technical Committee)
• Send announcement to mailing lists (RSNA Staff)
  • reminder copy of document that has been placed into field test
  • provide a link to the QIBA User Feedback Template
• Email comments to RSNA Staff (Commenters)
  • commenters include full range of users of the document, e.g., vendors/suppliers, clinical oncologists, imaging scientists, pharma, CRO’s, etc. However, only people who have used document are included.
  • please use the provided comment form
• Collate all comments into a spreadsheet (RSNA Staff)
  • Summarize range of feedback to the questions into a report for review by the technical committee provided by RSNA Staff.
  • Technical Committee decides what changes to make (if any) to the document and whether to refer it to the Modality Committee for re-publication of an updated version, whether to branch the document leaving the current version in place but spawning a derivative, or to retire the document
• Record resolution in spreadsheet
  • Regardless of recommended status for this document, ensure that the results are made available to related activities