Process Coordinating Committee

Chairs: Kevin O'Donnell, MASc (Chair); Michael Boss, PhD (Vice Chair)

RSNA Staff Support: Susan Stanfa

• Roster
• Attendance, Quorum and Voting Eligibility

Scope

The Process Committee facilitates the QIBA Coordinating Committees and their profiling activities by defining processes and tools to promote consistent quality work product.

• Develop common Templates & Processes
• Support & mentor adoption of above
• Drive consistent content & format of Profiles
• Support infrastructure for Committees

Meeting Agendas/Summaries

Biweekly Tcons, 1st and 3rd Tuesdays of each month @ 2-3pm Central Time (Contact qiba@rsna.org for details)

• Quarterly agenda topic: What to highlight during next CC call slots. See Process Education for potential topics.

Agendas:

• 2022-09-06 Process Committee Agenda
• 2022-08-16 Process Committee Agenda
• 2022-08-02 Process Committee Agenda
• 2022-07-05 Process Committee Agenda
• 2022-05-17 Process Committee Agenda

Process Committee Agenda Archive

Approved call summaries:

• November 1, 2022
• October 18, 2022
• October 4, 2022

Process Committee Call Summaries Archive

Working Documents

• Claim Guidance
• Assessment Procedure Guidance
• QIBA Profile Stages
• Process Education
• Profile Writing Guidance Ideas
• Wiki Use Cases

New Topics

Review, prioritize, and discuss which might merit a Task Force (to work between Process Cmte Meetings)

• Editors Meetings

• Put in Current work - consider TF - eventually will be a separate meeting track for the actual meetings anyway - would be nice to convene by start of summer
• Perhaps Quarterly
• 1) Identify the editors (See Dashboard - ask CC's to confirm Row 12 if their dashboards are up to date) - email to CC cochairs
• 2) Heads-up email to editors - explain intentions, ask for suggestions of topics/issues/questions and logistics/timing
• 3) Consider face-to-face time when possible (QIBA Annual Meeting, RSNA Annual Mtg) - happy hour

• Committee Expertise Roster

• Dashboard item to confirm representative(s) for major stakeholder groups

• Radiologists (incl. academic, private practice (might challenging in terms of time) but if we get ACR traction, and QIBs are billable, interest might grow), Physicists, Technologists, Scanner vendors, SW vendors, CRO (implications for recruitment, burden for conducting), Pharma (biological use of the metric, claim details), Statisticians, Regulatory (device regulators, biologic regulators), ...
• Q. If there are multiple, who do you list?

• Stakeholder representatives are liaisons responsible for engaging with their stakeholder community
• If the BC has a roster "gap", need extra engagement effort during public comment, or periodically on a focused call, invite to face-to-face meetings

• Q. What does the testing & conformance process look like for a Clinical Trial to be QIBA Profile Conformant? (Andy, Gudrun)

Current Work

Maintenance Process and Profile Edition & Version Guidance

• New Steering Committee Priority?
• Discuss some basic concepts then spawn TF if needed

• Conventions for Published Profile edition/version names

• Profile Editions
• Examples
• Address same-stage updates, not just stage transitions
• This is important - when in a stage, how do we update (change requirements) prior to moving to new stage.

• Conventions for internal filenames during Profile development

• Change Tracking and File Naming
• Lower priority but relevant

• Tracked Change Process

• Maintenance Process
• This is probably needed for the Profile Editions to ensure managed change not arbitrary.

• Do as one work item or divide into multiple - some complexity

• Key people - MRE (Kay), FDG (Gudrun will ask Jeff to recommend), DWI (Mike)

Streamlined Profile Template

• Mike & Kevin taking first pass at streamlining Profile document (some additional points in the draft template)

• Revisit the topic of dropping requirements post-TechConf to see how feedback informs
• Flavors of possible drops: (Some things are always being done, others are not affecting the claim even if not done)
• Reluctance to discarding information that took effort/time to develop (keep? preserve in appendix?)
• Streamlining is partly moving things around for better ordering, some is taking things out (or moving to appendix)
• Plan a small call on DWI - Mike/Kevin

• Are there structures we could think of to deal with the multi-organ usage of DWI? Optional modules? Multi-profiles with common parts?

• Once we vet it internally, we can make it available for voluntary use/conversion by committees.
• Don't want to distract committees, or force "unnecessary" work.
• Plan to publicize it in the CC meetings and promote the benefits/concepts.

• Don't want to blindside any BCs. Maybe start telegraphing the situation now... Reassure we're not going to force rewrites.
• Harmonize assessment procedures across profiles

Estimates of Precision (Wrap)

• TODO Need some guidance text (like Criteria page) for BCs to understand; keep as a high priority item. Need to tidy up following text
• Nancy will work on some guidance text in Word for later in March; after review can fold into Wiki.
• Homework for Nancy & Kevin & (Nancy designee)(Nick?)

• Estimates of Precision when BC Stages Profiles and when Sites Conform to Profiles

• This came out of issues raised by Nancy about when/where/who should do test/retest type estimates of precision
• The following rough notes are from the 4/5 meeting

• Address such Estimation as part of conformance assessment of each site and/or actor
• Discuss BC choice to include assessment of actors/site precision as profile conformance requirement
• Detail situations that would necessitate such profile requirements
• Made all BC put in a Section 4 procedure to test linearity and bias with phantoms to ensure estimate is under the claim.
• Q. Do we require the BC to do it once? Or do we require each site to do it as part of conformance?
• At Stage 3, we might not need it as BC, but if we suspect we’re going to need it, we should test the practicality.
• How much of a burden is test/retest? It varies depending on the amount of data required. When phantom data captures the variability you are looking for, it can be fairly easy (assuming access to the phantom is easy and the time is short).
• BC also chooses the pass/fail threshold that provides reasonable confidence in meeting the claim.
• If we see profiles mostly stopping at Stage 3, does it make sense to include a test/retest
• Given that Stage 4 is a lighter weight Stage 5, What if we had an even “lighter-weight” that is not enough to reach Stage 3 but is somehow better.
• Gap: we’re not doing groundwork with the procedures we have made in the profile (Basically that’s Stage 4) – it’s better if there have been Stage 1 groundwork that are roughly following the requirements, but most profiles do not have this.

• Key Point is Communication of what the conformance means at each level

• The profile might state at in/near the claim what conformance means
• There is value to all levels of conformance
• Stage 2 - clearly(?) the claim is off the table. Stage 4 - clearly the claim is on the table.
• Stage 3 is the current "danger" area.
• So conformance to Stage 3 does not require 95% testing, but if a site is willing to do that (and thus help the BC on it's way to getting the profile to Stage 4 where everyone who then conforms can invoke the claim) how do we document/acknowledge/credit/thank them for that.

• Guidance for how to design the Stage 3 to Stage 4 study. Note that the BC can decide to either tighten requirements or loosen claim if they two don't match. - NOTE this changes the design of the study. BC might need to weigh in which way they want to go before designing the study.
• Some profiles have very stringent testing (e.g. claim says wcv is 10%, the profile/actor must show with 95% CI that wcv <=10%) in that we are requesting they confirm the claim directly as part of conformance. Need to better distinguish what BC does for Stage 4 vs what a Profile requires for conformance.

Stage 3 Resolution Template (Wrap)

• Have the templates TODO need to finish cleaning up the Process Page, check above bullets, then close out.
• Homework - Kevin - links/pointers
• Map out how to collate practical feedback from sites into a sheet for resolution similar to public comments.
• Use the checklists - it's what they'd see in the field and it's "simple" - they can scan and we'll transcribe
• Additionally, the overall questions solicit broader comments which is really useful as well.
• Don't want identification being a barrier to feedback. But it's nice to be able to followup.
• Be VERY clear that we don't intend to broadcast the site details
• Do some clean up and run it around the coordinating/biomarker committees for feedback.
• Cover letter to solicit broad feedback, and the checklists both feed material into the resolution sheet.
• <See notes on Process Page for Tech Conf>
• Discussion Point: To what degree are we collecting feedback (survey) vs doing a dry run of site conformance process vs both

• Consider starting some Blog threads e.g. feedback on the Tech Conf

• Conformance Process

• What should go into a Conformance Record? - (e.g. findings from Technical Confirmation tests)
• Need a common view of what conformance entails, how it is achieved, what it looks like to a product, how it is documented, etc.
• Then we need to ensure that the QIBA Templates/Process supports those things clearly.
• Kevin will put some initial thoughts on a Wiki page and we can review/add
• How often should conformance assessment be done, who will assess, who will keep records?
• Should QIBA get into hosting the Conformance Statements on our site (and then we could link to them from the Profiles page which would be very handy)
• Consider the RSNA Image Share Conformance Assessment Program and whether we should do something similar for QIBA

• Process committee can think through the details of a process and confer with RSNA whether they would be interested

Priority (Next)

Profile Selection Process (SCP)

• Move into Current Work - Consider setting up a TF - gauge participants - below Version/Naming
• Review/update review criteria/evaluations for new BC proposals

• Harvest from past experiences

• Profile Success Factors and Challenges Survey Planning

• Given limited resources, how to approve not just good proposals but the best proposals

• It's not just about saying "Yes". It's about When should we say "No". When should we say "Not Yet"

• Evidence that the biomarker already has vendor and user adoption and just needs the "last mile" improvements that QIBA brings.

• Consider Prior Project completion when Steering Cmte reviews new Proposals from the same PI.

• timing overlap of prior cycle completion period with new proposal period needs to be one considered.
• possible solution would require final reports for prior COMPLETED funding cycles before considering a new project proposal

• Discuss which profiles have been most successful, which have hit bumps, which have foundered

• What characteristics of each contributed to the success/problems? Can we look for those during selection?
• As we refine criteria for new proposals, also see how they do/don't apply well to existing profile work

• Discuss Nancy's criteria for ultimate Claim Confirmation?

• BCs should understand what they are getting into and start planning for it
• If the profile will depend on a large test/retest being completed and it seems highly unlikely that would ever be doable, should that weigh on the SC approving the BC?

ToDo

Profile Format Harmonization

• Various profiles differ in their layout and content which makes it harder for users

• Use of New Templates has not been mandated, but rather left to committee discretion so as not to be disruptive
• Various committees have deviated from the template and/or introduced variations without discussing with the Process Cmte

• Consider starting with a table showing which template each profile is using

• Process makes table; Tags for Informal template; Formal#1, Formal#2 (Streamlined) and other divergence notes
• CCs get it populated; SC decides on priority/mandate? of harmonizing
• Use to inform "harmonization pressure" efforts?

Communicating Uncertainty to Radiologists

• "No measurement is complete without a statement of it's uncertainty" - NIST
• Measurement uncertainty is at the heart of what QIBA does
• If people don't recognize/appreciate measurement uncertainty they can't grasp the value of QIBA fully (or at all?)
• How do we get radiologists to appreciate the uncertainty of each measurement and incorporate it into their thinking and decision making?
• What should best practice be for communicating the uncertainty of each measurement to radiologists?

• Always give a confidence interval rather than a value?

On-boarding Process

• Goal: Help new committees, new members, new authors, new reviewers, new adopters
• Communication Committee might shoulder some of this too and only come to Process when gaps are found.
• Reviewed "How to develop a Profile" Guidance

• Email How-To guide to current authors and ask them to read/feedback
• Then email to the current community since it is helpful for all contributors

• Maybe update Profiles and Protocols page to focus less on UPICT and more on QIBA Profiles today.

• Linear layout
• Explain the mechanics of who does what with a profile (vendors, rads, techs, regulators, etc.)

• Maybe add some "How to Read a Profile" guidance (also useful for reviewers), e.g. focus on Spec tables. Look at your relevant Activities/Procedures. Read the Discussion sections for more background.

• Explain about the Claims, the Actors, the Requirements and the Assessment Procedures structure of a Profile
• Advice may vary depending on viewpoint of the reader.
• Consider/discuss relative merits of moving the checklists from Annex E into Section 5 of the template

• Joe's intro email, current template and stage docs, QIBA Fact Sheet

• Find a way to link directly into the "current" version of documents without having to update the links. Now we link to the Profile template page then you have to click a second time put pull up the template.

• Could use a "fresh eyes"/newcomer commentary - find a newcomer guinea pig

• Kevin will work with Joe to get it into a Google Doc for revision and linking from the Wiki.
• https://docs.google.com/document/d/1jB1CPy1Mll_v-yzvSZk7acYv_SfC3VDAgS_WJ3y2pm8/edit?usp=sharing
• Include link to How-To guide
• Some people might like videos/webinars

• Pictures and diagrams would help a lot for people reading what is "dry" text.
• Types of newcomers/stakeholders - link from an "intro to QIBA - mother page" - Stakeholder Viewpoints

• Practicing QIBA for Radiologists (Dan), QIBA for Physicists (Tom Chenevert, Sam Armato, Paul K.), QIBA for Technologists (Mike), QIBA for Imaging Vendors (Kevin), analysis SW vendors/developers (Cathy?), researchers, clinical trialists, pharma (Gudrun?), QIBA for Core Labs (Pat?), QIBA for Regulators (both device and drug?) (Nick), patients/advocates/lay public, government policy, payers, QIBA for Statisticians (Nancy)
• DONE targeted intro to each?groups? of those viewpoints

• What is QIBA basically about, what is the key value of QIBA to them, how do they "use"/"interact with" QIBA (some of them are "actors", some are not)
• easy language, a few paragraphs, and links to specific wiki pages/documents/slides that would be a good next step for them

• Maybe add a "First Steps" section for each page (see QIBA for Statisticians for example) telling them the first things they could/should do
• Might be good to also have "stories from your colleagues" who are already in QIBA and their experiences.

• Contact page on wiki? Point to qiba@rsna.org for questions/info on any committee, QIDW, etc.

Publication Process

• Ask Steering Cmte to determine whether having others publish is a necessity or not (Mike to draft some pro/con points - Kevin can help)

• Do the vendors and sites and trials that would be involved in using Profiles have an issue with these "standards" being published by QIBA (RSNA)?
• Would it be better if QIBA Profiles were balloted/published by another organization? IEC? IHE? AAPM? MITA?
• Would there be an impact in switching to a different document structure, formatting conventions, review/approval processes?

Clinical Confirmation Process

• Methodology/Study Design guidance/requirements for achieving "Clinically Confirmed"

QIBA Tooling

• How to use Google Docs?

• Issues with references, access restricted by federal agencies and some companies

• Round trip fidelity between Word-Google-Word

QIBA Wiki

• QIBA Wiki Improvements
• Wiki serves QIBA Committee work. Consider a new user oriented site (clinicians, administrators)?
• Wiki vs RSNA.org

• Address relationship between wiki content (current, easy to update) and website content (authoritative, stable)

• Likely RSNA staff will periodically mirror wiki content up onto the website; address this in publication process.

• Consider how to handle handoffs between RSNA website

• RSNA website is controlled by the RSNA Marketing Department, must meet certain style requirements, and require resources beyond QIBA staff
• QIBA would still like to find a way for the more formal publications (e.g. later stage profiles) to be accessible from the RSNA website.

Literature Search Guidance

• Process/best practices for review of literature around a new biomarker/profile (search is relatively straight forward)
• Assemble draft guidance to kick this off (Hendrik Laue, Ona Wu, Caroline? Chung)

• See existing material - http://qibawiki.rsna.org/index.php/Tools_and_recommendation_for_structured_literature_review

• Where do you store references, what kind of search works best, what exactly are you looking for?
• Also meta-analysis guidance to determine sources and levels of variance
• PDF/DWI - using Google Sheet to track/tabulate/organize - inherently input from many sources so nice to have centralized, commonly edited, accessible

Product Reference Guidelines

• Draft Guidelines for Reference Implementations and commercial products in Profiles

• e.g., the underlying algorithm should be open/public/available/documented, QIBA will list all comers, non-commercial?, etc)

• Level playing field, no favorites

Public Comment Attribution

• don't push for anonymity. Encourage people to identify themselves (deserve credit and we want to be able to followup)
• form can accept anonymous, but encourage constructive criticism that will be taken in the spirit given.

Hold

Todo items that are on hold pending some trigger:

Claim Guidance

• Hold pending people with questions/clarification needs on current text

Guidance Document Process

• Hold pending interest in "lightweight" QIBA output without claims/conformance
• Basically "best practice" recommendations
• Less rigor; Much easier to write,

Profile Retirement Process

• Hold pending a profile we want to retire
• When do we "withdraw" or at least stop maintaining/promoting a Profile
• Review/borrow from corresponding IHE Process - http://wiki.ihe.net/index.php/Evaluation_of_Published_Profiles
• Review DICOM retirement concept (stays available in archives, not maintained/promoted)

Profile Disclaimer

• Hold pending revist of Discriminatory Claims
• For now QIBA addresses technical performance. Getting into the clinical performance associated with Discriminatory Claims would open a lot of issues.

Statistical Assumption Testing Guidance

• Conformance with the statistical assumptions is required at the various QIBA profile stages.
• Need guidance on the assumptions in the profile, validating the assumptions and testing actors for meeting those assumptions.
• Symmetrical vs Assymetrical change intervals?
• Questions about the numbers in "Academic Radiology: ?? Paper"

Past Work

This isn't all of it but here's some notes...

Conflict of Interest Policy

• Should have a policy statement on the wiki on QIBA expectations (e.g. what kind of things might come up and how might handle, e.g. recusal from votes or ballots)

• Dan has extracted some example wording used elsewhere (e.g. protocol approval committees, other orgs, AAPM)
• Is there a personal (financial?) benefit to the individual
• This may link into the question of getting cochairs to sign RSNA conflict of interest policy

• AGREE - guidance for guidances.

• DONE - prepped language with examples and forwarded to SC/EC for approval and posting to the wiki

Public Comment Resolution

• DONE Set up Wiki page for holding all the comment resolution records from past public comment. - Profiles
• Work with RSNA on a boilerplate email to send to commenters (who gave their email on the comment form) pointing to the resolutions document for their submitted comments. (Staff will draft some proposed text for review)

• (DONE) Could build it into the automatic confirmation message that the confirmation form sends out.

• DONE Revised process to highlight circulating resolutions with ballots and posting by RSNA Staff when complete
• DONE Updated resolution page
• ONGOING get additional profiles to post their resolutions

Coordinating Committee Membership Criteria

• Review the current criteria Committee_Organization#Committee_Membership
• Feedback to SC/EC

• Remove Modality Scientific Liaison (can bring in SMEs as voting/non-voting)
• Remove explicit cap on voting membership
• Flag "additional voting" as "At Large Members - suggest 2-4 people"
• Keep all Co-chairs of BCs
• Suggest adding 3-year term (renewable) for CC xChairs - RSNA Staff will think through logistics - currently doing a CC membership review - will be some turnover. Staggered rotations is desirable.
• Process Cmte participation (periodic education agenda item like for BC) is useful, but can fit easily under the non-voting SME umbrella.

Profile Ballot Reminders

• Consider something like the following:

Please indicate below whether this Profile

• meets the criteria for this stage [link to criteria]
• conforms to profile guidelines [link to guidelines]
• is of sufficient quality to publish

Note: if you are unable to read through the profile, you may Abstain.

Requirement Criteria

• Making profiles shorter...

• Drop requirements that aren't critical
• DONE - devised, documented and communicated an approach to Vetting Requirements

• Impact Criteria Assessment: Asking existing profiles to go back and fill in a column may invoke some resistance to the effort required.
• One approach: Have the BC go through all the requirements and do an impact pruning just before Pub Cmte release

• Don't require long winded justifications, a short clear statement should be sufficient

• QIB Guidance documents (not Profiles) can have a role since they don't have a claim so dropping "requirements" is less of an issue

Driving Profile Adoption

• Adoption of Profiles

• Maybe a new TF to support Adoption (can include some Process people)

• Ed, Brad, Joe, Kevin, Dan, RSNA Webperson
• Start by looking at what is on the RSNA QIBA pages,

• Biomarker Adoption Steps - Framework for when/how site stakeholders are engaged

• Basis for reviewing organization/content/structure of current QIBA documents/deliverables

• Ultimately addressed by formation of QIBA Sustainability Implementation Committee (QSIC)

• For further details, see QSIC page
• Process Cmte can support QSIC when process questions/refinements come up.

Committee Leadership Process

• Address BC and CC leadership
• Continue with variations of co-chair/chair/vice-chair? - Maybe call everything below Steering Cmte as co-chairs and let them work out working patterns.
• Describe process for nominating & selecting that is transparent, allows time for nominations & self-nominations, encourages diversity

• Mostly current system has been working without major issues, but would like to have efficient flow rather than everyone re-thinking/re-inventing a process

• Process for recognizing cases where cochair availability/activity has decreased and need to call-the-question and trigger "replacement".

• Having "terms" would help trigger. E.g. 3-year terms, and limit of two terms to promote turnover, new faces, new ideas in some committees, on the other hand keeping experienced people and having continuity is good and there may be a limited supply of motivated leadership.
• CC term limits might be OK, but BC should maybe not have limits because they are intended to a have a limited life to start, and the chair is a key leader and change could falter
• If we stick with 3 year terms, might want to have a procedure where a cochair missing 8 meetings in a row triggers the secretariat doing a checkin to confirm that they still have the bandwidth/availability, and give them the opportunity to step aside if their other responsibilities have intruded.

• Stakeholder diversity (silent, recommend, or require)

• Having "only academic representation" is an adoption risk to QIBA, so there is a value to making some effort to have diversity
• Variability in the size of some committees so tough to enforce
• Lets Strongly Recommend (and maybe make it a review point in the Coord Cmte and the Steering Cmte to ask about attempts to improve?)
• Also consider this when forming a Biomarker Committee - require? At least formally build into the evaluation process.
• Consider handling committee stakeholder diversity in the committee formation and dashboard process, not the leadership process
• Define Stakeholder diversity; do we make user/vendor classes? or is one academic radiologist and one clinical trial radiologist the intended degree of diversity?

• For comparison - DICOM:

• For Working Groups it's required but unconstrained "Working Groups shall elect a chair or co-chairs."
• For the DSC (Steering Cmte) it's more constrained

• There shall be two co-chairs – one representing a producer and one representing a biomedical professional organization.
• Each co-chair shall serve a two-year term and may serve no more than two consecutive terms. The terms of the co-chairs shall expire in alternate years.
• At the meeting preceding the meeting before the last Committee meeting each year, the co-chairs shall appoint a three-member nominating subcommittee consisting of at least one representative of a producer , one representative of a biomedical professional organization and one former committee officer (if one is available). The former officer (if one is available) shall chair the nominating subcommittee. Members of the nominating subcommittee shall develop and submit to the secretariat a slate of candidates. The names of the candidates should be submitted to the secretariat no later than 30 days prior to the meeting preceding the last meeting of the year so that their names may be included in the published agenda. Additional nominations may be made from the floor at this meeting.
• Prior to the last meeting of the year the secretariat shall conduct a letter ballot. The name of the person elected shall be announced at the last meeting of the year. That person shall assume office immediately following the last meeting of the year.
• The Committee has the right to recall a co-chair.

• For comparison - IHE:

• There are general cochair guidelines in section 12.2.7 of the Governance: https://www.ihe.net/wp-content/uploads/2018/07/IHE-International-Principles-of-Governance.pdf
• The Board adds the constraint "The Board shall elect as co-chairs, two Board Members drawn from Member Organizations in the User category. They will serve as co-chairs for renewable two-year terms."
• The Domain Planning Cmtes adds the recommendation "It is recommended that one Domain Planning Committee co-chair be selected from the User member category and one from the Developer member category."

• For comparison - RSNA:

• A nomination announcement contains a link to an online nomination form and deadline

• A comment box provides for nominators or self-nominators to state why they wanted to serve/would be a good addition to the leadership

• Once the nomination period ended, another announcement contains a link to an eballot listing nominees and their qualifications

• Joe helpfully pointed out that we need to clearly define voting eligibility (i.e. do voting privileges apply to chair selection or can everyone vote)

See Also

This page is based on the Committee Template. Add any good ideas you have to the template.