Difference between revisions of "Process Coordinating Committee"
|Line 34:||Line 34:|
:* Biomarker Adoption Steps: https://docs.google.com/document/d/1YM0mreBRLNNQZicDatx1IE3Zi1NZeaojQJczrEIbvsk/edit?usp=sharing
:* Biomarker Adoption Steps: https://docs.google.com/document/d/1YM0mreBRLNNQZicDatx1IE3Zi1NZeaojQJczrEIbvsk/edit?usp=sharing
Revision as of 23:00, 21 April 2020
- Chairs: Kevin O'Donnell (Chair); Michael Boss, PhD (Vice Chair)
- RSNA Staff Support: Susan Stanfa
The Process Committee facilitates the QIBA Coordinating Committees and their profiling activities by defining processes and tools to promote consistent quality work product.
- Develop common Templates & Processes
- Support & mentor adoption of above
- Drive consistent content & format of Profiles
- Support infrastructure for Committees
- Upcoming: Biweekly Tcons, 1st and 3rd Wednesdays of each month @ 3-4pm Central Time (Contact Susan for details)
Approved call summaries:
- Wiki Use Cases - https://docs.google.com/spreadsheets/d/1bVP9pSjxtCVinlASCAeGDf5vjc2lCfj16qEg7hkft0Q/edit?usp=sharing
- Guidelines Page
- Fixes - how does this work in practice - sometimes not clearcut data for whether it impacts or how much of an impact.
- guidance for weighing?
- Academnicians are grounded in questioning assumptions and data
- due to lack of data might err on the side of caution/control?
- "didn't have time to write a short profile" because that would have taken many more groundwork projects...
- need to do more to do less; making it simple is more complicated
- involve techs in the drafting phase
- Editing Profiles Effectively
- 2 stages of profiling - earlier editors - long time, long ago. Newer profile - templates and process in place and whipped through it.
- Xavier, lock 4 people in a room for a weekend to get it done.
- Encourage BCs to have longer calls?
- [Components - longer calls, less boot up time, more focus, fatigue-driven consensus (not a bad thing), timeboxing, editor-driven offline work and progress, farming out subsections as homework]
- vague agenda issues? Lack of time to prepare? Presence of personalities
- RSNA Staff have tried getting BCs to have agendas prepared for calls, BC Leaders don't always respond - time limited for leaders to prep
- RSNA Staff tries to nudge toward agendas
- Coordinating Cmtes have all the BC chairs and might be good venues to promote productivity of the BCs.
- could have a standing item on Operational Reminders
- Profile writing basics 101? (not popular?)
- Build the culture. Agendas are important. Focusing on specific profile issues, offline work
- Onboarding of editors?
- is there room on the agenda? RSNA Staff - likely would have time
- is there BC chair participation? RSNA Staff - unfortunately true. BC representation has gaps.
- Alex - some need to light a fire, but factors like committee size, personality, come into play
- Plan to Pilot in May - Dan/Alex/Kevin - First Topic - goal/benefit of brevity and guidance on how to achieve it - Guidelines - Second Topic - lead editors and training
- Discussion of leaving in strawman requirements for PC then culling later.
- Highlight that you should expect some cutting, but don't want to always go for 2nd PC. What are the guidelines?
- Hard to say since small changes might have a big impact. The affected community should have a chance to input. Consultative process not approval.
- Biggest benefit
- finding blindspots, dealbreakers
- form of peer review/validation of the quality/content - need a bigger audience - get the criticisms when we can adapt to them
- lots of time writing - can loose the forest for the trees - process can build group-think/bias - need review to counteract that
- more technologist involvement - practical on the ground, key actors, understanding of site workflow - ARRT outreach?
- promotion/socialization of the material - like a pre-print - builds awareness - expands circle of community
- formalizes a window of entry for constructive criticism
- Biomarker Adoption Steps: https://docs.google.com/document/d/1YM0mreBRLNNQZicDatx1IE3Zi1NZeaojQJczrEIbvsk/edit?usp=sharing
- Basis for reviewing organization/content/structure of current QIBA documents/deliverables
- Conformance Process
- What should go into a Conformance Record? - (e.g. findings from Technical Confirmation tests)
- Then we need to ensure that the QIBA Templates/Process supports those things clearly.
- Kevin will put some initial thoughts on a Wiki page and we can review/add
- How often should conformance assessment be done, who will assess, who will keep records?
- Draft standing agenda item for Coordinating Cmtes to remind/confirm BC committees keeping their BC Wiki page up to date (especially links to current draft documents)
- Public Comment resolution
- DONE Set up Wiki page for holding all the comment resolution records from past public comment. - Profiles
- (DONE) Could build it into the automatic confirmation message that the confirmation form sends out.
- TODO get additional profiles to post their resolutions
- On-boarding process
- Goal: Help new committees, new members, new authors, new reviewers, new adopters
- Reviewed "How to develop a Profile" Guidance
- Email How-To guide to current authors and ask them to read/feedback
- Then email to the current community since it is helpful for all contributors
- TODO update Profiles and Protocols page to focus less on UPICT and more on QIBA Profiles today.
- Linear layout
- Explain the mechanics of who does what with a profile (vendors, rads, techs, regulators, etc.)
- TODO Add some "How to Read a Profile" guidance (also useful for reviewers), e.g. focus on Spec tables. Look at your relevant Activities/Procedures. Read the Discussion sections for more background.
- Explain about the Claims, the Actors, the Requirements and the Assessment Procedures structure of a Profile
- Advice may vary depending on viewpoint of the reader.
- Joe's intro email, current template and stage docs, QIBA Fact Sheet
- Could use a "fresh eyes"/newcomer commentary - find a newcomer guinea pig
- Kevin will work with Joe to get it into a Google Doc for revision and linking from the Wiki.
- Include link to How-To guide
- Pictures and diagrams would help a lot for people reading what is "dry" text.
- Types of newcomers/stakeholders - link from an "intro to QIBA - mother page" - Stakeholder Viewpoints
- Practicing QIBA for Radiologists (Dan), QIBA for Physicists (Tom Chenevert, Sam Armato, Paul K.), QIBA for Technologists (Mike), QIBA for Imaging Vendors (Kevin), analysis SW vendors/developers (Cathy?), researchers, clinical trialists, pharma (Gudrun?), QIBA for Core Labs (Pat?), QIBA for Regulators (both device and drug?) (Nick), patients/advocates/lay public, government policy, payers, QIBA for Statisticians (Nancy)
- DONE targeted intro to each?groups? of those viewpoints
- What is QIBA basically about, what is the key value of QIBA to them, how do they "use"/"interact with" QIBA (some of them are "actors", some are not)
- TODO First Steps for each page (see QIBA for Statisticians for example)
- TODO Might be good to also have "stories from your colleagues" who are already in QIBA and their experiences.
- Contact page on wiki? Point to email@example.com for questions/info on any committee, QIDW, etc.
- Profile Selection Process
- Review/update review criteria/evaluations for new BC proposals
- Harvest from past experiences
- Given limited resources, how to approve not just good proposals but the best proposals
- Consider Prior Project completion when Steering Cmte reviews new Proposals from the same PI.
- timing overlap of prior cycle completion period with new proposal period needs to be one considered.
- Committee Leadership Process
- Address BC and CC leadership
- Continue with variations of co-chair/chair/vice-chair? - Maybe call everything below Steering Cmte as co-chairs and let them work out working patterns.
- Describe process for nominating & selecting that is transparent, allows time for nominations & self-nominations, encourages diversity
- Mostly current system has been working without major issues, but would like to have efficient flow rather than everyone re-thinking/re-inventing a process
- Process for recognizing cases where cochair availability/activity has decreased and need to call-the-question and trigger "replacement".
- Having "terms" would help trigger. E.g. 3-year terms, and limit of two terms to promote turnover, new faces, new ideas in some committees, on the other hand keeping experienced people and having continuity is good and there may be a limited supply of motivated leadership.
- CC term limits might be OK, but BC should maybe not have limits because they are intended to a have a limited life to start, and the chair is a key leader and change could falter
- If we stick with 3 year terms, might want to have a procedure where a cochair missing 8 meetings in a row triggers the secretariat doing a checkin to confirm that they still have the bandwidth/availability, and give them the opportunity to step aside if their other responsibilities have intruded.
- Stakeholder diversity (silent, recommend, or require)
- Having "only academic representation" is an adoption risk to QIBA, so there is a value to making some effort to have diversity
- Variability in the size of some committees so tough to enforce
- Lets Strongly Recommend (and maybe make it a review point in the Coord Cmte and the Steering Cmte to ask about attempts to improve?)
- Also consider this when forming a Biomarker Committee - require? At least formally build into the evaluation process.
- Consiider handling committee stakeholder diversity in the committee formation and dashboard process, not the leadership process
- Define Stakeholder diversity; do we make user/vendor classes? or is one academic radiologist and one clinical trial radiologist the intended degree of diversity?
- For comparison - DICOM:
- For Working Groups it's required but unconstrained "Working Groups shall elect a chair or co-chairs."
- For the DSC (Steering Cmte) it's more constrained
- There shall be two co-chairs – one representing a producer and one representing a biomedical professional organization.
- Each co-chair shall serve a two-year term and may serve no more than two consecutive terms. The terms of the co-chairs shall expire in alternate years.
- At the meeting preceding the meeting before the last Committee meeting each year, the co-chairs shall appoint a three-member nominating subcommittee consisting of at least one representative of a producer , one representative of a biomedical professional organization and one former committee officer (if one is available). The former officer (if one is available) shall chair the nominating subcommittee. Members of the nominating subcommittee shall develop and submit to the secretariat a slate of candidates. The names of the candidates should be submitted to the secretariat no later than 30 days prior to the meeting preceding the last meeting of the year so that their names may be included in the published agenda. Additional nominations may be made from the floor at this meeting.
- Prior to the last meeting of the year the secretariat shall conduct a letter ballot. The name of the person elected shall be announced at the last meeting of the year. That person shall assume office immediately following the last meeting of the year.
- The Committee has the right to recall a co-chair.
- For comparison - IHE:
- There are general cochair guidelines in section 12.2.7 of the Governance: https://www.ihe.net/wp-content/uploads/2018/07/IHE-International-Principles-of-Governance.pdf
- The Board adds the constraint "The Board shall elect as co-chairs, two Board Members drawn from Member Organizations in the User category. They will serve as co-chairs for renewable two-year terms."
- The Domain Planning Cmtes adds the recommendation "It is recommended that one Domain Planning Committee co-chair be selected from the User member category and one from the Developer member category."
- For comparison - RSNA:
- A nomination announcement contains a link to an online nomination form and deadline
- A comment box provides for nominators or self-nominators to state why they wanted to serve/would be a good addition to the leadership
- Once the nomination period ended, another announcement contains a link to an eballot listing nominees and their qualifications
- Joe helpfully pointed out that we need to clearly define voting eligibility (i.e. do voting privileges apply to chair selection or can everyone vote)
- Communicating Uncertainty to Radiologists
- "No measurement is complete without a statement of it's uncertainty" - NIST
- Measurement uncertainty is at the heart of what QIBA does
- What should best practice be for communicating the uncertainty of each measurement to radiologists?
- Always give a confidence interval rather than a value?
- Adoption of Profiles
- Maybe a new TF to support Adoption (can include some Process people)
- Ed, Brad, Joe, Kevin, Dan, RSNA Webperson
- Start by looking at what is on the RSNA QIBA pages,
- Clinical Confirmation Process
- Methodology/Study Design guidance/requirements for achieving "Clinically Confirmed"
- QIBA Tooling
- How to use Google Docs?
- Issues with references, access restricted by federal agencies and some companies
- Round trip fidelity between Word-Google-Word
- QIBA Wiki
- QIBA Wiki Improvements
- Wiki serves QIBA Committee work. Consider a new user oriented site (clinicians, administrators).
- Wiki vs RSNA.org
- Consider how to handle handoffs between RSNA website
- Literature Search
- Assemble draft guidance to kick this off (Hendrik Laue, Ona Wu, Caroline? Chung)
- Where do you store references, what kind of search works best, what exactly are you looking for?
- Also meta-analysis guidance to determine sources and levels of variance
- Profile Maintenance Process
- Propose change tracking process
- Propose conventions for Profile editions/versions
- Guidelines for Reference Implementations and commercial products in Profiles
- (underlying algorithm available, list all comers, non-commercial?, etc)
- TODO add detail
- Public Comment attribution
Todo items that are on hold pending some trigger:
- Claim Guidance
- Hold pending people with questions/clarification needs on current text
- Profile Template
- Hold pending new suggestions/clarification needs
- Harmonize assessment procedures across profiles
- Profile Retirement Process
- Hold pending a profile we want to retire
- When do we "withdraw" or at least stop maintaining/promoting a Profile
- Review DICOM retirement concept (stays available in archives, not maintained/promoted)
- Profile Disclaimer
- Hold pending revist of Discriminatory Claims
- Statistical Assumption Testing Guidance
- Conformance with the statistical assumptions is required at the various QIBA profile stages.
- Symmetrical vs Assymetrical change intervals?
- Questions about the numbers in "Academic Radiology: ?? Paper"
This page is based on the Committee Template. Add any good ideas you have to the template.