Segmentation and Markup Formats

Standard formats that are in widespread commercial use are always preferable, but especially for larger scale experiments and in the Profiles themselves. Standards formats like DICOM address a number of issues that will be faced when we try to roll this out in a clinical environment: where do the segmentations get stored, how do we move them over the network, how do we exchange them on portable media, how do we associate them with the patient record, how do we find/retrieve them later for comparison, how do we identify the images from which they were derived, how do we identify the space in which they exist for later registration, how do we de-identify them for research purposes, etc.

Non-standard proprietary formats (from research or commercial software) may be tolerable for limited Groundwork activities and small experiments. The applications may already be installed and the researchers familiar with their use, allowing groundwork to proceed immediately. Such formats often contain no "context" or "header" information about which subject or exam date to which they apply or any information about when they were created, by whom and for what purpose, and are limited only to the "payload" of coordinates or contours or segmentation. They may or may not contain robust image references, or be dependent about assumptions about the image data they are derived from or intended to be applied to. They may or may not contain meta-data (categorical or quantitative) derived from the images. Managing the files manually may be tractable at small the scale of groundwork studies (as long as the files are grouped with the image data that they relate to, contain in a hierarchical folder organization or organized with some naming convention). Preferably the proprietary file format will be documented rather then dependent on a particular software program and version to be read.

DICOM Segmentation, Region of Interest and Other Annotation Formats

DICOM provides several ways to encode segmentations, regions of interest and other types of annotation.

Two groups of DICOM formats are defined, those that are preferred for QIBA use, and those that may be encountered and are not ideal but are likely better than any proprietary non-standard format.

All of the DICOM objects share with ordinary DICOM images a "header" containing the information about the patient (subject) and other management information that provides the "context" for use of the content, including globally unique identifiers. I.e., they are self-contained and not dependent on any file format, file organization or file naming convention.

NOTE: The DICOM Standard is referenced below. Individual supplement documents that may originally have defined a specific feature are not maintained after they have been incorporated into the DICOM Standard, and hence are unreliable references. The DICOM Standard may contain fixes and updates not in the original supplement.

Recommended DICOM Formats for QIBA Experiments

Summary: DICOM Structured Reports should ideally be used for all quantitative imaging experiments and trials, using 2D or 3D spatial coordinates within the SR or using references to DICOM Segmentation objects where a rasterized representation of segmented objects is required.

DICOM Structured Reports

The DICOM Structured Report (SR) mechanism allows encoding of a tree (or directed acyclic graph) of structured, coded and numerical information, which may include references to images, frames within a multi-frame image, coordinates in those images, and patient-relative 3D coordinates independent of images.

The structured content and codes used that are specific to a particular application, experiment or trial may be defined in a "template"; the re-use of such templates, or common sub-patterns and code sets within templates is encouraged. A library of templates is defined in DICOM PS 3.16.

Contours encoded in Structured Reports

Structured reports may contain content items (nodes in the tree) that are graphic outlines defined as closed polylines; these may be image (or frame) relative (2D) or patient relative (3D). They are required to be co-planar in either case. The tree structure of the SR may be used to group a set of such closed polylines to represent a single object (such as an entire lesion segmentation).

Other nodes in the SR tree may be used to describe properties of the contours or objects, including identifiers (human readable or unique), technique (such as how the segmentation was performed), and derived information (including quantitative information like volume, mean density, etc.).

Standard templates and codes are defined in DICOM PS 3.16 for many of these measurements, and have been specified for clinical use in such applications as obstetric ultrasound, vascular and cardiac ultrasound (echocardiography), quantitative cardiac CT and MRI, and various Computed Assisted Detection (CAD) applications (including mammography, chest and virtual colonoscopy CAD).

For further information see DICOM PS 3.3 C.18.6 Spatial Coordinates Macro and C.18.9 3D Spatial Coordinates Macro.

The coordinate and image references in DICOM SR can also include a reference to a DICOM Presentation State, which allows the window center and width and the pan/zoom state used when the segmentation was performed to be recorded and replayed.

Segmentations referenced from Structured Reports

Rather than encoding segmentations in-line as graphic contours, SRs may reference external objects that define the segmentation, yet still convey the context and the meta-data about such segmentations. For example, an SR object may define a set of lesions, and the volume and mean density of each, but reference separate objects that define the graphical information either as rasterized or surface mesh objects, as defined below.

DICOM Rasterized Segmentations

Those voxels (on a 3D image set) or pixels (on a 2D image set) representing a particular class can be encoded as a separate rasterized image, referred to as a "segmentation object".

Each segmentation object is a multi-frame image representing a classification of pixels in one or more referenced images.

The segmentation is not required to have the same spatial sampling or extent (or even orientation) as the referenced images, if a common 3D frame of reference is defined. For example, segmentations may be super-sampled or sub-sampled, and may consist of only a sub-region of the original image volume.

A single object contains the entire segmented region (i.e., it is a multi-frame image object). Segmented regions are not required to be contiguous (e.g., when encoding all regions of a tissue type), but may be (e.g., when encoding all voxels in a single lesion). More than one classification may be contained within a single object.

Segmentations are either binary or fractional. Fractional segmentations may be used to represent either the fraction of the voxel which is a part of the segmentation (partial volumes), or the fractional probability that the voxel belongs to the segmentation.

A degenerate case is a bit-mask corresponding to the original image voxels or pixels, in which there is a 1:1 correspondence between image voxels and segmentation object voxels and each segmentation object voxel is zero or one. This is functionally equivalent to an "overlay", but has specifically defined semantics.

A common use is to encode the extent of a single lesion segmentation in one segmentation object, and reference it from a DICOM Structured Report that contains the meta-data (esp. measurements) of the segmented lesion.

Another common use is to encode all the segmented tissue types of an entire region (e.g., anatomic regions of the brain), and reference each segment (identified by a segment number) from a DICOM Structured Report that contains the meta-data (esp. measurements) of the segmented tissues and regions, and additional derived information (esp. totals and quality of segmentation and registration metrics).

For further information see DICOM PS 3.3 C.8.20 Segmentation.

DICOM Surface Segmentations

Image-independent objects, patient-relative or patient-independent, can be encoded in a polygonal representation of the object's surface.

The Surface Segmentation object is not slice based, unlike rastrized (voxel or pixel) segmentations.

Surface Segmentation objects can be referenced from Structured Reports.

The surface mesh representation is shared with objects defined for surgical modeling and planning (such as implant definition).

For further information see DICOM PS 3.3 C.27.1 Surface Mesh Module and DICOM PS 3.17 Annex JJ Surface Mesh Representation.

DICOM Spatial Registrations

Segmentation use cases often require the use of registration of different 3D frames of reference, and accordingly DICOM defines separate Spatial Registration objects for rigid and non-rigid registration (e.g., to encode an affine transformation or deformation field), and these may be used in combination with the segmentation and SR objects. For further information see DICOM PS 3.3 C.20 Spatial Registration.

DICOM Radiotherapy Structure Sets

RT Structure Sets contain graphic outlines defined as contours with patient relative (3D) coordinates, which may or may not be coplanar.

Unlike the generic polyline mechanism of SR, the semantics are specific to 3D regions of interest. For example, a mechanism is defined for inner and outer contours to encode "holes".

A very limited set of quantitative (e.g., volume) and categorical information (e.g., ROI is ICRU50 Gross Tumor Volume (GTV)) may be encoded with the contour, designed for the intended purpose of radiotherapy planning. Unlike SR, no generic mechanism is present for encoding other structured content.

A rigid frame of reference transformation can be encoded in the same object.

Though not as flexible as DICOM SR, the use of contours in encoded in DICOM objects for radiotherapy planning purposes long predates the use of contours for other diagnostic or quantitative imaging, and many software tools exist that support the RT Structure Set object.

For further information see DICOM PS 3.3 C.8.8.5 Structure Set, C.8.8.6 ROI Contour and C.8.8.8 RT ROI Observations.

Other DICOM Formats

DICOM Presentation States with Graphic Annotations

Presentation States contain a means of encoding simple 2D image-relative vector graphics, and may be be used to encode, for example, isocontours of a region as a set of closed polylines. There is no standard or coded or structured means of describing their semantics, however.

This format is not recommended for QIBA purposes in view of the lack of such semantics.

However, since they are very widely implemented in commercial systems (in which the purpose is to capture consistent apperance rather than meaning), it is possible to use these by defining conventions for the users (creators) to follow (such as using the same text label for isocontours on successive slices to indicate the same lesion). The information can then be "converted" into DICOM SR objects once the convention-based semantics had been extracted.

DICOM Images or Presentation States with Bitmap Overlays

An annotation or contour or segmentation may be rasterized into a 2D bitmap and either encoded in unused high bits of the pixel data in a re-saved image, stored separately in an attribute of a re-saved image (Overlay Data (0x60xx,0x3000)), or stored in a separate Presentation State object.

Given the lack of a means to provide semantic information to relate contours, this use is deprecated in favor of DICOM Segmentation objects (referenced from DICOM SR) if a rasterized rather than contoured representation is required, and in favor of DICOM SR Spatial Coordinates if a contoured representation is required.

DICOM Images with Burned in (Rendered) Annotations

An application may burn in an annotation, outline or segmentation to the pixel data of an underlying image and re-save it as a secondary capture image or an encapsulated PDF object; this is suitable for human review only and the information is essentially lost for the purpose of further processing.

DICOM Images with Curves

A means of describing curves for graphics (and waveforms) was originally defined in the standard but has been removed and should not be used for QIBA purposes. Their function has been replaced by Graphic Annotations in Presentation State objects and time-based Waveform objects.

DICOM Format References

Proprietary Segmentation and Annotation Formats

DICOM Vendor Extensions

A (slightly) less objectionable form in which to store proprietary format annotations than a unique file format is by the addition of "private data elements" to standard DICOM objects, commonly images. Older commercial PACS and workstation software fill often encode annotations, including 2D regions of interest, in such private data elements, which may or may not be sufficiently well documented (such as in their DICOM Conformance Statement) or sufficiently obvious that they may be reverse engineered and the required information extracted.

Commercial Vendor Proprietary Formats

<<Include details of 3D Doctor and PLY formats here>>

Research and Open Source Tool Proprietary Formats

<<Include details of VTK and Osirix formats here>>

Project Specific Formats

Some research projects reinvent the wheel and define specific formats for a particular purpose, ostensibly to reduce the "complexity" of having to deal with standard formats (despite the availability of commercial and open source tools and tool kits to avoid such complexity). These formats are typically documented on the project's web site and/or in scientific articles.

An example is the Lung Image Database Consortium XML format

AIM

<<Include details of AIM and references here>>