NIMS archive

MR Steward, a key component of NIMS, is already installed at the CNI and is useful for organizing and archiving the data that you collect here.

MR Steward info

Development History

Version 0.1 - dicomxfer

Features:

• Command-line controlled search and transfer of studies from CNIMR to CNIRT.

Methods:

I made use of DCMTK package for findscu and movescu, shell commands that provide search and transfer functionality of dicom data. Both commands require verbose user input, so I wrapped them Python to allow more concise interaction. Information provided by findscu is parsed and used to rename and organize files upon download.

Problems:

• Nobody likes command-line tools.

Version 0.2 - dicomxfer

Features:

• GUI controlled search and transfer of studies from CNIMR to CNIRT.
• Transfer triggers creation of niftis, bvals, and bvecs files.

Methods:

The GUI is implemented with wxPython. A combination of the pydicom, nibabel, and numpy modules was used to create all derivative files.

Problems:

• No progress indicators.
• GUI hangs while transferring and processing data.
• Certain obscure dicom types cause the program to freeze (screen-saves).

Version 0.2a - dicomxfer

Features:

• Persistent GUI control (no hangs).
• Progress indicators for file transfer and processing.
• Handle previously error-prone dicom types.

Methods:

Restructured code to approximate model/view/controller model. The resulting modularity made it easier to make use of threading module. Progress is communicated between modules/threads using the Publisher available in the wxPython libraries.

Problems:

• No way to transfer incrementally during scans.
• Takes too long to transfer studies (related to above).
• Processing of large studies eats into swap space.

Version 0.3 - dirmonitor

Features:

• Passive incremental data transfer (get data immediately).
• Passive sorting into intuitively named folder structure.

Methods:

A portion of Jerzy Bodurka's real-time analysis code transfers dicoms from CNIMR to CNIRT using rsync. The directory structure it creates is fairly incomprehensible, so we linked the transferred files to a separate directory with a more comprehensible naming scheme. To accomplish this, we periodically compute a set of all files present in the original directory. New sets are diffed against previous sets to determine what files have been added (or removed), and performs an operation on each novel file. The current set is maintained in a text file for persistence.

Problems:

• File list becomes a memory hog. Danger of eventually eating swap space.
• No way to determine when a series is done transferring.
• No automatic generation of niftis, bvals, or bvecs (related to above).

Version 0.4 - dirmonitor

Features:

• Passive incremental data transfer (get data immediately).
• Detects separate acquisitions within series.
• Vastly reduced memory footprint.

Methods:

We abandoned Jerzy Bodurka's code. With the help of Gunnar Schaefer, CNIMR was set to export its dicom directory and we read all incoming data directly from that.

In addition to configuring the export directory, Gunnar helped devise a new, lightweight algorithm for tracking which files have been completed based on the observation that files appears in a nested directory structure of the form p(number)/e(number)/s(number)/(dicom_files) where number is 1 through n. The deepest folder level, s(number), represents series. Each incoming series is placed into such a folder, each an increment of 1 from the last.

The creation and modification of such folders allows us to track when files have finished being written to a series (e.g. when a folder of current_series_number + 1 is created, we're done working on current_series_number). Using this series number also gives us a very lightweight means of maintaining persistence between executions and reduces our memory footprint (no more keeping track of EVERY file processed, just the last series).

Problems:

• No derivative file creation.
• Space wasted by storing dicoms raw in folders.

Version 0.4a - dirmonitor

Features:

• Passive creation of derivative files (.png, .nii.gz, .bval, and .bvec).
• Passive compression of source dicoms into zip files.
• Logging implemented.

Methods:

Created dicomtools module to pass lists of files read in with pydicom and have all derivative file types automatically created. Threading was used to separate out detecting jobs and performing them, with the queue module used for inter-thread communication.

Problems:

• Pfiles aren't part of transfer protocol.

Version 0.4b - dirmonitor + pfilemonitor

Features:

• Passive transfer of pfiles into same folders.

Methods:

Created pfilemonitor module to independently handle transfer of pfiles. Created pfparse module to parse pfile headers. This is accomplished by running GE's print_raw_headers binary and parsing the text output to create a PfileHeader object.

Problems:

• Recons still have to be read independently.
• Physio data isn't part of transfer protocol.
• Program hangs when there are gaps in the incrementing series numbers (these exist?!)

Version 0.5 - mrsteward

Features:

• Passive dicom, pfile, and physio data transfer.
• Passive derivative file type creation and recon running.
• Improved persistence and flexibility for extensions.
• Config file for easy set up on different machines.
• Logging carried over.

Methods:

Redesigned program structure. Greater adherence to object-oriented principles. Unified previously separate modules and imposed shared structure between modules.

Physio data directory exported from CNIMR to CNIRT, and physiofinder module created to detect what physio files correspond to a given dicom or pfile, as well as compress them.

Created sqltools module with new JobQueue class to handle storing program state in an easy to access and manage sqlite database. Data stored here is available to external programs/modules, and thus provides a simple place for future modules to view the work being done in real-time. Opens the possibility for nice GUI additions to the tool. Such changes to the way we queue jobs allows better parallelization of jobs and tracking of completed tasks.

Makes use of ConfigParser module to simplify configuration for different machines.

Problems:

• Dependency on print_raw_headers for reading pfiles, rather than pure Python.

NIMS Info

MORE HERE!