imported>Bobd: Created page with "MR Steward, a key component of NIMS, is already installed at the CNI and is useful for organizing and archiving the data that you collect here. = MR Steward info = ==Development..."

2012-01-20T23:06:10Z

Created page with "MR Steward, a key component of NIMS, is already installed at the CNI and is useful for organizing and archiving the data that you collect here. = MR Steward info = ==Development..."

New page

MR Steward, a key component of NIMS, is already installed at the CNI and is useful for organizing and archiving the data that you collect here.

= MR Steward info =
==Development History==
===Version 0.1 - dicomxfer===
'''Features:'''
*Command-line controlled search and transfer of studies from CNIMR to CNIRT.

'''Methods:'''

I made use of DCMTK package for findscu and movescu, shell commands that
provide search and transfer functionality of dicom data. Both commands require
verbose user input, so I wrapped them Python to allow more concise interaction.
Information provided by findscu is parsed and used to rename and organize files
upon download.

'''Problems:'''
*Nobody likes command-line tools.

===Version 0.2 - dicomxfer===
'''Features:'''
*GUI controlled search and transfer of studies from CNIMR to CNIRT.
*Transfer triggers creation of niftis, bvals, and bvecs files.

'''Methods:'''

The GUI is implemented with wxPython. A combination of the pydicom, nibabel,
and numpy modules was used to create all derivative files.

'''Problems:'''
*No progress indicators.
*GUI hangs while transferring and processing data.
*Certain obscure dicom types cause the program to freeze (screen-saves).

===Version 0.2a - dicomxfer===
'''Features:'''
*Persistent GUI control (no hangs).
*Progress indicators for file transfer and processing.
*Handle previously error-prone dicom types.

'''Methods:'''

Restructured code to approximate model/view/controller model. The resulting
modularity made it easier to make use of threading module. Progress is
communicated between modules/threads using the Publisher available in the
wxPython libraries.

'''Problems:'''
*No way to transfer incrementally during scans.
*Takes too long to transfer studies (related to above).
*Processing of large studies eats into swap space.

===Version 0.3 - dirmonitor===
'''Features:'''
*Passive incremental data transfer (get data immediately).
*Passive sorting into intuitively named folder structure.

'''Methods:'''

A portion of Jerzy Bodurka's real-time analysis code transfers dicoms from
CNIMR to CNIRT using rsync. The directory structure it creates is fairly
incomprehensible, so we linked the transferred files to a separate directory
with a more comprehensible naming scheme. To accomplish this, we periodically
compute a set of all files present in the original directory. New sets are
diffed against previous sets to determine what files have been added (or
removed), and performs an operation on each novel file. The current set is
maintained in a text file for persistence.

'''Problems:'''
*File list becomes a memory hog. Danger of eventually eating swap space.
*No way to determine when a series is done transferring.
*No automatic generation of niftis, bvals, or bvecs (related to above).

===Version 0.4 - dirmonitor===
'''Features:'''
*Passive incremental data transfer (get data immediately).
*Detects separate acquisitions within series.
*Vastly reduced memory footprint.

'''Methods:'''

We abandoned Jerzy Bodurka's code. With the help of Gunnar Schaefer, CNIMR was
set to export its dicom directory and we read all incoming data directly from
that.

In addition to configuring the export directory, Gunnar helped devise a new,
lightweight algorithm for tracking which files have been completed based on the
observation that files appears in a nested directory structure of the form
p(number)/e(number)/s(number)/(dicom_files) where number is 1 through n. The
deepest folder level, s(number), represents series. Each incoming series is
placed into such a folder, each an increment of 1 from the last.

The creation and modification of such folders allows us to track when files
have finished being written to a series (e.g. when a folder of
current_series_number + 1 is created, we're done working on
current_series_number). Using this series number also gives us a very
lightweight means of maintaining persistence between executions and reduces our
memory footprint (no more keeping track of EVERY file processed, just the last
series).

'''Problems:'''
*No derivative file creation.
*Space wasted by storing dicoms raw in folders.

===Version 0.4a - dirmonitor===
'''Features:'''
*Passive creation of derivative files (.png, .nii.gz, .bval, and .bvec).
*Passive compression of source dicoms into zip files.
*Logging implemented.

'''Methods:'''

Created dicomtools module to pass lists of files read in with pydicom and have
all derivative file types automatically created. Threading was used to
separate out detecting jobs and performing them, with the queue module used for
inter-thread communication.

'''Problems:'''
*Pfiles aren't part of transfer protocol.

===Version 0.4b - dirmonitor + pfilemonitor===
'''Features:'''
*Passive transfer of pfiles into same folders.

'''Methods:'''

Created pfilemonitor module to independently handle transfer of pfiles.
Created pfparse module to parse pfile headers. This is accomplished by running
GE's print_raw_headers binary and parsing the text output to create a
PfileHeader object.

'''Problems:'''
*Recons still have to be read independently.
*Physio data isn't part of transfer protocol.
*Program hangs when there are gaps in the incrementing series numbers (these exist?!)

===Version 0.5 - mrsteward===
'''Features:'''
*Passive dicom, pfile, and physio data transfer.
*Passive derivative file type creation and recon running.
*Improved persistence and flexibility for extensions.
*Config file for easy set up on different machines.
*Logging carried over.

'''Methods:'''

Redesigned program structure. Greater adherence to object-oriented principles.
Unified previously separate modules and imposed shared structure between
modules.

Physio data directory exported from CNIMR to CNIRT, and physiofinder module
created to detect what physio files correspond to a given dicom or pfile, as
well as compress them.

Created sqltools module with new JobQueue class to handle storing program state
in an easy to access and manage sqlite database. Data stored here is available
to external programs/modules, and thus provides a simple place for future
modules to view the work being done in real-time. Opens the possibility for
nice GUI additions to the tool. Such changes to the way we queue jobs allows
better parallelization of jobs and tracking of completed tasks.

Makes use of ConfigParser module to simplify configuration for different
machines.

'''Problems:'''

*Dependency on print_raw_headers for reading pfiles, rather than pure Python.

= NIMS Info =

MORE HERE!

NIMS archive - Revision history

imported>Bobd: Created page with "MR Steward, a key component of NIMS, is already installed at the CNI and is useful for organizing and archiving the data that you collect here. = MR Steward info = ==Development..."