Data Access: Difference between revisions

From CNI Wiki
Jump to navigation Jump to search
← Older edit
imported>Bobd
No edit summary
imported>Lmperry
No edit summary
 
(46 intermediate revisions by 3 users not shown)
Line 1: Line 1:
The CNI is developing an integrated neurobiological image management system (NIMS) to facilitate the stewardship of all data collected at the CNI. NIMS is currently under active development and we plan to roll out a beta-test version in the Summer of 2011. In the mean time, MR Steward, a key component of NIMS, is already installed at the CNI and is useful for organizing and archiving the data that you collect here. However, until the full NIMS system is in place, CNI users are responsible for managing their data and will want to copy their data to their own file server.  
The CNI uses [https://flywheel.io Flywheel] for data management and access. Please see the [[ Flywheel | Flywheel ]] page for more information regarding data access via Flywheel.


For more information on the NIMS and MR Steward software development efforts, see the [[NIMS | CNI Software Development page]].
= Data Sharing =
This section describes the way in which data can be shared among researchers.


= Dirmonitor: Automatic data transfer to cnirt =
== IRB Considerations ==


A key part of NIMS is collating all data acquired in a scan session. A major aspect of that goal is already implemented and functional at the CNI. ''Dirmonitor'' and the related program ''pfilemonitor'' are two ever-running processes that manage images acquired on the MR scanner ('''cnimr'''). These processes automatically copy all imaging data acquired on '''cnimr''' to the linux box that sits to the right of the scanner ('''cnirt'''). On '''cnirt''', the data are organized by PatientID and exam date. All data acquired during an exam are put into a folder with a name of the form ''YYYYMMDD_EXAMNUM''. Dicoms are archived in zip files (one per series) and are also converted to a NIFTI file (also one per series), if possible. P-files are copied into this same folder. (Automatic recon of spiral data will be implemented soon!)
Before data can be shared across researchers the principal investigator responsible for the original data should make sure that provisions are in place within the approved IRB protocol that allow data sharing and assure that all requirements set forth in the protocol regarding data sharing are met.


Should data stop showing up at any time, a variety of things could be to blame: Perhaps the mount of the pfile or dicom directories reset or is absent, or someone may have reset the scanner (or maybe even '''cnirt'''). Debugging should generally happen on '''cnirt''':
== Sharing data within the Stanford community ==
Those seeking to gain access to data should FIRST contact the Stanford PI who is responsible for the data to request access. ''Requests should be made between PIs.''<br>
'''Persons seeking the data (PI or Lab Manager):'''
<ol>
# Find the data to be shared and note the experiment name, subject code, and session numbers.
# Contact the principal investigator and officially ask for the data to be shared. ''Contact should be between PIs.''  
# Once permission is granted you can log into [https://cni.flywheel.io Flywheel] and download the data.
</ol>
'''The persons responsible for the data (PI or Lab Manager):'''
<ol>
# Assure that the IRB protocol under which the data were collected allows for data sharing.
# Log into [https://cni.flywheel.io Flywheel]
</ol>


To check if dirmon is running, try:
== Sharing data with collaborators outside of Stanford ==
ps aux | grep dirmon
Collaborators seeking to gain access to data should FIRST contact the Stanford PI who is responsible for the data to request access. If the PI agrees to share their data, then the collaborator and PI should do the following steps.


You should see something like:
'''Outside collaborators'''
cni      31446  6.8 29.3 3769888 3616544 ?    Sl  May09 200:00 /usr/bin/python ./dirmonitor.py
<ol>
<li>Set up a SUNetID by visiting [http://accounts.stanford.edu Stanford Accounts] and choosing 'Create' -> 'I do not have a university ID' -> 'Continue without University ID'</li>
<li>Follow the steps to create a SUNET ID and pick a password, etc. </li>
:Your SUNETID must:
:*Be 3-8 characters long (lower-case letters and numbers only)
:*Start with a letter
:*We recommend: ''FirstinitialLastInstitution'' (e.g.: For Michael Perry @ Stanford sunetid = '''mperrysu''').  
:*For more info regarding SUNET IDs: http://itservices.stanford.edu/service/accounts/sunetids


To check if pfilemon is running, try:
<li> Once the SUNET ID is created, email your SUNET ID the collaborating Stanford PI; he/she must give you permission to access the data.</li>
ps aux | grep pfilemon
<li> After they have confirmed that they have granted you permission navigate to the Flywheel interface, find the experiment with the data and 'Download' those data </li>
</ol>


You should see something like:
cni      15878 97.2  0.0  41548  8648 ?        Rl  12:45 350:59 /usr/bin/python ./pfilemonitor.py


Pfiles are located on the mount:
'''Stanford PI'''
/net/cnimr/export/home/signa/research/mrraw/
<ol>
 
# After your collaborator has requested a SUNET ID, sponsor his/her account by visiting [http://sponsorship.stanford.edu the sponsorship page], entering the SUNET ID and following the prompts to sponsor a free account.  
Dicoms are located on the mount:
# Go to the NIMS database interface and choose -> 'Manage Experiments'
/net/cnimr/export/home1/sdc_image_pool/images
# Find the SUNET ID of your collaborator and drag it to the experiment containing the data you wish to share.  
 
# A popup will appear allowing you to choose the access level you wish to grant your collaborator.  We suggest you choose read-only or anon-read.
If you can ls both of those, you're in good shape.
</ol>
 
Assuming you want to reset whichever process is not working (be it dirmonitor or pfilemonitor), use a killall and re-initialize it.
 
If you aren't getting dicoms/niftis try:
killall -9 dirmonitor.py
  /home/cni/dirmonitor/dirmonitor.py &
 
If you aren't getting pfiles try:
killall -9 pfilemonitor.py
/home/cni/dirmonitor/pfilemonitor.py &

Latest revision as of 23:16, 4 December 2019

The CNI uses Flywheel for data management and access. Please see the Flywheel page for more information regarding data access via Flywheel.

Data Sharing

This section describes the way in which data can be shared among researchers.

IRB Considerations

Before data can be shared across researchers the principal investigator responsible for the original data should make sure that provisions are in place within the approved IRB protocol that allow data sharing and assure that all requirements set forth in the protocol regarding data sharing are met.

Sharing data within the Stanford community

Those seeking to gain access to data should FIRST contact the Stanford PI who is responsible for the data to request access. Requests should be made between PIs.
Persons seeking the data (PI or Lab Manager):

    1. Find the data to be shared and note the experiment name, subject code, and session numbers.
    2. Contact the principal investigator and officially ask for the data to be shared. Contact should be between PIs.
    3. Once permission is granted you can log into Flywheel and download the data.

The persons responsible for the data (PI or Lab Manager):

    1. Assure that the IRB protocol under which the data were collected allows for data sharing.
    2. Log into Flywheel

Sharing data with collaborators outside of Stanford

Collaborators seeking to gain access to data should FIRST contact the Stanford PI who is responsible for the data to request access. If the PI agrees to share their data, then the collaborator and PI should do the following steps.

Outside collaborators

  1. Set up a SUNetID by visiting Stanford Accounts and choosing 'Create' -> 'I do not have a university ID' -> 'Continue without University ID'
  2. Follow the steps to create a SUNET ID and pick a password, etc.
    3. Your SUNETID must:
  3. Once the SUNET ID is created, email your SUNET ID the collaborating Stanford PI; he/she must give you permission to access the data.
  4. After they have confirmed that they have granted you permission navigate to the Flywheel interface, find the experiment with the data and 'Download' those data


Stanford PI

    1. After your collaborator has requested a SUNET ID, sponsor his/her account by visiting the sponsorship page, entering the SUNET ID and following the prompts to sponsor a free account.
    2. Go to the NIMS database interface and choose -> 'Manage Experiments'
    3. Find the SUNET ID of your collaborator and drag it to the experiment containing the data you wish to share.
    4. A popup will appear allowing you to choose the access level you wish to grant your collaborator. We suggest you choose read-only or anon-read.
Retrieved from "http://cni.su.domains/wiki/index.php?title=Data_Access&oldid=1609"