Flywheel

Downloading Data

Sherlock and other Remotes

For those users who wish to transfer data to other remote compute resources, like Sherlock, we suggest that you use the Flywheel Command Line Interface (CLI) to do so. The CLI allows you to perform many tasks, one of which is downloading data, with the ability to restrict to only certain data types (e.g., NIfTI files only) or exclude specific data types (e.g., no PFiles or DICOMS).

Downloading and installing the CLI on Sherlock

To download and install the CLI on Sherlock we will use the `wget` tool, then `unzip` to extract the CLI resource package, and finally modify our `.bashrc` file to add the fw binary as an alias in our environment.

Step 1: Get the URL for the CLI package, which can be found on the "Profile" page within the Flywheel interface. To grab the URL, find the CLI section on the Profile page, right-click the Linux CLI Download link, and choose "Copy Link Address" (or similar). Once you have the download link address move to the next step.

Step 2: Log in to Sherlock and run the `wget` command, using the URL from Step 1 to download the CLI package.

 mkdir -p flywheel/cli
 cd flywheel/cli
 wget https://storage.googleapis.com/flywheel-dist/cli/7.1.0/fw-linux_amd64.zip

Step 3: Unpack the CLI archive and cleanup the downloaded package.

 unzip  fw-linux_amd64.zip
 mv linux_amd64/fw .
 rmdir linux_amd64
 rm -f fw-linux_amd64.zip

Step 4: Modify your `.bashrc` file to add the fw CLI command to your environment, and source it to make the alias active. Note that this only needs to be done once.

 echo -e "alias fw='$HOME/flywheel/cli/fw'" >> $HOME/.bashrc
 source $HOME/.bashrc

Step 5: Once the above steps are complete, you should be able to log in using the CLI and use it as described in the official documentation: https://flywheelio.zendesk.com/hc/en-us/sections/360001596834-Command-Line-Interface. The best way to do this is navigate to your profile page in Flywheel, make sure that you have generated an API Key, and use the login command text that is provided for you there.

 fw login <your API key>

Tips for Downloading data with the CLI

Download only the files you need

When downloading data from Flywheel using the CLI you can greatly speed up your downloads by excluding data types which are not needed for your analysis.

Example: Exclude pfile and DICOM data from a container download: Most users do not need to download the raw scanner files (PFILES) or raw DICOM data. You can exclude certain data types from your downloads by using the `-e` flag with your CLI download, like so:

 fw download "cni/testproject/subject1/session1" -e pfile -e dicom

This tells the CLI to exclude any pfile and dicom files in the container. Note that you can use consecutive -e flags to exclude multiple data types.

Example: Download only NIfTI, BVEC, and BVAL files:
Most users are only interested in the data that will be input to their analysis pipelines. This is most often limited to three data types (nifti, bvec, and bval). You can use the following command with multiple include flags (`-i`) to accomplish exactly that:

 fw download "cni/testproject/subject1/session1" -i nifti -i bvec -i bval

This will generate an archive (.tar) file containing the requested hierarchy with only those files you explicitly need.

Use quotes

Often times your source-path (that is the group/project/subejct/session' string the describes the location of your data in Flywheel) will have one or more spaces or special characters in it. To properly address that location using the CLI it's important to use quotes around the source path, like so:

 fw download "cni/testproject/subject1/session1"

Support

Michael can help with many, if not all, issues related to Flywheel. If further support is needed for Flywheel specific issues, not related to the CNI, please feel free to email support@flywheel.io.