CryoSPARC Guide
  • About CryoSPARC
  • Current Version
  • Licensing
    • Non-commercial license agreement
  • Setup, Configuration and Management
    • CryoSPARC Architecture and System Requirements
    • CryoSPARC Installation Prerequisites
    • How to Download, Install and Configure
      • Obtaining A License ID
      • Downloading and Installing CryoSPARC
      • CryoSPARC Cluster Integration Script Examples
      • Accessing the CryoSPARC User Interface
    • Deploying CryoSPARC on AWS
      • Performance Benchmarks
    • Using CryoSPARC with Cluster Management Software
    • Software Updates and Patches
    • Management and Monitoring
      • Environment variables
      • (Optional) Hosting CryoSPARC Through a Reverse Proxy
      • cryosparcm reference
      • cryosparcm cli reference
      • cryosparcw reference
    • Software System Guides
      • Guide: Updating to CryoSPARC v4
      • Guide: Installation Testing with cryosparcm test
      • Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)
      • Guide: Verify CryoSPARC Installation with the Extensive Workflow (≤v4.2)
      • Guide: Performance Benchmarking (v4.3+)
      • Guide: Download Error Reports
      • Guide: Maintenance Mode and Configurable User Facing Messages
      • Guide: User Management
      • Guide: Multi-user Unix Permissions and Data Access Control
      • Guide: Lane Assignments and Restrictions
      • Guide: Queuing Directly to a GPU
      • Guide: Priority Job Queuing
      • Guide: Configuring Custom Variables for Cluster Job Submission Scripts
      • Guide: SSD Particle Caching in CryoSPARC
      • Guide: Data Management in CryoSPARC (v4.0+)
      • Guide: Data Cleanup (v4.3+)
      • Guide: Reduce Database Size (v4.3+)
      • Guide: Data Management in CryoSPARC (≤v3.3)
      • Guide: CryoSPARC Live Session Data Management
      • Guide: Manipulating .cs Files Created By CryoSPARC
      • Guide: Migrating your CryoSPARC Instance
      • Guide: EMDB-friendly XML file for FSC plots
    • Troubleshooting
  • Application Guide (v4.0+)
    • A Tour of the CryoSPARC Interface
    • Browsing the CryoSPARC Instance
    • Projects, Workspaces and Live Sessions
    • Jobs
    • Job Views: Cards, Tree, and Table
    • Creating and Running Jobs
    • Low Level Results Interface
    • Filters and Sorting
    • View Options
    • Tags
    • Flat vs Hierarchical Navigation
    • File Browser
    • Blueprints
    • Workflows
    • Inspecting Data
    • Managing Jobs
    • Interactive Jobs
    • Upload Local Files
    • Managing Data
    • Downloading and Exporting Data
    • Instance Management
    • Admin Panel
  • Cryo-EM Foundations
    • Image Formation
      • Contrast in Cryo-EM
      • Waves as Vectors
      • Aliasing
  • Expectation Maximization in Cryo-EM
  • Processing Data in cryoSPARC
    • Get Started with CryoSPARC: Introductory Tutorial (v4.0+)
    • Tutorial Videos
    • All Job Types in CryoSPARC
      • Import
        • Job: Import Movies
        • Job: Import Micrographs
        • Job: Import Particle Stack
        • Job: Import 3D Volumes
        • Job: Import Templates
        • Job: Import Result Group
        • Job: Import Beam Shift
      • Motion Correction
        • Job: Patch Motion Correction
        • Job: Full-Frame Motion Correction
        • Job: Local Motion Correction
        • Job: MotionCor2 (Wrapper) (BETA)
        • Job: Reference Based Motion Correction (BETA)
      • CTF Estimation
        • Job: Patch CTF Estimation
        • Job: Patch CTF Extraction
        • Job: CTFFIND4 (Wrapper)
        • Job: Gctf (Wrapper) (Legacy)
      • Exposure Curation
        • Job: Micrograph Denoiser (BETA)
        • Job: Micrograph Junk Detector (BETA)
        • Interactive Job: Manually Curate Exposures
      • Particle Picking
        • Interactive Job: Manual Picker
        • Job: Blob Picker
        • Job: Template Picker
        • Job: Filament Tracer
        • Job: Blob Picker Tuner
        • Interactive Job: Inspect Particle Picks
        • Job: Create Templates
      • Extraction
        • Job: Extract from Micrographs
        • Job: Downsample Particles
        • Job: Restack Particles
      • Deep Picking
        • Guideline for Supervised Particle Picking using Deep Learning Models
        • Deep Network Particle Picker
          • T20S Proteasome: Deep Particle Picking Tutorial
          • Job: Deep Picker Train and Job: Deep Picker Inference
        • Topaz (Bepler, et al)
          • T20S Proteasome: Topaz Particle Picking Tutorial
          • T20S Proteasome: Topaz Micrograph Denoising Tutorial
          • Job: Topaz Train and Job: Topaz Cross Validation
          • Job: Topaz Extract
          • Job: Topaz Denoise
      • Particle Curation
        • Job: 2D Classification
        • Interactive Job: Select 2D Classes
        • Job: Reference Based Auto Select 2D (BETA)
        • Job: Reconstruct 2D Classes
        • Job: Rebalance 2D Classes
        • Job: Class Probability Filter (Legacy)
        • Job: Rebalance Orientations
        • Job: Subset Particles by Statistic
      • 3D Reconstruction
        • Job: Ab-Initio Reconstruction
      • 3D Refinement
        • Job: Homogeneous Refinement
        • Job: Heterogeneous Refinement
        • Job: Non-Uniform Refinement
        • Job: Homogeneous Reconstruction Only
        • Job: Heterogeneous Reconstruction Only
        • Job: Homogeneous Refinement (Legacy)
        • Job: Non-uniform Refinement (Legacy)
      • CTF Refinement
        • Job: Global CTF Refinement
        • Job: Local CTF Refinement
        • Job: Exposure Group Utilities
      • Conformational Variability
        • Job: 3D Variability
        • Job: 3D Variability Display
        • Job: 3D Classification
        • Job: Regroup 3D Classes
        • Job: Reference Based Auto Select 3D (BETA)
        • Job: 3D Flexible Refinement (3DFlex) (BETA)
      • Postprocessing
        • Job: Sharpening Tools
        • Job: DeepEMhancer (Wrapper)
        • Job: Validation (FSC)
        • Job: Local Resolution Estimation
        • Job: Local Filtering
        • Job: ResLog Analysis
        • Job: ThreeDFSC (Wrapper) (Legacy)
      • Local Refinement
        • Job: Local Refinement
        • Job: Particle Subtraction
        • Job: Local Refinement (Legacy)
      • Helical Reconstruction
        • Helical symmetry in CryoSPARC
        • Job: Helical Refinement
        • Job: Symmetry search utility
        • Job: Average Power Spectra
      • Utilities
        • Job: Exposure Sets Tool
        • Job: Exposure Tools
        • Job: Generate Micrograph Thumbnails
        • Job: Cache Particles on SSD
        • Job: Check for Corrupt Particles
        • Job: Particle Sets Tool
        • Job: Reassign Particles to Micrographs
        • Job: Remove Duplicate Particles
        • Job: Symmetry Expansion
        • Job: Volume Tools
        • Job: Volume Alignment Tools
        • Job: Align 3D maps
        • Job: Split Volumes Group
        • Job: Orientation Diagnostics
      • Simulations
        • Job: Simulate Data (GPU)
        • Job: Simulate Data (Legacy)
    • CryoSPARC Tools
    • Data Processing Tutorials
      • Case study: End-to-end processing of a ligand-bound GPCR (EMPIAR-10853)
      • Case Study: DkTx-bound TRPV1 (EMPIAR-10059)
      • Case Study: Pseudosymmetry in TRPV5 and Calmodulin (EMPIAR-10256)
      • Case Study: End-to-end processing of an inactive GPCR (EMPIAR-10668)
      • Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716)
      • Case Study: Exploratory data processing by Oliver Clarke
      • Tutorial: Tips for Membrane Protein Structures
      • Tutorial: Common CryoSPARC Plots
      • Tutorial: Negative Stain Data
      • Tutorial: Phase Plate Data
      • Tutorial: EER File Support
      • Tutorial: EPU AFIS Beam Shift Import
      • Tutorial: Patch Motion and Patch CTF
      • Tutorial: Float16 Support
      • Tutorial: Particle Picking Calibration
      • Tutorial: Blob Picker Tuner
      • Tutorial: Helical Processing using EMPIAR-10031 (MAVS)
      • Tutorial: Maximum Box Sizes for Refinement
      • Tutorial: CTF Refinement
      • Tutorial: Ewald Sphere Correction
      • Tutorial: Symmetry Relaxation
      • Tutorial: Orientation Diagnostics
      • Tutorial: BILD files in CryoSPARC v4.4+
      • Tutorial: Mask Creation
      • Case Study: Yeast U4/U6.U5 tri-snRNP
      • Tutorial: 3D Classification
      • Tutorial: 3D Variability Analysis (Part One)
      • Tutorial: 3D Variability Analysis (Part Two)
      • Tutorial: 3D Flexible Refinement
        • Installing 3DFlex Dependencies (v4.1–v4.3)
      • Tutorial: 3D Flex Mesh Preparation
    • Webinar Recordings
  • Real-time processing in cryoSPARC Live
    • About CryoSPARC Live
    • Prerequisites and Compute Resources Setup
    • How to Access cryoSPARC Live
    • UI Overview
    • New Live Session: Start to Finish Guide
    • CryoSPARC Live Tutorial Videos
    • Live Jobs and Session-Level Functions
    • Performance Metrics
    • Managing a CryoSPARC Live Session from the CLI
    • FAQs and Troubleshooting
  • Guides for v3
    • v3 User Interface Guide
      • Dashboard
      • Project and Workspace Management
      • Create and Build Jobs
      • Queue Job, Inspect Job and Other Job Actions
      • View and Download Results
      • Job Relationships
      • Resource Manager
      • User Management
    • Tutorial: Job Builder
    • Get Started with CryoSPARC: Introductory Tutorial (v3)
    • Tutorial: Manually Curate Exposures (v3)
  • Resources
    • Questions and Support
Powered by GitBook
On this page
  • Introduction
  • Requirements
  • Use Cases
  • A. Moving only raw particle, micrograph or movie data already imported into CryoSPARC
  • B. Moving Only CryoSPARC Project Directories (and all jobs inside them)
  • C. Moving the CryoSPARC database
  • D. Hosting CryoSPARC on another machine
  1. Setup, Configuration and Management
  2. Software System Guides

Guide: Migrating your CryoSPARC Instance

A guide to moving CryoSPARC from one location to another.

PreviousGuide: Manipulating .cs Files Created By CryoSPARCNextGuide: EMDB-friendly XML file for FSC plots

Last updated 6 months ago

Introduction

There may come a time when you want to move your CryoSPARC instance from one location to another. This may be between folders, different network storage locations, or even different host machines entirely. There are four main areas we will focus on:

  1. The paths of any raw particle, micrograph, or movie data imported into CryoSPARC

  2. All CryoSPARC project directories

  3. The CryoSPARC database and its (new) location

  4. The identities/hostnames of compute nodes or the master node and the CryoSPARC binaries

All four of the above areas can be taken care of in isolation, but if combined, will amount to a full-out migration of your CryoSPARC instance.

Requirements

We will be using a combination of the shell as well as an interactive python session to complete this migration. You will need access to the master node in which the CryoSPARC system is hosted.

It is also recommended that a database backup is created before starting anything. More details are at:

Use Cases

A. Moving only raw particle, micrograph or movie data already imported into CryoSPARC

When raw data is imported into a CryoSPARC project, rather than copy the data into the project directory, symlinks are created inside the import job directories pointing to the original data files. Read for more details.

If you're moving data that you used an "Import Particles", "Import Micrographs" or "Import Movies" job to bring into CryoSPARC, you will need to repair these jobs. When CryoSPARC imports these three types of data, it creates to each file inside the job's imported directory. These symlinks may become broken if the original path to the file no longer exists. You can check the status of the symlinks by running ls -l inside the imported directory of the job. Note: The "Import Templates" and "Import Volumes" jobs copy the specified files directly into the job directory.

Step One - Identify Project/Job symlinks

Start up an interactive python session

cryosparcm icli

Use the cli to find all the symlinks for an entire project or a single job.

>>> cli.get_project_symlinks(’P1’)

or

>>> cli.get_job_symlinks(’P1’, ‘J3’)

[{'exists': True,
'link_path': '/bulk8/data/dev_nwong_projects/P1/J2/imported/004525579726026751633_14sep05c_c_00003gr_00014sq_00011hl_00003es.frames.tif',
'link_target': '/bulk8/data/dev_nwong_projects/testdata/empiar_10025_subset/14sep05c_c_00003gr_00014sq_00011hl_00003es.frames.tif'},
…]

where

  • link_path is the path to the symlink file

  • link_target is the file the symlink points to

  • exists indicates if the target file exists

Start up an interactive python session

cryosparcm icli

Execute a MongoDB query for all potentially affected "import" jobs

>>> jobs = list(db.jobs.find({'deleted': False, 'job_type': {'$in': ['import_particles', 'import_movies', 'import_micrographs']}}, {'_id': 0, 'project_uid': 1, 'uid': 1, 'job_type': 1}))
>>> print(jobs)
[{'job_type': 'import_movies', 'project_uid': 'P1', 'uid': 'J3'},
{'job_type': 'import_movies', 'project_uid': 'P2', 'uid': 'J3'},
{'job_type': 'import_movies', 'project_uid': 'P1', 'uid': 'J41'},
{'job_type': 'import_particles', 'project_uid': 'P1', 'uid': 'J42'},
{'job_type': 'import_movies', 'project_uid': 'P2', 'uid': 'J29'},
{'job_type': 'import_particles', 'project_uid': 'P2', 'uid': 'J51'},
{'job_type': 'import_movies', 'project_uid': 'P2', 'uid': 'J55'},
{'job_type': 'import_movies', 'project_uid': 'P2', 'uid': 'J64'},
...]

From this point, you can take a look into each list job's imported directory

>>> cli.get_job_dir_abs('P1', 'J3')
'/data/cryosparc_projects/P1/J3'
    
>>> !ls -l /data/cryosparc_projects/P1/J3/imported
total 99
lrwxrwxrwx 1 cryosparcuser cryosparcuser 83 Jan 31  2018 14sep05c_00024sq_00003hl_00002es.frames.mrc -> /data/EMPIAR/10025/data/14sep05c_raw_196/14sep05c_00024sq_00003hl_00002es.frames.mrc
lrwxrwxrwx 1 cryosparcuser cryosparcuser 83 Jan 31  2018 14sep05c_00024sq_00003hl_00005es.frames.mrc -> /data/EMPIAR/10025/data/14sep05c_raw_196/14sep05c_00024sq_00003hl_00005es.frames.mrc
lrwxrwxrwx 1 cryosparcuser cryosparcuser 83 Jan 31  2018 14sep05c_00024sq_00004hl_00002es.frames.mrc -> /data/EMPIAR/10025/data/14sep05c_raw_196/14sep05c_00024sq_00004hl_00002es.frames.mrc
lrwxrwxrwx 1 cryosparcuser cryosparcuser 83 Jan 31  2018 14sep05c_00024sq_00006hl_00003es.frames

Step Two - Modify One or Many Links

Use the command cli.job_import_replace_symlinks(project_uid, job_uid, prefix_cut, prefix_new) where prefix_cut is the beginning of the link you'd like to cut (e.g. /data/EMPIAR) and where prefix_new is what you'd like to replace it with (e.g. /data). This function will loop through every file inside the job directory, find all symlinks, and only modify them only if they start with prefix_cut. The function returns the number of links it modified. Below it is used in a loop to modify all jobs across all projects all at once.

>>> failed_jobs = []
>>> for job in jobs:
        try:
            print("Repairing %s %s" % (job['project_uid'], job['uid']))
            modified_count = cli.job_import_replace_symlinks(job['project_uid'], job['uid'], '/data/EMPIAR', '/data')
            print("Finished. Modified %d links." % (modified_count))
        except Exception as e:
            failed_jobs.append((job['project_uid'], job['uid']))
            print("Failed to repair %s %s: %s" % (job['project_uid'], job['uid'], str(e)))
...
    
>>> failed_jobs
[]

B. Moving Only CryoSPARC Project Directories (and all jobs inside them)

If you're moving the locations of the projects and their jobs, you will need to point CryoSPARC to the new directory where the projects reside. Jobs inside CryoSPARC are referenced by their relative location to their project directory. This allows a user to specify a new location for the project directory only, rather than each job.

Update a Single Project

cryosparcm cli "update_project('PXX', {'project_dir' : '/new/abs/path/PXX'})"

  • Where 'PXX' is the project UID and '/new/abs/path/PXX' is the new directory.

Updating Multiple Projects

Step One - Identify All Project Directories

Start up an interactive python session

cryosparcm icli

Execute a MongoDB query to list all project directories

>>> projects = list(db['projects'].find({}, {'uid': 1, 'project_dir': 1, '_id': 0}))
>>> projects
    [{'project_dir': '/data/cryosparc_projects/P1', 'uid': 'P1'},
     {'project_dir': '/data/cryosparc_projects/P2', 'uid': 'P2'},
     {'project_dir': '/data/cryosparc_projects/P3', 'uid': 'P3'},
     {'project_dir': '/data/cryosparc_projects/P4', 'uid': 'P4'},
    ...]

Step Two - Modify One or Many Project Directories

Use the command update_project(project_uid, attrs, operation='$set') where attrs is a dictionary whose keys correspond to the fields in the project document to update. In the following example, update_project used in a loop to modify all project directory paths.

>>> failed_projects = []
>>> new_parent_dir = '/cryoem/cryosparc_projects'
>>> for project in projects:
        new_project_dir = os.path.join(new_parent_dir, os.path.basename(project['project_dir']))
        try:
            print("Modifying project directory for %s: %s --> %s" % (project['uid'], project['project_dir'], new_project_dir))
            cli.update_project(project['uid'], {'project_dir': new_project_dir})
        except Exception as e:
            failed_projects.append(project['uid'])
            print("Failed to update %s: %s" % (project['uid'], str(e)))
...

>>> failed_projects
[]

C. Moving the CryoSPARC database

The CryoSPARC database doesn't necessarily have to be in the same location as the CryoSPARC binaries. To move the database, specify the new location in the cryosparc_master/config.sh file.

A copy of the database should only be used with CryoSPARC project directories whose contents have not changed after the database copy has started. Otherwise, CryoSPARC may malfunction and the affected project directory may get corrupted.

Step One - Turn off CryoSPARC

cryosparcm stop

Step Two - Move the Database

rsync -r --links /data/cryosparc/cryosparc_database/* /cryoem/cryosparc/cryosparc_database

Step Three - Modify Configurations

Navigate to the cryosparc_master directory

cd /data/cryosparc/cryosparc_master

Modify config.sh to contain the new directory path to the database

nano config.sh
    
#modify the line below
export CRYOSPARC_DB_PATH="/cryoem/cryosparc/cryosparc_database"

Step Four - Restart CryoSPARC

cryosparcm start

D. Hosting CryoSPARC on another machine

Assuming that the CryoSPARC instance is located on a shared storage layer, if you want to host it on another machine, (i.e. server1:39000 —> server2:39000 you just need to specify the new hostname to CryoSPARC master. Read further if the new machine doesn't have access to the same file system.

Step Zero - Reinstall CryoSPARC on new machine (Optional)

NOTE: Skip this step if you're using a shared filesystem (e.g. a remote storage server that is hosted on all machines). This means that the CryoSPARC binaries, database and project directories are already accessible on the new machine.

Step One - Turn off CryoSPARC

cryosparcm stop

Step Two - Modify Configurations

Navigate to the cryosparc_master directory

cd /data/cryosparc/cryosparc_master

Modify config.sh to list the new master node

nano config.sh
    
#modify the line below
export CRYOSPARC_MASTER_HOSTNAME="newnode"

Step Three - Restart CryoSPARC

On the new machine, start CryoSPARC. Ensure you start CryoSPARC using the same user.

cryosparcm start

The information in this section applies to CryoSPARC ≤v3.3. For instructions on moving cryoSPARC project directories in v4.0+, see

First, follow all previous parts of this guide in order. Read this entire guide thoroughly before starting anything. You will need your old instance started and working, so make sure you still have access to it. Then, follow the to install CryoSPARC normally using the migrated database path. After this step, you are done.

install guide
Setup, Configuration and Management
symlinks
#7.-imported-data-and-symlinks-in-project-directories
Use Case: Moving a project directory from one storage location to another