Managing a CryoSPARC Live Session from the CLI

This guide details how to automate the creation, configuration and overall management of a CryoSPARC Live session in Python via the CryoSPARC API.

A future release of CryoSPARC will break the code shown on this page.

Overview

This guide details how to automate the creation, configuration and overall management of a CryoSPARC Live session in Python via the CryoSPARC API.

The instructions are valid for CryoSPARC versions from 3.3 through 4.6.

Setup

To run the following commands, you can use the built-in python interpreter provided by CryoSPARC by running cryosparcm icli.

You can also issue one-off calls directly to CryoSPARC by running cryosparcm cli "<function_name()>" or cryosparcm rtpcli “<function_name()>” for command_core and command_rtp functions, respectively.

This guide will use the interactive python shell provided by CryoSPARC to create a CryoSPARC Live Session. When you run cryosparcm icli, you will have access to the command_core API via cli, the command_rtp API via rtp, and the database via db.

Using a Script

You can also put these functions inside a bash script or a python script. To get access to the CryoSPARC API (cli and rtp) inside a python script, add the following lines to your script, e.g., called automate_cryosparc_live.py:

import os
from cryosparc_compute import client

host = os.environ['CRYOSPARC_MASTER_HOSTNAME']
command_core_port = int(os.environ['CRYOSPARC_COMMAND_CORE_PORT'])
command_rtp_port = int(os.environ['CRYOSPARC_COMMAND_RTP_PORT'])

cli = client.CommandClient(host=host, port=command_core_port)
rtp = client.CommandClient(host=host, port=command_rtp_port)

# ...

Use CryoSPARC’s provided python interpreter to execute your script (required):

# Enter these in the command line
eval $(cryosparcm env)
export PYTHONPATH="${CRYOSPARC_ROOT_DIR}"
python automate_cryosparc_live.py

Automate a CryoSPARC Live Session

Create A Project [Optional]

You will need a project to create your Live session. If you already have a project, you can skip this step.

1. Get your CryoSPARC user ID using your CryoSPARC email address

   email_address = '[e-mail]'
   user_id = cli.get_id_by_email(email_address)

2. Create the project

   Note that CryoSPARC will create a folder for the project inside the project_container_dir path you specify named after the UID of the project created (e.g., P12).

   The definition of the project_description variable and the corresponding desc parameter are optional.

   project_container_dir = '/data/cryoSPARC_projects/'
   project_title = 'EMPIAR-10025'
   project_description = 'T20s Processing Project'
   
   project_uid = cli.create_empty_project(owner_user_id=user_id, project_container_dir=project_container_dir, title=project_title, desc=project_description)

Create a CryoSPARC Live Session

1. Get your CryoSPARC user ID using your CryoSPARC email address

   email_address = '[e-mail]'
   user_id = cli.get_id_by_email(email_address)

2. Create the CryoSPARC Live Session

   Note that the title and desc arguments are optional. project_uid can be defined explicitly for pre-existing projects, or by using the value returned by cli.create_empty_project(<..>)(see above).

   project_uid = 'P12' # set this variable if you didn't already create a project in the above step
   session_title = 'EMPIAR-10025 Live Processing Session'
   session_description = 'Simulated live session'
   
   session_uid = rtp.create_new_live_workspace(project_uid=project_uid, created_by_user_id=user_id, title=session_title, desc=session_description)

Configure your CryoSPARC Live Session

1. Set the session’s compute configuration

   phase_one_lane = 'gpu_lane_name'
   phase_one_gpus = 2 # optional
   phase_two_lane = 'gpu_lane_name'
   phase_two_ssd = True # optional
   auxiliary_lane = 'gpu_lane_name'
   auxiliary_ssd = True # optional
   
   rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='phase_one_lane', value=phase_one_lane)
   rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='phase_one_gpus', value=phase_one_gpus)
   rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='phase_two_lane', value=phase_two_lane)
   rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='phase_two_ssd', value=phase_two_ssd)
   rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='auxiliary_lane', value=auxiliary_lane)
   rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='auxiliary_ssd', value=auxiliary_ssd)

2. Set the session’s exposure group parameters

   Note that the following lines change the exposure group parameters for the first exposure group (exp_group_id=1).

   file_engine_watch_path_abs = '/data/EMPIAR/10025/data/14sep05c_raw_196/'
   file_engine_filter = '*.frames.mrc'
   gainref_path = '/data/EMPIAR/10025/data/14sep05c_raw_196/norm-amibox05-0.mrc'
   
   rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=1, name='file_engine_watch_path_abs', value=file_engine_watch_path_abs)
   rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=1, name='file_engine_filter', value=file_engine_filter)
   rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=1, name='gainref_path', value=gainref_path)
   
   rtp.exposure_group_finalize_and_enable(project_uid=project_uid, session_uid=session_uid, exp_group_id=1)

3. Set the session’s processing parameters

   Note that the following parameters are required to be set before starting a session.

   To see the full list of parameters available to be set, see below.

   rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='mscope_params', param_name='psize_A', value=0.6575)
   rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='mscope_params', param_name='accel_kv', value=300)
   rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='mscope_params', param_name='cs_mm', value=2.7)
   rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='mscope_params', param_name='total_dose_e_per_A2', value=53)
   
   rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='blob_pick', param_name='diameter', value=100)
   rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='blob_pick', param_name='diameter_max', value=200)
   
   rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='extraction', param_name='box_size_pix', value=440)

Start the CryoSPARC Live Session

1. Start the session

   rtp.start_session(project_uid=project_uid, session_uid=session_uid, user_id=user_id)

   If you’ve configured your session properly, CryoSPARC Live will start to find movies and process data. At this point, you can navigate to the user interface and watch as exposures are processed. The next step would be to start Streaming 2D Classification and Streaming Homogeneous Refinement. You can also use the user interface to manually pick particles and use the template picker to identify particle locations.

2. Start Streaming 2D Classification

   rtp.phase2_class2D_set_param(project_uid=project_uid, session_uid=session_uid, par='class2D_K', value=200)
   rtp.phase2_class2D_start(project_uid=project_uid, session_uid=session_uid)

3. Start Ab-Initio Reconstruction

   1. Create a new initial volume

      rtp.phase2_abinit_set_param(project_uid=project_uid, session_uid=session_uid, par='abinit_K', value=3)
      rtp.phase2_abinit_start(project_uid=project_uid, session_uid=session_uid)

   2. Use an existing job’s volume output as the initial volume

      rtp.phase2_abinit_set_job(project_uid=project_uid, session_uid=session_uid, abinit_juid='J43')

4. Start Streaming Homogeneous Refinement

   Note that an initial volume must be set for the session before starting Streaming Homogeneous Refinement.

   rtp.phase2_refine_set_param(project_uid=project_uid, session_uid=session_uid, par='refine_symmetry', value='D7')
   rtp.phase2_refine_start(project_uid=project_uid, session_uid=session_uid)

Stop the CryoSPARC Live Session

1. Pause the entire CryoSPARC Live Session

   rtp.pause_session(project_uid=project_uid, session_uid=session_uid)

2. Clear the CryoSPARC Live Session

   rtp.clear_session(project_uid=project_uid, session_uid=session_uid)

Export Exposures and Particles from the CryoSPARC Live Session

1. Export all exposures from the session

   rtp.dump_exposures(project_uid=project_uid, session_uid=session_uid)

2. Export all particles from the session

   rtp.dump_particles(project_uid=project_uid, session_uid=session_uid)

Appendix

As of v3.3, these are the parameters you can set for a live session:

[{'param_sec': 'mscope_params',
  'param_name': 'gainref_flip_x',
  'param_type': 'boolean',
  'base_value': False,
  'title': 'Flip gain ref in X?',
  'desc': 'Flip gain ref left-to-right (in X axis)'},
 {'param_sec': 'mscope_params',
  'param_name': 'gainref_flip_y',
  'param_type': 'boolean',
  'base_value': False,
  'title': 'Flip gain ref in Y?',
  'desc': 'Flip gain ref top-to-bottom (in Y axis)'},
 {'param_sec': 'mscope_params',
  'param_name': 'gainref_rotate_num',
  'param_type': 'number',
  'base_value': 0,
  'title': 'Rotate gain ref?',
  'desc': 'Rotate gain ref counter-clockwise by 90 degrees this many times'},
 {'param_sec': 'mscope_params',
  'param_name': 'psize_A',
  'param_type': 'number',
  'base_value': None,
  'title': 'Raw pixel size (A)',
  'desc': 'Pixel size of the raw movie data in Angstroms'},
 {'param_sec': 'mscope_params',
  'param_name': 'accel_kv',
  'param_type': 'number',
  'base_value': None,
  'title': 'Accelerating voltage (kV)',
  'desc': ''},
 {'param_sec': 'mscope_params',
  'param_name': 'cs_mm',
  'param_type': 'number',
  'base_value': None,
  'title': 'Spherical aberration (mm)',
  'desc': ''},
 {'param_sec': 'mscope_params',
  'param_name': 'total_dose_e_per_A2',
  'param_type': 'number',
  'base_value': None,
  'title': 'Total exposure dose (e/A^2)',
  'desc': ''},
 {'param_sec': 'mscope_params',
  'param_name': 'phase_plate',
  'param_type': 'boolean',
  'base_value': False,
  'title': 'Phase plate',
  'desc': 'Were the images collected using a phase plate?'},
 {'param_sec': 'mscope_params',
  'param_name': 'neg_stain',
  'param_type': 'boolean',
  'base_value': False,
  'title': 'Negative stain',
  'desc': 'Are the samples negative stain (True) or cryo (False)?'},
 {'param_sec': 'mscope_params',
  'param_name': 'eer_upsampfactor',
  'param_type': 'number',
  'base_value': 2,
  'title': 'EER upsampling factor',
  'desc': 'EER upsampling factor (applies to .eer/.ecc format data only.'},
 {'param_sec': 'mscope_params',
  'param_name': 'eer_numfractions',
  'param_type': 'number',
  'base_value': 40,
  'title': 'EER number of fractions',
  'desc': 'EER number of fractions (applies to .eer/.ecc format data only.'},
 {'param_sec': 'motion_settings',
  'param_name': 'res_max_align',
  'base_value': 5,
  'title': 'Maximum alignment resolution (A)',
  'param_type': 'number',
  'desc': 'Maximum resolution (in A) to consider when aligning frames. Generally, betwen 5A and 3A is best.'},
 {'param_sec': 'motion_settings',
  'param_name': 'bfactor',
  'base_value': 500,
  'title': 'B-factor during alignment',
  'param_type': 'number',
  'desc': 'B-factor that blurs frames before aligning. Generally 500 to 100 is best.'},
 {'param_sec': 'motion_settings',
  'param_name': 'frame_start',
  'base_value': 0,
  'title': 'Start frame (included, 0-based)',
  'param_type': 'number',
  'desc': 'Which frame number, starting at zero, to begin motion correction from. This value controls how many early frames are dropped from the motion corrected result. This value will also be used in local motion correction.'},
 {'param_sec': 'motion_settings',
  'param_name': 'frame_end',
  'base_value': None,
  'title': 'End frame (excluded, 0-based) ',
  'param_type': 'number',
  'desc': 'Which frame number, starting at zero, to not include in motion correction, also excluding all frames after this one. Generally this does not improve results, as later frames are downweighted during dose weighting in local motion correction.'},
 {'param_sec': 'motion_settings',
  'param_name': 'output_fcrop_factor',
  'base_value': 1,
  'title': 'Output F-crop factor',
  'param_type': 'number',
  'desc': 'Output Fourier cropping factor. 1.0 means no cropping, 0.5 means crop to 1/2 the resolution, etc. Only 1, 0.75, 0.5, 0.25 are allowed values'},
 {'param_sec': 'motion_settings',
  'param_name': 'override_total_exp',
  'base_value': None,
  'title': 'Override e/A^2',
  'param_type': 'number',
  'desc': 'Override the dose (in total e/A^2 over the exposure) that was given at import time but can be overridden here.'},
 {'param_sec': 'motion_settings',
  'param_name': 'variable_dose',
  'base_value': False,
  'title': 'Allow Variable Dose',
  'param_type': 'boolean',
  'desc': 'Enable correct processing when frames have variable dose fractionation'},
 {'param_sec': 'motion_settings',
  'param_name': 'smooth_lambda_cal',
  'base_value': 0.5,
  'title': 'Calibrated smoothing',
  'param_type': 'number',
  'desc': 'Calibrated smoothing constant applied to trajectories (None to autotune)'},
 {'param_sec': 'motion_settings',
  'param_name': 'override_K_Z',
  'base_value': None,
  'title': 'Override knots Z',
  'param_type': 'number',
  'desc': 'Override automatically selected spline order for Z dimension (time)'},
 {'param_sec': 'motion_settings',
  'param_name': 'override_K_Y',
  'base_value': None,
  'title': 'Override knots Y',
  'param_type': 'number',
  'desc': 'Override automatically selected spline order for Y dimension (vertical)'},
 {'param_sec': 'motion_settings',
  'param_name': 'override_K_X',
  'base_value': None,
  'title': 'Override knots X',
  'param_type': 'number',
  'desc': 'Override automatically selected spline order for X dimension (horizontal)'},
 {'param_sec': 'motion_settings',
  'param_name': 'optimize_for_gpu_memory',
  'base_value': False,
  'title': 'Low-memory mode',
  'param_type': 'boolean',
  'desc': 'If running out of GPU memory, this option can be used to prioritize memory use at the expense of speed (BETA). The results are unchanged.'},
 {'param_sec': 'ctf_settings',
  'param_name': 'amp_contrast',
  'base_value': 0.1,
  'title': 'Amplitude Contrast',
  'param_type': 'number',
  'desc': 'Amplitude constrast to use. Typically 0.07 or 0.1 for cryo-EM data.'},
 {'param_sec': 'ctf_settings',
  'param_name': 'res_min_align',
  'base_value': 25,
  'title': 'Minimum resolution (A)',
  'param_type': 'number',
  'desc': 'Minimum resolution (in A) to consider when estimating CTF.'},
 {'param_sec': 'ctf_settings',
  'param_name': 'res_max_align',
  'base_value': 4,
  'title': 'Maximum resolution (A)',
  'param_type': 'number',
  'desc': 'Maximum resolution (in A) to consider when estimating CTF.'},
 {'param_sec': 'ctf_settings',
  'param_name': 'df_search_min',
  'base_value': 1000,
  'title': 'Minimum search defocus (A)',
  'param_type': 'number',
  'desc': 'Defocus range for gridsearch.'},
 {'param_sec': 'ctf_settings',
  'param_name': 'df_search_max',
  'base_value': 40000,
  'title': 'Maximum search defocus (A)',
  'param_type': 'number',
  'desc': 'Defocus range for gridsearch.'},
 {'param_sec': 'ctf_settings',
  'param_name': 'do_phase_shift_search_refine',
  'base_value': False,
  'title': 'Do phase shift search',
  'param_type': 'boolean',
  'desc': 'Whether to carry out search and refinement over phase shift.'},
 {'param_sec': 'ctf_settings',
  'param_name': 'phase_shift_min',
  'base_value': 0,
  'title': 'Min. search phase-shift (rad)',
  'param_type': 'number',
  'desc': 'Phase-shift range for gridsearch.'},
 {'param_sec': 'ctf_settings',
  'param_name': 'phase_shift_max',
  'base_value': 3.141592653589793,
  'title': 'Max. search phase-shift (rad)',
  'param_type': 'number',
  'desc': 'Phase-shift range for gridsearch.'},
 {'param_sec': 'ctf_settings',
  'param_name': 'do_phase_shift_refine_only',
  'base_value': False,
  'title': 'Do phase refine only',
  'param_type': 'boolean',
  'desc': 'Whether to carry out refinement over phase shift only'},
 {'param_sec': 'ctf_settings',
  'param_name': 'override_K_Y',
  'base_value': None,
  'title': 'Override knots Y',
  'param_type': 'number',
  'desc': 'Override automatically selected spline order for Y dimension (vertical)'},
 {'param_sec': 'ctf_settings',
  'param_name': 'override_K_X',
  'base_value': None,
  'title': 'Override knots X',
  'param_type': 'number',
  'desc': 'Override automatically selected spline order for X dimension (horizontal)'},
 {'param_sec': 'ctf_settings',
  'param_name': 'classic_mode',
  'base_value': False,
  'title': 'Classic mode',
  'param_type': 'boolean',
  'desc': 'Uses the old Patch CTF algorithm (cryoSPARC v.2.15 and earlier) intead of the new one.'},
 {'param_sec': 'particle_picking',
  'param_name': 'current_picker',
  'base_value': 'blob',
  'title': 'Current active picker',
  'param_type': 'enum',
  'enum_keys': ['blob', 'template', 'deep'],
  'desc': 'Which picker type to use on future exposures.'},
 {'param_sec': 'blob_pick',
  'param_name': 'diameter',
  'base_value': None,
  'title': 'Minimum particle diameter (A)',
  'param_type': 'number',
  'desc': 'Min Particle diameter (A)'},
 {'param_sec': 'blob_pick',
  'param_name': 'diameter_max',
  'base_value': None,
  'title': 'Maximum particle diameter (A)',
  'param_type': 'number',
  'desc': 'Max Particle diameter (A)'},
 {'param_sec': 'blob_pick',
  'param_name': 'use_circle',
  'base_value': True,
  'title': 'Use circular blob',
  'param_type': 'boolean',
  'desc': ''},
 {'param_sec': 'blob_pick',
  'param_name': 'use_ellipse',
  'base_value': False,
  'title': 'Use elliptical blob',
  'param_type': 'boolean',
  'desc': ''},
 {'param_sec': 'blob_pick',
  'param_name': 'use_ring',
  'base_value': False,
  'title': 'Use ring blob',
  'param_type': 'boolean',
  'desc': ''},
 {'param_sec': 'blob_pick',
  'param_name': 'lowpass_res_template',
  'base_value': 20,
  'title': 'Lowpass filter to apply to templates (A)',
  'param_type': 'number',
  'desc': 'Lowpass filter to apply to templates, (A)s'},
 {'param_sec': 'blob_pick',
  'param_name': 'lowpass_res',
  'base_value': 20,
  'title': 'Lowpass filter to apply (A)',
  'param_type': 'number',
  'desc': 'Lowpass filter to apply, (A)s'},
 {'param_sec': 'blob_pick',
  'param_name': 'angular_spacing_deg',
  'base_value': 5,
  'title': 'Angular sampling (degrees)',
  'param_type': 'number',
  'desc': 'Angular sampling of templates in degrees. Lower value will mean finer rotations.'},
 {'param_sec': 'blob_pick',
  'param_name': 'use_ctf',
  'base_value': False,
  'title': 'Use CTFs to filter the templates',
  'param_type': 'boolean',
  'desc': 'Whether to use micrograph CTF to filter the templates'},
 {'param_sec': 'blob_pick',
  'param_name': 'min_distance',
  'base_value': 1.0,
  'title': 'Min. separation dist (diameters)',
  'param_type': 'number',
  'desc': 'Minimum distance between particles in units of particle diameter (min diameter for blob picker). The lower this value, the more and closer particles it picks.'},
 {'param_sec': 'blob_pick',
  'param_name': 'num_process',
  'base_value': None,
  'title': 'Number of mics to process',
  'param_type': 'number',
  'desc': 'Number of micrographs to process. None means all.'},
 {'param_sec': 'blob_pick',
  'param_name': 'num_plot',
  'base_value': 10,
  'title': 'Number of mics to plot',
  'param_type': 'number',
  'desc': 'Number of micrographs to plot.'},
 {'param_sec': 'blob_pick',
  'param_name': 'max_num_hits',
  'base_value': 4000,
  'title': 'Maximum number of local maxima to consider',
  'param_type': 'number',
  'desc': 'Maximum number of local maxima (peaks) considered.'},
 {'param_sec': 'template_pick',
  'param_name': 'diameter',
  'base_value': None,
  'title': 'Particle diameter (A)',
  'param_type': 'number',
  'desc': 'Particle diameter (A)'},
 {'param_sec': 'template_pick',
  'param_name': 'lowpass_res_template',
  'base_value': 20,
  'title': 'Lowpass filter to apply to templates (A)',
  'param_type': 'number',
  'desc': 'Lowpass filter to apply to templates, (A)s'},
 {'param_sec': 'template_pick',
  'param_name': 'lowpass_res',
  'base_value': 20,
  'title': 'Lowpass filter to apply (A)',
  'param_type': 'number',
  'desc': 'Lowpass filter to apply, (A)s'},
 {'param_sec': 'template_pick',
  'param_name': 'angular_spacing_deg',
  'base_value': 5,
  'title': 'Angular sampling (degrees)',
  'param_type': 'number',
  'desc': 'Angular sampling of templates in degrees. Lower value will mean finer rotations.'},
 {'param_sec': 'template_pick',
  'param_name': 'use_ctf',
  'base_value': True,
  'title': 'Use CTFs to filter the templates',
  'param_type': 'boolean',
  'desc': 'Whether to use micrograph CTF to filter the templates'},
 {'param_sec': 'template_pick',
  'param_name': 'min_distance',
  'base_value': 0.5,
  'title': 'Min. separation dist (diameters)',
  'param_type': 'number',
  'desc': 'Minimum distance between particles in units of particle diameter. The lower this value, the more and closer particles it picks.'},
 {'param_sec': 'template_pick',
  'param_name': 'num_process',
  'base_value': None,
  'title': 'Number of mics to process',
  'param_type': 'number',
  'desc': 'Number of micrographs to process. None means all.'},
 {'param_sec': 'template_pick',
  'param_name': 'num_plot',
  'base_value': 10,
  'title': 'Number of mics to plot',
  'param_type': 'number',
  'desc': 'Number of micrographs to plot.'},
 {'param_sec': 'template_pick',
  'param_name': 'max_num_hits',
  'base_value': 4000,
  'title': 'Maximum number of local maxima to consider',
  'param_type': 'number',
  'desc': 'Maximum number of local maxima (peaks) considered.'},
 {'param_sec': 'template_pick',
  'param_name': 'templates_from_job',
  'base_value': None,
  'title': 'Tempates from job (PX.JXX)',
  'param_type': 'string'},
 {'param_sec': 'template_pick',
  'param_name': 'templates_selected',
  'base_value': None,
  'title': 'Templates selected (comma sep)',
  'param_type': 'string'},
 {'param_sec': 'extraction',
  'param_name': 'thresh_score_min',
  'base_value': None,
  'title': 'Score threshold min',
  'param_type': 'number',
  'desc': 'Minimum picking score threshold'},
 {'param_sec': 'extraction',
  'param_name': 'thresh_power_min',
  'base_value': None,
  'title': 'Power threshold min',
  'param_type': 'number',
  'desc': 'Minimum picking power threshold'},
 {'param_sec': 'extraction',
  'param_name': 'thresh_power_max',
  'base_value': None,
  'title': 'Power threshold max',
  'param_type': 'number',
  'desc': 'Maximum picking power threshold'},
 {'param_sec': 'extraction',
  'param_name': 'box_size_pix',
  'base_value': None,
  'title': 'Extraction box size (pix)',
  'param_type': 'number',
  'desc': 'Size of box to be extracted from micrograph.'},
 {'param_sec': 'extraction',
  'param_name': 'bin_size_pix',
  'base_value': None,
  'title': 'Fourier crop to box size (pix)',
  'param_type': 'number',
  'desc': 'Size of particle boxes after they have been extracted. None means use the same as the extraction box size'}]