New Live Session: Start to Finish Guide
Last updated
Last updated
This guide covers the end-to-end workflow for using cryoSPARC Live.
In addition to the below, we recommend checking out the cryoSPARC Live Walkthrough:
From the CryoSPARC Live Sessions View, create and configure new sessions and view a summary of existing sessions.
➡️Navigate to the Sessions View by clicking on the CryoSPARC Live icon on the sidebar.
➡️Click on the New Session button in the header, which opens a panel in the sidebar.
➡️Select or enter the cryoSPARC Project number where you would like this Session to be created. Enter a Title for the Session.
The project you select must already exist. You may need to create a new Project in the cryoSPARC interface, if one does not already exist. Recommended scenarios for creating new Projects/Sessions:
Collecting new data for the first time on a new target molecule: Create a new Project and a new Session within it.
Collecting data a second time on the same sample/target (potentially the same or different grid from the same batch, potentially on a different day): Use the existing Project and existing Session where you processed the first set of images. In the existing Session, create a new exposure group (see Configure New Session, below) and start the Session again, causing it to read in the new set of images.
Collecting data on a new sample/grid/preparation of the same target molecule: Use the existing Project, but create a new Session. This allows easy re-use of 3D volumes, 2D templates, and easy combining of particle images downstream. You can create multiple Sessions within a project, for example if collecting/processing new data from a similar sample. The recommended workflow is to create a new Project for each new unrelated sample on which you are collecting data.
Only users who own a particular Project or have a Project shared with them can see cryoSPARC Live Sessions within those Projects. To add a user to a Project, navigate to the Project Details Panel in your regular cryoSPARC instance and click Share With Users to select the user you wish to give access.
➡️Click Create Session. This will open your New Session. All New Sessions by default are set to Paused status.
On the Configuration Tab, enter required parameters (or load a saved Configuration Profile) and select compute resources.
New Sessions will open on the Configuration Tab by default.
In order to Start this New Session, you must first type in (or load from a profile - see below) a few required parameters in three sections (Configuration, Parameters, and Exposure Groups). Required parameters are outlined in red and are also summarized in the Start Checklist.
These are parameters relating to the GPU resources on which cryoSPARC Live jobs will run. For more details on hardware requirements for Live, please see: Prerequisites and Compute Resources Setup
➡️Select/enter the following required parameters
Preprocessing Lane: Select a Preprocessing Lane and a Number of Preprocessing GPU Workers. This is effectively the number of GPU workers that will carry out motion correction, CTF estimation, particle picking and extraction in parallel concurrently.
It's possible to adjust the number of GPU workers allocated for the preprocessing stage throughout the lifecycle of the session. For example, at the start of the session, you can allocate four GPUs to quickly extract particles from exposures, then lower the number of workers to two or one when resources are needed for the reconstruction stage.
For information on the minimum number of GPU workers you need to assign, see: Prerequisites and Compute Resources Setup
Reconstruction Lane: Select a Reconstruction Lane where Streaming 2D classification and Streaming Refinement jobs will be launched.
Auxiliary Lane: Select an Auxiliary Lane where Ab-initio Reconstruction and Generate Templates jobs will be launched.
(Optional) Use SSD: If the disk you are using for the project directory is already an SSD, you don't need to copy the files to another SSD. Turn off Use SSD if you do not wish to copy files over (for larger datasets, it can take some time to write all files over to the SSD before processing can stream in new collected particles).
(Optional) Priority: Unless specified, all jobs in the Live Session will run according to the Session-level priority. For details on NEW Priority Queuing in cryoSPARC, please see: Priority Queuing Tutorial
Jobs launched through Live can be viewed by clicking on the 'x Active Jobs' button in the footer of the cryoSPARC Live application. You can also see Live jobs in the Resource Manager tab in your regular cryoSPARC instance. To quickly navigate to the Resource Manager, click on any job in the cryoSPARC Live application footer.
This will open the Active Jobs modal.
To adjust the Number of Preprocessing GPU workers or any other Compute Resources settings while a Session is Running, you will need to first Pause the Session, update the configuration and then Start the Session again.
These are data processing parameters that will be engaged across Microscope/Camera, Motion Correction, CTF Estimation, Blob Picker, Template Picker, and Particle Extraction once the Session is Started.
➡️Step 1: At a minimum, you need to enter the following 7 required parameters which are outlined in red.
Microscope/Camera Parameter
Raw pixel size (A): The raw pixel size of the input movie data. For super-resolution data, this should be the super-resolution pixel size (i.e. not the camera native pixel size). For EER data, this should be the camera native pixel size, and you should also modify the corresponding EER parameters for upsampling and dose fractionation.
Accelerating voltage (kV): The accelerating voltage of the microscope collecting the data.
Spherical abberation (mm): The spherical aberration of the microscope collecting the data. This should be zero if there is a Cs-corrector.
Total exposure dose (e/A^2): This should be the total electron dose across each movie (i.e. not the per frame dose).
Blob Picker
Minimum particle diameter (A): This can initially be set as the minimum dimension of the particle expected. This can be fine-tuned later on the Picking tab.
Maximum particle diameter (A): This should be set equal to or slightly larger than the maximum dimension of the particle expected. This can be fine-tuned later on the Picking tab.
Particle Extraction
Extraction box size (pix): This should be a box size in pixels for extracting particles. Generally, this should be set to about twice the particle diameter, and should be a set to a number that has 2,3,5,7 as its prime factors. Typical box sizes are between 256 and 640 pixels. Note that the box size is in pixels after Fourier-cropping (if enabled in motion correction). Mathematically good numbers are:
If you would like to save micrographs or particles in float16 format , you can enable the toggle titled "Save results in 16-bit floating point" under "Motion correction", "Particle extraction", or both. (CryoSPARC v4.4+)
To view advanced parameters, click Show Advanced. You can also filter by a parameter name.
➡️Step 2: Apply (i.e., Save) your parameters
Click Apply to All to save your parameter entries for all exposures (existing and incoming) in the Session. This is the recommended course of action when starting a New Session.
If your Session is already running and you wish to have new parameter changes apply only to new exposures coming in after that point, then you may instead wish to click Apply to Future.
This section covers parameters that will tell cryoSPARC Live where to find new movie files, which files to read, whether to apply a gain reference and/or defect file (if available), and how to handle multiple Exposure Groups, if applicable.
By default, there is always at least one Exposure Group in a Session. Exposure groups are collections of exposures that have the same optical parameters. You can use exposure groups for multiple purposes including:
A new exposure group per grid
A new exposure group for each beam-shift position in a template
A new exposure group for different squares on a grid
A new exposure group for a new data collection session, perhaps on a different day
After enabling an exposure group, you can choose to continuously listen to the specified directory and filename wildcard filter for new exposures, or ignore the group if you no longer want to include a set of exposures for processing.
➡️Step 1: Edit Parameters for Exposure Group 1
Enable continuous import: Toggling this on will allow new exposures added to the directory, to be processed as they are found. Be default, this is enabled. If you are creating more than one exposure group and the initial exposure groups are not expected to contain any new images that Live has not already found, you can disable continuous import from the older exposure groups to save disk operations needed to check for new files periodically.
(Optional) Ignore exposures from group: This is useful only if you have multiple Exposure Groups configured, and you would like to ignore all exposures from a particular Group. This might be used if you find that all the images from a group are of poor quality, for example.
Directory to watch: The file system location where the exposures are being written or are already saved. This is a directory on the filesystem, not a wildcard path.
(Optional) File name wildcard filter: You can optionally filter by wildcard to select files within the Directory to watch, that match a specific extension, e.g., *.mrc. This is important if multiple types of files will be saved in the same directory.
(Optional) Search recursively: This option will traverse all subdirectories within the directory specified in Directory to watch that matches the File name wildcard filter value. This can be helpful if your data collection software writes files to multiple sub-folders, for example. Leaving this option turned off will tell the file engine to only search for files matching the File name wildcard filter value in the top level directory. If using Search Recursively, then in Directory to watch, ensure you choose the highest-level folder to which movies are being written for the data collection session. For example, for EPU, specifying ../Images_Disc1/Data as the Directory to watch value and FoilHole_*_Data_*_*_*_*.mrc for the filter value when Search Recursively is enabled allows you to get around creating an exposure group for every grid square.
Gain Reference Path: Enter or select an absolute path to the gain reference, if the gain reference file is available. The gain reference should be in .mrc or .gain (for EER data) format.
Defect File Path: Enter or select an absolute path to the defect file, if the defect file is available. Note that zeros in the gain reference are also treated as defects and corrected in the same way.
To add a new Exposure Group, click 'New',
➡️Step 2: Click Enable (i.e., Save) for each Exposure Group added
You must click Enable for each newly added Exposure Group. This saves the parameters you entered. Once the exposure group is enabled, its parameters are locked in and cannot be changed unless the Session is cleared. For more information on clearing a session, please see: Live Jobs and Session-Level Functions
To speed up the configuration a new Live session, you can save all or some of the parameters of an existing session into a configuration profile and apply the profile to future sessions.
Configuration profiles can store:
Compute resource preferences
Exposure groups
Parameter sections such as microscope params
In any Live session that has been configured with all required parameters, you can choose to save a subset or all of that configuration as a profile.
➡️Navigate to the 'Configuration' section.
➡️Click the Profiles dropdown, then 'Save current configuration'.
➡️Choose a title, and one or more sections to save to that profile.
➡️Navigate to the 'Configuration' section.
➡️Click the Profiles button to see a list of saved profiles.
➡️Click View and Apply to preview. If a profile contains exposure groups, it will create a new exposure group in the session.
➡️Click Apply Profile to load the profile into the current Session.
As the session progresses, view diagnostic plots and metadata for each exposure. Start Manual Picking anytime, and optionally Reject or Reprocess individual exposures.
Once you have configured and saved all required parameters and exposure groups, the Session can be started anytime.
➡️Step 1: Start Session
➡️Click Start Session (from the Configuration Tab, or from the Header).
As exposures are captured, detected, and loaded into cryoSPARC Live, a visualization/thumbnail for each appears sequentially from left to right in the top bar. Near the bottom of each thumbnail, a blue progress bar indicates the status of pre-processing (motion correction, CTF estimation, picking, extraction, etc). Rejected exposures are indicated with a red “X”.
Exposures are processed in they order they are found, as follows: patch motion correction (global and local) > patch CTF estimation > particle picking > particle extraction. Exposures being processed are indicated with a flashing blue outline and a blue progress indicator labelled with the current processing stage (e.g., Motion Correction).
Once a session is started, you can see sessions statistics at a glance in the sidebar on the left. This includes the number of exposures found, processed, accepted, rejected, and the rates at which exposures are being found and processed.
➡️Step 2: View Individual Exposures information
Clicking on the Individual Exposure Tab will bring up diagnostic plots for the exposure that is currently selected in the Exposure Feed.
You can traverse the Exposure Feed by clicking on a thumbnail or using the navigation buttons at the top. The arrows allow for traversing through exposures including the ability to follow the latest exposure, or navigate to a specific exposure using its ID.
Clicking on the |<-- button will follow the latest incoming exposure.
On the right side of the Individual Exposure Tab is the Exposure Viewer. You can adjust the Low Pass Filter (LP Filter) slider (top right of the exposure canvas) as required. Buttons are available in the bottom left for zooming, panning and resetting the view.
If an exposure has failed processing for any reason, the corresponding traceback/error message will be displayed in the Individual Exposure Tab.
➡️Step 3: Reject individual exposures and/or reset failed exposures
If you wish to Reject an individual exposure, click on the dropdown menu above the Exposure Viewer and click 'Reject'. Alternatively, use the keyboard shortcut by pressing "R" on the keyboard. A red 'M' (for manually rejected) will appear on the thumbnail of any manually rejected exposure.
To un-reject an individual exposure, or to reset an exposure that has failed, click 'Unreject' or 'Reprocess exposure' from the same dropdown menu.
➡️Step 4: Modify Exposure Processing Priority
The priority in which exposures are processed by the preprocessing GPU worker(s) can be modified at any time. For more information, see the following section:
Live Jobs and Session-Level FunctionsThe Overview Tab contains time-series plots depicting the evolution of computed data attributes. Use these to optionally exclude groups of exposures based on various thresholds.
The Overview Tab contains a number of plots that are useful for assessing the quality of processed images. You can hover over any individual exposure (dot) on any graph to view a small thumbnail and other details of the exposure. Clicking on any dot will cause the selected exposure to be displayed in the exposure viewer and selected in the feed.
Use the threshold sliders or the input fields to optionally exclude groups of exposures that do not meet desired criteria. For example, you may wish to exclude exposures with a calculated CTF Fit (Å) value greater than 4Å from downstream processing in Live (i.e., further particle picking, inclusion of the corresponding particles in 2D classification and refinement). The exposures that are rejected in this way will still be available for later downstream processing in cryoSPARC or for export - they are not deleted.
You can apply, adjust and clear thresholds at any time during a Live Session. Particles from accepted micrographs will be used for downstream 2D and 3D processing. If a micrograph is rejected, its particles will be removed from consideration downstream and outputs that depend on particles such as the 3D refinement density map and FSC curves will be re-calculated at the next update.
➡️There are two ways to adjust a threshold on a particular attribute of micrographs. Graphically: Click on 'Select Thresholds' in the bar above the overview plots to set the plot mode to selection. Then, click and drag over a vertical selection of the plot for the attribute you wish to filter, to select a range that includes the data you wish to keep. When you release the mouse button after dragging, you will see your selection appear on the graph as a green highlight, and the threshold slider above the attribute will change to the new values you have selected. Manually: Use the threshold sliders or input boxes to enter the exact values you wish to use for filtering for a given attribute. You will see a highlighted region appear on the graph in green corresponding with the slider position.
➡️Once you see a region highlighted in green, you can finalize your selection by clicking 'Set Threshold' for that attribute. Otherwise you can click 'Cancel'. Once you set the threshold, all existing micrographs outside the set range will be threshold rejected as will any new micrographs that are processed.
In the exposure feed, rejected exposures will have a reject icon.
You can apply thresholds on multiple attributes, e.g., on both CTF Fit (Å) and on Defocus Range (Å). Exposures that are rejected for any reason will show as red dots in all graphs.
➡️To clear thresholds, click 'Clear' on any attribute and this will reset the exposures that were previously rejected. It is possible to both manually reject and threshold reject an exposure.
The Browse Tab contains various exposure statistics that can be plotted, compared and downloaded.
At any time during a session, you can view exposure statistics, filter values and download a .csv file containing all of the data.
➡️Navigate to the Browse Tab to view statistics and scatter plots comparing any two attributes.
Perform manual, blob or template-based particle picking from the Picking Tab as the session progresses, with the option to set new particle score thresholds and re-extract particles during preprocessing, which will be fed into downstream steps.
The Blob Picker is the default picker enabled in cryoSPARC Live for a new session. The cryoSPARC Live GPU Workers will perform blob-based picking and extraction on all incoming movies using the Minimum particle diameter (A), Maximum particle diameter (A) and Extraction box size (pix) that you provided on when configuring the Session, until and unless Blob Picker settings are adjusted or the active picker is changed.
➡️Step 1: View Blob Picks
Blob picks "B" are displayed in yellow in the exposure viewer. Picking statistics are displayed on the Blob Picker button and can also be viewed by clicking on the dropdown.
To hide or view blob picks, click on the Blob Picker button at the top of the Exposure Viewer.
By default, cryoSPARC Live displays pick locations as dots. Shift + Click on the Blob Picker button at the top of the Exposure Viewer to cycle through circular, square and dot pick markers. You can also adjust how picks are displayed using the expansion menu on the right of the picker buttons.
For a full list of keyboard shortcuts, click on the Main Menu in the top left corner of any Session.
➡️Step 2: Adjust Blob Picking Parameters
As more exposures are processed, you may wish to adjust Blob Picker parameters to obtain better picks.
You can adjust the Minimum particle diameter (A), Maximum particle diameter (A), blob shape (circular, elliptical, blob, or a combination of them), Lowpass filter to apply (A) and the Min. separation distance (diameters) at any time during a session.
➡️Adjust the value(s) as desired and then click Activate for All or Activate for Future in order to trigger re-picking and re-extraction based on your changes. Or, use Test Adjustments, explained next.
Activate for All: Applies the thresholds and. any changed parameters to all exposures (i.e., will cause all exposures to be re-processed with the changes)
Activate for Future: Applies the thresholds and any changed parameters to all exposures that have not yet been processed. Useful if you do not want your changes to trigger reprocessing.
➡️Step 3: Test Adjustments (Optional)
If you are not sure how your parameter changes might affect processing or if you would like to experiment, you can use 'Test Adjustments' which will cause only the active (currently selected) exposure to be re-processed with the new picking parameter changes, and re-extracted.
After clicking 'Test Adjustments' you may have to wait a few seconds or minutes until one of the cryoSPARC Live GPU Workers becomes available to pick up the test micrograph. Once this process is complete, the new pick locations will appear on the active micrograph. Exposures to which a Test parameter has been applied are indicated with a purple "T" on their respective thumbnails. You can apply 'Test Adjustments' on as many individual exposures as you like.
➡️Once satisfied with the new picker settings, click 'Activate for All' or 'Activate for Future' as desired. This will trigger re-picking and re-extraction. Unless one of the Activate buttons is clicked following Test, the exposure on which Test was run, will simply be excluded from any further processing (i.e., from particle extraction and steps downstream). To undo Test mode on a particular exposure, i.e., to reset it so that it can be included in further processing, click on the dropdown above the Exposure Viewer and click Reprocess exposure.
➡️Step 4: Filter and Extract Blob Picks
Along with picker parameters, you will need to adjust the Normalized Cross-Correlation (NCC) and Power Score thresholds, similar to Inspect Picks in cryoSPARC, to adjust and exclude poor quality picks.
➡️Adjust the thresholds or input new values as desired and then click Confirm Thresholds or Cancel if you wish to revert to the previous setting. As you drag the sliders, you will see picks appear and disappear in real time on the exposure viewer. Typically, you will want to adjust the NCC and power sliders until only "true" particles visually remain. For the Blob Picker, the power scores are often most useful. Removing low-power picks will exclude picks in empty patches of ice, smaller contaminants, and broken particles. Removing high-power picks will exclude carbon edges, ice crystals, gold particles, overlapping particles, etc.
➡️Click Extract for All or Extract for Future to trigger re-extraction. The extraction step will attempt to extract all particles falling within the new thresholds, and will remove particles that are too close to the edges of the exposures.
Note that if you make changes to 'Adjust Blob Picker' parameters and 'Filter and Extract Blob Picks' extraction thresholds, both 'Activate for All' and 'Extract for All' buttons will become available. Clicking either button will cause both new settings to be applied.
The Manual Picker can be engaged anytime during a Session. To pick particles, navigate to the Exposure Viewer, click the "arrow+" button in the bottom right and begin selecting particles by left-clicking. To remove manual picks, either right-click, or click the "arrow-" button and then left-click over the picks you wish to remove.
Manual picks are displayed in a table on the Manual Picker tab. Clicking on a table row will navigate to the exposure indicated.
➡️To extract Manual Picks, click Start Manual Extraction on the Manual Picker tab. These picks can be fed into the Template Picker (by generating templates, below), or simply exported.
The Template Picker can be engaged anytime after starting a Session. To do so, you can either choose to generate templates from available blob picks or manual picks in the session, or load in available templates from any existing cryoSPARC Project/Job.
➡️Step 1: Create or Load Templates
You can either generate templates using 2D Classification from Latest Exposures or Load Existing Templates.
➡️To generate templates, specify:
Classes (number of classes for 2D classification)
Exposures (particles from this many exposures will be included)
Classify (whether to classify the existing Blob Picks, Template Picks or Manual Picks)
➡️Click Generate Templates, which will start a 2D Classification job on the Auxiliary Lane that will classify the particles. Once complete, you will be able to select the desired classes below to use as templates for the Template Picker.
➡️To Load Templates that you already have available, enter the cryoSPARC Project ID and Job ID corresponding to the available particles/templates. These will be loaded and then they can be selected. If you have already started Streaming 2D Classification in cryoSPARC Live (eg using blob picks) you can enter the project ID and job ID of the streaming 2D class job to use the current templates to instantiate the template picker.
➡️Step 2: Select Templates
Select as many templates as desired by clicking on each class.
➡️Step 3: Adjust Template Picker
➡️Enter (or adjust) the Particle Diameter (A) (required). If desired, adjust the Lowpass filter to apply (A), Ang. sampling (degrees) and Min. separation dist (diameters). The particle diameter should be set equal or larger than the longest expected dimension of the particle.
You can Test Adjustments if desired, and then click Apply to All or Apply to Future to trigger template-based picking using your parameters.
➡️Step 4: Filter and Extract Templates
You will need to adjust the Normalized Cross-Correlation (NCC) and Power Score threshholds, similar to Inspect Picks in cryoSPARC, to adjust and exclude poor quality picks. See the similar section for Blob picking here.
➡️Adjust the thresholds or input new values as desired and then click Confirm Thresholds or Cancel if you wish to revert to the previous setting.
➡️Click Extract for All or Extract for Future to trigger re-extraction. The extraction step will attempt to extract all particles falling within the new thresholds, and will remove particles that are too close to the edges of the exposures.
2D classification is performed seamlessly while a session is in progress. Newly available particles can be classified into existing classes as they are extracted, or 2D classification can be re-started at any time.
You can start Streaming 2D Classification at any time once you have started the session. You may wish to start 2D Classification early on into a session to see more about the visual quality of the particles extracted thus far and make choices about tuning picking parameters or about the sample itself.
➡️Step 1: Start Streaming 2D Classification
➡️On the 2D Classification Tab, click the Gear icon and enter the number of Classes.
Alternatively, you can Build with Custom Parameters by clicking on the "hammer" button. This will create a new Streaming 2D Classification job in the cryoSPARC Project where the Live Session is housed.
Navigate to the cryoSPARC Project and find the new job (set to Building status). Enter any custom parameters you wish to change in the Job Builder. DO NOT launch the job from the regular cryoSPARC interface.
Return to the cryoSPARC Live interface > 2D Classification Tab and click Queue to launch the job.
➡️Click Start. The job will start and take in all the particles that have been extracted so far. The most recently extracted set of particles from each micrograph will be used. Therefore if a micrograph was originally picked with the blob picker, then re-picked with the template picker, the template picked particles will be used.
➡️Step 2: Select 2D Classes
Once the first 20 iterations of classification are complete, the job will display interactive 2D classes that can be selected. These displayed classes will update as new particles arrive.
➡️Click to select the desired class averages. Use the buttons above the class averages to sort and select, or right click on any class average to display a menu of actions. All particles falling within the selected class averages will be used for Ab-Initio Reconstruction and Streaming Refinement.
2D Classes will continue to update with new particles every few minutes, as new particles arrive.
The rate of updating depends on how fast new particles are coming and, and how long it takes to update classes. Once 10,000 new particles are seen, Streaming 2D classification will also go back and re-classify all existing particles into the updated classes, which can take several minutes. Since the 2D templates that are resolved will change only slowly as new particles are seen, the selection of 2D classes that are made will persist over updates of streaming 2D classification. However, it can be a good idea to return to the streaming 2D class tab periodically to check if any classes should be newly selected or unselected.
While waiting for new particles, the Streaming 2D Classification job will enter Waiting status.
➡️Step 3: Stop, Re-Start or Attempt Resume (Optional)
Stop
At any time during 2D Classification, you can also choose to Stop the job. This will kill the Streaming 2D Classification job. If you wish to start another Streaming 2D job from scratch, you will need to configure the number of classes and click Start.
Re-Start
After streaming 2D class has been stopped or killed, you can "force" Re-Start 2D Classification. This will start off 2D classification from scratch without reference to previous results.
Attempt Resume
Alternatively, if the number of classes has not changed, you can attempt to resume streaming 2D classification from the previous results that are currently displayed. This will work if the previous 2D class job was killed or stopped after writing out some results, and if the particle box size, pixel size, and number of classes has not changed. If resuming fails, 2D class will start from scratch.
On the Ab-Initio Tab, generate an initial model from available particles or load an initial model to use for refinement.
Once you have run Streaming 2D Classification and selected some classes, the particles falling into those class averages are available for Ab-Initio Reconstruction.
➡️Step 1: Configure Ab-Initio Reconstruction or Load Volume
➡️Click on the Gear icon to configure Ab-Initio Reconstruction.
➡️Enter the Number of Classes if you wish to resolve more than 1 class, Symmetry (optional, we recommend using the default C1 as it is not necessary/recommended to enforce symmetry during ab-initio reconstruction), and the number of Particles you wish to use for Ab-Initio (also optional, default 100,000). Note that if you use multiple classes at this stage, you can select one as the initial model for streaming 3D refinement, but all particles will be used for refinement. Streaming heterogeneous refinement is not currently available in Live.
Alternatively, you can Build with Custom Parameters by clicking on the "hammer" button. This will create a new Ab-Initio Reconstruction job in the cryoSPARC Project where the Live Session is housed.
Navigate to the cryoSPARC Project and find the new job (set to Building status). Enter any custom parameters you wish to change in the Job Builder. DO NOT launch the job from the regular cryoSPARC interface.
Return to the cryoSPARC Live interface > Ab-Initio Tab and click Queue to launch the job.
➡️Click Queue to launch the job. Volume slices will display in the cryoSPARC Live interface and the 3D volume will be available to view in the Volume Viewer as the reconstruction progresses.
You can also view the progress of any job in Live by clicking on the job number in the sidebar or on the relevant tab, to expand the streamlog view. For example, we clicked into the Ab-Initio job from the sidebar:
Alternatively, you can load an existing volume by entering the Project UID and Job UID corresponding to the location of the initial model in your cryoSPARC instance, and then click Load.
The volume viewer can be rotated and zoomed by dragging with the left mouse button or scrolling respectively. Holding shift and dragging with the left mouse will pan the view.
To download the volume, click 'Download map' on the bottom right hand corner of the Volume Viewer. Alternatively, you can find the volume in the cryoSPARC Project where the Live Session is housed.
➡️Step 2: Select a Volume for Refinement
➡️Select one volume from Ab-Initio to use for Streaming Refinement. Click on the volume slices image to select the class.
Refinement in cryoSPARC Live operates in a streaming manner, taking into account new particles that become available after preprocessing.
Once the Ab-Initio job has been completed (or a volume has been loaded) and one volume has been selected in the Ab-Initio Tab, you can start a Streaming Refinement job.
➡️Step 1: Configure Streaming Refinement
➡️Navigate to the Refinement Tab. Click on the Gear icon to configure refinement parameters.
➡️Specify the Symmetry for refinement if known/required. Initial volumes will be automatically aligned to the symmetry axes if symmetry is specified.
Alternatively, you can Build with Custom Parameters by clicking on the "hammer" button. This will create a new Streaming Refinement job in the cryoSPARC Project where the Live Session is housed.
Navigate to the cryoSPARC Project and find the new job (set to Building status). Enter any custom parameters you wish to change in the Job Builder. DO NOT launch the job from the regular cryoSPARC interface.
Return to the cryoSPARC Live interface > Refinement Tab and click Queue to launch the job.
➡️Click Queue to launch the job. Various plots will display in the cryoSPARC Live interface and the refined 3D volume will be available to view in the Volume Viewer as the reconstruction progresses.
To download the volume, click 'Download map' on the bottom right hand corner of the Volume Viewer. Alternatively, you can find the volume in the cryoSPARC Project where the Live Session is housed.
As new particles are picked, extracted and classified, they will be picked up by the Streaming Refinement job so that the refinement volume will update over time in the Volume Viewer.
Streaming 3D refinement will update the 3D map, FSC resolution estimate, and diagnostic plots repeatedly as new images are collected. When sufficient new particles are available that pass the previous filtering stages and 2D class selection, refinement will backtrack to a lower resolution map, and re-perform refinement of all particles until convergence. The rate of updating will depend on the number of particles already collected and the current resolution.
While waiting for new particles, the Streaming Refinement job will enter Waiting status.
What to do once you have finished processing in Live.
Once you have completed processing in cryoSPARC Live for the day/dataset, you can pause the session to free up compute resources by clicking Pause Session in the header. Pausing will retain all Session configuration and parameters and all results. Running preprocessing workers will be allowed to finish their current exposure and then gracefully exit. Running 2D or 3D streaming jobs will be killed and marked as completed so that their latest results become available to use for further processing in cryoSPARC or resuming in cryoSPARC Live.
If you have finished with a Session and do not plan to return to it, you may wish to Mark as Complete from the header. This will set the session to completed status and separate it from running and paused sessions in the browse sessions page. A completed session can still be started again and no results are lost.
For more information on Session-Level Functions, please see: Live Jobs and Session-Level Functions
Interacting with outputs and performing advanced processing.
For any Session, you can navigate into the cryoSPARC Project where the Live Session is housed and use outputs from Streaming 2D Classification or Streaming Refinement (e.g., particles, volumes, templates, etc), directly for further processing. Using particles from Streaming 2D classification will allow using the 2D alignments and assignments downstream (e.g., for select 2D or re-centering extraction). Using the particles from Streaming Refinement will allow using the 3D alignments downstream (e.g., for reconstruction, 3D Variability, CTF refinement, local refinement, etc).
Note that if you create a new job in cryoSPARC and use the outputs of a Live streaming job that is still running or waiting as input, the new job will remain Queued until the streaming job enters completed status. You can force this to happen by stopping the streaming job from the Live interface. The streaming job will be killed and marked as completed. You can then start streaming processing again from Live and this will create a new streaming job.
At any time during a Session, you can navigate to the Details Tab and click Export Exposures or Export Particles to cause the available exposures and/or particles to be made available in cryoSPARC.
No data is copied in this process. Rather, you will see a new job or jobs appear in the workspace in cryoSPARC corresponding to your Live session.
These jobs will have outputs pointing to the data from cryoSPARC Live, and can be used as any other cryoSPARC job output for connecting to new jobs for more processing.
CryoSPARC Live Exposure Export Job: This job will run in the cryoSPARC Project where the Live Session is housed and will separately output accepted, rejected, and manually rejected exposures for each Exposure Group in the Session.
CryoSPARC Live Particle Export Job: This job will export all particles from the Live session that pass the threshold tests in picking and have been extracted. These are the same particles that would be seen by streaming 2D classification. These particles will not come with alignment or class assignment information, but do contain location, pick_stats, blob and ctf outputs.
You can continue processing in cryoSPARC from the outputs of the above jobs.
The Details Tab contains session history and a handy notes features.
The Details Tab contains information about the user, session directory, session-level functions and start/pause history for the session along with a notes feature that includes checklists.
