Comment on page
About CryoSPARC Live
From microscope to structure, in minutes. The fastest, streamlined path to high-resolution structures, during data collection or afterwards.
CryoSPARC Live is a software platform that enables:
- Real-time cryo-EM data quality assessment
- Decision making based on 2D and 3D results during live data collection
- An expedited, streamlined workflow for processing previously collected data
- Direct seamless interoperation with CryoSPARC for advanced processing
CryoSPARC Live is built to enable experimentation with parameters on the fly, while the software efficiently manages the reprocessing that is necessary in order to test or effect parameter changes, and while maintaining the overall progress of the Live session.
Processing EMPIAR-10288 in cryoSPARC Live.
CryoSPARC Live is built for:
- Data collection facilities, cryo-EM cores and and microscope operators, who want to make the most of microscope time with real-time data quality assessment and collection;
- Facilities and service providers, who want to provide their users with expedited information about sample quality and/or delivery of 3D maps; and
- Individual users, who wish to gain insights about data quality by performing 3D reconstructions in real-time with data collection, or on previously collected data in a seamless and "first cut" manner.
CryoSPARC Live can be used by microscope operators during a collection session. We recommend the exposures are written to fast disks as I/O can become a bottleneck.
CryoSPARC Live can be engaged to read files that are being uploaded to the user, e.g., via AWS or other similar service. Live will read in new exposures as they are found.
Finally, we recommend using the Live workflow for expedited preprocessing (see below) and a first-cut look at data quality in 2D and 3D, for all datasets including those already collected.
CryoSPARC Live watches specified directories for new files and processes them as they become available. CryoSPARC Live preprocessing includes four steps: motion correction, CTF estimation, particle picking and extraction. CryoSPARC Live can sustain a throughput of 450 or more exposures per hour, per GPU, for K3 data. On a 4-GPU machine, that can scale to 1800+ exposures per hour! For K2 or Falcon data, performance can be even higher, upwards of 650 exposures per hour per GPU.
Particles from preprocessing are seamlessly transitioned into 2D classification and 3D reconstruction, and preprocessed exposures can also be exported for further processing in CryoSPARC or other software.
For extensive benchmarks and throughput statistics, please see:
- Microscope/camera and job parameters: Any of these can be adjusted over the course of a session, and combinations of parameters can be saved as "Configuration Profiles" for use in future sessions. For example, microscope parameters such as the
pixel size,spherical aberrationandaccelerating voltageare likely to be consistent for a given instrument and can be applied quickly using Configuration Profiles at the start of a new Session. If working on multiple samples of the same protein or complex, it may be useful to save picking and extraction parameters to save time. - Exposure groups: It is possible to add, remove, and ignore exposures from one or more Exposure Groups (collections of exposures with the same optical parameters). This makes it possible to process subsets of a larger dataset and make comparisons.
Learn more about Configuration Profiles in the UI Overview:
3_overview_threshold_1.png?alt=media&token=b2c96e17-bae6-47ff-86fb-93abbc98b6ce)
Users can review several useful statistics (e.g., CTF Fit, Defocus, Total Motion, etc) across a dataset and apply thresholds to automatically reject incoming and already-processed exposures that do not fall within the desired range for one or more parameters. Exposures can also be rejected manually. Rejected exposures are excluded from further processing unless thresholds are changed, which can be done at any time during a Live session. On export, exposures are split into different output groups (e.g., rejected and accepted) for further advanced processing outside of the Live workflow.
It is possible to have finalized a picking strategy within the first few hundred exposures, which can then be left to run on the remaining exposures as they come in. Blob-based picking is active by default at the start of any Live session and manual picking can be engaged at any time. Blob/manual picks can be curated and used to generate templates for template-based picking, or available templates can be loaded directly into the Session. Pick scores (NCC and Power Threshold) can be used to include or exclude picks. These and particle extraction parameters can be tested on a single or few exposures before being applied to the entire dataset, and can be updated as many times as necessary. Updated particle locations will be fed automatically into later stages of the pipeline (e.g., Streaming 2D Classification).
Real time streaming 2D classification enables assessing sample quality, preferred orientation issues and presence/absence of the expected target and/or ligands, as well as large conformational variability.
2D Classification in CryoSPARC Live picks up newly extracted particles from incoming exposures and automatically updates 2D classes every few minutes using a new streaming method. This means that after starting a Streaming 2D Classification job, class averages will update automatically as new particles become available from upstream. Class averages can be selected as soon as the initial classification is complete, and these selections will be retained, enabling Streaming 3D Refinement to take in new particles from the selected classes and update the reconstruction in real time.
Early feedback in 2D (and 3D, see below) can confirm whether a collection should continue, or whether upstream steps such as sample preparation may require improvement. Within a few hours of starting a Live session, it is possible to make a "go/no go" decision about the sample and assess issues that may result in a poor 3D reconstruction. Thus, it is possible to save on microscope time, or, to at least inform the user ahead of time about the quality of result they may be able to expect.
Learn more about CryoSPARC Live Jobs and Session-Level Functions:
The ability to generate a refinement during data collection is important as a diagnostic during collection, as a first-cut structure from which to continue further processing, and in many cases is comparable to the highest resolution structure(s) that can be generated after extensive advanced processing in CryoSPARC. Streaming 3D Refinement in CryoSPARC Live also will pick up newly available particles from Streaming 2D Classification so that over the course of a collection, the 3D structure updates every few minutes.
For a detailed walkthrough of setting up your own Live session, please see:
Over the course of a CryoSPARC Live Session, it is possible to adjust the compute resources dedicated to processing. The
Number of Preprocessing GPU Workers can be increased or decreased during a session in order to free up compute resources or add more parallelization capacity for preprocessing. Additionally, the compute lanes being used can be adjusted.
Learn more about GPU requirements for Live:
CryoSPARC Live is tightly integrated with CryoSPARC. Each CryoSPARC Live Session is housed within a CryoSPARC Project, so the results of live processing can always be used seamlessly for further advanced processing in CryoSPARC as well as for export.
It is common to take the final map(s), exposures and particle stack(s) from CryoSPARC Live and hand them off directly to e.g., users of a microscope facility or lab, or to use the motion-corrected, CTF-estimated exposures directly for advanced processing without redoing preprocessing steps.
CryoSPARC Live Sessions and jobs can also be controlled via the command line:
"CryoSPARC Live enables the Pacific Northwest Cryo-EM Center to keep up with data coming from all 5 of our microscopes, a data volume that has at times exceeded 15TB (30,000+ images) per day. We can monitor not only standard image pre-processing metrics, but also directly observe improvements to 3D maps concurrent with data collection! In many cases we’ve stopped data collection with a 2.2-3.5Å map already in hand. We can also quickly identify pathologies such as poor orientational distributions, or tune imaging conditions to improve alignment of difficult particles."
- Craig Yoshioka, Center Co-Director, Pacific Northwest Center for Cryo-EM (PNCC)
In our facility, CryoSPARC Live has completely revolutionised the way we collect and process cryoEM data. Initially, we commissioned CryoSPARC Live with the primary goal of feedback on micrograph quality through on-the-fly motion correction and CTF estimation. But it has turned out to be so much more than we hoped for. We now use it for all initial processing of data coming through our facility. Our latest GPU workstation was specifically specced out for rapid CryoSPARC processing as it has become our primary in-house processing tool. Some of the aspects we love about CryoSPARC Live include: the GUI is intuitive, easy to use, and easy to learn for new users; processing is extremely fast; parameter changes are propagated throughout the workflow in a streaming fashion; easy linkages with ‘regular’ CryoSPARC make continued processing simple; picking tools are easy to use and work for almost all targets, and micrograph curation is powerful and easy to apply. CryoSPARC has become such an essential part of our data collection and analysis workflow; we could not imagine working without it. Highly valuable microscope time is now used much more efficiently and our microscope users are very satisfied.
- Simon HJ Brown, PhD, Customer Solutions Expert, Cryo Electron Microscopy - Molecular Horizons, University of Wollongong
"CryoSPARC live may literally make your jaw drop. The speed of processing combined with the capability to make adjustments in real time are remarkable. It allows you to make go/no go decisions for each experiment using 2D and 3D results generated almost as fast as the images can be collected. Perhaps most impressive is the ability to go back and change almost any parameter on the fly, and it will immediately reprocess the images from that step, and key metrics such as motion and CTF fit can be examined in interactive plots where ranges can be set and immediately applied for image curation. Together these features take a large step toward the future in reducing both the time and resources that are needed to achieve a successful outcome for each data collection."
- Jeff Speir, Director of Operations, NanoImaging Services Inc. (NIS)
"My group and I have been using CryoSPARC Live beta for a little over a year with great successes. We currently only run CryoSPARC Live for our in house projects and it has been of great help. It is easy to set up, very easy to use and very fast. There is no need to create scripts or any slightly complicated task. On most projects with very limited human interaction and no knowledge of the target of interest we can reach high resolution overnight before the run on the microscope is finished. The CryoSPARC-live features allow us to monitor the quality of the acquisition and make modifications in real time if necessary, whether it is a sample problem (e.g. ice thickness) or an issue with the alignment of the microscope. We are looking forward to extend the use to the core facility users allowing them to get real time feedback on their runs."
- Eric Hanssen, Head, Advanced Microscopy Facility and Associate Professor, University of Melbourne
"CryoSPARC Live is a wonderful tool that not only gets researchers excited about their cryoEM experiments but lets microscope operators know they are acquiring high-quality processable data. With on-the-fly feedback users and staff are able to engage with each other to identify bottlenecks and modify data collection strategies in real-time to conduct optimized experiments for a sample. 3D feedback is critical because it ensures we are able to collect a full dataset of their macromolecule of interest, thereby allowing our users to accelerate their biomedical research."
- Edward Eng, Manager, New York Structural Biology Center (NYSBC)
"Having used CryoSPARC Live at both New York Structural Biology Center (USA) and The Hospital for Sick Children (Canada), it has become an invaluable tool in real time assessment of samples for a frequent user like me. By having the movie frames aligned, CTF estimated, particles picked, 2D classification and ab initio done on the fly allows me to quickly judge which grid/sample is worth collecting on. As all the statistics can also be neatly presented as an overview, it is straightforward too to pick out trends and exceptions in the data: For instance I have noticed cases where a bunch of micrographs had poor estimated resolution, only to pinpoint the problem to a single grid square, allowing me to avoid similar grid squares subsequently. The final big advantage of CryoSPARC Live is that the resulting data can easily be passed onto the conventional CryoSPARC pipeline for further processing – saving both time and energy!"
- Yong Zi Tan, Postdoctoral Fellow, The Hospital for Sick Children
CryoSPARC Live was first released as a private beta in May 2019. Based on extensive beta testing at dozens of facilities and labs globally, we have incorporated feedback and worked to improve the workflow, with several iterations already released. CryoSPARC Live will continue to evolve with advancements in data collection, user feedback and automation of the cryo-EM workflow.
In 2022, we collaborated with Thermo Fisher Scientific Inc. to make available Embedded CryoSPARC Live, a version of CryoSPARC Live that is designed to seamlessly integrated with Thermo ScientificTM cryo-transmission electron microscope systems. To learn more about the collaboration, please visit: https://cryosparc.com/embedded-cryosparc-live