Interactive Jobs
Last updated
Last updated
Interactive jobs represent the points in the CryoSPARC data processing pipeline where user interaction and intervention occur. These are either required interactions to build outputs necessary in sequential steps or optional interventions to operate on data manually in order to secure better results downstream.
Interactive jobs always run on the master node and when queued will run until reaching a stage of necessary intervention, which is represented by the “waiting” status. At this point the job will gain a new “Interactive” tab inside the job dialog. This tab will be the default view when opening an interactive job with the “waiting” status, and will be switched over to automatically if the dialog is already open when it becomes available.
All major interactive jobs share the same two column layout. The overview column is on the lefthand side and the inspection column is on the righthand side. Certain panels within these columns are foundational systems for interacting with these jobs and are present across interactive job types.
The exposure plot is a generalized plot with two view options (scatter or histogram) that shows all of the exposures present in the job with configurable axes. This plot is designed to allow for quickly visualizing patterns and trends in your data that you can further act on with the other tools available in the job.
The plot type is configurable using the “type” selection menu in the top left of the panel above the plot. Both plot types have a download button located in the top right of the panel. This button allows you to download an image of the plot to share or keep for your records.
The scatter plot option shows all exposures across two configurable axes. Each point in the plot can be hovered to reveal an exposure preview that opens offset on the righthand side of the plot. These previews show a number of pertinent fields from the exposure and an exposure preview at the top if available. Each point in the scatterplot is also an actionable button, and will select the corresponding exposure, scroll to it in the micrographs table, and set it in the micrograph viewer when clicked.
All interactive jobs feature a configurable exposure plot with histogram option to view the distribution of a particular attribute. Exposure curation supports a 'split histogram' chart which displays distributions of accepted and rejected exposures as separate traces.
The micrographs table shows all micrographs present in the job with a number of sortable columns for pertinent fields (eg. DF Average, Relative Ice Thickness, CTF Fit).
The table header has three actionable items. The scroll target button allows you to quickly scroll back to a selected table row in case you have scrolled the table to another location. The filter menu allows you to select subsets of micrographs based on status (eg. accepted, rejected, exposures with picks, exposures without picks). The final item is a CSV download button that allows you to download a CSV of the entire table data with any sorting or filter options applied.
The micrograph viewer will be immediately familiar to anyone that has used CryoSPARC Live, and has nearly identical functionality. It consists of a viewing plane with a number of overlayed navigation and action palettes, as well as some data modules with contextual information.
The micrograph viewer control bar contains a variety of options for displaying and controlling the data visualization options on the micrograph. These include toggling particle diameter and box size on or off, as well as adjusting their respective sizes; also available are options for controlling the contrast intensity override and changing the pick or box colour. Advanced controls are contained in the triple dot overflow menu button.
The micrograph viewer will initialize with the micrograph fit to the viewer window. Your cursor will initially be set to drag mode which will allow you to click and hold on the micrograph to drag it around the viewer. Scrolling in the viewer will zoom into the micrograph at the location of your cursor.
Scale Bar: The top right corner of the viewer houses the scale bar. This bar shows a dynamic scale that changes as you zoom in or out, and will switch colours to maintain visibility over your micrograph.
Lowpass Filter: This slider allows you to easily change the lowpass filter (in angstroms) to gain better contrast of the micrograph.
Download Button: This button allows you to download an image of the micrograph with any overlaid information present. This includes picks and boxes set over particles.
Scale Palette: The scale palette is located in the bottom left corner and allows you to quickly refit the micrograph to the viewer display area. It also gives multiple preset scale option buttons to quickly jump to a specific zoom level without scrolling.
Navigation Palette: The navigation palette is located in the bottom right corner and is split into two modules. One controls scroll behaviour, and allows you to set scrolling to either zoom or pan the viewer. The other controls the click behaviour and allows you to switch between the measurement tool and dragging tool. The measurement tool allows you to click and hold on the micrograph to drag out a circular measurement ruler. This ruler will show a red circle with a tooltip indicating the circle’s diameter in both pixels and angstroms.
Action Palette: This palette is only available in some jobs and allows actions to be taken on the micrograph for data processing purposes (eg. for the manual picker you can change between click options for adding and removing manual picks).
The manual picker job allows you to pick particles interactively and extract particles for generating templates.
The control bar at the top of the job allows you to see a quick statistical overview of the running job, including the total number of exposures, total exposures with picks, total exposures without picks, and the ratio between the set particle diameter and box size. The “Done Picking | Extract Particles” button sits at the far right side of the control bar and will complete the job and extract all particles when clicked.
The main content area breaks down into two panels, the overview panel, and the inspection panel.
The manual picker overview panel contains the standard configurable scatterplot at the top and the micrograph table below. These are covered in more detail in the previous section.
The inspection panel includes a micrograph viewer with various overlayed controls, and a control bar above the viewer that allows you to configure picking and viewing parameters.
The main bar includes stacked particle diameter and box size configuration inputs. Each of these allows you to toggle on or off the picks that appear on the exposure (this is a visual toggle only and will not remove picks), and an input and slider that allow you to adjust the size of the picks. The particle diameter is used solely for visualization purposes, while the box size is used for extraction. If the provided maximum value on the slider is not sufficient for your required selections, you can adjust the number in the input to any value and the slider will automatically update its bounds to accommodate that number.
The control bar also includes an actions button in its top right corner that will open a menu with additional parameters when clicked. Included are the “Contrast intensity override” and options to adjust colour of the picks or boxes that appear on the micrograph, as well as a toggle to hide or display the scale bar in the exposure viewer.
The inspect particle picks job allows you to inspect and modify picks using various thresholds following auto-picking.
The inspect picks overview panel contains the standard configurable scatterplot at the top and the micrograph table at the bottom. These are covered in more detail in the previous section.
This job additionally contains a Power Histogram panel which includes an image of the power histogram with Power Score and NCC Score input sliders colocated along its relevant axes for clarity and ease of visualization. The filament histograms (Local Curvature and Filament Sinuosity) also include colocated input sliders along their relevant axes.
Along with configuring a particle diameter to view picks, you can set a box size and choose to overlay it within the exposure viewer. This functionality is virtually identical to that of the manual picker, except that the selections are not used for any outputs, but solely for visualization purposes.
As of CryoSPARC v4.6+, Inspect Particle Picks has a mode to cluster particles based on their particle picking scores (NCC and power score) and automatically select a cluster that will ideally contain true particle picks. This feature allows Inspect Particle Picks to be run fully automatically, without user input.
The auto clustering mode is designed to work with particle picks that were picked using denoised micrographs from the Job: Micrograph Denoiser (BETA). It operates by first separating particles into five defocus bins spanning the defocus range of micrographs in the dataset. Within each bin, clustering is performed on the pick stats (NCC and power score) and the cluster with the average power score closest to the "Target power score" parameter (default 50) is selected as the cluster to keep. The selected clusters from the five defocus bins are combined to produce the final output.
To enable the Auto-Clustering mode, turn on the Auto Cluster parameter. It may be necessary to adjust the target power score parameter for certain datasets depending on e.g. the particle size. The target power score settings should be re-usable across multiple datasets of the same or similar particle.
Manually curate exposures enables a user to visually inspect and curate a set of micrographs or movies using either population-level statistics and adjustable thresholds, or individual inspection of diagnostic plots.
The overall goal of this interactive job is to aid in cleaning up your data so that low-quality micrographs, and their associated particles do not detract from achieving the highest possible resolution refined structures.
After pre-processing: After Patch Motion Correction and Patch CTF estimation, you can use Manually Curate Exposures to help you identify and exclude micrographs with sub-optimal characteristics, e.g., broken or thick ice, carbon edges, too much motion, etc.
After particle picking and local motion correction: Once you have picked particles and estimated location motion trajectories, you can use this information in Manually Curate Exposures to identify and exclude micrographs associated with those picks whose local motion trajectories deviate from the norm for your dataset. In this case, the job will output a filtered set of micrographs as well as particles.
After using the Junk Detector: If micrographs have junk annotations, the labeled areas will be displayed and colored by junk type. Micrographs can be filtered by the percentage of their total area that is covered by junk of various categories.
The control bar at the top of the job gives a quick overview of relevant statistics on the lefthand side, and houses the “Done” button for completing the job on the righthand side.
The main panels are arranged similarly the Manual Picker and Inspect Picks jobs, with the overview panel on the lefthand side of the main content area, and the action panel on the righthand side.
The same functionality exists in the overview panel as in the aforementioned jobs, with a configurable scatterplot at the top and a micrographs table of all available exposures below (in this case thumbnail previews and metadata shown when hovering plot points are loaded from Motion Correction). Exposure Curation also supports a separate cards view that can be used to quickly examine and manually reject/accept individual micrographs.
The exposure card view is available by clicking the card view button on the control bar above the Micrographs Table. This will transition the table to a grid of exposure image thumbnails.
The grid automatically fills each row with as many exposure cards as possible while maintaining aspect ratio and maximum resolution.
From the top left and clockwise, each exposure card shows the exposure’s index, defocus average in micrometers, the rejection status if applicable (an “M” represents manual rejection and a lack of one represents threshold rejection), and a reject/accept button visible on hover (clicking this button will toggle the exposure’s rejection status). The index and defocus average tags can be hidden using the “Details” toggle on the control bar.
Clicking on an exposure card will select it and switch the action panel to the Micrograph tab (revealing the Exposure Viewer). Once an exposure has been selected the keyboard can be used to navigate between exposures using the left and right arrow keys. Pressing the “R” key will toggle the exposure’s status between accepted and manually rejected. Hovering over an exposure card will reveal the exposure preview rendered directly beside the grid with a variety of pertinent information.
The action panel is configured with a main action bar at the top, housing tabs for changing the panel view, as well as navigation arrows for moving between individual exposures, and a rejection button for manually rejecting a selected exposure.
“Thresholds” is the default panel view and allows you to filter out exposures by dragging a selection box over an assortment of exposures points on any individual exposure plot (or by adjusting the slider or inputs above the plot). Clicking the “Select Thresholds” confirmation button will filter your exposures by rejecting any outside of these bounds. The threshold can be cleared by simply clicking the “Clear thresholds” button above the individual plot and controls.
Threshold selections are additive, and any exposures that do not fit within the combined selection will be rejected. Rejected exposures will show up as red points on all scatterplots (including the overview panel scatterplot) and will be differentiated in the table with red “Index” cells.
The other tabs in the action panel are “Individual” and “Micrograph”. In order to view data in these tabs you must first select an exposure, either through a scatterplot or the micrographs table.
Individual: When an exposure is selected the “Individual” tab will be populated with a variety of information about that exposure, including its 2D CTF, Global Motion Trajectory, Local Motion Trajectory, and 1D CTF. Each of these detail panels within this view can be expanded by clicking the image, and the image can be downloaded by clicking the download button in the bottom right corner of each panel.
Micrograph: The “Micrograph” tab houses an exposure viewer with functionality for zooming, panning, measuring, adjusting the lowpass filter, and downloading the image.
When an exposure has been selected, the rejection button in the top control bar can be clicked to individually reject that exposure.
Once thresholding and curation is complete, the “Done” button in the job control bar can be clicked to complete the job and output the relevant selections of exposures.
In some cases it may be preferable to curate exposures based on a set of pre-defined thresholds. If one or more parameters within the Auto Thresholds section is set (numerical value range in the form of min,max), upon launching the job will skip the interactive process, apply the threshold automatically and generate relevant outputs. If an auto-threshold parameter is set but that attribute does not exist in the exposure input, the threshold will be skipped.
This job allows you to interactively select 2D classes from the output of a 2D Classification job. If particles (from the same 2D Classification job) are included in the inputs, then particles will also be partitioned based on their assignments to the selected class averages.
The Select 2D Classes job most commonly follows any 2D Classification job in which there are one or more "junk" classes. Removing "junk" particles from the particle dataset is an important step in particle curation, and will help increase the quality of 3D reconstructions and refinements. Note that Select 2D Classes can also be used on the output for any job that creates templates (e.g. on the templates generated in a Create Templates job).
At the top right of the panel the number of currently selected particles as well as the total number of particles in the dataset are shown.
Each desired class can be selected by clicking on it. To improve the quality of your particle dataset, avoid selecting classes that contain only a partial particle, two or more particles, or a non-particle junk image (e.g. ice crystals). You can use both the number of particles and the provided class resolution score to identify good classes of particles. There are several ways to sort and filter the classes in ascending or descending order, shown along the top of the panel:
Filter : Filter the classes by selected or unselected.
# of particles: Sort by the total number of particles in each class.
Resolution: Sort by the relative resolution of all particles in the class (Å).
ECA: Sort by the number of Effective Classes Assigned (ECA).
In addition, for dealing with large numbers of classes, you may want to select all classes that meet a given threshold in particle count, resolution, or ECA value. To do this, first choose a class that has the desired threshold value. Next, right click on the chosen class or click the triple dot overflow menu button to reveal a context menu with options to select all classes that meet or fail this threshold.
The "Select All", "Select None", and "Invert Selection" buttons along the top of the panel can be used to quickly select all classes, clear all selections, and invert the current selection, respectively. These can be useful for dealing with a large set of class averages.
Filtering the classes by "Selected" or "Unselected" can be helpful to group all of these classes together in order to inspect them collectively and allow sorting on only these subsets.
