Creating and Running Jobs

Working with the Job Builder, Job Cart and Job Quick Actions.

Jobs are the atomic processing unit within CryoSPARC. You progress through each stage of the cryo-EM data processing pipeline by building and connecting jobs together. Each job contains a set of inputs and parameters, and generates outputs to feed into other jobs.

Data processing workflows are often iterative and non-linear, and CryoSPARC jobs are designed to fit well within these workflows.

Job Builder: New Job

The job builder is the primary method of creating new jobs in CryoSPARC. It is available through the ‘Builder’ tab on the sidebar.

You can activate the job builder through the spotlight by typing command + k and typing “Job builder” or “New job”.

When the builder opens the search bar is automatically focused, making it easy to filter the list for a job you would like to build. While the search bar is in focus, you can use the up and down arrow keys to navigate through the list or skip to the top or bottom of the list with the shortcut command + up/down.

The new job list is organized into sections based on job category, and individual jobs are marked with helpful tags:

• New: this job has recently been introduced or upgraded

• Beta: this job is actively in development and may not be as stable as others

• Import: this job allows you to bring external data into CryoSPARC

• Utility: this job performs a useful action to augment or transform the inputs

• Interactive: this job spawns an process you can interact with to view and filter data

• GPU: this job is single GPU accelerated

• Multi-GPU: this job is GPU accelerated and can run on one or more GPUs

• Live: this job is used in CryoSPARC Live Sessions

You can further refine the list of jobs presented by expanding the filter options and selecting tags.

The list of jobs reduced to only those which can run on multiple GPUs

To view legacy jobs within the builder, expand the filter and toggle the ‘Show legacy jobs’ option. Keep in mind these jobs may not be stable or available in future versions of CryoSPARC.

Legacy jobs in CryoSPARC v4.0.0

Job Cart

A new method of creating jobs in CryoSPARC v4 is through the job cart. Rather than choose from a list of jobs, the cart allows you to compile a set of job outputs that can be used as inputs for another job. The job cart automatically filters the list of new jobs based on their input requirements.

Job cart outputs can be populated from the job card outputs view, accessible in two ways. The first is from the job card header using the outputs action button. Clicking this will toggle the card view, hiding any images and information in the main area, and showing instead a list of the job’s outputs.

A list of job cards with the outputs mode active. All output groups will be listed alongside their item count. Clicking any group will add it to the cart.

The other way to activate this view is by holding down the shift key on your keyboard. This will toggle all job cards to the outputs view, and allow viewing and selecting these outputs as long as the key is held down.

Clicking on these output options will add them to the job cart. From here you have options to clear the selected outputs, modify the number of outputs of a type being used, and view jobs which have inputs matching the selected outputs. For each output that is added to the cart, a preview image and count will be displayed.

Similarly to the job list on the ‘Builder’ tab, you can use the arrow keys to navigate the list and press enter to create the job. Selecting a new job from the job cart will automatically create that job and connect all of its inputs with the available outputs.

With an output group added to the cart, the list of applicable jobs is automatically filtered. You can further filter this list via the search input and hover over any job to view additional details. Once you click to select or press enter, the job will be created and all groups within the cart will automatically be connected to the new job.

As projects are standalone containers of jobs, you can only add output groups from the same project into the cart. You must also be in a workspace of the relevant project to create a job from the cart.

Job Quick Actions

In addition to the builder and job cart, you can quickly create jobs from a list of common actions available in the context menu of a job card. For example, a common action after running a 2D Classification job is to select certain classes via the Select 2D Classes interactive job. With job quick actions, you can simply right click on the job card and select ‘Queue Select 2D Classes’ to build and queue the interactive job without any manual intervention.

Right-clicking on a 2D Classification job presents two quick actions. Clicking ‘Queue Select 2D Classes’ will automatically create and queue this interactive job.

Quick actions are available for almost all job types in CryoSPARC. Most created jobs will enter building status and wait to be queued manually, while interactive jobs will automatically queue and run until they enter the waiting status (where user intervention is required).

Some jobs such as Ab-initio Reconstruction contain quick actions that allow for multiple jobs to be created, one for each class:

Right-clicking a multi-class Ab-initio Reconstruction presents an option to build a refinement job for each class that was reconstructed.

Building a Job

When you create a new job, the “Builder” sidebar tab displays the building (or editing) view of a job. Here, you can connect inputs, configure the job’s parameters and queue it to be run.

Jobs active in the builder tab are sticky, meaning that you can continue to select and view other jobs while being able to add inputs and edit parameters of the building job. This allows you to view another job and drag and drop the output groups into the input groups of the building job.

To activate the job builder on any building job, press ‘b’ while the card is selected or click on the ‘Build’ button at the bottom of the card.

To stop building a particular job, press ‘b’ or click on the deselect button in the sidebar.

The job builder sidebar tab active with an Inspect Particle Picks job in building status

Inputs and Parameters

Enter any required parameters, and connect any required inputs. Inputs to a job are the data or files that will be processed by the job. For example, the input to a Patch CTF Estimation job are micrographs. Inputs for a job can come from another job inside the Workspace in which you are actively working, or any other Workspace within your current Project.

See All Job Types in CryoSPARC for detailed information about the inputs, parameters and outputs for each job type.

Drag and Drop to Connect

Connect the required input(s) by dragging and dropping the output(s) from another job.

If you have successfully connected the output(s), you will see the job number appear below the input, as well as an option to Remove if you have accidentally connected the wrong output:

For more details how how CryoSPARC inputs and outputs work, along with more advanced features of the job builder, refer to the following page: Tutorial: Job Builder

Default and Custom Parameters

Most job parameters in CryoSPARC have a default value. When you change a parameter to a value other than the default, it is marked as a custom parameter and will be highlighted in green. At any time, you can click the ‘Reset’ button to cycle between resetting the parameter to its default value or clearing the parameter value altogether.

At the top of the ‘Parameters’ section you can easily view how many parameters have been customized and toggle between displaying all, only defaults or only custom.

Each custom parameter is also listed directly on the job card, making it easier to distinguish many building jobs from one another.

Viewing Advanced Parameters

By default, the job builder will display a set of parameters for a job that are more commonly edited. To view the full list of parameters available for a job, select the ‘Advanced’ toggle at the top of the ‘Parameters’ section:

Queueing a Job

When you are satisfied with a job’s inputs and parameters, you can queue the job. Click 'Queue' on the Job Builder. This will bring up the Queue Modal.

1. Select the lane where the job should run. By default, new jobs you create will run in the current active workspace. If you wish to run the job and link it into a different workspace, you can also specify this under 'Run Job in'.

2. Optionally, you can set the Job Priority. See: Priority Queuing.

3. Click 'Create' to run the job.

When a job is queued, it will be cleared from the job builder and available to view via the details panel or main content area.

Queuing Chains of Jobs that Run Automatically

You can queue a chain of jobs, which will each commence as soon as their respective required inputs (i.e., the outputs from jobs earlier in the series) become available.

For example, while an Import Movies job is running, open the job card, and drag and drop the imported-movies output into a new Patch Motion Correction job in the Job Builder. Queue the Patch Motion Correction job and it will appear in "Queued - waiting because inputs are not ready" state until the imported movies become available, at which point it will start running automatically.

See also, Management and Monitoring if you wish to create and launch jobs through the command line interface.

Running a Job

When a job is queued, it will be launched on a processing node or cluster. To view details of a job, select it and press the spacebar to inspect it.

Queued jobs no longer automatically open like in CryoSPARC v3.

As a job runs, more information will display in the browse view. When viewing jobs in the cards view or tree view, a preview image of results and info tags will display on the relevant job card, providing a summary of results without having to inspect the job.

The next section covers more information on how to inspect job data and interpret results.