Managing Jobs
Last updated
Last updated
This is the hub for managing jobs and viewing instance usage. The Current Jobs panel shows all jobs in progress in a grid of their containing lanes.
The control bar at the top of the view contains a view toggle for filtering between all jobs, queued jobs, or active jobs. On the right side of the bar, instance token usage is visible (if applicable). Token usage is broken down by each active instance and summarized in total.
The main area is composed of lanes and their attached jobs. This view has been condensed from earlier versions of CryoSPARC by moving all inactive lanes into a single tracker pinned to the left side of the viewing area. Only lanes that are currently active will be displayed separately. Each lane is composed of a three parts.
Header
The header shows the lane title, type, and the number of jobs currently attached to the lane. The job number will change if viewing jobs filtered by active or queued, and will show what number out of the total are currently being displayed.
The header also houses additional information visible when clicked to expand. This will reveal the title, name, and description of that lane.
Jobs
The jobs section houses unique job cards with features specific to managing the instance.
The card header shows the status colour indicator followed by the job’s project ID and individual job ID. On the right side is the number of GPU tokens being used by the job, its running priority, and its current status. The job type is displayed below.
The job body includes information about who created it and how long it has been attached to the lane in its current status. Below is an action button to kill the job, and a trigger button for the job’s quick access menu.
The job card can also be selected and inspected in the sidebar, and opened from the “View Job” button at the bottom of the sidebar (to be noted: this will close the manage dialog as only one dialog can be open at a time, but it is possible to go back to the manage dialog by simply pressing the native browser back button).
Footer
The lane footer contains a “View Details” button that will navigate to the flattened jobs browse section set to the table view, and will set filters for the relevant lane and job statuses of “Queued, Launched, Started, Waiting, and Running”. This allows for more detailed and dynamic inspection of jobs running on the lane.
The app footer is designed to give contextual information about the jobs currently active in the instance at a glance. It is highly dynamic and will size up or down depending on the space available in the window. The footer is made up of two sections, the active job section and the active target section.
Active Jobs: The active job section is composed of a details button which is followed by job pill buttons. The details button shows a count of how many jobs are currently queued and/or active. This button will open up the “Current Jobs” dialog when clicked. The job pill buttons show the job’s project ID, job ID, and a status indicator. Hovering a job button will show the full job type in a tooltip, and clicking on a job will open that job’s inspection dialog (this will not navigate you away from your current page).
Active Targets: The active targets section has its own details button which similarly opens the “Current Jobs” dialog. The target buttons show the applicable lane’s name and can be clicked to open a target dialog with a small grid of job cards representing the jobs running on that target. Each card has nearly identical functionality to those in the “Current Jobs” dialog.
All jobs have variety of actions available for both processing data as well as doing general management and maintenance. These actions can be accessed from two distinct locations, the quick actions menu, and the sidebar actions panel.
Sidebar Actions Panel
The sidebar actions panel is analogous to the actions section in the jobs sidebar of previous versions of CryoSPARC. This panel can be expanded by clicking the “Actions” button on the footer of the job sidebar. The actions panel overlays the main sidebar area, and allows you to continue to see and scroll through all of the information available in the sidebar. This panel is focused on managing the current job’s data and only includes core job actions (these are detailed below). This panel will remain open until you decide to close it, which allows for ease of iteration while processing data within a job whether the job dialog is open or closed.
Quick Actions Menu
The quick actions menu is a context menu that can be opened on job cards or rows and gives a wide range general actions and job specific actions. In order to access this menu you must either right click on a job card or row, or click the triple dot menu button on the job card header. The quick actions menu has five sections: the core actions section, multi actions section, quick actions section, general actions section, and navigation section. The quick actions section is populated by common next steps in a processing chain, and allows you to quickly build subsequent jobs with the inputs already connected. The general actions section includes the tags menu where you can add or remove tags. The navigation section has options to enter the job’s containing project and view all workspaces inside, or enter the job’s workspace and see all jobs inside. The core actions and multi actions sections are covered below.
Queue Job
Queuing jobs will connect them to a selected lane and set their status to queued until the necessary resources become available to run them. This is covered further in the “Creating and Running Jobs” guide section.
Link Job
To avoid redundant processing, it's possible to link jobs across multiple workspaces to continue processing in a new or existing workspace. Any changes you make to the job in one workspace will be reflected in the other, as both workspaces will be displaying the exact same job. If a linked job is deleted from any workspace, all other instances of it in other workspaces will also be deleted. You can unlink a job from a workspace if needed.
Move Job
When re-organizing your workspaces, you may find it necessary to move a job from one workspace to another. This action will cause the job to disappear from its original workspace and move to the new one.
Kill Job
You can kill a job while it is running. A pop-up message will allow you to confirm the action. Killed jobs will display as orange and will remain as part of the workspace until you delete them. Killed jobs will not have their intermediate results expunged until they are cleared.
Any job that is in a killable or clearable state can be restarted. This action will run through a process of automatically killing the job, clearing the job, and then queueing the job to the same lane with the same parameters it was initially run with.
Clear Job
When a job has been killed, has failed, or has been completed, it can be cleared. This means that its results and outputs are all erased from the file system, but its connections to other jobs and parameters are retained. This makes it easy to correct a mistakenly set parameter, or to re-run a failed job. Clearing also removes queued jobs from the queue and resets jobs to Building status.
Clear Intermediate Results
Unused intermediate results can be cleared from iterative jobs to save space. This action can be executed at a job level (by clicking "Clear Intermediate Results" in the quick actions menu or actions panel) or at a project level for every job (by clicking "Clear Intermediate Results" in the project level quick actions menu or actions panel). This function will remove all unused outputs created by iterative jobs that save raw data at every iteration. Final results for every result slot will be retained, whether they have been used elsewhere or not.
Export Job
Any individual job can be exported, for sharing, manipulation, or archiving. Jobs must be exported manually in order to create a "consolidated" exported-job directory which is then importable. For more details, please see: Exporting a job.
Clone Job
Cloning is particularly useful when you wish to process subsequent jobs in a new workspace for better organization of your experiment, or if you wish to quickly replicate a job (to try another setting of parameters) without dragging and dropping inputs into the Job Builder. Select the job you wish to clone, and from the quick action menu or sidebar actions panel, click 'Clone job'. By default, the job will be cloned in the current workspace. Click 'Queue' to launch the job. Note that you can always edit the parameters of a cloned job before Queuing. You can also change inputs by removing connected inputs and dragging in new inputs.
Mark Job as Complete
This option allows you to mark failed or killed jobs in CryoSPARC as completed in order to allow their latest outputs to be used for further processing. For more details, please see: Data Management in CryoSPARC.
Delete Job
While selected, you can delete a job from the quick actions menu or sidebar actions panel. A pop-up message will ask you to confirm the deletion. Once deleted, the job will disappear from the workspace (except in the tree view, if it has children that need to be displayed). Note that job IDs within a workspace are unique, so a new job in the same workspace will not be assigned the same ID as a previously run or previously deleted job. Deleted jobs are cleared before deletion, meaning that their intermediate and final results are erased from disk.
Multi actions are actions that only appear when multiple jobs have been selected. This can be done by holding the command key and clicking on job cards or rows to select multiple, or by holding the command key and pressing or holding either the left or right arrow keys with one or more jobs already selected.
Once multiple jobs have been selected a new multi-actions menu will become available in place of the default actions menu when right clicking on job cards or rows. A multi-actions palette is also available and can be activated by clicking the “Actions” button at the bottom of the multi-select sidebar.
Jobs can be removed from the selection by either clicking the “clear” button in the top right corner of the job cards in the multi-select sidebar, or by holding the command key and clicking selected job cards or rows.
All selected jobs can be cleared by either clicking the “clear” button on the righthand side of the multi-select sidebar header, clicking the background of the cards view, or pressing the escape key.
This action allows you to select multiple applicable jobs and choose a lane to queue them too. All selected jobs will be queued to the chosen lane, with the exception of interactive jobs and master direct import jobs, which will automatically be queued to the default lane even if a different lane was chosen.
This action will kill all applicable selected jobs, and works identically to the singular “Kill Job” action.
This action will clear all applicable selected jobs, and works identically to the singular “Clear Job” action.
This action allows you to restart any selection of jobs that are in a killable or clearable state. It will automatically kill any running jobs, clear all stopped jobs, and then queue each job to the same lane with the same parameters it was initially run with.
Only a single restart process can be performed at a time in order to prevent jobs states from becoming inconsistent during the operation.
This action will mark all applicable selected jobs as complete, and works identically to the singular “Mark Jobs as Complete” action.
This action allows you to link all selected jobs to another workspace. When selecting this option a confirmation alert will be shown with a list of all selected jobs for review as well as options to choose the workspace where the selected jobs will be linked, or create a new workspace to link the jobs to. When the link is confirmed the jobs will be linked to the chosen or new workspace and you will be navigated into the workspace with the linked jobs selected.
This action functions similarly to the “Link Jobs” action and allows you to move all selected jobs into another workspace. When selecting this option a confirmation alert will be shown with a list of all selected jobs for review as well as options to choose the workspace where the selected jobs will be moved, or create a new workspace to move the jobs into. When the move is confirmed the jobs will be moved into the chosen or new workspace and you will be navigated into the workspace with the moved jobs selected.
This action allows you to clone a variable number of jobs based on how many you have selected. When this option is chosen an alert will open showing you the jobs to be cloned with an option to clone them into either the current workspace, another workspace, or a new workspace. When cloning is confirmed the jobs will be cloned into the chosen or new workspace, if the chosen workspace is different from the one you are currently in you will be navigated into it. The cloned jobs will be selected automatically after the action is complete.
When selecting “Clone Jobs” the alert will open with the “Clone” confirmation button automatically focused. If you wish to clone the selected jobs into the current workspace, you can simply press the enter key to do this in an expedient manner.
This action allows you to clone a chain of jobs between two selections. This can only be done with a valid chain of jobs, if unrelated jobs are selected as the start and end points you will be alerted that the chain is not valid. When this option is selected a dialog with the same options as the “Clone Jobs” action will open. Here you can see all of the jobs that would be created in the chain (this list is all of the jobs between the two selected points), and options to select a workspace to clone into or create a new workspace to clone into. When cloning is confirmed the job chain will be cloned into the chosen or new workspace, if the chosen workspace is different from the one you are currently in you will be navigated into it.
This action will clear the intermediate results of all applicable selected jobs, and works identically to the singular “Clear Intermediate Results” action.
This action will export all applicable selected jobs, and works identically to the singular “Export Job” action.
This action will unlink all selected linked jobs from the current workspace, and works identically to the singular “Unlink Job” action.
This action allows you to multi-select any number of jobs and delete all of them. A confirmation alert will open once this option is selected with a list of all of the jobs to be deleted available for review. Once confirmed, this action will permanently delete all selected jobs.
