cryosparcm cli reference
Call methods described in this module directly as arguments to cryosparcm cli or referenced from the cli object in cryosparcm icli.
  • cli example
1
cryosparcm cli "enqueue_job(project_uid='P3', job_uid='J42', lane='cryoem1')"
Copied!
  • icli example
1
$ cryosparcm icli
2
3
connecting to cryoem5:61002 ...
4
cli, rtp, db, gfs and tools ready to use
5
6
In [1]: cli.enqueue_job(project_uid='P3', job_uid='J42', lane='cryoem1')
7
8
In [2]:
Copied!

add_project_user_access(project_uid, requester_user_id, add_user_id)

Allows the owner of a project to grant access to another user to view and edit the project
  • Parameters
  • project_uid (str) -- the uid of the project to grant access to
  • requester_user_id (str) -- the _id of the user requesting a new user to be added to the project
  • add_user_id (str) -- the uid of the user to add to the project
  • Raises
    AssertionError

add_scheduler_lane(name, lanetype, title=None, desc='')

Adds a new lane to the master scheduler
  • Parameters
  • name (str) -- name of the lane
  • lanetype (str) -- type of lane ("cluster" or "node")
  • title (str) -- optional title of the lane
  • desc (str) -- optional description of the lane

add_scheduler_target_node(hostname, ssh_str, worker_bin_path, num_cpus, cuda_devs, ram_mb, has_ssd, cache_path=None, cache_quota=None, cache_reserve=10000, monitor_port=None, lane='default', title=None, desc=None)

Adds a worker node to the master scheduler
  • Parameters
  • hostname (str) -- the hostname of the target worker node
  • ssh_str (str) -- the ssh connection string of the worker node
  • worker_bin_path (str) -- the absolute path to 'cryosparc_package/cryosparc_worker/bin/cryosparcw' on the worker node
  • num_cpus (int) -- total number of CPU threads available on the worker node
  • cuda_devs (list) -- total number of cuda-capable devices (GPUs) on the worker node, represented as a list (i.e. range(4))
  • ram_mb (float) -- total available physical ram in MB
  • has_ssd (bool) -- if an ssd is available or not
  • cache_path (str) -- path on SSD that can be used for the cryosparc cache
  • cache_quota (int) -- the max size (in MB) to use for the cache on the SSD
  • cache_reserve (int) -- size (in MB) to initially reserve for the cache on the SSD
  • lane (str) -- the scheduler lane to add the worker node to
  • title (str) -- an optional title to give to the worker node
  • desc (str) -- an optional description of the worker node
  • Returns
    configuration parameters for new worker node
  • Return type
    dict
  • Raises
    AssertionError

clear_job(project_uid, job_uid, nofail=False, isdelete=False)

Clear a job to get it back to building state (do not clear params or inputs)
  • Parameters
  • project_uid (str) -- the uid of the project that contains the job to clear
  • job_uid (str) -- the uid of the job to clear
  • Raises
    AssertionError

clone_job(project_uid, workspace_uid, job_uid, created_by_user_id, created_by_job_uid=None)

Creates a new job as a clone of the provided job
  • Parameters
  • project_uid (str) -- uid of project
  • workspace_uid (str) -- the uid of the workspace
  • job_uid (str) -- the uid of the job to copy
  • created_by_user_id (str) -- the id of the user creating the clone
  • created_by_job_uid (str) -- the uid of the job creating the clone (default = None)
  • Returns
    the job uid of the newly created clone
  • Return type
    str

complete_patch()

Called automatically after a patch installation with cryosparcm patch

create_empty_project(owner_user_id, project_container_dir, title=None, desc=None, project_dir=None, export=True)

Creates a new project and project directory and creates a new document in the project collection
NOTE: project_container_dir is an absolute path that the user guarantees is available everywhere.
  • Parameters
  • owner_user_id (str) -- the _id of the user that owns the new project (which is also the user that requests to create the project)
  • project_container_dir (str) -- the path to the "root" directory in which to create the new project directory
  • title (str) -- the title of the new project to create [optional] [default = None]
  • desc (str) -- the description of the new project to create [optional] [default = None]
  • Returns
    the new uid of the project that was created
  • Return type
    str

create_new_job(job_type, project_uid, workspace_uid, created_by_user_id, created_by_job_uid=None, title=None, desc=None, do_layout=True, dry_run=False, enable_bench=False, priority=None)

Creates a new job
NOTE: This request is run in its own thread, but locks on the "jobs" lock, so only one create can happen at a time.This means that the builder functions below can take as much time as they like to do stuff in their own thread (releasing GIL like IO or numpy etc) and command can still respond to other requests that don't require this lock.
  • Parameters
  • job_type (str) -- the name of the job to create
  • project_uid (str) -- the uid of the project
  • workspace_uid (str) -- the uid of the workspace
  • created_by_user_id (str) -- the id of the user creating the job
  • created_by_job_uid (str) -- the uid of the job that created this job
  • title (str) -- optional title of the job
  • desc (str) -- optional description of the job
  • do_layout (bool) -- specifies if the job tree layout should be recalculated (default = True)
  • enable_bench (bool) -- should performance be measured for this job (benchmarking)
  • Returns
    the uid of the newly created job
  • Return type
    str
  • Raises
    AssertionError

create_user(created_by_user_id, email, password, username, first_name, last_name, admin=False)

Creates a new cryosparc user
  • Parameters
  • created_by_user_id (str) -- identity of user who is creating the new user
  • email (str) -- email of user to create
  • password (str) -- password of user to create
  • admin (bool) -- specifies if user should have an administrator role or not
  • Returns
    mongo _id of user that was created
  • Return type
    str

delete_job(project_uid, job_uid, relayout=True, nofail=False)

Deletes a Job after killing it (if running), clearing it, setting the "deleted" property, and recalculating the tree layout
  • Parameters
  • project_uid (str) -- the uid of the project that contains the job to delete
  • job_uid (str) -- the uid of the job to delete
  • relayout (bool) -- specifies if the tree layout should be recalculated or not

delete_project(project_uid, request_user_id, all_jobs_in_project, all_workspaces_in_project)

Iterates through each job and workspace associated with the project to be deleted and deletes both, and then disables the project itself
  • Parameters
  • project_uid (str) -- the uid of the project to be deleted
  • request_user_id (str) -- the uid of the user that is requesting the project to be deleted
  • all_jobs_in_project (list) -- list of all jobs to be fully deleted
  • all_workspaces_in_project (list) -- list of all workspaces that will be deleted by association
  • Returns
    string confirmation that the workspace has been deleted
  • Return type
    str
  • Raises
    AssertionError -- if for some reason a workspace isn't deleted, this will be thrown

delete_project_user_access(project_uid, requester_user_id, delete_user_id)

Removes a user's access from a project.
  • Parameters
  • project_uid (str) -- the uid of the project to revoke access to
  • requester_user_id (str) -- the _id of the user requesting a user to be removed from the project
  • delete_user_id (str) -- the uid of the user to remove from the project
  • Raises
    AssertionError

delete_user(email, requesting_user_email, requesting_user_password)

Remove a user from the cryoSPARC. Only administrators may do this
  • Parameters
  • email (str) -- user's email
  • requesting_user_email (str) -- your cryoSPARC login email
  • requesting_user_password (str) -- your cryoSPARC password
  • Returns
    confirmation message
  • Return type
    str

delete_workspace(project_uid, workspace_uid, request_user_id, jobs_inside_one_workspace, jobs_inside_multiple_workspaces)

Iterates through each job associated with the workspace to be deleted and either deletes or updates the job, and then disables the workspace itself
  • Parameters
  • project_uid (str) -- the uid of the project containing the workspace to be deleted
  • workspace_uid (str) -- the uid of the workspace to be deleted
  • request_user_id (str) -- the uid of the user that is requesting the workspace to be deleted
  • jobs_inside_one_workspace (list) -- list of all jobs to be fully deleted
  • jobs_inside_multiple_workspaces (list) -- list of all jobs to be updated so that the workspace uid is removed from its list of "workspace_uids"
  • Returns
    string confirmation that the workspace has been deleted
  • Return type
    str

enqueue_job(project_uid, job_uid, lane=False, hostname=False, gpus=False, no_check_inputs_ready=False)

Add the job in the given project to the queue for the given worker lane (default lane if not specified)
  • Parameters
  • project_uid (str) -- uid of the project containing the job to queue
  • job_uid (str) -- job uid to queue
  • lane (str, optional) -- name of the worker lane onto which to queue., defaults to False
  • hostname (str, optional) -- the hostname of the target worker node
  • gpus (list, optional) -- list of GPU indexes on which to queue the given job., defaults to False
  • no_check_inputs_ready (bool, optional) -- if True, forgoes checking whether inputs are ready (not recommended), defaults to False
  • Returns
    job status (e.g., 'launched' if success)
  • Return type
    str
  • Example
1
enqueue_job('P3', 'J42', lane='cryoem1')
Copied!

get_job(project_uid, job_uid, *args, **kwargs)

Get a job object with optionally only a few fields specified in \*args
  • Parameters
  • project_uid (str) -- uid of project
  • job_uid (str) -- uid of job
  • Returns
    mongo result set
  • Return type
    dict
1
get_job('P1','J1','status','abc','def')
Copied!

get_project(project_uid)

Returns the information about a single project
  • Parameters
    project_uid (str) -- the id of the project
  • Returns
    the information related to the project thats stored in the database
  • Return type
    str

get_scheduler_lanes()

Returns a list of lanes that are registered with the master scheduler
  • Returns
    list of dicts -- information about each lane

get_scheduler_target_cluster_info(name)

Returns cluster information based on the passed cluster name
  • Parameters
    name (str) -- name of cluster/lane (lane must be of type 'cluster')
  • Returns
    json string containing cluster information
  • Return type
    str
  • Raises
    AssertionError

get_scheduler_target_cluster_script(name)

Returns cluster job submission template for specified cluster
  • Parameters
    name (str) -- name of cluster/lane (lane must be of type 'cluster')
  • Returns
    job scheduler script template
  • Return type
    str
  • Raises
    AssertionError

get_scheduler_targets()

Returns a list of worker nodes that are registered with the master scheduler
  • Returns
    list of dicts -- information about each worker node

get_system_info()

Returns system-related information related to the cryosparc app
  • Returns
    dictionary listing information about cryosparc environment
  • Return type
    dict

get_worker_nodes()

Returns a list of worker nodes registered in the master scheduler
  • Returns
    list of dicts -- information about each worker node

import_project(owner_user_id, abs_path_export_project_dir)

Allows a user to import any project that was exported by cryoSPARC
  • Parameters
  • owner_user_id (str) -- the mongo object id ("_id") of the user requesting to import the project
  • abs_path_export_project_dir (str) -- the absolute project directory containing the jobs to import
  • abs_path_database_export_dir (str) -- the absolute database export directory (created by cryoSPARC's export_project()) to import

job_add_to_workspace(project_uid, job_uid, workspace_uid)

Adds a job to the specified workspace
  • Parameters
  • project_uid (str) -- the id of the project
  • job_uid (str) -- the id of the job to add
  • workspace_uid (str) -- the id of the workspace
  • Returns
    the number of modified documents
  • Return type
    int

job_remove_from_workspace(project_uid, job_uid, workspace_uid)

Removes a job from the specified workspace
  • Parameters
  • project_uid (str) -- the id of the project
  • job_uid (str) -- the id of the job to remove
  • workspace_uid (str) -- the id of the workspace
  • Returns
    the number of modified documents
  • Return type
    int

kill_job(project_uid, job_uid, killed_by_user_id=None)

Kill the given running job
  • Parameters
  • project_uid (str) -- uid of the project that contain the job to kill
  • job_uid (str) -- job uid to kill
  • killed_by_user_id (str) -- ID of user that killed the job, optional
  • Raises
    AssertionError

list_projects()

Returns a list of all projects available
  • Returns
    all projects available in the database
  • Return type
    list

list_users()

Show a list of all cryosparc users

list_workspaces(project_uid=None)

List all workspaces inside a given project
  • Parameters
    project_uid (str, optional) -- the uid of the project (e.g. P1), defaults to None
  • Returns
    list of workpaces in project or all projects if not specified
  • Return type
    list

make_job(job_type, project_uid, workspace_uid, user_id, created_by_job_uid=None, title=None, params={}, input_group_connects={}, enable_bench=False, priority=None)

Create a new job with the given type in the given project/workspace
To see all available job types, see cryosparc_compute/jobs/register.py (look for the value of the contains keys).
To see what parameters are available for a job and what values are available, reference the build.py file in cryosparc_compute/jobs that pertains to the desired job type.
  • Parameters
  • job_type (str) -- Type of job
  • project_uid (str) -- project ID, e.g., P3
  • workspace_uid (str) -- workspace ID, e.g., W1
  • user_id (str) -- ID of user
  • created_by_job_uid (str, optional) -- ID of the parent job that created this, defaults to None
  • title (str, optional) -- Descriptive title for what this job is for, defaults to None
  • params (dict, optional) -- Parameter settings for the job, defaults to {}
  • input_group_connects (dict, optional) -- Connected input groups, which each key is name and each value has format JXX.output_group_name, defaults to {}
  • Returns
    ID of new job
  • Return type
    str
  • Example
The following makes a 3-class ab-initio job with particles connected from a Select 2D Classes job:
1
juid = cli.make_job(
2
job_type='homo_abinit',
3
project_uid='P3',
4
workspace_uid='W1',
5
user_id=cli.GetUser('[email protected]')['_id'],
6
params={'abinit_K': 3},
7
input_group_connects={'particles': 'J41.particles_selected'})
8
9
print(juid) # "J42"
Copied!

refresh_job_types()

Reload the available parameter types on jobs created prior to a cryoSPARC update. Run this following a cryoSPARC update to ensure jobs run with the correct parameters.
  • Returns
    list of dicts -- the results of the mongodb query

remove_scheduler_lane(name)

Removes the specified lane and any targets assigned under the lane in the master scheduler
NOTE: This will remove any worker node associated with the specified lane.
  • Parameters
    name (str) -- the name of the lane to remove

remove_scheduler_target_cluster(name)

Removes the specified cluster/lane and any targets assigned under the lane in the master scheduler
NOTE: This will remove any worker node associated with the specified cluster/lane.
  • Parameters
    name (str) -- the name of the cluster/lane to remove
  • Returns
    "True" if successful
  • Return type
    bool

remove_scheduler_target_node(hostname)

Removes a target worker node from the master scheduler
  • Parameters
    hostname (str) -- the hostname of the target worker node to remove

request_delete_project(project_uid, request_user_id)

Allows the user to confirm what jobs and workspaces they will potentially be deleting when requesting to delete a project.
  • Parameters
  • project_uid (str) -- the uid of the project to be deleted
  • request_user_id (str) -- the uid of the user that is requesting the project to be deleted
  • Returns
    If the project does not have any jobs or workspaces related to it, this function will go ahead and "disable" the project and return a string confirmation.
  • Return type
    str
  • Returns
    If the project has jobs or workspaces associated with it, it will return 2 lists. First list are all non-deleted jobs that exist within the project, and second list are all workspaces within the project to be deleted.
  • Return type
    lists

request_delete_workspace(project_uid, workspace_uid, request_user_id)

Allows the user to confirm what jobs they will potentially be deleting when requesting to delete a workspace.
  • Parameters
  • project_uid (str) -- the uid of the project containing the workspace to be deleted
  • workspace_uid (str) -- the uid of the workspace to be deleted
  • Returns
    If the workspace does not have any jobs related to it, this function will go ahead and "disable" the workspace and return a string confirmation.
  • Return type
    str
  • Returns
    If the workspace has jobs associated with it, it will return 2 lists. First list are jobs that can be deleted outright, and second list are jobs that are a part of other workspaces.
  • Return type
    lists

reset_password(email, newpass)

Reset a cryosparc user's password
  • Parameters
  • email (str) -- the user's email address
  • password (str) -- the user's new password
  • Returns
    the number of modified documents
  • Return type
    str

set_project_owner(project_uid, user_id)

Updates a project's owner
  • Parameters
  • project_uid (str) -- the id of the project to delete
  • user_id (str) -- the owner's user id

set_scheduler_target_node_cache(hostname, cache_reserve=None, cache_quota=None)

Sets the cache reserve and cache quota for a target worker node
  • Parameters
  • hostname (str) -- the hostname of the target worker node
  • cache_reserve (int) -- the size (in MB) to reserve on the SSD for the cryosparc cache
  • cache_quota (int) -- the max size (in MB) to use on the SSD for the cryosparc cache
  • Raises
    AssertionError

set_scheduler_target_node_lane(hostname, lane)

Sets the lane of a target worker node
  • Parameters
  • hostname (str) -- the hostname of the target worker node
  • lane (str) -- the name of the lane to assign to the target worker node:
  • Returns
    target worker node's updated configurations
  • Return type
    list
  • Raises
    AssertionError

set_scheduler_target_property(hostname, key, value)

Sets a property for the target worker node
  • Parameters
  • hostname (str) -- the hostname of the target worker node
  • key (str) -- the key of the property whose value is being modified
  • value (str) -- the actual value to set for the property
  • Returns
    list of dicts -- information about all targets
  • Raises
    AssertionError

set_user_allowed_prefix_dir(user_id, allowed_prefix)

Sets directories that users are allowed to query from the file browser
  • Parameters
  • user_id (str) -- the mongo id of the user to update
  • allowed_prefix (str) -- the path of the directory the user can query inside (must start with "/", and must be an absolute path)
  • Returns
    True if successful
  • Return type
    bool
  • Raises
    AssertionError

update_job(project_uid, job_uid, attrs, operation='$set')

Updates a specific job's document with the provided attributes
  • Parameters
  • project_uid (str) -- uid of project
  • job_uid (str) -- uid of job
  • attrs (dict) -- the attribute to modify in the job document
  • operation (str) -- the operation to perform on the document (default: '$set')

update_project(project_uid, attrs, operation='$set', export=True)

Updates a project's attributes
  • Parameters
  • project_uid (str) -- the id of the project to update
  • attrs (dict) -- the key to update and its value
  • operation (str) -- the type of mongodb operation to perform
  • Example
1
update_project(project_uid, {'project_dir' : new_abs_path})
Copied!

update_project_directory(project_uid, new_project_dir)

Safely updates the project directory of a project given a directory. Checks if the directory exists, is readable, and writeable.

Parameters

  • project_uid (str) -- the uid of the project to update
  • new_project_dir (str) -- the absolute path of the new directory

update_project_root_dir(project_uid, new_project_dir_container)

Updates the root directory of a project (creates a new project (PXXX) folder within the new root directory and updates the project document)
NOTE: the root project directory passed cannot contain a folder with the same project (PXXX) folder name
  • Parameters
  • project_uid (str) -- the uid of the project to update
  • new_project_dir_container (str) -- the new "root" path where the project (PXXX) folder is to be created

update_project_size(project_uid)

Calculates the size of the project. Similar to running du -sL inside the project dir

update_user(email, password, username=None, first_name=None, last_name=None, admin=None)

Updates a cryosparc user's details. Email and password are required, other params will only be set if they are not empty.
  • Parameters
  • email (str) -- the user's email address
  • password (str) -- the user's password
  • username (str) -- new username of the user, optional
  • first_name (str) -- new given name of the user, optional
  • last_name (str) -- new surname of the user, optional
  • admin (bool) -- whether the user should be admin or not, optional
  • Returns
    confirmation message if successful or not
  • Return type
    str

verify_cluster(name)

Used to ensure cluster has been properly configured by executing a generic 'info' command
  • Parameters
    name (str) -- name of cluster/lane (lane must be of type 'cluster')
  • Returns
    the result of the command execution
  • Return type
    str
  • Raises
    AssertionError
Copy link
Contents
add_project_user_access(project_uid, requester_user_id, add_user_id)
add_scheduler_lane(name, lanetype, title=None, desc='')
add_scheduler_target_node(hostname, ssh_str, worker_bin_path, num_cpus, cuda_devs, ram_mb, has_ssd, cache_path=None, cache_quota=None, cache_reserve=10000, monitor_port=None, lane='default', title=None, desc=None)
clear_job(project_uid, job_uid, nofail=False, isdelete=False)
clone_job(project_uid, workspace_uid, job_uid, created_by_user_id, created_by_job_uid=None)
complete_patch()
create_empty_project(owner_user_id, project_container_dir, title=None, desc=None, project_dir=None, export=True)
create_new_job(job_type, project_uid, workspace_uid, created_by_user_id, created_by_job_uid=None, title=None, desc=None, do_layout=True, dry_run=False, enable_bench=False, priority=None)
create_user(created_by_user_id, email, password, username, first_name, last_name, admin=False)
delete_job(project_uid, job_uid, relayout=True, nofail=False)
delete_project(project_uid, request_user_id, all_jobs_in_project, all_workspaces_in_project)
delete_project_user_access(project_uid, requester_user_id, delete_user_id)
delete_user(email, requesting_user_email, requesting_user_password)
delete_workspace(project_uid, workspace_uid, request_user_id, jobs_inside_one_workspace, jobs_inside_multiple_workspaces)
enqueue_job(project_uid, job_uid, lane=False, hostname=False, gpus=False, no_check_inputs_ready=False)
get_job(project_uid, job_uid, *args, **kwargs)
get_project(project_uid)
get_scheduler_lanes()
get_scheduler_target_cluster_info(name)
get_scheduler_target_cluster_script(name)
get_scheduler_targets()
get_system_info()
get_worker_nodes()
import_project(owner_user_id, abs_path_export_project_dir)
job_add_to_workspace(project_uid, job_uid, workspace_uid)
job_remove_from_workspace(project_uid, job_uid, workspace_uid)
kill_job(project_uid, job_uid, killed_by_user_id=None)
list_projects()
list_users()
list_workspaces(project_uid=None)
make_job(job_type, project_uid, workspace_uid, user_id, created_by_job_uid=None, title=None, params={}, input_group_connects={}, enable_bench=False, priority=None)
refresh_job_types()
remove_scheduler_lane(name)
remove_scheduler_target_cluster(name)
remove_scheduler_target_node(hostname)
request_delete_project(project_uid, request_user_id)
request_delete_workspace(project_uid, workspace_uid, request_user_id)
reset_password(email, newpass)
set_project_owner(project_uid, user_id)
set_scheduler_target_node_cache(hostname, cache_reserve=None, cache_quota=None)
set_scheduler_target_node_lane(hostname, lane)
set_scheduler_target_property(hostname, key, value)
set_user_allowed_prefix_dir(user_id, allowed_prefix)
update_job(project_uid, job_uid, attrs, operation='$set')
update_project(project_uid, attrs, operation='$set', export=True)
update_project_directory(project_uid, new_project_dir)
update_project_root_dir(project_uid, new_project_dir_container)
update_project_size(project_uid)
update_user(email, password, username=None, first_name=None, last_name=None, admin=None)
verify_cluster(name)