Guide: Configuring Custom Variables for Cluster Job Submission Scripts
The information in this section applies to CryoSPARC v4.1+.
User-set variables can be injected into cluster submission scripts before a job is queued. This may be useful for running jobs with different submission parameters on-the-fly, such as adjusting memory requirements in accordance with job resource demands, or keeping track of user quotas.
Custom variables can be configured at the instance level, the target level, and the job level.
Instance-wide and per-target level custom variables for cluster job submissions can be configured in the "Cluster Configuration" tab of the Admin Panel.
The Cluster Configuration tab of the Admin Panel
At the instance level, custom variables and their values will be applied to all targets which include the respective custom variables in their templates. If a target's submission script does not contain an instance level custom variable, that custom variable will not apply to the target.
At the target level, template variable names are parsed from the cluster submission script. Variables parsed from a target's script can be assigned values specific to that target. Any values for custom variables applied at the target level will override those that are set at the instance level. For example, if the custom instance level variable
var is set to 'x' and the custom target level variable var is set to 'y', the value 'y' will be applied when the job is queued to the cluster.At the job level, the values of instance level and target level custom variables can be modified before the job is queued. Modifications of the custom variable values at this level will override the default values set at the target level and instance level. Similarly to the previous example, if there is an instance level custom variable
var set to 'x', there is a target level custom variable var set to 'y', and var is set to 'z' at the job level, then the value 'z' will be applied when the job is queued to the cluster.To edit a custom variable at the job level, a dropdown menu will appear after selecting a cluster lane while queueing a job. Instance level and target level custom variables will appear here, as well as editable variables provided by CryoSPARC.
Below is a list of internal variables set by CryoSPARC that cannot be used as custom variables or be overridden by custom variables.
{{ run_cmd }} - the complete command string to run the job
{{ num_cpu }} - the number of CPUs needed
{{ num_gpu }} - the number of GPUs needed.
Note: the code will use this many GPUs starting from dev id 0
the cluster scheduler or this script have the responsibility
of setting CUDA_VISIBLE_DEVICES so that the job code ends up
using the correct cluster-allocated GPUs.
{{ ram_gb }} - the amount of RAM needed in GB
{{ job_dir_abs }} - absolute path to the job directory
{{ project_dir_abs }} - absolute path to the project dir
{{ job_log_path_abs }} - absolute path to the log file for the job
{{ worker_bin_path }} - absolute path to the cryosparc worker command
{{ run_args }} - arguments to be passed to cryosparcw run
{{ project_uid }} - uid of the project
{{ job_uid }} - uid of the job
{{ job_creator }} - name of the user that created the job (may contain spaces)
{{ cryosparc_username }} - cryosparc username of the user that created the job (usually an email)
{{ job_type }} - CryoSPARC job type
num_cpu, num_gpu, and ram_gb are script variables provided by CryoSPARC specifying the job's estimated required resources needed to run. Sometimes before submitting jobs to a cluster, these values may need to be adjusted depending on the accuracy of the job's required resources estimation. Custom cluster script variables can be used in this case to modify the values requested of the cluster.For example, below is a SLURM cluster configuration where the value for the amount of RAM required is modified by a
ram_gb_multiplier custom variable.#!/usr/bin/env bash
#SBATCH --job-name=cryosparc_{{ project_uid }}_{{ job_uid }}
#SBATCH -n {{ num_cpu }}
#SBATCH --gres=gpu:{{ num_gpu }}
#SBATCH --partition=slurmpar
## Example of a modifier variable
#SBATCH --mem={{ (ram_gb|float * ram_gb_multiplier|float)|int }}G
#SBATCH --output={{ job_log_path_abs }}
#SBATCH --error={{ job_log_path_abs }}
available_devs=""
for devidx in $(seq 0 15);
do
if [[ -z $(nvidia-smi -i $devidx --query-compute-apps=pid --format=csv,noheader) ]] ; then
if [[ -z "$available_devs" ]] ; then
available_devs=$devidx
else
available_devs=$available_devs,$devidx
fi
fi
done
echo "AVAILABLE DEVICES: $available_devs"
export CUDA_VISIBLE_DEVICES=$available_devs
echo {{ instance_test_var }}
{{ run_cmd }}
Last modified 3mo ago