CryoSPARC Guide
  • About CryoSPARC
  • Current Version
  • Licensing
    • Non-commercial license agreement
  • Setup, Configuration and Management
    • CryoSPARC Architecture and System Requirements
    • CryoSPARC Installation Prerequisites
    • How to Download, Install and Configure
      • Obtaining A License ID
      • Downloading and Installing CryoSPARC
      • CryoSPARC Cluster Integration Script Examples
      • Accessing the CryoSPARC User Interface
    • Deploying CryoSPARC on AWS
      • Performance Benchmarks
    • Using CryoSPARC with Cluster Management Software
    • Software Updates and Patches
    • Management and Monitoring
      • Environment variables
      • (Optional) Hosting CryoSPARC Through a Reverse Proxy
      • cryosparcm reference
      • cryosparcm cli reference
      • cryosparcw reference
    • Software System Guides
      • Guide: Updating to CryoSPARC v4
      • Guide: Installation Testing with cryosparcm test
      • Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)
      • Guide: Verify CryoSPARC Installation with the Extensive Workflow (≤v4.2)
      • Guide: Performance Benchmarking (v4.3+)
      • Guide: Download Error Reports
      • Guide: Maintenance Mode and Configurable User Facing Messages
      • Guide: User Management
      • Guide: Multi-user Unix Permissions and Data Access Control
      • Guide: Lane Assignments and Restrictions
      • Guide: Queuing Directly to a GPU
      • Guide: Priority Job Queuing
      • Guide: Configuring Custom Variables for Cluster Job Submission Scripts
      • Guide: SSD Particle Caching in CryoSPARC
      • Guide: Data Management in CryoSPARC (v4.0+)
      • Guide: Data Cleanup (v4.3+)
      • Guide: Reduce Database Size (v4.3+)
      • Guide: Data Management in CryoSPARC (≤v3.3)
      • Guide: CryoSPARC Live Session Data Management
      • Guide: Manipulating .cs Files Created By CryoSPARC
      • Guide: Migrating your CryoSPARC Instance
      • Guide: EMDB-friendly XML file for FSC plots
    • Troubleshooting
  • Application Guide (v4.0+)
    • A Tour of the CryoSPARC Interface
    • Browsing the CryoSPARC Instance
    • Projects, Workspaces and Live Sessions
    • Jobs
    • Job Views: Cards, Tree, and Table
    • Creating and Running Jobs
    • Low Level Results Interface
    • Filters and Sorting
    • View Options
    • Tags
    • Flat vs Hierarchical Navigation
    • File Browser
    • Blueprints
    • Workflows
    • Inspecting Data
    • Managing Jobs
    • Interactive Jobs
    • Upload Local Files
    • Managing Data
    • Downloading and Exporting Data
    • Instance Management
    • Admin Panel
  • Cryo-EM Foundations
    • Image Formation
      • Contrast in Cryo-EM
      • Waves as Vectors
      • Aliasing
  • Expectation Maximization in Cryo-EM
  • Processing Data in cryoSPARC
    • Get Started with CryoSPARC: Introductory Tutorial (v4.0+)
    • Tutorial Videos
    • All Job Types in CryoSPARC
      • Import
        • Job: Import Movies
        • Job: Import Micrographs
        • Job: Import Particle Stack
        • Job: Import 3D Volumes
        • Job: Import Templates
        • Job: Import Result Group
        • Job: Import Beam Shift
      • Motion Correction
        • Job: Patch Motion Correction
        • Job: Full-Frame Motion Correction
        • Job: Local Motion Correction
        • Job: MotionCor2 (Wrapper) (BETA)
        • Job: Reference Based Motion Correction (BETA)
      • CTF Estimation
        • Job: Patch CTF Estimation
        • Job: Patch CTF Extraction
        • Job: CTFFIND4 (Wrapper)
        • Job: Gctf (Wrapper) (Legacy)
      • Exposure Curation
        • Job: Micrograph Denoiser (BETA)
        • Job: Micrograph Junk Detector (BETA)
        • Interactive Job: Manually Curate Exposures
      • Particle Picking
        • Interactive Job: Manual Picker
        • Job: Blob Picker
        • Job: Template Picker
        • Job: Filament Tracer
        • Job: Blob Picker Tuner
        • Interactive Job: Inspect Particle Picks
        • Job: Create Templates
      • Extraction
        • Job: Extract from Micrographs
        • Job: Downsample Particles
        • Job: Restack Particles
      • Deep Picking
        • Guideline for Supervised Particle Picking using Deep Learning Models
        • Deep Network Particle Picker
          • T20S Proteasome: Deep Particle Picking Tutorial
          • Job: Deep Picker Train and Job: Deep Picker Inference
        • Topaz (Bepler, et al)
          • T20S Proteasome: Topaz Particle Picking Tutorial
          • T20S Proteasome: Topaz Micrograph Denoising Tutorial
          • Job: Topaz Train and Job: Topaz Cross Validation
          • Job: Topaz Extract
          • Job: Topaz Denoise
      • Particle Curation
        • Job: 2D Classification
        • Interactive Job: Select 2D Classes
        • Job: Reference Based Auto Select 2D (BETA)
        • Job: Reconstruct 2D Classes
        • Job: Rebalance 2D Classes
        • Job: Class Probability Filter (Legacy)
        • Job: Rebalance Orientations
        • Job: Subset Particles by Statistic
      • 3D Reconstruction
        • Job: Ab-Initio Reconstruction
      • 3D Refinement
        • Job: Homogeneous Refinement
        • Job: Heterogeneous Refinement
        • Job: Non-Uniform Refinement
        • Job: Homogeneous Reconstruction Only
        • Job: Heterogeneous Reconstruction Only
        • Job: Homogeneous Refinement (Legacy)
        • Job: Non-uniform Refinement (Legacy)
      • CTF Refinement
        • Job: Global CTF Refinement
        • Job: Local CTF Refinement
        • Job: Exposure Group Utilities
      • Conformational Variability
        • Job: 3D Variability
        • Job: 3D Variability Display
        • Job: 3D Classification
        • Job: Regroup 3D Classes
        • Job: Reference Based Auto Select 3D (BETA)
        • Job: 3D Flexible Refinement (3DFlex) (BETA)
      • Postprocessing
        • Job: Sharpening Tools
        • Job: DeepEMhancer (Wrapper)
        • Job: Validation (FSC)
        • Job: Local Resolution Estimation
        • Job: Local Filtering
        • Job: ResLog Analysis
        • Job: ThreeDFSC (Wrapper) (Legacy)
      • Local Refinement
        • Job: Local Refinement
        • Job: Particle Subtraction
        • Job: Local Refinement (Legacy)
      • Helical Reconstruction
        • Helical symmetry in CryoSPARC
        • Job: Helical Refinement
        • Job: Symmetry search utility
        • Job: Average Power Spectra
      • Utilities
        • Job: Exposure Sets Tool
        • Job: Exposure Tools
        • Job: Generate Micrograph Thumbnails
        • Job: Cache Particles on SSD
        • Job: Check for Corrupt Particles
        • Job: Particle Sets Tool
        • Job: Reassign Particles to Micrographs
        • Job: Remove Duplicate Particles
        • Job: Symmetry Expansion
        • Job: Volume Tools
        • Job: Volume Alignment Tools
        • Job: Align 3D maps
        • Job: Split Volumes Group
        • Job: Orientation Diagnostics
      • Simulations
        • Job: Simulate Data (GPU)
        • Job: Simulate Data (Legacy)
    • CryoSPARC Tools
    • Data Processing Tutorials
      • Case study: End-to-end processing of a ligand-bound GPCR (EMPIAR-10853)
      • Case Study: DkTx-bound TRPV1 (EMPIAR-10059)
      • Case Study: Pseudosymmetry in TRPV5 and Calmodulin (EMPIAR-10256)
      • Case Study: End-to-end processing of an inactive GPCR (EMPIAR-10668)
      • Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716)
      • Case Study: Exploratory data processing by Oliver Clarke
      • Tutorial: Tips for Membrane Protein Structures
      • Tutorial: Common CryoSPARC Plots
      • Tutorial: Negative Stain Data
      • Tutorial: Phase Plate Data
      • Tutorial: EER File Support
      • Tutorial: EPU AFIS Beam Shift Import
      • Tutorial: Patch Motion and Patch CTF
      • Tutorial: Float16 Support
      • Tutorial: Particle Picking Calibration
      • Tutorial: Blob Picker Tuner
      • Tutorial: Helical Processing using EMPIAR-10031 (MAVS)
      • Tutorial: Maximum Box Sizes for Refinement
      • Tutorial: CTF Refinement
      • Tutorial: Ewald Sphere Correction
      • Tutorial: Symmetry Relaxation
      • Tutorial: Orientation Diagnostics
      • Tutorial: BILD files in CryoSPARC v4.4+
      • Tutorial: Mask Creation
      • Case Study: Yeast U4/U6.U5 tri-snRNP
      • Tutorial: 3D Classification
      • Tutorial: 3D Variability Analysis (Part One)
      • Tutorial: 3D Variability Analysis (Part Two)
      • Tutorial: 3D Flexible Refinement
        • Installing 3DFlex Dependencies (v4.1–v4.3)
      • Tutorial: 3D Flex Mesh Preparation
    • Webinar Recordings
  • Real-time processing in cryoSPARC Live
    • About CryoSPARC Live
    • Prerequisites and Compute Resources Setup
    • How to Access cryoSPARC Live
    • UI Overview
    • New Live Session: Start to Finish Guide
    • CryoSPARC Live Tutorial Videos
    • Live Jobs and Session-Level Functions
    • Performance Metrics
    • Managing a CryoSPARC Live Session from the CLI
    • FAQs and Troubleshooting
  • Guides for v3
    • v3 User Interface Guide
      • Dashboard
      • Project and Workspace Management
      • Create and Build Jobs
      • Queue Job, Inspect Job and Other Job Actions
      • View and Download Results
      • Job Relationships
      • Resource Manager
      • User Management
    • Tutorial: Job Builder
    • Get Started with CryoSPARC: Introductory Tutorial (v3)
    • Tutorial: Manually Curate Exposures (v3)
  • Resources
    • Questions and Support
Powered by GitBook
On this page
  • Extracting recentered particles
  • Setup
  • Manipulating a .cs File with Python
  • Full Example Python Script
  1. Setup, Configuration and Management
  2. Software System Guides

Guide: Manipulating .cs Files Created By CryoSPARC

A short guide on reading and modifying CryoSPARC dataset (.cs) files.

PreviousGuide: CryoSPARC Live Session Data ManagementNextGuide: Migrating your CryoSPARC Instance

Last updated 2 years ago

For CryoSPARC v4.1 or newer, cryosparc-tools may also be used to modify job results, including datasets. .

There may be an instance where you need to manually open a .cs (CryoSPARC) file to manipulate your data's attributes for a specific data processing workflow or question you may have. In this tutorial, you will learn how to open, view, manipulate and save .cs files using Python, allowing you to craft complex workflows within CryoSPARC (e.g., , , etc.).

Extracting recentered particles

In the following example, we edit a particle dataset in preparation for re-extraction after centering particle locations based on the shifts calculated by a 2D Classification job.

Setup

Use a subset of raw movies from and picked particle locations from a Template Picker job with templates resembling the top and side views of the protein. Then extract the particles and run them through a 2D Classification job, which creates templates. Filter the classified particles with a Select 2D job.

Manipulating a .cs File with Python

Option 1: Use the built-in interactive shell

To use CryoSPARC's built-in interactive Python shell, run the following command:

cryosparcm icli

This starts an interactive Python command shell where you can import the required modules and copy+paste the commands below.

Option 2: Write your own Python script

You can also create a .py Python script and execute it non-interactively. First evaluate all of CryoSPARC's environment variables into your current shell session's path with this command:

eval $(cryosparcm env)

Then use CryoSPARC's provided Python interpreter to execute your script.

To import cryosparc_compute.datasetin your Python script, set the PYTHONPATH environment variable to cryoSPARC's root execution directory:

# Enter these in the command line
eval $(cryosparcm env)
export PYTHONPATH="${CRYOSPARC_ROOT_DIR}"
python recenter_particles.py

The full script is provided at the end of this guide.

Import the Required Modules

First, import the two modules required for this example:

import numpy as n

# "dataset" is the main module required to interact with cryoSPARC .cs files
from cryosparc_compute import dataset

Loading the Dataset File

Use the .cs file path from exporting the particle result group:

# load the dataset into memory from file
dataset_path = '/bulk/data/cryosparc_projects/P28/exports/groups/P28_J283_particles_selected/P28_J283_particles_selected_exported.cs'
particle_dataset = dataset.Dataset.load(dataset_path)

Manipulate the data

[('uid', '<u8'),
 ('blob/path', '|O'),
 ('blob/idx', '<u4'),
 ('blob/shape', '<u4', (2,)),
 ('blob/psize_A', '<f4'),
 ('blob/sign', '<f4'),
 ('alignments2D/split', '<u4'),
 ('alignments2D/shift', '<f4', (2,)),
 ('alignments2D/pose', '<f4'),
 ('alignments2D/psize_A', '<f4'),
 ('alignments2D/error', '<f4'),
 ('alignments2D/error_min', '<f4'),
 ('alignments2D/resid_pow', '<f4'),
 ('alignments2D/slice_pow', '<f4'),
 ('alignments2D/image_pow', '<f4'),
 ('alignments2D/cross_cor', '<f4'),
 ('alignments2D/alpha', '<f4'),
 ('alignments2D/alpha_min', '<f4'),
 ('alignments2D/weight', '<f4'),
 ('alignments2D/pose_ess', '<f4'),
 ('alignments2D/shift_ess', '<f4'),
 ('alignments2D/class_posterior', '<f4'),
 ('alignments2D/class', '<u4'),
 ('alignments2D/class_ess', '<f4'),
 ('ctf/type', '|O'),
 ('ctf/exp_group_id', '<u4'),
 ('ctf/accel_kv', '<f4'),
 ('ctf/cs_mm', '<f4'),
 ('ctf/amp_contrast', '<f4'),
 ('ctf/df1_A', '<f4'),
 ('ctf/df2_A', '<f4'),
 ('ctf/df_angle_rad', '<f4'),
 ('ctf/phase_shift_rad', '<f4'),
 ('ctf/scale', '<f4'),
 ('ctf/scale_const', '<f4'),
 ('ctf/shift_A', '<f4', (2,)),
 ('ctf/tilt_A', '<f4', (2,)),
 ('ctf/trefoil_A', '<f4', (2,)),
 ('ctf/tetra_A', '<f4', (4,)),
 ('ctf/anisomag', '<f4', (4,)),
 ('ctf/bfactor', '<f4'),
 ('location/micrograph_uid', '<u8'),
 ('location/exp_group_id', '<u4'),
 ('location/micrograph_path', '|O'),
 ('location/micrograph_shape', '<u4', (2,)),
 ('location/center_x_frac', '<f4'),
 ('location/center_y_frac', '<f4'),
 ('pick_stats/ncc_score', '<f4'),
 ('pick_stats/power', '<f4'),
 ('pick_stats/template_idx', '<u4'),
 ('pick_stats/angle_rad', '<f4')]

Calculate new particle locations for our dataset by applying the shift values from a 2D Classification job. These shifts, when added to existing particle (x,y) coordinates, yield a "recentered" position based on where the 2D Classification algorithm calculated the "centre of mass" of the particle to be.

First convert the raw particle location into pixels, then angstroms (Å) (the shifts are stored in angstroms relative to the particle image). CryoSPARC stores particle coordinates as ratios of the centre of the particle relative to the original micrograph's shape. Transpose the array, then multiply it by the length of the micrograph in each dimension. Finally, multiply it by the original micrograph's pixel size (set manually on line 1).

# organize information about the original micrographs these particles were extracted from
micrograph_pixel_size = 0.6575 # also can be obtained from mscope_params/psize_A
micrograph_ny = particle_dataset['location/micrograph_shape'][0][0]
micrograph_nx = particle_dataset['location/micrograph_shape'][0][1]

# get the shift values obtained from 2D classification in angstroms (Å)
pixel_size = particle_dataset['alignments2D/psize_A'].reshape(-1,1)
shifts = particle_dataset['alignments2D/shift'] * pixel_size

# calulate the current locations of the particles in angstroms (Å)
location_xs = (particle_dataset['location/center_x_frac'].reshape(-1,1) * # fraction relative to micrograph
               micrograph_nx *
               micrograph_pixel_size)

location_ys = (particle_dataset['location/center_y_frac'].reshape(-1,1) * # fraction relative to micrograph
               micrograph_ny *
               micrograph_pixel_size)

# calculate the new locations of the particles, after applying the shifts
recentered_locations = n.concatenate((location_xs, location_ys), 1) - shifts

# convert the new locations back to fractions relative to the micrograph
recentered_fraction_xs = (recentered_locations[:,0] / 
               micrograph_nx /
               micrograph_pixel_size)

recentered_fraction_ys = (recentered_locations[:,1] / 
               micrograph_ny /
               micrograph_pixel_size)

Storing the Data in the Dataset

After recentering particle locations by adding the shifts, convert the locations back to fractions. Save these back to the dataset.

# save the new fractions back to the dataset
particle_dataset['location/center_x_frac'] = recentered_fraction_xs
particle_dataset['location/center_y_frac'] = recentered_fraction_ys

# reset the shifts now that they've already been applied to the particle locations
particle_dataset['alignments2D/shift'] = 0

Save the Dataset To Disk

Save the dataset back to disk with the save(string)function.

# save the dataset back to disk
# Note: this will overwrite the currently exported dataset
particle_dataset.save(dataset_path)

If using the interactive Python shell, exit with ctrl+D. If using a script, save it and run it with python my_script.py.

Import the Dataset into CryoSPARC

Re-import the dataset into the workspace via the Import Result Groups job. Use the Extract From Micrographs job to extract recentered particle locations.

In the Import Result Group job, specify the path to the .csg file that was next to the modified .cs file on disk to re-import the dataset.

Connect the particles_selected group (or similar) to an Extract From Micrographs job to re-extract the recentered particles.

Full Example Python Script

recenter_particles.py
import numpy as n

# dataset is the main module required to interact with cryoSPARC .cs files
from cryosparc_compute import dataset

# load the dataset into memory from file
dataset_path = '/bulk6/data/devv2stephan_projects_bulk6/P28/exports/groups/P28_J283_particles_selected/P28_J283_particles_selected_exported.cs'
particle_dataset = dataset.Dataset.load(dataset_path)

# organize information about the original micrographs these particles were extracted from
micrograph_pixel_size = 0.6575 # also can be obtained from mscope_params/psize_A
micrograph_ny = particle_dataset['location/micrograph_shape'][0][0]
micrograph_nx = particle_dataset['location/micrograph_shape'][0][1]

# get the shift values obtained from 2D classification in angstroms (Å)
pixel_size = particle_dataset['alignments2D/psize_A'].reshape(-1,1)
shifts = particle_dataset['alignments2D/shift'] * pixel_size

# calulate the current locations of the particles in angstroms (Å)
location_xs = (particle_dataset['location/center_x_frac'].reshape(-1,1) * # fraction relative to micrograph
               micrograph_nx *
               micrograph_pixel_size)

location_ys = (particle_dataset['location/center_y_frac'].reshape(-1,1) * # fraction relative to micrograph
               micrograph_ny *
               micrograph_pixel_size)

# calculate the new locations of the particles, after applying the shifts
recentered_locations = n.concatenate((location_xs, location_ys), 1) - shifts

# convert the new locations back to fractions relative to the micrograph
recentered_fraction_xs = (recentered_locations[:,0] / 
               micrograph_nx /
               micrograph_pixel_size)

recentered_fraction_ys = (recentered_locations[:,1] / 
               micrograph_ny /
               micrograph_pixel_size)

# save the new fractions back to the dataset
particle_dataset['location/center_x_frac'] = recentered_fraction_xs
particle_dataset['location/center_y_frac'] = recentered_fraction_ys

# reset the shifts now that they've already been applied to the particle locations
particle_dataset['alignments2D/shift'] = 0

# save the dataset back to disk
# Note: this will overwrite the currently exported dataset
particle_dataset.save(dataset_path)

Press the "Export" on the particles_selected output result group in the "Outputs" tab of the Select 2D job. This consolidates the newly created dataset with the passthrough outputs (datasets created by ancestor jobs that were not modified during processing, but are passed along the workflow). For more information on Exporting data in CryoSPARC, read the

Navigate back to the "Overview" tab to find the location of the exported .cs (CryoSPARC file) and .csg (CryoSPARC group file) at the end of the streamlog. Keep this path; we use it to open the dataset file on disk to modify it. For more information on .cs and .csg files,

Here, you need a Python environment with modules required to open and edit .cs files. You may use CryoSPARC's own built-in interactive Python shell or a running a Python 3.7 environment.

You can access any column in a dataset as you would a Python dictionary: First use index access syntax (i.e., dset[column] for some value of column). This returns a of all values inside the column. In this example, these are the available columns (also known as "fields") and their types:

Data Management Guide.
see our FAQ here.
Jupyter Notebook
Numpy array
Read the documentation
reverse symmetry expansion for a particle set
sorting particles above a certain angle and shift threshold
EMPIAR-10025
Tree-view diagram of the workspace
Press the "Export" button under "Actions" to export an output result group.
This is the path to the .cs file you will manipulate, then import back into cryoSPARC.