How to manage the data created by your cryoSPARC Live Sessions via the user interface and data management API.
Last updated
The images in this section depict CryoSPARC ≤v3.3. For CryoSPARC v4.0+, please see: Managing Data
Live Session Data Management tools are available in CryoSPARC v3.0.0+.
To access the Session Management page, click on the "Manage Data" button at the top of the Browse Sessions Page.
You will be greeted with a table that shows an overview of all Projects and Live Sessions including:
Project and Session UID
Project or Session title
Status of the Session (e.g., Running, Paused, Marked Completed)
File sizes of the following data categories (Projects show the total across all Sessions within that project)
Raw import data
Motion corrected micrographs
Exposure thumbnails
Extracted particles
Metadata
Date and time the Project or Session was created
Date and time the session was last Paused (if applicable)
Date and time the session was Marked Completed (if applicable)
Project actions:
Refresh statistics for all sessions within a Project (update file sizes)
Download a list of all Project statistics (all files linked to each session, organized into groups)
Session actions:
Refresh statistics for a particular Session (update file sizes)
Navigate to the Live Session interface
Additional actions than can be performed on a category of data within a session is described below
Upon first visit to the Data Management page, you will need to click the "Refresh Project Stats" or "Refresh Session Stats" action to populate the file sizes of each category.
You can only perform session actions on sessions that are in Completed status. To mark a session as completed, navigate to the session, and click the Mark Completed button in the Session Information sub-tab under the Configuration tab.
You can download a list of file paths (JSON format) of all sessions within a project (organized into categories) by clicking on the 'Download Project Stats' button in the 'Action' column.
You can perform actions on all five categories in completed sessions by clicking a cell in the table:
Download a list of file paths (JSON format) for a particular category (such as motion corrected micrographs) via the browser for use in an external archiving utility
Available for all categories except for thumbnails
Mark a category as 'archived', 'archiving', or 'active'
Available for all categories except for thumbnails
Mark a category as 'deleted' (for when you have used an external tool to delete all files associated with that category)
Available for all categories except for thumbnails
Delete data from a particular category (CryoSPARC will delete all files associated with the category)
Available for all categories except for raw data
A confirmation dialog will be presented before any data is deleted:
Right-clicking over any cell other than file size data will present will three options:
Refresh table data
Expand all projects (show all Sessions across all Projects)
Collapse all Projects (show only Project totals)
As of v3.3+, CryoSPARC Live Data Management supports executing a script upon a datatype's state change.
To use this feature, add the following environment variables to cryosparc_master/config.sh:
Then, restart CryoSPARC (cryosparcm stop && cryosparcm start).
The trigger will execute the script that you've specified and pass the following arguments: project_uid, session_uid, datatype, status
For example, the "Mark as Archiving" button is clicked for micrographs in P45 S1, the script will be triggered with 'P45', 'S1', 'micrographs', 'archiving'
Your script can look something like this (the following script reacts to "archiving" state changes, does some work, then updates the status of the datatype to "archived" once completed):
Here is the rest of the CryoSPARC Live Data Management API that is available via cryosparcm rtpcli or cryosparcm icli:
get_datatype_file_paths(project_uid, session_uid, datatype)
Get all the file paths associated with a specific datatype inside a session as a json dictionary.
delete_live_datatype(project_uid, session_uid, datatype, filepaths_to_delete=None, user_id=None, asynchronous=True)
Delete a specific datatype inside a session. Alternatively, provide a json dict of file paths to delete.
get_datatype_size(project_uid, session_uid, datatype)
Get the total size of a datatype inside a session in bytes.
update_session_datatype_sizes(project_uid, session_uid)
Updates the session's 'data_management' top-level key with the current size of each datatype. Additionally returns the entire size of the session's datatypes.
update_all_sessions_datatype_sizes(project_uid)
Loops through each session in the project and updates all datatype sizes in each session document.
get_data_management_stats(project_uid) **
Returns a json formatted dictionary that includes the data_management dictionary of all sessions in the project.
change_session_data_management_state(project_uid, session_uid, datatype, status)
Modify the data management status for a specific datatype inside a session. Note that this function will call trigger_live_data_management_script once completed
trigger_live_data_management_script(project_uid, session_uid, datatype, status=None, script_location=None, script_log_path_abs=None)
Execute the specified script using the parameters sent. Will log all stdout & stderr into a file. Will only execute if environment variable CRYOSPARC_LIVE_DATA_MANAGEMENT_SCRIPT_ENABLE is set in cryosparc_master/config.sh
All CryoSPARC users are able to:
View the data management table and refresh file sizes via the 'Action' column → 'Refresh Project Stats'/'Refresh Session Stats'
Download the JSON list of files for a category within a session or across a project
In order to mark a category as active/archiving/archived/deleted or delete data, users must have access enabled in the admin panel (in the main cryoSPARC web application):
Admin users can click on the button within the 'Live Data Management' column to toggle the ability for a user to modify Live session data:
