CryoSPARC Guide
  • About CryoSPARC
  • Current Version
  • Licensing
    • Non-commercial license agreement
  • Setup, Configuration and Management
    • CryoSPARC Architecture and System Requirements
    • CryoSPARC Installation Prerequisites
    • How to Download, Install and Configure
      • Obtaining A License ID
      • Downloading and Installing CryoSPARC
      • CryoSPARC Cluster Integration Script Examples
      • Accessing the CryoSPARC User Interface
    • Deploying CryoSPARC on AWS
      • Performance Benchmarks
    • Using CryoSPARC with Cluster Management Software
    • Software Updates and Patches
    • Management and Monitoring
      • Environment variables
      • (Optional) Hosting CryoSPARC Through a Reverse Proxy
      • cryosparcm reference
      • cryosparcm cli reference
      • cryosparcw reference
    • Software System Guides
      • Guide: Updating to CryoSPARC v4
      • Guide: Installation Testing with cryosparcm test
      • Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)
      • Guide: Verify CryoSPARC Installation with the Extensive Workflow (≤v4.2)
      • Guide: Performance Benchmarking (v4.3+)
      • Guide: Download Error Reports
      • Guide: Maintenance Mode and Configurable User Facing Messages
      • Guide: User Management
      • Guide: Multi-user Unix Permissions and Data Access Control
      • Guide: Lane Assignments and Restrictions
      • Guide: Queuing Directly to a GPU
      • Guide: Priority Job Queuing
      • Guide: Configuring Custom Variables for Cluster Job Submission Scripts
      • Guide: SSD Particle Caching in CryoSPARC
      • Guide: Data Management in CryoSPARC (v4.0+)
      • Guide: Data Cleanup (v4.3+)
      • Guide: Reduce Database Size (v4.3+)
      • Guide: Data Management in CryoSPARC (≤v3.3)
      • Guide: CryoSPARC Live Session Data Management
      • Guide: Manipulating .cs Files Created By CryoSPARC
      • Guide: Migrating your CryoSPARC Instance
      • Guide: EMDB-friendly XML file for FSC plots
    • Troubleshooting
  • Application Guide (v4.0+)
    • A Tour of the CryoSPARC Interface
    • Browsing the CryoSPARC Instance
    • Projects, Workspaces and Live Sessions
    • Jobs
    • Job Views: Cards, Tree, and Table
    • Creating and Running Jobs
    • Low Level Results Interface
    • Filters and Sorting
    • View Options
    • Tags
    • Flat vs Hierarchical Navigation
    • File Browser
    • Blueprints
    • Workflows
    • Inspecting Data
    • Managing Jobs
    • Interactive Jobs
    • Upload Local Files
    • Managing Data
    • Downloading and Exporting Data
    • Instance Management
    • Admin Panel
  • Cryo-EM Foundations
    • Image Formation
      • Contrast in Cryo-EM
      • Waves as Vectors
      • Aliasing
  • Expectation Maximization in Cryo-EM
  • Processing Data in cryoSPARC
    • Get Started with CryoSPARC: Introductory Tutorial (v4.0+)
    • Tutorial Videos
    • All Job Types in CryoSPARC
      • Import
        • Job: Import Movies
        • Job: Import Micrographs
        • Job: Import Particle Stack
        • Job: Import 3D Volumes
        • Job: Import Templates
        • Job: Import Result Group
        • Job: Import Beam Shift
      • Motion Correction
        • Job: Patch Motion Correction
        • Job: Full-Frame Motion Correction
        • Job: Local Motion Correction
        • Job: MotionCor2 (Wrapper) (BETA)
        • Job: Reference Based Motion Correction (BETA)
      • CTF Estimation
        • Job: Patch CTF Estimation
        • Job: Patch CTF Extraction
        • Job: CTFFIND4 (Wrapper)
        • Job: Gctf (Wrapper) (Legacy)
      • Exposure Curation
        • Job: Micrograph Denoiser (BETA)
        • Job: Micrograph Junk Detector (BETA)
        • Interactive Job: Manually Curate Exposures
      • Particle Picking
        • Interactive Job: Manual Picker
        • Job: Blob Picker
        • Job: Template Picker
        • Job: Filament Tracer
        • Job: Blob Picker Tuner
        • Interactive Job: Inspect Particle Picks
        • Job: Create Templates
      • Extraction
        • Job: Extract from Micrographs
        • Job: Downsample Particles
        • Job: Restack Particles
      • Deep Picking
        • Guideline for Supervised Particle Picking using Deep Learning Models
        • Deep Network Particle Picker
          • T20S Proteasome: Deep Particle Picking Tutorial
          • Job: Deep Picker Train and Job: Deep Picker Inference
        • Topaz (Bepler, et al)
          • T20S Proteasome: Topaz Particle Picking Tutorial
          • T20S Proteasome: Topaz Micrograph Denoising Tutorial
          • Job: Topaz Train and Job: Topaz Cross Validation
          • Job: Topaz Extract
          • Job: Topaz Denoise
      • Particle Curation
        • Job: 2D Classification
        • Interactive Job: Select 2D Classes
        • Job: Reference Based Auto Select 2D (BETA)
        • Job: Reconstruct 2D Classes
        • Job: Rebalance 2D Classes
        • Job: Class Probability Filter (Legacy)
        • Job: Rebalance Orientations
        • Job: Subset Particles by Statistic
      • 3D Reconstruction
        • Job: Ab-Initio Reconstruction
      • 3D Refinement
        • Job: Homogeneous Refinement
        • Job: Heterogeneous Refinement
        • Job: Non-Uniform Refinement
        • Job: Homogeneous Reconstruction Only
        • Job: Heterogeneous Reconstruction Only
        • Job: Homogeneous Refinement (Legacy)
        • Job: Non-uniform Refinement (Legacy)
      • CTF Refinement
        • Job: Global CTF Refinement
        • Job: Local CTF Refinement
        • Job: Exposure Group Utilities
      • Conformational Variability
        • Job: 3D Variability
        • Job: 3D Variability Display
        • Job: 3D Classification
        • Job: Regroup 3D Classes
        • Job: Reference Based Auto Select 3D (BETA)
        • Job: 3D Flexible Refinement (3DFlex) (BETA)
      • Postprocessing
        • Job: Sharpening Tools
        • Job: DeepEMhancer (Wrapper)
        • Job: Validation (FSC)
        • Job: Local Resolution Estimation
        • Job: Local Filtering
        • Job: ResLog Analysis
        • Job: ThreeDFSC (Wrapper) (Legacy)
      • Local Refinement
        • Job: Local Refinement
        • Job: Particle Subtraction
        • Job: Local Refinement (Legacy)
      • Helical Reconstruction
        • Helical symmetry in CryoSPARC
        • Job: Helical Refinement
        • Job: Symmetry search utility
        • Job: Average Power Spectra
      • Utilities
        • Job: Exposure Sets Tool
        • Job: Exposure Tools
        • Job: Generate Micrograph Thumbnails
        • Job: Cache Particles on SSD
        • Job: Check for Corrupt Particles
        • Job: Particle Sets Tool
        • Job: Reassign Particles to Micrographs
        • Job: Remove Duplicate Particles
        • Job: Symmetry Expansion
        • Job: Volume Tools
        • Job: Volume Alignment Tools
        • Job: Align 3D maps
        • Job: Split Volumes Group
        • Job: Orientation Diagnostics
      • Simulations
        • Job: Simulate Data (GPU)
        • Job: Simulate Data (Legacy)
    • CryoSPARC Tools
    • Data Processing Tutorials
      • Case study: End-to-end processing of a ligand-bound GPCR (EMPIAR-10853)
      • Case Study: DkTx-bound TRPV1 (EMPIAR-10059)
      • Case Study: Pseudosymmetry in TRPV5 and Calmodulin (EMPIAR-10256)
      • Case Study: End-to-end processing of an inactive GPCR (EMPIAR-10668)
      • Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716)
      • Case Study: Exploratory data processing by Oliver Clarke
      • Tutorial: Tips for Membrane Protein Structures
      • Tutorial: Common CryoSPARC Plots
      • Tutorial: Negative Stain Data
      • Tutorial: Phase Plate Data
      • Tutorial: EER File Support
      • Tutorial: EPU AFIS Beam Shift Import
      • Tutorial: Patch Motion and Patch CTF
      • Tutorial: Float16 Support
      • Tutorial: Particle Picking Calibration
      • Tutorial: Blob Picker Tuner
      • Tutorial: Helical Processing using EMPIAR-10031 (MAVS)
      • Tutorial: Maximum Box Sizes for Refinement
      • Tutorial: CTF Refinement
      • Tutorial: Ewald Sphere Correction
      • Tutorial: Symmetry Relaxation
      • Tutorial: Orientation Diagnostics
      • Tutorial: BILD files in CryoSPARC v4.4+
      • Tutorial: Mask Creation
      • Case Study: Yeast U4/U6.U5 tri-snRNP
      • Tutorial: 3D Classification
      • Tutorial: 3D Variability Analysis (Part One)
      • Tutorial: 3D Variability Analysis (Part Two)
      • Tutorial: 3D Flexible Refinement
        • Installing 3DFlex Dependencies (v4.1–v4.3)
      • Tutorial: 3D Flex Mesh Preparation
    • Webinar Recordings
  • Real-time processing in cryoSPARC Live
    • About CryoSPARC Live
    • Prerequisites and Compute Resources Setup
    • How to Access cryoSPARC Live
    • UI Overview
    • New Live Session: Start to Finish Guide
    • CryoSPARC Live Tutorial Videos
    • Live Jobs and Session-Level Functions
    • Performance Metrics
    • Managing a CryoSPARC Live Session from the CLI
    • FAQs and Troubleshooting
  • Guides for v3
    • v3 User Interface Guide
      • Dashboard
      • Project and Workspace Management
      • Create and Build Jobs
      • Queue Job, Inspect Job and Other Job Actions
      • View and Download Results
      • Job Relationships
      • Resource Manager
      • User Management
    • Tutorial: Job Builder
    • Get Started with CryoSPARC: Introductory Tutorial (v3)
    • Tutorial: Manually Curate Exposures (v3)
  • Resources
    • Questions and Support
Powered by GitBook
On this page
  • Parameters
  • Specifying Model
  • Using a provided training model
  • Using a model trained by a user
  • Using a model previously trained by a user
  • Interpreting Denoising Results
  1. Processing Data in cryoSPARC
  2. All Job Types in CryoSPARC
  3. Deep Picking
  4. Topaz (Bepler, et al)

Job: Topaz Denoise

Topaz Denoise job available via wrapper in CryoSPARC.

The Topaz Denoise job can be used to remove noise from micrographs using the Topaz provided model. A new model can also be trained using the job, which can then also be used. This job has the following inputs and outputs.

Inputs

  • Micrographs

  • Denoise Model

  • Training Micrographs

Outputs

  • Denoised Micrographs

  • Topaz Denoise Model

These inputs and outputs are pertinent in selecting which kind of model to use for denoising the micrographs. How the inputs and outputs affect the model are detailed in the Specifying Model section.

Parameters

The key parameters are detailed below:

  • General Settings

    • Path to Topaz Executable

      • The absolute path to the Topaz executable that will run the denoise job.

    • Number of Plots to Show

      • The number of side-by-side micrograph comparisons to show at the end of the job.

    • Number of Parallel Threads

      • The number of threads to run the denoising job on. This parameter is used only if the provided Topaz model is used rather than a self-trained model. This parameter decreases the preprocessing time by a factor approximately equal to the input value. Values less than 2 will default to single thread. These threads can also be distributed amongst GPUs, the number of which can be set with the Number of GPUs for Parallel Threads parameter.

    • Number of GPUs for Parallel Threads

      • The number of GPUs to distribute parallel threads over. The specific GPUs to use can be set using the CryoSPARC scheduler when queuing the job.

  • Denoising Parameters

    • Normalize Micrographs

      • Specify whether to normalize the micrographs prior to denoising.

    • Shape of Split Micrographs

      • The shape of micrographs after they been split into patches. The shape of the split micrographs will be (x, x) where x is the input parameter.

    • Padding around Each Split Micrograph

      • The padding to set around each split micrograph.

  • Training Parameters

    • Learning Rate

      • The value that determines how quickly the training approaches an optimum. Higher values will result with training approaching an optimum faster but may prevent the model from reaching the optimum itself, resulting with potentially worse final accuracy.

    • Minibatch Size

      • The number of examples that are used within each batch during training. Lower values will improve model accuracy at the cost of significantly increased training time.

    • Number of Epochs

      • The number of iterations through the entire dataset the training performs. Higher number of epochs will naturally lead to longer training times. The number of epochs does not have to be optimized as the train and cross validation jobs will automatically output the model from the epoch with the highest precision.

    • Criteria

      • The number of updates that occur each epoch. Increasing this value will increase the amount of training performed in each epoch in exchange for slower training speed.

    • Crop Size

      • The size of each micrograph after random cropping is performed during data augmentation.

    • Number of Loading Threads

      • The number of threads to use for loading data during training.

  • Pretrained Parameters

    • Model Architecture

      • U-Net (unet) is a convolutional neural network architecture that convolves the input information until a suitable bottleneck shape and then deconvolves the data, concatenating the opposite convolution outputs during the deconvolutions.

      • U-Net Small (unet-small) is the same as a U-Net except with less layers.

      • FCNN (fcnn) stands for fully-convolutional neural network and is the standard neural network architecture used in many computer vision tasks.

      • Affine (affine) applies an affine transformation by a single convolution.

      • Prior to Topaz version 0.2.3, only the L2 model architecture is available.

  • Compute Settings

    • Use CPU for Training

      • Specify whether to only use CPU for training.

Specifying Model

To denoise micrographs using the Topaz Denoise job, users have the option of using the provided pretrained model or to train a model for immediate and future use. Thus the user must select which model to use from three general categories. These categories are:

  1. Provided pretrained model

  2. Model to be trained by user

  3. Model previously trained by user

Specifying which approach to use depends on the job inputs and the build parameters. However, the Topaz Denoise job requires the micrographs that will be denoised to be input into the micrographs input slot regardless of model specification. The job inputs and build parameters required to select each category are summarized in the table below and detailed further below the table.

Model Category

"denoise_model" Input Slot

"training_micrographs" Input Slot

Provided pretrained model

False

False

Model to be trained by user

False

True

Model previously trained by user

True

False

Using a provided training model

To use the provided training model, the denoise_model and training_micrographs input slots must be empty.

Using a model trained by a user

To train a model for immediate and future use, imported movies that were not pre-processed must be input into the training_micrographs input slot and the denoise_model input slot must be empty.

When the job is complete, it will output the trained model through the topaz_denoise_model output, allowing the the trained model to be used in other Topaz Denoise jobs. How to use this output is specified in the Using model previously trained by user section below.

Using a model previously trained by a user

To use a previously trained model, pass the topaz_denoise_model output from the Topaz Denoise job with the trained model into the denoise_model input slot. The training_micrographs input slot must remain empty.

Interpreting Denoising Results

Once the Topaz Denoise job is complete, the job will output micrograph comparisons, the amount of which is dependent on the Number of plots to show build parameter. Each comparison features two micrographs on the same row. The micrograph to the left is the original micrograph prior to denoising and the micrograph to the right is the denoised version of the micrograph. This side-by-side comparison serves to inform the user of the effect of the denoising.

When a Topaz Denoise job is used to train a model, a plot of the training and validation losses will also be shown. The plots for both losses should be descending overtime. If the plot for the training loss is decreasing while the plot for the validation loss is increasing, this indicates that the model has overfit and training parameters must be tuned. The simplest approach to resolving overfitting is to reduce the learning rate.

PreviousJob: Topaz ExtractNextParticle Curation

Last updated 2 years ago

Side-by-Side Micrograph Denoising Comparison
Topaz Denoising Loss Plot