> For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-junk-masking-during-preprocessing-v5.1+.md).

# Tutorial: Junk masking during preprocessing (v5.1+)

Automatically remove junk features in micrographs and improve downstream results

## What is junk masking?

Some undesirable features commonly found in cryo-EM micrographs, for example, crystalline ice, cracks, transfer ice, hydrocarbon contaminants, denatured and aggregated proteins, and even carbon or gold supports (i.e., hole edges) can interfere with preprocessing steps (especially CTF estimation, denoising and picking) and therefore ultimately impact the quality of downstream processing and 3D reconstruction. It would be ideal for the effect of these “**junk**” features in a micrograph to be avoided, however standard cryo-EM preprocessing algorithms treat the entire micrograph the same regardless of which areas contain usable signal or junk.

In CryoSPARC v4.7, we introduced the [Micrograph Junk Detector](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta) job which uses a pre-trained neural network model to label regions of a micrograph as different [categories of junk](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta#description). After using Micrograph Junk Detector, you can use [Manually Curate Exposures](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/interactive-job-manually-curate-exposures) to curate micrographs based on how much junk they contain. You can also use Micrograph Junk Detector to reject particles based on how close they are to regions labelled as junk.

**In v5.1, we introduce the ability to perform “junk masking” in preprocessing steps**. Junk masking uses the junk labels produced by the Micrograph Junk Detector to ignore or specially handle junk features in [Patch CTF Estimation](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation), [Micrograph Denoiser](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-denoiser-beta), [Blob](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-blob-picker) and [Template Picker](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-template-picker#common-parameters). **In addition, the new “junk filling” option** in [Extract from Micrographs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs#notes-and-limitations) and [Reference Based Motion correction](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta) fills remaining junk regions in particles images with random noise so that the junk feature is erased.

<figure><img src="https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FxMxKpcBVWdgGXPHrXtq1%2Fv5-1-0-junk-mask-v3.avif?alt=media&amp;token=b29a6268-7ca1-4de5-adb7-a9483c4424cd" alt=""><figcaption><p><strong>Figure 1.</strong> Junk masking application in a data processing pipeline using CryoSPARC v5.1.</p></figcaption></figure>

Together, these changes mean you can now:

* Process cryo-EM data where the presence of junk (e.g., gold edges) previously caused preprocessing to fail, such a [HexAuFoi&#x6C;**®** gold grids](https://www.quantifoil.com/products/hexaufoil)
* Improve the quality of CTF estimation for micrographs that contain any junk
* Retain micrographs that would previously have been discarded during exposure curation due to poor preprocessing statistics
* Automatically improve the quality of micrograph denoising and particle picking
* Retain particles that previously would have to be discarded due to proximity to junk features, (e.g., near the edge of a hole) and improve quality of their downstream 2D and 3D pose assignment

### When can junk masking and junk filling make a difference?

Junk masking can be particularly helpful for the following cases:

* Datasets that were collected on gold grids where the gold is visible in the micrograph images, especially for data that do not benefit from Local CTF Refinement. For datasets collected on [HexAuFoi&#x6C;**®** gold grids](https://www.quantifoil.com/products/hexaufoil), exclusion of junk regions can substantially improve CTF estimation, micrograph denoising and picking, making it possible to process datasets that previously may have been unusable.
* Datasets where the extraction box is large enough to include junk regions often, particularly if the junk content is high, or the particles are sparse.
* Datasets where particles near the edges of holes or other junk are important to retain (e.g. because particles in those areas tend to adopt a particular orientation)

In our testing, we have not encountered scenarios where performing the junk masking steps described in this tutorial were detrimental.

## How to apply junk masking during preprocessing

We demonstrate the junk masking pipeline on a rabbit aldolase dataset kindly provided by Dr Talya Levitz at the Dana-Farber Cancer Institute, Boston. The data was collected using a Thermo Scientific Tundra microscope operated at 100 kV, and [HexAuFoi&#x6C;**®** gold grids.](https://www.quantifoil.com/products/hexaufoil)

{% hint style="info" %}
**By default in CryoSPARC v5.1,** junk masking is “off” and so you will need to go through the below steps to enable junk masking in each relevant job.
{% endhint %}

### 1. Use the Micrograph Junk Detector to generate junk masks

The [Micrograph Junk Detector](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta) takes in motion corrected micrographs, and detects and labels junk regions in the following [categories](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta#description): Carbon/Gold, Intrinsic Ice defects and Extrinsic Ice defects. These **junk labels** (also known as junk annotations) are output as a result called `annotation_blob` along with per-micrographs stats in a result called `junk_stats`.

**In v5.1, the Micrograph Junk Detector job newly produces a junk mask result**, called `junk_mask_params`, which is carried along with the labelled micrographs to all subsequent preprocessing jobs. Junk masks (Figure 2) are defined by which kinds of junk are to be masked out, a dilation distance (default 20Å), and a softening distance (default 50Å). These parameters are further described below.

<figure><img src="https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FOQM5pl8TdRtgbnBS8lAW%2Fv5-1-0-junk-mask-fig-2.png?alt=media&amp;token=7ac70de3-e374-4e03-b605-e1906abb9ec6" alt=""><figcaption><p><strong>Figure 2.</strong> An example output micrograph from the Micrograph Junk Detector. A) the micrograph is coloured by detected junk labels. B) The micrograph is coloured according to the junk mask produced using default settings with an inset diagram showing the distances that the job parameters control.</p></figcaption></figure>

{% hint style="info" %}
If you have already run Micrograph Junk Detector in a previous version of CryoSPARC, those output micrographs can be input to a new Micrograph Junk Detector job in CryoSPARC v5.1+ to rapidly generate the `junk_mask_params` output.
{% endhint %}

<figure><img src="https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FMQIOtGlz3rbkF8djcrXt%2Fv5-1-0-junk-mask-fig-3.png?alt=media&amp;token=c7eb1c3c-e59f-469b-ac31-59d718026d09" alt=""><figcaption><p><strong>Figure 3.</strong> Job building for the Micrograph Junk Detector job in CryoSPARC v5.1. New or updated parameters indicated.</p></figcaption></figure>

Figure 3 shows the parameters of the Micrograph Junk Detector job that have changed in CryoSPARC v5.1:

* **Reject regions/particles near \[Category]**
  * You can control which [junk categories](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta#description) are included in the mask using the toggles for `Reject regions/particles near Carbon/Grid`, `Reject regions/particles near Intrinsic Ice Defect`, and `Reject regions/particles near Extrinsic Ice Defect`.
  * Figure 4 shows how the junk mask changes with junk category selection. In cases where desirable features of the micrograph (e.g. filaments) are incorrectly labelled as a type of junk, it can help to exclude that type of junk from the mask.

    ![Figure 4. An example micrograph showing the effects of different junk category selections on the resulting junk mask.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2F6Tqz0XYZp3TSwcf00sob%2Fv5-1-0-junk-mask-fig-4.png?alt=media\&token=488ce3d1-b7e4-4ebc-9b59-324a242f208d)
* **Junk mask dilation (default 20Å)**
  * The hard mask extension can be altered with the `Junk mask dilation (+)/erosion (-) (A)` parameter which expands the junk region when it is given a positive value, and shrinks the junk region when given a negative value. If you find the junk annotation generously includes your junk regions, then a negative value might allow you to make use of more of the micrograph and to keep particles that are a bit closer to the junk. Examples are shown in Figure 5.

    ![Figure 5. An example micrograph showing the effects of different junk mask dilation.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2F6ibieN2nbAhylCbjii0f%2Fv5-1-0-junk-mask-fig-5.png?alt=media\&token=5a16ecab-a344-4f0d-9b01-5167397d3664)
* **Junk mask soften width (default 50Å)**
  * The `Junk mask soften width (A)` can be altered to change the width of the **junk mask** soft edge (examples are shown in Figure 6).
  * The soft edge width becomes particularly important at the Extract From Micrographs and Reference-Based Motion Correction stages, if you plan to use junk filling. Note that a width narrower than 50 Å might lead to a more visible transition between the non-junk and junk regions in extracted particle images.

    ![Figure 6. An example micrograph showing the effects of different mask soften width settings.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FVwdNPml6ffocyVWd3n9k%2Fv5-1-0-junk-mask-fig-6.png?alt=media\&token=87a7ffbf-453f-462a-8212-172376904722)

### 2. Exclude junk during Patch CTF Estimation

The **junk mask** produced in the previous step can now be used to exclude regions of junk from consideration during [Patch CTF Estimation](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation). This can be especially useful for datasets that contain a significant amount of gold support that can otherwise interfere with estimation of defocus and astigmatism.

![Figure 7. CTF Estimation for an example micrograph using default settings in Patch CTF Estimation.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FDrgZ5VfoaHKd3cqnFCht%2Fv5-1-0-junk-mask-fig-7.png?alt=media\&token=7fbe685a-f9fd-4ad3-af72-6988ce6b7122)

For example, in Figure 7 (above), we show an example micrograph that was processed using Patch CTF Estimation (**without** junk masking). The defocus profile for this micrograph is heavily distorted at the corners due to the presence of gold support, and the CTF estimation is poor. Many users would choose to reject this micrograph at the curation stage due to the low CTF maximum resolution fit. Using the junk mask in Patch CTF Estimation can overcome this problem.&#x20;

![Figure 8. The job builder for Patch CTF Estimation in CryoSPARC v5.1. The new lower-level slot and new parameters are indicated.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2F2zFz8IoCEoicksphysEb%2Fv5-1-0-junk-mask-fig-8.png?alt=media\&token=b01df3f9-c798-41db-8244-d55b3d79a2f4)

Figure 8 displays the changes to Patch CTF Estimation parameters in v5.1:

* **Use junk mask**
  * To enable exclusion of junk-masked regions from consideration, turn on the `Use junk mask` toggle. Note that this toggle will only be visible if the input micrographs have a junk mask (`junk_mask_params`) and junk labels (`annotation_blob`) associated with them (see [Section 1](https://app.notion.com/p/WIP-Tutorial-Junk-masking-during-preprocessing-3d14324a3a7b804d901bcbd791812736?pvs=21) and Figure 8).
* **Max junk fraction in patch**
  * To control the inclusion / exclusion of patches that contain junk, you can adjust the maximum junk content for each patch based on the **junk mask hard edge**, by altering the parameter `Max junk fraction in patch` (see Figure 8). We found that values of 0.3 to 0.5 work well, meaning that patches with a junk content of >30-50% are excluded from CTF estimation.
  * For patches that are excluded from CTF estimation, their defocus values are extrapolated on the basis of the fitted patches, so that if particles are later extracted within those patches, they can still be used.
  * Micrographs that have too few remaining patches for CTF estimation will produce a message in the job log, and those micrographs will be output as `micrographs_rejected` without undergoing CTF estimation. If you wish to use these micrographs with a very high junk content downstream, they can be input to a further Patch CTF estimation job with the `Use junk mask` option disabled.

#### Impact of junk masking in Patch CTF Estimation

Returning to the example shown in Figure 7, we see that by enabling the `Use junk mask` parameter, the defocus profile became flatter, and the maximum CTF resolution fit improved from 25 Å to 4.3 Å. Additionally, adjusting `Max junk fraction in patch` to 0.3 further improved the defocus profile flatness, and the CTF maximum resolution fit improved to 3.9 Å, indicating that this is likely a good quality image worth retaining in the downstream pipeline.

![Figure 9. An example micrograph with results shown from Patch CTF. In each row, Left: micrograph and included patches, Middle: defocus profile, Right: CTF fit. A) Patch CTF without using junk masks. B) Patch CTF using junk masks with the default  of 0.5. C) Patch CTF using junk masks with a  of 0.3.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2Fl05YVslZNgN7DEy08jWv%2Fv5-1-0-junk-mask-fig-9.png?alt=media\&token=f3982204-610b-476f-94d6-a6438d67a0fe)

For this dataset, when Patch CTF estimation was run without excluding junk regions, only 57% of the micrographs were fitted with resolutions better than 5Å, but by excluding the patches that contain >30 % junk, the number of micrographs fit to a resolution of better than 5Å increased to 86%. This improvement means that the defocus values are likely more accurate (which is especially important for small particles that do not benefit from Local CTF Refinement), and furthermore more micrographs are likely to be retained after a Manually Curate Exposures job.

### 3. Exclude junk during Micrograph Denoiser

Large, high contrast junk features such as gold support, or many large ice crystals, can lead to poor denoising results. Using the previously generated **junk mask,** we can exclude the regions inside of the junk mask from denoising (during both training and inference). Example images of a micrograph denoised with, and without applying the junk mask are shown in Figure 10. To enable exclusion of junk masked regions from the denoiser, turn on the `Use junk mask` toggle.

#### Impact of junk masking on Micrograph Denoiser

Junk masking can make a dramatic difference to the quality of denoised micrographs. When junk is not masked, the denoiser can tend to focus on the junk features and erase or dim the particles.

![Figure 10. Example denoising results from a micrograph. A) Denoising without using the junk mask and B) Denoising using the junk mask.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FBXW6ul0uALXDm2ZUedoN%2Fv5-1-0-junk-mask-fig-10.png?alt=media\&token=c95d9afa-8e48-4ec0-aa68-c766373476ba)

### 4. Junk masking in particle picking (Blob Picker, Template Picker)

In CryoSPARC v5.1, Blob and Template Picker jobs can optionally exclude particle picks using the junk mask. Specifically, particle picks whose centers fall within the **junk mask hard edge** (see Figure 2) will be excluded, when the `Use junk mask` toggle is on.

{% hint style="info" %}
Note that when picking on micrographs that underwent denoising with junk masking on, it is best to turn on `Use junk mask` in picking as well, to avoid picks in the masked regions of the micrographs.
{% endhint %}

#### Impact of junk masking in particle picking

In Figure 11 we show example images of particle picks and the NCC score / power score plot for Blob Picker, and Template Picker on denoised micrographs with, and without **junk masks** applied. Junk masking can dramatically improve the quality of particle picking, and help avoid the need to manually exclude picks on junk regions.

![Figure 11. Example micrograph images from particle picking and Inspect Picks.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FA8QqFDGm8dwDccW5J1Sy%2Fv5-1-0-junk-mask-fig-11.png?alt=media\&token=de35a640-0842-4612-ab99-eb4ccdef5ace)

### 5. Junk masking and particle rejection in Extract From Micrographs

[Extract from Micrographs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs) extracts particle boxes from the original, non-denoised micrographs which are not masked by the junk mask. This means that extracted particle boxes near junk regions may still contain junk features, even if junk masking was used up to the point of extraction. This is especially possible if particles are prone to clustering near junk, or if the box size is relatively large compared to the particle.

In CryoSPARC v5.1, we introduce “**junk filling**” during particle extractoin, which uses the junk mask to replace junk features that may be leftover within the extraction box with random noise. The noise is tuned to match the statistics of the noise in the particle images. Junk filling allows particles near junk regions to be retained and used in downstream processing, where previously the junk features would have interfered with particle alignment. Junk filling can help with retention of particles through curation in 2D and 3D, as well as improving pose assignment of particles that reside close to junk features. To enable this option, turn on `Noise-fill junk regions` in Extract from Micrographs.

in v5.1, Extract from Micrographs also provides an option `Reject particles near junk` that, similar to the Blob and Template Picker jobs, rejects particles that are too close to the junk region. Specifically, rejection is based on distance to the **junk mask soft edge** in Extract from Micrographs jobs. This option is important to set to exclude particles that are too close to the junk regions to be junk-filled without also erasing the target protein signal. You can adjust the `Rejection distance from junk mask (A)` as appropriate for your target molecule (see below section and Figure 13 and 14).

An image of the job builder for Extract From Micrographs, and a diagram explaining what happens to particles and junk mask regions is shown in Figure 12.

![Figure 12. Image of the new job parameters in Extract From Micrographs and a diagram explaining the boundaries for noise-filling of junk regions, and rejection of particles.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FHSnFZSIFYjnVOCnJUmrC%2Fv5-1-0-junk-mask-fig-12.png?alt=media\&token=9b48cc83-286a-442a-9f44-ba76f3ddd6a4)

#### Selecting the correct particle rejection distance from the junk mask

It is important to consider the diameter of your particle when using junk filling. To avoid retaining particles that are so near to junk features that the target protein signal itself would get erased by junk filling, the `Rejection distance from junk mask (A)` parameter should ideally be set to the maximum radius of your particle plus a buffer of \~10 Å to account for slightly off-centre coordinates.

{% hint style="info" %}
A histogram plot is produced by the Extract From Micrographs job, to indicate the quantity of noise-filled pixels in the extracted particles (see Figure 14) which can help guide the choice of rejection distance for particles close to junk.
{% endhint %}

By default, the `Rejection distance from junk mask (A)` is set to 100 Å, which is a conservative value that might be appropriate for a large particle with a diameter of \~ 180-190 Å. Figure 13 and 14 show what happens during junk filling if the rejection distance is set to:

* **A:** slightly larger than the target particle’s radius (ideal for junk filling)
* **B:** larger than half the width of the extraction box (junk filling has minimal effect)
* **C**: smaller than the particle radius (protein signal is erased by junk filling)

![Figure 13. Diagrams showing the effect of different rejection distances on the quantity of noise-filled junk regions in extracted boxes. The particle extraction box is indicated with a black box, and the particle radius is indicated with a circle and white arrow. A) When the rejection distance is slightly larger than the particle radius, so that particles close to junk can be extracted with noise filling the junk mask region. B) When the rejection distance is larger than half of the box width, only particles with very little or no overlap with the junk mask are extracted. C) When the rejection distance is smaller than the particle radius, parts of the particle signal might be replaced with random noise.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FffeVKS6j1hZZSVpv49pM%2Fv5-1-0-junk-mask-fig-13.png?alt=media\&token=e11f16d4-c533-4b87-b84a-f3bf64e0f5cd)

![Figure 14. Example outputs from Extract From Micrographs. A) Junk regions filled with noise, and particles rejected < 60 Å from the junk mask soft edge. B) Junk regions filled with noise, and particles rejected < 140 Å from the junk mask soft edge. C) Junk regions filled with noise, and particles rejected < 10 Å from the junk mask soft edge.](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FwQ8czl85STNvrjsjocM4%2Fv5-1-0-junk-mask-fig-14.png?alt=media\&token=fc036bd7-e7d2-45ea-ab73-9ba242461f47)

### 6. Junk masking in Reference-Based Motion Correction (BETA)

For the same reasons as for Extract From Micrographs, when particles are re-extracted during [Reference Based Motion Correction](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta), there may be scenarios where filling the junk region with noise might help retain more particles that are near to junk, and mean that particles near to junk are less prone to alignment being biased by the high contrast junk features.

As in Extract from Micrographs, you can:

1. Fill the junk mask area with noise, by switching on the toggle for `Noise-fill junk regions`
2. Reject particles close to the **junk mask** **soft edge by** switching on the `Reject particles near junk` toggle and adjust the `Rejection distance from junk mask (A)` parameter as appropriate for your target molecule.

The same considerations apply for the rejection distance in Reference-Based Motion Correction as those outlined in Section 5.

## Acknowledgements

We would like to thank Dr. Talya Levitz at the Dana-Farber Cancer Institute, Boston, for generously sharing data with us that was used in the example images of this tutorial.