itemhas a type, for example an exposure, particle, volume, mask, etc. Every
itemalso has properties, with each property having a name (eg.
ctf) and sub-properties containing actual metadata values (eg.
ctf/astigmatism, etc). Collections of
items with the same type and same properties, constitute a
datasetis essentially a table, where each row is a single item, and columns are properties/sub-properties. Every job can only input and output
datasets. Therefore every type of data/metadata is stored in a
dataset. On disk,
datasets are stored in the
.csfile format, which is a binary
numpyformat descibed in a later section.
result. For example, a CTF estimation job would output a
ctfresult, containing sub-properties like defocus, astigmatism, spherical abberation, etc. The
results that a job outputs are the basic component of what gets connected to other jobs.
result-groups - each one is a set of
results that describe the same type of
item. Thus a job can output a
result-groupdefining particles, with two
input-groups each allowing a certain type of
itemlike particles, volumes, etc.
slots, each taking in a particular kind of
result. For example, an
input-grouptaking in particles may have a
ctfand another for
input-groupalso defines the number of different
result-groups that can be connected to it. In general all the items from all
result-groups that are connected are appended together to make one larger
datasetthat forms the input to the job. So for example, connecting two particle stacks to a single
input-groupwill cause those stacks to be appended together.
result-groups, etc is so that in CryoSPARC, most connections between jobs can be made simply at the
grouplevel, without having to specify particular files, paths, columns or rows in tables or text files. Subsets of
datasets can be easily defined and passed around, and different subsets can be joined together as inputs to a further job. For advanced uses, however, the lower-level
results allow a user to connect only certain metadata about an item from one job to another, or override the metadata for certain properties in a
result-group. Examples of how and when to use this capability follow.
results that it doesn't actually need, and then output those
results as "passthrough" metadata that is not read or modified by the job, but just passed along in its output so that subsetquent jobs can use it without needing to be manually connected to an earlier output in the chain.
uid(a 64-bit integer) that is used to maintain correspondences across chains of processing jobs and to ensure that regardless of the order that a job outputs items, the properties of each items are always correctly assigned to the correct item.
uidis an array of
Cstructures, implemented using
numpystructured arrays. These arrays are stored in memory and on disk in the same format. On disk, we store these arrays in binary format in
.csfile in CryoSPARC contains a single table. Each row corresponds to a single item. A
.csfile must contain a column for the
uidof each item, and further columns define properties/sub-properties of that item. Multiple
.csfiles therefore can be used in aggregate to define all the properties of a set of items, since the rows in every table all have a
uidthat can be used to join the tables. In general, when multiple tables are used to specify a dataset, the dataset contains only the intersection of items included in each table.
half_map_Binputs from different jobs. The example below outlines the three step process of using one input group in the local resolution estimation job builder to populate volume data from three separate jobs.
particles.blobinput slot with the non-downsampled data that you previously connected as an input group.