7. Metadata Tables¶

Mass spectra and other entities described in mzPeak are complex, multi-faceted entities. Each facet is described by a data_kind associated with the entity_type. The columns of a metadata table can vary widely based upon context, but they have the following rules:

For data_kind == metadata: - The first column MUST be named index and be an integer value that MUST be unique. It SHOULD also be sorted and incremented by 1 for each new entity. This is akin to a primary key in relational databases.

For all other kinds: - The first column MUST be source_index and be an integer value that references the index of the metadata table for the instance that this row is associated. There may be zero or more rows with the same source_index value. This is akin to a foreign key in relational databases. - There MAY be another column or a set of columns that function as a primary or composite primary key. This will be defined by the particular file's schema.

In addition, all data_kind must support the following: - There MAY be a parameters column that is a list column as described 7.3.1 below in the 7.3 Controlled Vocabulary Terms section. If there are nested columns, each struct-like group may have its own parameters column. - Additional columns described in the index file's column mapping may be present that map a controlled vocabulary term to a column as described in 7.3 Controlled Vocabulary Terms section. - Additional columns may be defined specifically for a combination of entity_type and data_kind. - If a column value is null, treat its value as absent from that row. The index or source_index columns' values MUST not be null.

7.1 Column naming¶

When a column is not otherwise controlled by the schema, it is assumed to map to a parameter. The schemas that follow metadata tables in this document will use specific names associated with CURIEs. These are the canonical names for those parameters, and writers SHOULD use them and readers SHOULD expect them. While the correct meaning can always be recovered by using the 7.3.3 column mapping process described below, some readers may be treating mzPeak component files like generic Parquet files, and will expect a stable, consistent schema. Column names should be produced from controlled vocabulary terms by replacing any characters matching the regex /[^a-zA-Z0-9_\\-]+/ with a space, prefixing non-standardized names with opt_.

7.2 Examples¶

7.2.1 Example 1: entity_type=spectrum data_kind=metadata¶

This is the primary metadata table for mass spectra. There is an index column that uniquely identifies rows.

7.2.2 Example 2: entity_type=spectrum data_kind=selected_ions¶

This is a secondary metadata file, it has source_index column. This identifies the index value of the spectrum row in the table above in 7.2.1 Example 1. It has a second key, precursor_index denoting the index of the precursor spectrum that the selected ion was isolated in. When there is only one selected ion per spectrum, this will look unique, but this isn't guaranteed!

7.3 Controlled Vocabulary Terms¶

Like mzML, mzPeak makes heavy use of controlled vocabularies to represent rich metadata. mzPeak uses CV terms in three ways:

1. As columns. When a term is used as a column name, that column's values are either the defined value of the term's expected type (e.g. via has_value_type) or a CURIE for a child of the column-name term. For example:
   • The column spectrum_representation (MS:1000525) holds CURIEs for a child term — MS:1000127 "centroid spectrum" or MS:1000128 "profile spectrum" — as appropriate for the spectrum in that row. If the CURIE is null, then no value is added for that term. These 7.3.3 column mappings MUST set term_marker to true and use a string or large string value type for storage. For standardized columns defined in this document, the term_marker MAY be omitted.
   • If multiple instances of a particular parent term without a has_value_type relationship are needed to describe the same row. Map both terms to columns, use a column mapped to the second term itself with a boolean value. A true value there indicates the presence of the value-less term and false or null indicate its absence. These 7.3.3 column mappings MUST set term_marker to true.
     • This is useful for cases like dissociation_method (MS:1000044) being used to express the primary dissociation method MS:1000598|electron transfer dissociation but also need to record a supplementary dissociation method MS:1002679|supplemental collision-induced dissociation.
   • The column ms_level (MS:1000511) holds an integer value.
2. As structural elements. In several places — such as the array index — CURIEs reference named concepts that explain the semantics of a data structure without changing its shape.
3. As pluggable metadata carriers in parameters arrays, analogous to <cvParam/> in mzML. Every schema facet of a metadata table may carry a parameters column. Writers MUST NOT write the same parameter to both the parameters list and as a column to avoid differences in interpretation depending upon which copy is promoted, or if the two values would need to be considered in combination to understand the semantics.

7.3.1 The parameters list¶

The parameters column may be present in any struct-like group of a metadata table. It MUST be a list of the following schema:

group field_id=-1 parameters (List) {
  repeated group field_id=-1 list {
    group field_id=-1 item {
      group field_id=-1 value {
        optional int64   field_id=-1 integer;
        optional double  field_id=-1 float;
        optional binary  field_id=-1 string (String);
        optional boolean field_id=-1 boolean;
      }
      optional binary field_id=-1 accession (String);
      binary field_id=-1 name (String);
      optional binary field_id=-1 unit (String);
    }
  }
}

The parameters.list.item.value group has a slot for each data type, so a parameter can take exactly one of these value types. Unused slots MUST be null. A small number of PSI-MS-defined parameters may have carry a list of values instead of a scalar. When list-valued parameters are NOT promoted to columns, their values MUST be serialized as JSON strings. Parquet's variant type is still experimental and not widely supported. As the variant type's support matures, value storage for parameters may change to use it in the future.

A parameter MAY carry a unit, given as a CV-term CURIE in parameters.list.item.unit. Units are currently drawn from the Unit Ontology (UO) and the PSI-MS controlled vocabulary. As with mzML's <userParam/>, an uncontrolled parameter may be stored simply by leaving parameters.list.item.accession empty.

7.3.2 Typing parameter values¶

When a term has a value and it is stored in the parameters list, the value MUST use one of the value types in the fixed schema above. When a term is instead promoted to its own column, the value may use any Parquet data type. This flexibility lets writers pick an appropriate size and precision, but it can create redundant reader code — a column expected to hold an integer might be written as an 8-, 16-, 32-, or 64-bit integer, signed or unsigned. Some languages handle this naturally with dynamic typing or generics; others need a separate implementation per physical type.

Recommended physical types:

• Integral values that are not indices or identifiers — prefer signed 32- or 64-bit, as needed to cover the value's domain.
• Indices or identifiers represented as integers — prefer unsigned 32- or 64-bit. Dictionary encoding reduces most cases to ~8 bits per variant on disk anyway if the number of unique values is relatively small.
• Floating-point values — prefer 64-bit doubles unless precision is truly of no concern. For repetitive values (e.g. collision energy), dictionary encoding drops the on-disk cost well below 32 bits per value.
• Strings and lists — prefer 64-bit offsets ("large strings"/"large lists") when defining the embedded Arrow schema, but write code that supports both 32- and 64-bit offsets. This matters especially for strings, where the offset is a byte offset, not an item offset.

7.3.3 Column mapping¶

When a CV-term concept is represented as a column, the column name SHOULD unless otherwise stated be named according to the following rules and have a corresponding entry in the file's file index's column mappings.

1. If defined by the schema in this standard, match the expected name, often derived from the CV-term's name.
2. Otherwise, begin with the prefix opt_ followed by a unique name that is descriptive of the value being stored.
   1. If the value is a CV-term, the term's name with non-identifier-safe characters (/[^a-zA-Z0-9_\\-]+/) replaced with _
   2. If not, provide as succinct unique name in the column name after the opt_ prefix, with a more complete name defined in the column mapping.

7.3.3.1 For Writers: Matching Rules¶

Some column mappings produce wide rules matching multiple terms, while others match a narrow set of terms. While it is not reasonable to define a complete topological sorting of column mapping rules, writers are encouraged to match narrow column mappings to controlled vocabulary terms before matching wider rules. Care is needed when dealing with overlapping term trees such as with MS:1000044|dissociation method. See Example 5 for what this might look like.

7.3.3.2 Traversing a column mapping instruction¶

Column mappings' .path attribute tells the reader where to find the column corresponding to the parameter. As described in the index documentation, the path skips over the layers of the file schema denoting lists and elements. For top-level objects, this is the column name itself.

Here are two scans table column mappings (JSON)

{
  "name": "preset scan configuration",
  "path": "preset_scan_configuration",
  "accession": "MS:1000616"
},
{
  "name": "scan window lower limit",
  "path": "scan_windows.scan_window_lower_limit",
  "accession": "MS:1000501",
  "unit": "MS:1000040"
}

They refer to this Parquet schema:

required group scan_schema {
  int64 source_index (Int(bitWidth=64, isSigned=false));
  optional int64 scan_index (Int(bitWidth=64, isSigned=false));
  optional double scan_start_time;
  optional int32 preset_scan_configuration (Int(bitWidth=32, isSigned=false)); <<< first mapping, `preset_scan_configuration`
  optional binary filter_string (String);
  optional float ion_injection_time;
  optional double ion_mobility_value;
  optional binary ion_mobility_type (String);
  optional int32 instrument_configuration_id (Int(bitWidth=32, isSigned=false));
  optional binary spectrum_reference (String);
  ...
  optional group scan_windows (List) {
    repeated group list {
      optional group item {
        optional float scan_window_lower_limit; <<< second mapping, `scan_windows.scan_window_lower_limit` skips the `list.item` tokens.
        optional float scan_window_upper_limit;
      }
    }
  }
}

With the table rendered in example 3.

7.3.3.2.1 Example 3: entity_type=spectrum data_kind=scans¶

In the first case, preset scan configuration (MS:1000616) maps to preset_scan_configuration with values [1, 2, 3, 4, ...]. The second is more complex as it maps scan lower limit (MS:1000501) to a column nested under of the scan_windows list with values [200, 200, 210, 220, ...]. This arrangement encourages readers to use some form of tree traversal approach.

7.3.3.2.2 Example 4: entity_type=spectrum data_kind=metadata¶

This example returns to the main spectrum metadata table, adding an example term_marker column. The opt_calibration_spectrum column is mapped to the term MS:1000928|calibration spectrum. When the column contains the value true, this term MUST be present in that row's spectrum, otherwise it MUST be absent, not indicated through any other method. In this example, the denoted spectra are used solely to measure the lock mass for post-acquisition mass calibration, and contain no analytes of interest otherwise and could be ignored by another piece of software not interested in calibration spectra.

[
  {
    "name": "ms level",
    "path": "ms_level",
    "accession": "MS:1000511"
  },
  {
    "name": "scan polarity",
    "path": "scan_polarity",
    "accession": "MS:1000465"
  },
  {
    "name": "spectrum representation",
    "path": "spectrum_representation",
    "accession": "MS:1000525",
    "term_marker": true
  },
  {
    "name": "spectrum type",
    "path": "spectrum_type",
    "accession": "MS:1000559",
    "term_marker": true
  },
  {
    "name": "calibration spectrum",
    "path": "opt_calibration_spectrum",
    "accession": "MS:1000928",
    "term_marker": true
  }
]

7.3.3.2.3 Example 5: entity_type=spectrum data_kind=precursors¶

This example visits a subset of the precursor metadata table for a triggered EThcD run. It shows examples of term_marker columns denoting child terms in activation.dissociation_method and term_marker columns denoting presence/absence in activation.opt_MS_1002678_suppl_beam_disc. The former tells the reader what kind of dissociation method was used on each scan, alternating between MS:1000133|collision-induced dissociation, MS:1000422| beam-type collision-induced dissociation and MS:1000598|electron transfer dissociation, the primary dissociation method. The latter marks which rows have the MS:1002678|supplemental beam-type collision-induced dissociation in addition to the dissociation_method column's value. Because MS:1002678 is a child term of MS:1000044, it needs to be defined using term_marker with a boolean value to avoid having two columns tied to MS:1000044 directly. This also lets the reader now quickly filter on rows which contain a supplemental dissociation method directly. Without it the supplemental dissociation method term would need to be stored in the parameters list, and be invisible to most basic query mechanisms. Some but not all query engines support list traversal, but such operations are more complicated to specify and have to deal with the less efficient storage model of the parameters list in any case. Both of these columns are nested under the activation. prefix, which is a group in the Parquet schema. This is an organizational nicety for Parquet, but when translated to Arrow, this produces a struct type column with multiple children. Care MUST be taken to support these when reading column statistics.

[
  {
    "name": "dissociation method",
    "path": "activation.dissociation_method",
    "accession": "MS:1000044",
    "term_marker": true
  },
  {
    "name": "collision energy",
    "path": "activation.collision_energy",
    "accession": "MS:1000045",
    "unit": "UO:0000266"
  },
  {
    "name": "supplemental beam-type collision-induced dissociation",
    "path": "activation.opt_MS_1002678_suppl_beam_disc",
    "accession": "MS:1002678",
    "term_marker": true
  }
]

7.4 Column statistics¶

Parquet can store statistics, including the minimum value, maximum value, and null count at the row group and data page level for each column. Writers MUST include them. This makes it easy to determine the minimum and maximum value for any given parameter. For instance, inferring the m/z ranges covered by an acquisition can be done by reading the minimum lowest observed m/z (MS:1000528) and maximum highest observed m/z (MS:1000527) statistics without needing to traverse the entire list of values in both columns.