Add processing:version and processing:datetime#32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
m-mohr merged 3 commits into from
May 9, 2024
Merged

Add processing:version and processing:datetime #32

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bde9911
Add processing:version and processing:datetime
m-mohr Apr 4, 2024
a6f6b15
Merge branch 'main' into add-version-time
m-mohr Apr 4, 2024
9746a0f
Various clarifications, added new rel type
m-mohr May 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `processing:version` field to describe the primary software version of workflow version that produced the data
- `processing:datetime` field to describe when the processing happened
- `processing-execution` relation type to link to the processing execution that produced the data.
- `processing-software` relation type to link to the processing execution that produced the data.

### Changed

### Deprecated

### Removed

### Fixed

## [v1.1.0] - 2022-01-07

Expand Down
55 changes: 47 additions & 8 deletions README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -22,38 +22,76 @@ and therefore are shared across all items, it is recommended adding the fields t
- [JSON Schema](json-schema/schema.json)
- [Changelog](./CHANGELOG.md)

## Item Properties and Collection Provider Fields
## Fields

| Field Name | Type | Description |
| ----------------------- | ------------------- | ----------- |
| processing:expression | [Expression Object](#expression-object) | An expression or processing chain that describes how the data has been processed. Alternatively, you can also link to a processing chain with the relation type `processing-expression` (see below). |
| processing:lineage | string | Lineage Information provided as free text information about the how observations were processed or models that were used to create the resource being described [NASA ISO](https://wiki.earthdata.nasa.gov/display/NASAISO/Lineage+Information). For example, `GRD Post Processing` for "GRD" product of Sentinel-1 satellites. [CommonMark 0.29](https://commonmark.org/) syntax MAY be used for rich text representation. |
| processing:level | string | The name commonly used to refer to the processing level to make it easier to search for product level across collections or items. The short name must be used (only `L`, not `Level`). See the [list of suggested processing levels](#suggested-processing-levels). |
| processing:facility | string | The name of the facility that produced the data. For example, `Copernicus S1 Core Ground Segment - DPA` for product of Sentinel-1 satellites. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more softwares that produced the data. For example, `"Sentinel-1 IPF":"002.71"` for the software that produces Sentinel-1 satellites data. |
| processing:datetime | string | Processing date and time of the corresponding data formatted according to [RFC 3339, section 5.6](https://tools.ietf.org/html/rfc3339#section-5.6), in UTC. |
| processing:version | string | The version of the primary processing software or processing chain that produced the data. For example, this could be the processing baseline for the Sentinel missions. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more applications or libraries that were involved during the production of the data for provenance purposes. |
Comment thread
m-mohr marked this conversation as resolved.

These fields can be used in a variety of places:
The fields in the table above can be used in these parts of STAC documents:
- [ ] Catalogs
- [ ] Collections
- [x] [Collection Provider](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
- [x] Item Properties (incl. Summaries in Collections)
- [x] Assets (for both Collections and Items, incl. Item Asset Definitions in Collections)
- [ ] Links

In more detail, the following restrictions apply:

1. Items:
- The fields are placed in the properties. At least one field is required to be present.
- The fields are usually placed in the properties. At least one field is required to be present.
- Additionally, STAC allows all fields to be used in the Asset Object.

2. Collections:
- The fields are usually placed in the [Provider Objects](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
for the `providers` that have the role `producer` or `processor` assigned.
They don't need to be provided for all providers of the respective role.
- The fields can also be used in `summaries`, Collection `assets` or Item asset definitions (`item_assets`).
Please note that the JSON Schema is not be able to validate the values of Collection summaries.

If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.
Please note that the JSON Schema is not be able to validate the values of Collection summaries.
If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.

### Processing Date Time

The time of the processing is directly specified via the `created` properties of the target asset as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time)
The time of the processing can be specified as a global field in `processing:datetime`,
but it can also be specified directly and individually via the `created` properties of the target asset
as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time).

`created` in Item properties describes the STAC metadata creation and in Assets it describes the creation of the data files.
Thus the timestamps provided in Item Properties for `created` and `processing:datetime` may differ.
As Item properties are easier to be indexed and used for filtering purposes, `processing:datetime` exists.
`created` and `processing:datetime` should usually be the same value in Assets and as such `processing:datetime`
can usually be omitted.

### Version Numbers

Three fields exist for version numbers:
- `processing:software`
- `processing:version`
- `version` (in the [Version extension](https://github.com/stac-extensions/version))

The different fields exist to give data providers more flexibility depending on their needs.

In Item Properties:
- `processing:version` is useful if a single version number is available for the metadata or data that users should be able to filter on.
A popular example for this is the processing baseline in Sentinel missions.
- `processing:software` is used if the software libraries/tools are important to know, but it's not important to filter on them.
They are mostly informative and important to be complete for reporducibility purposes.
Thus, the values in the object can not just be version numbers, but also be e.g. tag names, commit hashes or similar.
For example, you could expose a simplified version of the `Pipfile.lock` (Python) or `package-lock.json` (NodeJS).
If you need more information, you could also link to such files via the relation type `processing-software`.
- `version` is usually not used in the context of processing and describes the version of the metadata.

### Linking the Items

In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing. They could be used to trace back the processing history of the dataset.
In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing.
They could be used to trace back the processing history of the dataset.

### Suggested Processing Levels

Expand DownExpand Up@@ -99,6 +137,7 @@ The following types should be used as applicable `rel` types in the
| derived_from | URL to a STAC Item that was used as input data in the creation of this Item. |
| processing-expression | A processing chain (or script) that describes how the data has been processed. |
| processing-execution | URL to any resource representing the processing execution (e.g. OGC Process API). |
| processing-software | URL to any resource that identifies the software and versions used for processing the data, e.g. a `Pipfile.lock` (Python) or `package-lock.json` (NodeJS). |

## Contributing

Expand Down
10 changes: 4 additions & 6 deletions examples/collection.json
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -17,11 +17,9 @@
],
"url": "https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi",
"processing:lineage": "Generation of Level-1C User Product",
"processing:level": "L1C",
"processing:level": "L1",
"processing:facility": "Copernicus S2 Processing and Archiving Facility",
"processing:software": {
"IPF-S2L1C": "02.06"
}
"processing:version": "02.06"
},
{
"name": "Processing Corp.",
Expand DownExpand Up@@ -82,8 +80,8 @@
60
],
"processing:level": [
"L1C",
"L2A"
"L1",
"L2"
]
},
"links": [
Expand Down
5 changes: 3 additions & 2 deletions examples/item.json
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -29,11 +29,12 @@
],
"sar:product_type": "GRD",
"processing:lineage": "GRD Post Processing",
"processing:level": "L1C",
"processing:level": "L1",
"processing:facility": "Copernicus S1 Core Ground Segment - DPA",
"processing:software": {
"Sentinel-1 IPF": "002.71"
}
},
"processing:datetime": "2016-08-23T00:30:33Z"
},
"links": [
{
Expand Down
56 changes: 32 additions & 24 deletions json-schema/schema.json
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -3,6 +3,18 @@
"$id": "https://stac-extensions.github.io/processing/v1.1.0/schema.json#",
"title": "Processing Extension",
"description": "STAC Processing Extension for STAC Items and STAC Collections.",
"type": "object",
"required": [
"stac_extensions"
],
"properties": {
"stac_extensions": {
"type": "array",
"contains": {
"const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
}
}
},
"anyOf": [
{
"$comment": "This is the schema for STAC Items.",
Expand DownExpand Up@@ -33,12 +45,7 @@
"$ref": "#/definitions/fields"
}
}
},
"allOf": [
{
"$ref": "#/definitions/stac_extensions"
}
]
}
},
{
"$comment": "This is the schema for STAC Collections.",
Expand DownExpand Up@@ -72,11 +79,6 @@
}
}
},
"allOf": [
{
"$ref": "#/definitions/stac_extensions"
}
],
"anyOf": [
{
"$comment": "Requires at least one provider to contain processing fields.",
Expand DownExpand Up@@ -170,18 +172,6 @@
],
"definitions": {
"stac_extensions": {
"type": "object",
"required": [
"stac_extensions"
],
"properties": {
"stac_extensions": {
"type": "array",
"contains": {
"const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
}
}
}
},
"require_provider_role": {
"type": "object",
Expand All@@ -206,7 +196,9 @@
{"type": "object", "required": ["processing:lineage"]},
{"type": "object", "required": ["processing:level"]},
{"type": "object", "required": ["processing:facility"]},
{"type": "object", "required": ["processing:software"]}
{"type": "object", "required": ["processing:software"]},
{"type": "object", "required": ["processing:version"]},
{"type": "object", "required": ["processing:datetime"]}
]
},
"fields": {
Expand DownExpand Up@@ -257,6 +249,22 @@
"Copernicus S1 Core Ground Segment - DPA"
]
},
"processing:version": {
"title": "Processing Version",
"type": "string",
"examples": [
"0.2.0"
]
},
"processing:datetime": {
"title": "Processing Datetime",
"type": "string",
"format": "date-time",
"pattern": "(\\+00:00|Z)$",
"examples": [
"2020-01-05T12:34:55Z"
]
},
"processing:software": {
"title": "Processing Software Name / version",
"type": "object",
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Add processing:version and processing:datetime#32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

m-mohr
merged 3 commits into

from

May 9, 2024
 Merged


Add processing:version and processing:datetime
#32
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
bde9911

Add processing:version and processing:datetime


m-mohr Apr 4, 2024
a6f6b15

Merge branch 'main' into add-version-time


m-mohr Apr 4, 2024
9746a0f

Various clarifications, added new rel type


m-mohr May 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


11 changes: 11 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `processing:version` field to describe the primary software version of workflow version that produced the data
- `processing:datetime` field to describe when the processing happened
- `processing-execution` relation type to link to the processing execution that produced the data.
- `processing-software` relation type to link to the processing execution that produced the data.

### Changed

### Deprecated

### Removed

### Fixed

## [v1.1.0] - 2022-01-07

Expand Down

55 changes: 47 additions & 8 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -22,38 +22,76 @@ and therefore are shared across all items, it is recommended adding the fields t
- [JSON Schema](json-schema/schema.json)
- [Changelog](./CHANGELOG.md)

## Item Properties and Collection Provider Fields
## Fields

| Field Name | Type | Description |
| ----------------------- | ------------------- | ----------- |
| processing:expression | [Expression Object](#expression-object) | An expression or processing chain that describes how the data has been processed. Alternatively, you can also link to a processing chain with the relation type `processing-expression` (see below). |
| processing:lineage | string | Lineage Information provided as free text information about the how observations were processed or models that were used to create the resource being described [NASA ISO](https://wiki.earthdata.nasa.gov/display/NASAISO/Lineage+Information). For example, `GRD Post Processing` for "GRD" product of Sentinel-1 satellites. [CommonMark 0.29](https://commonmark.org/) syntax MAY be used for rich text representation. |
| processing:level | string | The name commonly used to refer to the processing level to make it easier to search for product level across collections or items. The short name must be used (only `L`, not `Level`). See the [list of suggested processing levels](#suggested-processing-levels). |
| processing:facility | string | The name of the facility that produced the data. For example, `Copernicus S1 Core Ground Segment - DPA` for product of Sentinel-1 satellites. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more softwares that produced the data. For example, `"Sentinel-1 IPF":"002.71"` for the software that produces Sentinel-1 satellites data. |
| processing:datetime | string | Processing date and time of the corresponding data formatted according to [RFC 3339, section 5.6](https://tools.ietf.org/html/rfc3339#section-5.6), in UTC. |
| processing:version | string | The version of the primary processing software or processing chain that produced the data. For example, this could be the processing baseline for the Sentinel missions. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more applications or libraries that were involved during the production of the data for provenance purposes. |
Comment thread
m-mohr marked this conversation as resolved.


These fields can be used in a variety of places:
The fields in the table above can be used in these parts of STAC documents:
- [ ] Catalogs
- [ ] Collections
- [x] [Collection Provider](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
- [x] Item Properties (incl. Summaries in Collections)
- [x] Assets (for both Collections and Items, incl. Item Asset Definitions in Collections)
- [ ] Links

In more detail, the following restrictions apply:

1. Items:
 - The fields are placed in the properties. At least one field is required to be present.
 - The fields are usually placed in the properties. At least one field is required to be present.
 - Additionally, STAC allows all fields to be used in the Asset Object.

2. Collections:
 - The fields are usually placed in the [Provider Objects](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
 for the `providers` that have the role `producer` or `processor` assigned.
 They don't need to be provided for all providers of the respective role.
 - The fields can also be used in `summaries`, Collection `assets` or Item asset definitions (`item_assets`).
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.

 If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.
If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.

### Processing Date Time

The time of the processing is directly specified via the `created` properties of the target asset as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time)
The time of the processing can be specified as a global field in `processing:datetime`,
but it can also be specified directly and individually via the `created` properties of the target asset
as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time).

`created` in Item properties describes the STAC metadata creation and in Assets it describes the creation of the data files.
Thus the timestamps provided in Item Properties for `created` and `processing:datetime` may differ.
As Item properties are easier to be indexed and used for filtering purposes, `processing:datetime` exists.
`created` and `processing:datetime` should usually be the same value in Assets and as such `processing:datetime`
can usually be omitted.

### Version Numbers

Three fields exist for version numbers:
- `processing:software`
- `processing:version`
- `version` (in the [Version extension](https://github.com/stac-extensions/version))

The different fields exist to give data providers more flexibility depending on their needs.

In Item Properties:
- `processing:version` is useful if a single version number is available for the metadata or data that users should be able to filter on.
 A popular example for this is the processing baseline in Sentinel missions.
- `processing:software` is used if the software libraries/tools are important to know, but it's not important to filter on them.
 They are mostly informative and important to be complete for reporducibility purposes.
 Thus, the values in the object can not just be version numbers, but also be e.g. tag names, commit hashes or similar.
 For example, you could expose a simplified version of the `Pipfile.lock` (Python) or `package-lock.json` (NodeJS).
 If you need more information, you could also link to such files via the relation type `processing-software`.
- `version` is usually not used in the context of processing and describes the version of the metadata.

### Linking the Items

In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing. They could be used to trace back the processing history of the dataset.
In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing.
They could be used to trace back the processing history of the dataset.

### Suggested Processing Levels

Expand DownExpand Up@@ -99,6 +137,7 @@ The following types should be used as applicable `rel` types in the
| derived_from | URL to a STAC Item that was used as input data in the creation of this Item. |
| processing-expression | A processing chain (or script) that describes how the data has been processed. |
| processing-execution | URL to any resource representing the processing execution (e.g. OGC Process API). |
| processing-software | URL to any resource that identifies the software and versions used for processing the data, e.g. a `Pipfile.lock` (Python) or `package-lock.json` (NodeJS). |

## Contributing

Expand Down

10 changes: 4 additions & 6 deletions
examples/collection.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,11 +17,9 @@
 ],
 "url": "https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi",
 "processing:lineage": "Generation of Level-1C User Product",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S2 Processing and Archiving Facility",
 "processing:software": {
 "IPF-S2L1C": "02.06"
 }
 "processing:version": "02.06"
 },
 {
 "name": "Processing Corp.",
Expand DownExpand Up@@ -82,8 +80,8 @@
 60
 ],
 "processing:level": [
 "L1C",
 "L2A"
 "L1",
 "L2"
 ]
 },
 "links": [
Expand Down

5 changes: 3 additions & 2 deletions
examples/item.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,11 +29,12 @@
 ],
 "sar:product_type": "GRD",
 "processing:lineage": "GRD Post Processing",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S1 Core Ground Segment - DPA",
 "processing:software": {
 "Sentinel-1 IPF": "002.71"
 }
 },
 "processing:datetime": "2016-08-23T00:30:33Z"
 },
 "links": [
 {
Expand Down

56 changes: 32 additions & 24 deletions
json-schema/schema.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -3,6 +3,18 @@
 "$id": "https://stac-extensions.github.io/processing/v1.1.0/schema.json#",
 "title": "Processing Extension",
 "description": "STAC Processing Extension for STAC Items and STAC Collections.",
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 },
 "anyOf": [
 {
 "$comment": "This is the schema for STAC Items.",
Expand DownExpand Up@@ -33,12 +45,7 @@
 "$ref": "#/definitions/fields"
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ]
 }
 },
 {
 "$comment": "This is the schema for STAC Collections.",
Expand DownExpand Up@@ -72,11 +79,6 @@
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ],
 "anyOf": [
 {
 "$comment": "Requires at least one provider to contain processing fields.",
Expand DownExpand Up@@ -170,18 +172,6 @@
 ],
 "definitions": {
 "stac_extensions": {
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 }
 },
 "require_provider_role": {
 "type": "object",
Expand All@@ -206,7 +196,9 @@
 {"type": "object", "required": ["processing:lineage"]},
 {"type": "object", "required": ["processing:level"]},
 {"type": "object", "required": ["processing:facility"]},
 {"type": "object", "required": ["processing:software"]}
 {"type": "object", "required": ["processing:software"]},
 {"type": "object", "required": ["processing:version"]},
 {"type": "object", "required": ["processing:datetime"]}
 ]
 },
 "fields": {
Expand DownExpand Up@@ -257,6 +249,22 @@
 "Copernicus S1 Core Ground Segment - DPA"
 ]
 },
 "processing:version": {
 "title": "Processing Version",
 "type": "string",
 "examples": [
 "0.2.0"
 ]
 },
 "processing:datetime": {
 "title": "Processing Datetime",
 "type": "string",
 "format": "date-time",
 "pattern": "(\\+00:00|Z)$",
 "examples": [
 "2020-01-05T12:34:55Z"
 ]
 },
 "processing:software": {
 "title": "Processing Software Name / version",
 "type": "object",
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add processing:version and processing:datetime#32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

m-mohr
merged 3 commits into

from

May 9, 2024
 Merged


Add processing:version and processing:datetime
#32
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
bde9911

Add processing:version and processing:datetime


m-mohr Apr 4, 2024
a6f6b15

Merge branch 'main' into add-version-time


m-mohr Apr 4, 2024
9746a0f

Various clarifications, added new rel type


m-mohr May 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


11 changes: 11 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `processing:version` field to describe the primary software version of workflow version that produced the data
- `processing:datetime` field to describe when the processing happened
- `processing-execution` relation type to link to the processing execution that produced the data.
- `processing-software` relation type to link to the processing execution that produced the data.

### Changed

### Deprecated

### Removed

### Fixed

## [v1.1.0] - 2022-01-07

Expand Down

55 changes: 47 additions & 8 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -22,38 +22,76 @@ and therefore are shared across all items, it is recommended adding the fields t
- [JSON Schema](json-schema/schema.json)
- [Changelog](./CHANGELOG.md)

## Item Properties and Collection Provider Fields
## Fields

| Field Name | Type | Description |
| ----------------------- | ------------------- | ----------- |
| processing:expression | [Expression Object](#expression-object) | An expression or processing chain that describes how the data has been processed. Alternatively, you can also link to a processing chain with the relation type `processing-expression` (see below). |
| processing:lineage | string | Lineage Information provided as free text information about the how observations were processed or models that were used to create the resource being described [NASA ISO](https://wiki.earthdata.nasa.gov/display/NASAISO/Lineage+Information). For example, `GRD Post Processing` for "GRD" product of Sentinel-1 satellites. [CommonMark 0.29](https://commonmark.org/) syntax MAY be used for rich text representation. |
| processing:level | string | The name commonly used to refer to the processing level to make it easier to search for product level across collections or items. The short name must be used (only `L`, not `Level`). See the [list of suggested processing levels](#suggested-processing-levels). |
| processing:facility | string | The name of the facility that produced the data. For example, `Copernicus S1 Core Ground Segment - DPA` for product of Sentinel-1 satellites. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more softwares that produced the data. For example, `"Sentinel-1 IPF":"002.71"` for the software that produces Sentinel-1 satellites data. |
| processing:datetime | string | Processing date and time of the corresponding data formatted according to [RFC 3339, section 5.6](https://tools.ietf.org/html/rfc3339#section-5.6), in UTC. |
| processing:version | string | The version of the primary processing software or processing chain that produced the data. For example, this could be the processing baseline for the Sentinel missions. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more applications or libraries that were involved during the production of the data for provenance purposes. |
Comment thread
m-mohr marked this conversation as resolved.


These fields can be used in a variety of places:
The fields in the table above can be used in these parts of STAC documents:
- [ ] Catalogs
- [ ] Collections
- [x] [Collection Provider](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
- [x] Item Properties (incl. Summaries in Collections)
- [x] Assets (for both Collections and Items, incl. Item Asset Definitions in Collections)
- [ ] Links

In more detail, the following restrictions apply:

1. Items:
 - The fields are placed in the properties. At least one field is required to be present.
 - The fields are usually placed in the properties. At least one field is required to be present.
 - Additionally, STAC allows all fields to be used in the Asset Object.

2. Collections:
 - The fields are usually placed in the [Provider Objects](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
 for the `providers` that have the role `producer` or `processor` assigned.
 They don't need to be provided for all providers of the respective role.
 - The fields can also be used in `summaries`, Collection `assets` or Item asset definitions (`item_assets`).
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.

 If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.
If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.

### Processing Date Time

The time of the processing is directly specified via the `created` properties of the target asset as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time)
The time of the processing can be specified as a global field in `processing:datetime`,
but it can also be specified directly and individually via the `created` properties of the target asset
as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time).

`created` in Item properties describes the STAC metadata creation and in Assets it describes the creation of the data files.
Thus the timestamps provided in Item Properties for `created` and `processing:datetime` may differ.
As Item properties are easier to be indexed and used for filtering purposes, `processing:datetime` exists.
`created` and `processing:datetime` should usually be the same value in Assets and as such `processing:datetime`
can usually be omitted.

### Version Numbers

Three fields exist for version numbers:
- `processing:software`
- `processing:version`
- `version` (in the [Version extension](https://github.com/stac-extensions/version))

The different fields exist to give data providers more flexibility depending on their needs.

In Item Properties:
- `processing:version` is useful if a single version number is available for the metadata or data that users should be able to filter on.
 A popular example for this is the processing baseline in Sentinel missions.
- `processing:software` is used if the software libraries/tools are important to know, but it's not important to filter on them.
 They are mostly informative and important to be complete for reporducibility purposes.
 Thus, the values in the object can not just be version numbers, but also be e.g. tag names, commit hashes or similar.
 For example, you could expose a simplified version of the `Pipfile.lock` (Python) or `package-lock.json` (NodeJS).
 If you need more information, you could also link to such files via the relation type `processing-software`.
- `version` is usually not used in the context of processing and describes the version of the metadata.

### Linking the Items

In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing. They could be used to trace back the processing history of the dataset.
In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing.
They could be used to trace back the processing history of the dataset.

### Suggested Processing Levels

Expand DownExpand Up@@ -99,6 +137,7 @@ The following types should be used as applicable `rel` types in the
| derived_from | URL to a STAC Item that was used as input data in the creation of this Item. |
| processing-expression | A processing chain (or script) that describes how the data has been processed. |
| processing-execution | URL to any resource representing the processing execution (e.g. OGC Process API). |
| processing-software | URL to any resource that identifies the software and versions used for processing the data, e.g. a `Pipfile.lock` (Python) or `package-lock.json` (NodeJS). |

## Contributing

Expand Down

10 changes: 4 additions & 6 deletions
examples/collection.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,11 +17,9 @@
 ],
 "url": "https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi",
 "processing:lineage": "Generation of Level-1C User Product",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S2 Processing and Archiving Facility",
 "processing:software": {
 "IPF-S2L1C": "02.06"
 }
 "processing:version": "02.06"
 },
 {
 "name": "Processing Corp.",
Expand DownExpand Up@@ -82,8 +80,8 @@
 60
 ],
 "processing:level": [
 "L1C",
 "L2A"
 "L1",
 "L2"
 ]
 },
 "links": [
Expand Down

5 changes: 3 additions & 2 deletions
examples/item.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,11 +29,12 @@
 ],
 "sar:product_type": "GRD",
 "processing:lineage": "GRD Post Processing",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S1 Core Ground Segment - DPA",
 "processing:software": {
 "Sentinel-1 IPF": "002.71"
 }
 },
 "processing:datetime": "2016-08-23T00:30:33Z"
 },
 "links": [
 {
Expand Down

56 changes: 32 additions & 24 deletions
json-schema/schema.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -3,6 +3,18 @@
 "$id": "https://stac-extensions.github.io/processing/v1.1.0/schema.json#",
 "title": "Processing Extension",
 "description": "STAC Processing Extension for STAC Items and STAC Collections.",
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 },
 "anyOf": [
 {
 "$comment": "This is the schema for STAC Items.",
Expand DownExpand Up@@ -33,12 +45,7 @@
 "$ref": "#/definitions/fields"
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ]
 }
 },
 {
 "$comment": "This is the schema for STAC Collections.",
Expand DownExpand Up@@ -72,11 +79,6 @@
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ],
 "anyOf": [
 {
 "$comment": "Requires at least one provider to contain processing fields.",
Expand DownExpand Up@@ -170,18 +172,6 @@
 ],
 "definitions": {
 "stac_extensions": {
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 }
 },
 "require_provider_role": {
 "type": "object",
Expand All@@ -206,7 +196,9 @@
 {"type": "object", "required": ["processing:lineage"]},
 {"type": "object", "required": ["processing:level"]},
 {"type": "object", "required": ["processing:facility"]},
 {"type": "object", "required": ["processing:software"]}
 {"type": "object", "required": ["processing:software"]},
 {"type": "object", "required": ["processing:version"]},
 {"type": "object", "required": ["processing:datetime"]}
 ]
 },
 "fields": {
Expand DownExpand Up@@ -257,6 +249,22 @@
 "Copernicus S1 Core Ground Segment - DPA"
 ]
 },
 "processing:version": {
 "title": "Processing Version",
 "type": "string",
 "examples": [
 "0.2.0"
 ]
 },
 "processing:datetime": {
 "title": "Processing Datetime",
 "type": "string",
 "format": "date-time",
 "pattern": "(\\+00:00|Z)$",
 "examples": [
 "2020-01-05T12:34:55Z"
 ]
 },
 "processing:software": {
 "title": "Processing Software Name / version",
 "type": "object",
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add processing:version and processing:datetime#32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

m-mohr
merged 3 commits into

from

May 9, 2024
 Merged


Add processing:version and processing:datetime
#32
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
bde9911

Add processing:version and processing:datetime


m-mohr Apr 4, 2024
a6f6b15

Merge branch 'main' into add-version-time


m-mohr Apr 4, 2024
9746a0f

Various clarifications, added new rel type


m-mohr May 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


11 changes: 11 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `processing:version` field to describe the primary software version of workflow version that produced the data
- `processing:datetime` field to describe when the processing happened
- `processing-execution` relation type to link to the processing execution that produced the data.
- `processing-software` relation type to link to the processing execution that produced the data.

### Changed

### Deprecated

### Removed

### Fixed

## [v1.1.0] - 2022-01-07

Expand Down

55 changes: 47 additions & 8 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -22,38 +22,76 @@ and therefore are shared across all items, it is recommended adding the fields t
- [JSON Schema](json-schema/schema.json)
- [Changelog](./CHANGELOG.md)

## Item Properties and Collection Provider Fields
## Fields

| Field Name | Type | Description |
| ----------------------- | ------------------- | ----------- |
| processing:expression | [Expression Object](#expression-object) | An expression or processing chain that describes how the data has been processed. Alternatively, you can also link to a processing chain with the relation type `processing-expression` (see below). |
| processing:lineage | string | Lineage Information provided as free text information about the how observations were processed or models that were used to create the resource being described [NASA ISO](https://wiki.earthdata.nasa.gov/display/NASAISO/Lineage+Information). For example, `GRD Post Processing` for "GRD" product of Sentinel-1 satellites. [CommonMark 0.29](https://commonmark.org/) syntax MAY be used for rich text representation. |
| processing:level | string | The name commonly used to refer to the processing level to make it easier to search for product level across collections or items. The short name must be used (only `L`, not `Level`). See the [list of suggested processing levels](#suggested-processing-levels). |
| processing:facility | string | The name of the facility that produced the data. For example, `Copernicus S1 Core Ground Segment - DPA` for product of Sentinel-1 satellites. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more softwares that produced the data. For example, `"Sentinel-1 IPF":"002.71"` for the software that produces Sentinel-1 satellites data. |
| processing:datetime | string | Processing date and time of the corresponding data formatted according to [RFC 3339, section 5.6](https://tools.ietf.org/html/rfc3339#section-5.6), in UTC. |
| processing:version | string | The version of the primary processing software or processing chain that produced the data. For example, this could be the processing baseline for the Sentinel missions. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more applications or libraries that were involved during the production of the data for provenance purposes. |
Comment thread
m-mohr marked this conversation as resolved.


These fields can be used in a variety of places:
The fields in the table above can be used in these parts of STAC documents:
- [ ] Catalogs
- [ ] Collections
- [x] [Collection Provider](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
- [x] Item Properties (incl. Summaries in Collections)
- [x] Assets (for both Collections and Items, incl. Item Asset Definitions in Collections)
- [ ] Links

In more detail, the following restrictions apply:

1. Items:
 - The fields are placed in the properties. At least one field is required to be present.
 - The fields are usually placed in the properties. At least one field is required to be present.
 - Additionally, STAC allows all fields to be used in the Asset Object.

2. Collections:
 - The fields are usually placed in the [Provider Objects](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
 for the `providers` that have the role `producer` or `processor` assigned.
 They don't need to be provided for all providers of the respective role.
 - The fields can also be used in `summaries`, Collection `assets` or Item asset definitions (`item_assets`).
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.

 If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.
If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.

### Processing Date Time

The time of the processing is directly specified via the `created` properties of the target asset as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time)
The time of the processing can be specified as a global field in `processing:datetime`,
but it can also be specified directly and individually via the `created` properties of the target asset
as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time).

`created` in Item properties describes the STAC metadata creation and in Assets it describes the creation of the data files.
Thus the timestamps provided in Item Properties for `created` and `processing:datetime` may differ.
As Item properties are easier to be indexed and used for filtering purposes, `processing:datetime` exists.
`created` and `processing:datetime` should usually be the same value in Assets and as such `processing:datetime`
can usually be omitted.

### Version Numbers

Three fields exist for version numbers:
- `processing:software`
- `processing:version`
- `version` (in the [Version extension](https://github.com/stac-extensions/version))

The different fields exist to give data providers more flexibility depending on their needs.

In Item Properties:
- `processing:version` is useful if a single version number is available for the metadata or data that users should be able to filter on.
 A popular example for this is the processing baseline in Sentinel missions.
- `processing:software` is used if the software libraries/tools are important to know, but it's not important to filter on them.
 They are mostly informative and important to be complete for reporducibility purposes.
 Thus, the values in the object can not just be version numbers, but also be e.g. tag names, commit hashes or similar.
 For example, you could expose a simplified version of the `Pipfile.lock` (Python) or `package-lock.json` (NodeJS).
 If you need more information, you could also link to such files via the relation type `processing-software`.
- `version` is usually not used in the context of processing and describes the version of the metadata.

### Linking the Items

In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing. They could be used to trace back the processing history of the dataset.
In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing.
They could be used to trace back the processing history of the dataset.

### Suggested Processing Levels

Expand DownExpand Up@@ -99,6 +137,7 @@ The following types should be used as applicable `rel` types in the
| derived_from | URL to a STAC Item that was used as input data in the creation of this Item. |
| processing-expression | A processing chain (or script) that describes how the data has been processed. |
| processing-execution | URL to any resource representing the processing execution (e.g. OGC Process API). |
| processing-software | URL to any resource that identifies the software and versions used for processing the data, e.g. a `Pipfile.lock` (Python) or `package-lock.json` (NodeJS). |

## Contributing

Expand Down

10 changes: 4 additions & 6 deletions
examples/collection.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,11 +17,9 @@
 ],
 "url": "https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi",
 "processing:lineage": "Generation of Level-1C User Product",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S2 Processing and Archiving Facility",
 "processing:software": {
 "IPF-S2L1C": "02.06"
 }
 "processing:version": "02.06"
 },
 {
 "name": "Processing Corp.",
Expand DownExpand Up@@ -82,8 +80,8 @@
 60
 ],
 "processing:level": [
 "L1C",
 "L2A"
 "L1",
 "L2"
 ]
 },
 "links": [
Expand Down

5 changes: 3 additions & 2 deletions
examples/item.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,11 +29,12 @@
 ],
 "sar:product_type": "GRD",
 "processing:lineage": "GRD Post Processing",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S1 Core Ground Segment - DPA",
 "processing:software": {
 "Sentinel-1 IPF": "002.71"
 }
 },
 "processing:datetime": "2016-08-23T00:30:33Z"
 },
 "links": [
 {
Expand Down

56 changes: 32 additions & 24 deletions
json-schema/schema.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -3,6 +3,18 @@
 "$id": "https://stac-extensions.github.io/processing/v1.1.0/schema.json#",
 "title": "Processing Extension",
 "description": "STAC Processing Extension for STAC Items and STAC Collections.",
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 },
 "anyOf": [
 {
 "$comment": "This is the schema for STAC Items.",
Expand DownExpand Up@@ -33,12 +45,7 @@
 "$ref": "#/definitions/fields"
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ]
 }
 },
 {
 "$comment": "This is the schema for STAC Collections.",
Expand DownExpand Up@@ -72,11 +79,6 @@
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ],
 "anyOf": [
 {
 "$comment": "Requires at least one provider to contain processing fields.",
Expand DownExpand Up@@ -170,18 +172,6 @@
 ],
 "definitions": {
 "stac_extensions": {
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 }
 },
 "require_provider_role": {
 "type": "object",
Expand All@@ -206,7 +196,9 @@
 {"type": "object", "required": ["processing:lineage"]},
 {"type": "object", "required": ["processing:level"]},
 {"type": "object", "required": ["processing:facility"]},
 {"type": "object", "required": ["processing:software"]}
 {"type": "object", "required": ["processing:software"]},
 {"type": "object", "required": ["processing:version"]},
 {"type": "object", "required": ["processing:datetime"]}
 ]
 },
 "fields": {
Expand DownExpand Up@@ -257,6 +249,22 @@
 "Copernicus S1 Core Ground Segment - DPA"
 ]
 },
 "processing:version": {
 "title": "Processing Version",
 "type": "string",
 "examples": [
 "0.2.0"
 ]
 },
 "processing:datetime": {
 "title": "Processing Datetime",
 "type": "string",
 "format": "date-time",
 "pattern": "(\\+00:00|Z)$",
 "examples": [
 "2020-01-05T12:34:55Z"
 ]
 },
 "processing:software": {
 "title": "Processing Software Name / version",
 "type": "object",
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Add processing:version and processing:datetime#32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

m-mohr
merged 3 commits into

from

May 9, 2024
 Merged


Add processing:version and processing:datetime
#32
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
bde9911

Add processing:version and processing:datetime


m-mohr Apr 4, 2024
a6f6b15

Merge branch 'main' into add-version-time


m-mohr Apr 4, 2024
9746a0f

Various clarifications, added new rel type


m-mohr May 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


11 changes: 11 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `processing:version` field to describe the primary software version of workflow version that produced the data
- `processing:datetime` field to describe when the processing happened
- `processing-execution` relation type to link to the processing execution that produced the data.
- `processing-software` relation type to link to the processing execution that produced the data.

### Changed

### Deprecated

### Removed

### Fixed

## [v1.1.0] - 2022-01-07

Expand Down

55 changes: 47 additions & 8 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -22,38 +22,76 @@ and therefore are shared across all items, it is recommended adding the fields t
- [JSON Schema](json-schema/schema.json)
- [Changelog](./CHANGELOG.md)

## Item Properties and Collection Provider Fields
## Fields

| Field Name | Type | Description |
| ----------------------- | ------------------- | ----------- |
| processing:expression | [Expression Object](#expression-object) | An expression or processing chain that describes how the data has been processed. Alternatively, you can also link to a processing chain with the relation type `processing-expression` (see below). |
| processing:lineage | string | Lineage Information provided as free text information about the how observations were processed or models that were used to create the resource being described [NASA ISO](https://wiki.earthdata.nasa.gov/display/NASAISO/Lineage+Information). For example, `GRD Post Processing` for "GRD" product of Sentinel-1 satellites. [CommonMark 0.29](https://commonmark.org/) syntax MAY be used for rich text representation. |
| processing:level | string | The name commonly used to refer to the processing level to make it easier to search for product level across collections or items. The short name must be used (only `L`, not `Level`). See the [list of suggested processing levels](#suggested-processing-levels). |
| processing:facility | string | The name of the facility that produced the data. For example, `Copernicus S1 Core Ground Segment - DPA` for product of Sentinel-1 satellites. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more softwares that produced the data. For example, `"Sentinel-1 IPF":"002.71"` for the software that produces Sentinel-1 satellites data. |
| processing:datetime | string | Processing date and time of the corresponding data formatted according to [RFC 3339, section 5.6](https://tools.ietf.org/html/rfc3339#section-5.6), in UTC. |
| processing:version | string | The version of the primary processing software or processing chain that produced the data. For example, this could be the processing baseline for the Sentinel missions. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more applications or libraries that were involved during the production of the data for provenance purposes. |
Comment thread
m-mohr marked this conversation as resolved.


These fields can be used in a variety of places:
The fields in the table above can be used in these parts of STAC documents:
- [ ] Catalogs
- [ ] Collections
- [x] [Collection Provider](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
- [x] Item Properties (incl. Summaries in Collections)
- [x] Assets (for both Collections and Items, incl. Item Asset Definitions in Collections)
- [ ] Links

In more detail, the following restrictions apply:

1. Items:
 - The fields are placed in the properties. At least one field is required to be present.
 - The fields are usually placed in the properties. At least one field is required to be present.
 - Additionally, STAC allows all fields to be used in the Asset Object.

2. Collections:
 - The fields are usually placed in the [Provider Objects](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
 for the `providers` that have the role `producer` or `processor` assigned.
 They don't need to be provided for all providers of the respective role.
 - The fields can also be used in `summaries`, Collection `assets` or Item asset definitions (`item_assets`).
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.

 If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.
If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.

### Processing Date Time

The time of the processing is directly specified via the `created` properties of the target asset as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time)
The time of the processing can be specified as a global field in `processing:datetime`,
but it can also be specified directly and individually via the `created` properties of the target asset
as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time).

`created` in Item properties describes the STAC metadata creation and in Assets it describes the creation of the data files.
Thus the timestamps provided in Item Properties for `created` and `processing:datetime` may differ.
As Item properties are easier to be indexed and used for filtering purposes, `processing:datetime` exists.
`created` and `processing:datetime` should usually be the same value in Assets and as such `processing:datetime`
can usually be omitted.

### Version Numbers

Three fields exist for version numbers:
- `processing:software`
- `processing:version`
- `version` (in the [Version extension](https://github.com/stac-extensions/version))

The different fields exist to give data providers more flexibility depending on their needs.

In Item Properties:
- `processing:version` is useful if a single version number is available for the metadata or data that users should be able to filter on.
 A popular example for this is the processing baseline in Sentinel missions.
- `processing:software` is used if the software libraries/tools are important to know, but it's not important to filter on them.
 They are mostly informative and important to be complete for reporducibility purposes.
 Thus, the values in the object can not just be version numbers, but also be e.g. tag names, commit hashes or similar.
 For example, you could expose a simplified version of the `Pipfile.lock` (Python) or `package-lock.json` (NodeJS).
 If you need more information, you could also link to such files via the relation type `processing-software`.
- `version` is usually not used in the context of processing and describes the version of the metadata.

### Linking the Items

In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing. They could be used to trace back the processing history of the dataset.
In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing.
They could be used to trace back the processing history of the dataset.

### Suggested Processing Levels

Expand DownExpand Up@@ -99,6 +137,7 @@ The following types should be used as applicable `rel` types in the
| derived_from | URL to a STAC Item that was used as input data in the creation of this Item. |
| processing-expression | A processing chain (or script) that describes how the data has been processed. |
| processing-execution | URL to any resource representing the processing execution (e.g. OGC Process API). |
| processing-software | URL to any resource that identifies the software and versions used for processing the data, e.g. a `Pipfile.lock` (Python) or `package-lock.json` (NodeJS). |

## Contributing

Expand Down

10 changes: 4 additions & 6 deletions
examples/collection.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,11 +17,9 @@
 ],
 "url": "https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi",
 "processing:lineage": "Generation of Level-1C User Product",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S2 Processing and Archiving Facility",
 "processing:software": {
 "IPF-S2L1C": "02.06"
 }
 "processing:version": "02.06"
 },
 {
 "name": "Processing Corp.",
Expand DownExpand Up@@ -82,8 +80,8 @@
 60
 ],
 "processing:level": [
 "L1C",
 "L2A"
 "L1",
 "L2"
 ]
 },
 "links": [
Expand Down

5 changes: 3 additions & 2 deletions
examples/item.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,11 +29,12 @@
 ],
 "sar:product_type": "GRD",
 "processing:lineage": "GRD Post Processing",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S1 Core Ground Segment - DPA",
 "processing:software": {
 "Sentinel-1 IPF": "002.71"
 }
 },
 "processing:datetime": "2016-08-23T00:30:33Z"
 },
 "links": [
 {
Expand Down

56 changes: 32 additions & 24 deletions
json-schema/schema.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -3,6 +3,18 @@
 "$id": "https://stac-extensions.github.io/processing/v1.1.0/schema.json#",
 "title": "Processing Extension",
 "description": "STAC Processing Extension for STAC Items and STAC Collections.",
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 },
 "anyOf": [
 {
 "$comment": "This is the schema for STAC Items.",
Expand DownExpand Up@@ -33,12 +45,7 @@
 "$ref": "#/definitions/fields"
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ]
 }
 },
 {
 "$comment": "This is the schema for STAC Collections.",
Expand DownExpand Up@@ -72,11 +79,6 @@
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ],
 "anyOf": [
 {
 "$comment": "Requires at least one provider to contain processing fields.",
Expand DownExpand Up@@ -170,18 +172,6 @@
 ],
 "definitions": {
 "stac_extensions": {
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 }
 },
 "require_provider_role": {
 "type": "object",
Expand All@@ -206,7 +196,9 @@
 {"type": "object", "required": ["processing:lineage"]},
 {"type": "object", "required": ["processing:level"]},
 {"type": "object", "required": ["processing:facility"]},
 {"type": "object", "required": ["processing:software"]}
 {"type": "object", "required": ["processing:software"]},
 {"type": "object", "required": ["processing:version"]},
 {"type": "object", "required": ["processing:datetime"]}
 ]
 },
 "fields": {
Expand DownExpand Up@@ -257,6 +249,22 @@
 "Copernicus S1 Core Ground Segment - DPA"
 ]
 },
 "processing:version": {
 "title": "Processing Version",
 "type": "string",
 "examples": [
 "0.2.0"
 ]
 },
 "processing:datetime": {
 "title": "Processing Datetime",
 "type": "string",
 "format": "date-time",
 "pattern": "(\\+00:00|Z)$",
 "examples": [
 "2020-01-05T12:34:55Z"
 ]
 },
 "processing:software": {
 "title": "Processing Software Name / version",
 "type": "object",
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add processing:version and processing:datetime#32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

m-mohr
merged 3 commits into

from

May 9, 2024
 Merged


Add processing:version and processing:datetime
#32
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
bde9911

Add processing:version and processing:datetime


m-mohr Apr 4, 2024
a6f6b15

Merge branch 'main' into add-version-time


m-mohr Apr 4, 2024
9746a0f

Various clarifications, added new rel type


m-mohr May 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


11 changes: 11 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `processing:version` field to describe the primary software version of workflow version that produced the data
- `processing:datetime` field to describe when the processing happened
- `processing-execution` relation type to link to the processing execution that produced the data.
- `processing-software` relation type to link to the processing execution that produced the data.

### Changed

### Deprecated

### Removed

### Fixed

## [v1.1.0] - 2022-01-07

Expand Down

55 changes: 47 additions & 8 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -22,38 +22,76 @@ and therefore are shared across all items, it is recommended adding the fields t
- [JSON Schema](json-schema/schema.json)
- [Changelog](./CHANGELOG.md)

## Item Properties and Collection Provider Fields
## Fields

| Field Name | Type | Description |
| ----------------------- | ------------------- | ----------- |
| processing:expression | [Expression Object](#expression-object) | An expression or processing chain that describes how the data has been processed. Alternatively, you can also link to a processing chain with the relation type `processing-expression` (see below). |
| processing:lineage | string | Lineage Information provided as free text information about the how observations were processed or models that were used to create the resource being described [NASA ISO](https://wiki.earthdata.nasa.gov/display/NASAISO/Lineage+Information). For example, `GRD Post Processing` for "GRD" product of Sentinel-1 satellites. [CommonMark 0.29](https://commonmark.org/) syntax MAY be used for rich text representation. |
| processing:level | string | The name commonly used to refer to the processing level to make it easier to search for product level across collections or items. The short name must be used (only `L`, not `Level`). See the [list of suggested processing levels](#suggested-processing-levels). |
| processing:facility | string | The name of the facility that produced the data. For example, `Copernicus S1 Core Ground Segment - DPA` for product of Sentinel-1 satellites. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more softwares that produced the data. For example, `"Sentinel-1 IPF":"002.71"` for the software that produces Sentinel-1 satellites data. |
| processing:datetime | string | Processing date and time of the corresponding data formatted according to [RFC 3339, section 5.6](https://tools.ietf.org/html/rfc3339#section-5.6), in UTC. |
| processing:version | string | The version of the primary processing software or processing chain that produced the data. For example, this could be the processing baseline for the Sentinel missions. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more applications or libraries that were involved during the production of the data for provenance purposes. |
Comment thread
m-mohr marked this conversation as resolved.


These fields can be used in a variety of places:
The fields in the table above can be used in these parts of STAC documents:
- [ ] Catalogs
- [ ] Collections
- [x] [Collection Provider](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
- [x] Item Properties (incl. Summaries in Collections)
- [x] Assets (for both Collections and Items, incl. Item Asset Definitions in Collections)
- [ ] Links

In more detail, the following restrictions apply:

1. Items:
 - The fields are placed in the properties. At least one field is required to be present.
 - The fields are usually placed in the properties. At least one field is required to be present.
 - Additionally, STAC allows all fields to be used in the Asset Object.

2. Collections:
 - The fields are usually placed in the [Provider Objects](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
 for the `providers` that have the role `producer` or `processor` assigned.
 They don't need to be provided for all providers of the respective role.
 - The fields can also be used in `summaries`, Collection `assets` or Item asset definitions (`item_assets`).
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.

 If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.
If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.

### Processing Date Time

The time of the processing is directly specified via the `created` properties of the target asset as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time)
The time of the processing can be specified as a global field in `processing:datetime`,
but it can also be specified directly and individually via the `created` properties of the target asset
as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time).

`created` in Item properties describes the STAC metadata creation and in Assets it describes the creation of the data files.
Thus the timestamps provided in Item Properties for `created` and `processing:datetime` may differ.
As Item properties are easier to be indexed and used for filtering purposes, `processing:datetime` exists.
`created` and `processing:datetime` should usually be the same value in Assets and as such `processing:datetime`
can usually be omitted.

### Version Numbers

Three fields exist for version numbers:
- `processing:software`
- `processing:version`
- `version` (in the [Version extension](https://github.com/stac-extensions/version))

The different fields exist to give data providers more flexibility depending on their needs.

In Item Properties:
- `processing:version` is useful if a single version number is available for the metadata or data that users should be able to filter on.
 A popular example for this is the processing baseline in Sentinel missions.
- `processing:software` is used if the software libraries/tools are important to know, but it's not important to filter on them.
 They are mostly informative and important to be complete for reporducibility purposes.
 Thus, the values in the object can not just be version numbers, but also be e.g. tag names, commit hashes or similar.
 For example, you could expose a simplified version of the `Pipfile.lock` (Python) or `package-lock.json` (NodeJS).
 If you need more information, you could also link to such files via the relation type `processing-software`.
- `version` is usually not used in the context of processing and describes the version of the metadata.

### Linking the Items

In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing. They could be used to trace back the processing history of the dataset.
In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing.
They could be used to trace back the processing history of the dataset.

### Suggested Processing Levels

Expand DownExpand Up@@ -99,6 +137,7 @@ The following types should be used as applicable `rel` types in the
| derived_from | URL to a STAC Item that was used as input data in the creation of this Item. |
| processing-expression | A processing chain (or script) that describes how the data has been processed. |
| processing-execution | URL to any resource representing the processing execution (e.g. OGC Process API). |
| processing-software | URL to any resource that identifies the software and versions used for processing the data, e.g. a `Pipfile.lock` (Python) or `package-lock.json` (NodeJS). |

## Contributing

Expand Down

10 changes: 4 additions & 6 deletions
examples/collection.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,11 +17,9 @@
 ],
 "url": "https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi",
 "processing:lineage": "Generation of Level-1C User Product",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S2 Processing and Archiving Facility",
 "processing:software": {
 "IPF-S2L1C": "02.06"
 }
 "processing:version": "02.06"
 },
 {
 "name": "Processing Corp.",
Expand DownExpand Up@@ -82,8 +80,8 @@
 60
 ],
 "processing:level": [
 "L1C",
 "L2A"
 "L1",
 "L2"
 ]
 },
 "links": [
Expand Down

5 changes: 3 additions & 2 deletions
examples/item.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,11 +29,12 @@
 ],
 "sar:product_type": "GRD",
 "processing:lineage": "GRD Post Processing",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S1 Core Ground Segment - DPA",
 "processing:software": {
 "Sentinel-1 IPF": "002.71"
 }
 },
 "processing:datetime": "2016-08-23T00:30:33Z"
 },
 "links": [
 {
Expand Down

56 changes: 32 additions & 24 deletions
json-schema/schema.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -3,6 +3,18 @@
 "$id": "https://stac-extensions.github.io/processing/v1.1.0/schema.json#",
 "title": "Processing Extension",
 "description": "STAC Processing Extension for STAC Items and STAC Collections.",
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 },
 "anyOf": [
 {
 "$comment": "This is the schema for STAC Items.",
Expand DownExpand Up@@ -33,12 +45,7 @@
 "$ref": "#/definitions/fields"
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ]
 }
 },
 {
 "$comment": "This is the schema for STAC Collections.",
Expand DownExpand Up@@ -72,11 +79,6 @@
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ],
 "anyOf": [
 {
 "$comment": "Requires at least one provider to contain processing fields.",
Expand DownExpand Up@@ -170,18 +172,6 @@
 ],
 "definitions": {
 "stac_extensions": {
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 }
 },
 "require_provider_role": {
 "type": "object",
Expand All@@ -206,7 +196,9 @@
 {"type": "object", "required": ["processing:lineage"]},
 {"type": "object", "required": ["processing:level"]},
 {"type": "object", "required": ["processing:facility"]},
 {"type": "object", "required": ["processing:software"]}
 {"type": "object", "required": ["processing:software"]},
 {"type": "object", "required": ["processing:version"]},
 {"type": "object", "required": ["processing:datetime"]}
 ]
 },
 "fields": {
Expand DownExpand Up@@ -257,6 +249,22 @@
 "Copernicus S1 Core Ground Segment - DPA"
 ]
 },
 "processing:version": {
 "title": "Processing Version",
 "type": "string",
 "examples": [
 "0.2.0"
 ]
 },
 "processing:datetime": {
 "title": "Processing Datetime",
 "type": "string",
 "format": "date-time",
 "pattern": "(\\+00:00|Z)$",
 "examples": [
 "2020-01-05T12:34:55Z"
 ]
 },
 "processing:software": {
 "title": "Processing Software Name / version",
 "type": "object",
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add processing:version and processing:datetime#32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

m-mohr
merged 3 commits into

from

May 9, 2024
 Merged


Add processing:version and processing:datetime
#32
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
bde9911

Add processing:version and processing:datetime


m-mohr Apr 4, 2024
a6f6b15

Merge branch 'main' into add-version-time


m-mohr Apr 4, 2024
9746a0f

Various clarifications, added new rel type


m-mohr May 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


11 changes: 11 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `processing:version` field to describe the primary software version of workflow version that produced the data
- `processing:datetime` field to describe when the processing happened
- `processing-execution` relation type to link to the processing execution that produced the data.
- `processing-software` relation type to link to the processing execution that produced the data.

### Changed

### Deprecated

### Removed

### Fixed

## [v1.1.0] - 2022-01-07

Expand Down

55 changes: 47 additions & 8 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -22,38 +22,76 @@ and therefore are shared across all items, it is recommended adding the fields t
- [JSON Schema](json-schema/schema.json)
- [Changelog](./CHANGELOG.md)

## Item Properties and Collection Provider Fields
## Fields

| Field Name | Type | Description |
| ----------------------- | ------------------- | ----------- |
| processing:expression | [Expression Object](#expression-object) | An expression or processing chain that describes how the data has been processed. Alternatively, you can also link to a processing chain with the relation type `processing-expression` (see below). |
| processing:lineage | string | Lineage Information provided as free text information about the how observations were processed or models that were used to create the resource being described [NASA ISO](https://wiki.earthdata.nasa.gov/display/NASAISO/Lineage+Information). For example, `GRD Post Processing` for "GRD" product of Sentinel-1 satellites. [CommonMark 0.29](https://commonmark.org/) syntax MAY be used for rich text representation. |
| processing:level | string | The name commonly used to refer to the processing level to make it easier to search for product level across collections or items. The short name must be used (only `L`, not `Level`). See the [list of suggested processing levels](#suggested-processing-levels). |
| processing:facility | string | The name of the facility that produced the data. For example, `Copernicus S1 Core Ground Segment - DPA` for product of Sentinel-1 satellites. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more softwares that produced the data. For example, `"Sentinel-1 IPF":"002.71"` for the software that produces Sentinel-1 satellites data. |
| processing:datetime | string | Processing date and time of the corresponding data formatted according to [RFC 3339, section 5.6](https://tools.ietf.org/html/rfc3339#section-5.6), in UTC. |
| processing:version | string | The version of the primary processing software or processing chain that produced the data. For example, this could be the processing baseline for the Sentinel missions. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more applications or libraries that were involved during the production of the data for provenance purposes. |
Comment thread
m-mohr marked this conversation as resolved.


These fields can be used in a variety of places:
The fields in the table above can be used in these parts of STAC documents:
- [ ] Catalogs
- [ ] Collections
- [x] [Collection Provider](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
- [x] Item Properties (incl. Summaries in Collections)
- [x] Assets (for both Collections and Items, incl. Item Asset Definitions in Collections)
- [ ] Links

In more detail, the following restrictions apply:

1. Items:
 - The fields are placed in the properties. At least one field is required to be present.
 - The fields are usually placed in the properties. At least one field is required to be present.
 - Additionally, STAC allows all fields to be used in the Asset Object.

2. Collections:
 - The fields are usually placed in the [Provider Objects](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
 for the `providers` that have the role `producer` or `processor` assigned.
 They don't need to be provided for all providers of the respective role.
 - The fields can also be used in `summaries`, Collection `assets` or Item asset definitions (`item_assets`).
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.

 If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.
If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.

### Processing Date Time

The time of the processing is directly specified via the `created` properties of the target asset as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time)
The time of the processing can be specified as a global field in `processing:datetime`,
but it can also be specified directly and individually via the `created` properties of the target asset
as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time).

`created` in Item properties describes the STAC metadata creation and in Assets it describes the creation of the data files.
Thus the timestamps provided in Item Properties for `created` and `processing:datetime` may differ.
As Item properties are easier to be indexed and used for filtering purposes, `processing:datetime` exists.
`created` and `processing:datetime` should usually be the same value in Assets and as such `processing:datetime`
can usually be omitted.

### Version Numbers

Three fields exist for version numbers:
- `processing:software`
- `processing:version`
- `version` (in the [Version extension](https://github.com/stac-extensions/version))

The different fields exist to give data providers more flexibility depending on their needs.

In Item Properties:
- `processing:version` is useful if a single version number is available for the metadata or data that users should be able to filter on.
 A popular example for this is the processing baseline in Sentinel missions.
- `processing:software` is used if the software libraries/tools are important to know, but it's not important to filter on them.
 They are mostly informative and important to be complete for reporducibility purposes.
 Thus, the values in the object can not just be version numbers, but also be e.g. tag names, commit hashes or similar.
 For example, you could expose a simplified version of the `Pipfile.lock` (Python) or `package-lock.json` (NodeJS).
 If you need more information, you could also link to such files via the relation type `processing-software`.
- `version` is usually not used in the context of processing and describes the version of the metadata.

### Linking the Items

In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing. They could be used to trace back the processing history of the dataset.
In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing.
They could be used to trace back the processing history of the dataset.

### Suggested Processing Levels

Expand DownExpand Up@@ -99,6 +137,7 @@ The following types should be used as applicable `rel` types in the
| derived_from | URL to a STAC Item that was used as input data in the creation of this Item. |
| processing-expression | A processing chain (or script) that describes how the data has been processed. |
| processing-execution | URL to any resource representing the processing execution (e.g. OGC Process API). |
| processing-software | URL to any resource that identifies the software and versions used for processing the data, e.g. a `Pipfile.lock` (Python) or `package-lock.json` (NodeJS). |

## Contributing

Expand Down

10 changes: 4 additions & 6 deletions
examples/collection.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,11 +17,9 @@
 ],
 "url": "https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi",
 "processing:lineage": "Generation of Level-1C User Product",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S2 Processing and Archiving Facility",
 "processing:software": {
 "IPF-S2L1C": "02.06"
 }
 "processing:version": "02.06"
 },
 {
 "name": "Processing Corp.",
Expand DownExpand Up@@ -82,8 +80,8 @@
 60
 ],
 "processing:level": [
 "L1C",
 "L2A"
 "L1",
 "L2"
 ]
 },
 "links": [
Expand Down

5 changes: 3 additions & 2 deletions
examples/item.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,11 +29,12 @@
 ],
 "sar:product_type": "GRD",
 "processing:lineage": "GRD Post Processing",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S1 Core Ground Segment - DPA",
 "processing:software": {
 "Sentinel-1 IPF": "002.71"
 }
 },
 "processing:datetime": "2016-08-23T00:30:33Z"
 },
 "links": [
 {
Expand Down

56 changes: 32 additions & 24 deletions
json-schema/schema.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -3,6 +3,18 @@
 "$id": "https://stac-extensions.github.io/processing/v1.1.0/schema.json#",
 "title": "Processing Extension",
 "description": "STAC Processing Extension for STAC Items and STAC Collections.",
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 },
 "anyOf": [
 {
 "$comment": "This is the schema for STAC Items.",
Expand DownExpand Up@@ -33,12 +45,7 @@
 "$ref": "#/definitions/fields"
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ]
 }
 },
 {
 "$comment": "This is the schema for STAC Collections.",
Expand DownExpand Up@@ -72,11 +79,6 @@
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ],
 "anyOf": [
 {
 "$comment": "Requires at least one provider to contain processing fields.",
Expand DownExpand Up@@ -170,18 +172,6 @@
 ],
 "definitions": {
 "stac_extensions": {
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 }
 },
 "require_provider_role": {
 "type": "object",
Expand All@@ -206,7 +196,9 @@
 {"type": "object", "required": ["processing:lineage"]},
 {"type": "object", "required": ["processing:level"]},
 {"type": "object", "required": ["processing:facility"]},
 {"type": "object", "required": ["processing:software"]}
 {"type": "object", "required": ["processing:software"]},
 {"type": "object", "required": ["processing:version"]},
 {"type": "object", "required": ["processing:datetime"]}
 ]
 },
 "fields": {
Expand DownExpand Up@@ -257,6 +249,22 @@
 "Copernicus S1 Core Ground Segment - DPA"
 ]
 },
 "processing:version": {
 "title": "Processing Version",
 "type": "string",
 "examples": [
 "0.2.0"
 ]
 },
 "processing:datetime": {
 "title": "Processing Datetime",
 "type": "string",
 "format": "date-time",
 "pattern": "(\\+00:00|Z)$",
 "examples": [
 "2020-01-05T12:34:55Z"
 ]
 },
 "processing:software": {
 "title": "Processing Software Name / version",
 "type": "object",
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Add processing:version and processing:datetime#32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

m-mohr
merged 3 commits into

from

May 9, 2024
 Merged


Add processing:version and processing:datetime
#32
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
bde9911

Add processing:version and processing:datetime


m-mohr Apr 4, 2024
a6f6b15

Merge branch 'main' into add-version-time


m-mohr Apr 4, 2024
9746a0f

Various clarifications, added new rel type


m-mohr May 8, 2024

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


11 changes: 11 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,7 +8,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- `processing:version` field to describe the primary software version of workflow version that produced the data
- `processing:datetime` field to describe when the processing happened
- `processing-execution` relation type to link to the processing execution that produced the data.
- `processing-software` relation type to link to the processing execution that produced the data.

### Changed

### Deprecated

### Removed

### Fixed

## [v1.1.0] - 2022-01-07

Expand Down

55 changes: 47 additions & 8 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -22,38 +22,76 @@ and therefore are shared across all items, it is recommended adding the fields t
- [JSON Schema](json-schema/schema.json)
- [Changelog](./CHANGELOG.md)

## Item Properties and Collection Provider Fields
## Fields

| Field Name | Type | Description |
| ----------------------- | ------------------- | ----------- |
| processing:expression | [Expression Object](#expression-object) | An expression or processing chain that describes how the data has been processed. Alternatively, you can also link to a processing chain with the relation type `processing-expression` (see below). |
| processing:lineage | string | Lineage Information provided as free text information about the how observations were processed or models that were used to create the resource being described [NASA ISO](https://wiki.earthdata.nasa.gov/display/NASAISO/Lineage+Information). For example, `GRD Post Processing` for "GRD" product of Sentinel-1 satellites. [CommonMark 0.29](https://commonmark.org/) syntax MAY be used for rich text representation. |
| processing:level | string | The name commonly used to refer to the processing level to make it easier to search for product level across collections or items. The short name must be used (only `L`, not `Level`). See the [list of suggested processing levels](#suggested-processing-levels). |
| processing:facility | string | The name of the facility that produced the data. For example, `Copernicus S1 Core Ground Segment - DPA` for product of Sentinel-1 satellites. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more softwares that produced the data. For example, `"Sentinel-1 IPF":"002.71"` for the software that produces Sentinel-1 satellites data. |
| processing:datetime | string | Processing date and time of the corresponding data formatted according to [RFC 3339, section 5.6](https://tools.ietf.org/html/rfc3339#section-5.6), in UTC. |
| processing:version | string | The version of the primary processing software or processing chain that produced the data. For example, this could be the processing baseline for the Sentinel missions. |
| processing:software | Map<string, string> | A dictionary with name/version for key/value describing one or more applications or libraries that were involved during the production of the data for provenance purposes. |
Comment thread
m-mohr marked this conversation as resolved.


These fields can be used in a variety of places:
The fields in the table above can be used in these parts of STAC documents:
- [ ] Catalogs
- [ ] Collections
- [x] [Collection Provider](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
- [x] Item Properties (incl. Summaries in Collections)
- [x] Assets (for both Collections and Items, incl. Item Asset Definitions in Collections)
- [ ] Links

In more detail, the following restrictions apply:

1. Items:
 - The fields are placed in the properties. At least one field is required to be present.
 - The fields are usually placed in the properties. At least one field is required to be present.
 - Additionally, STAC allows all fields to be used in the Asset Object.

2. Collections:
 - The fields are usually placed in the [Provider Objects](https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#provider-object)
 for the `providers` that have the role `producer` or `processor` assigned.
 They don't need to be provided for all providers of the respective role.
 - The fields can also be used in `summaries`, Collection `assets` or Item asset definitions (`item_assets`).
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.

 If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.
 Please note that the JSON Schema is not be able to validate the values of Collection summaries.
If the extension is given in the `stac_extensions` list, at least one of the fields must be specified in any of the given places listed above.

### Processing Date Time

The time of the processing is directly specified via the `created` properties of the target asset as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time)
The time of the processing can be specified as a global field in `processing:datetime`,
but it can also be specified directly and individually via the `created` properties of the target asset
as specified in the [STAC Common metadata](https://github.com/radiantearth/stac-spec/blob/master/item-spec/common-metadata.md#date-and-time).

`created` in Item properties describes the STAC metadata creation and in Assets it describes the creation of the data files.
Thus the timestamps provided in Item Properties for `created` and `processing:datetime` may differ.
As Item properties are easier to be indexed and used for filtering purposes, `processing:datetime` exists.
`created` and `processing:datetime` should usually be the same value in Assets and as such `processing:datetime`
can usually be omitted.

### Version Numbers

Three fields exist for version numbers:
- `processing:software`
- `processing:version`
- `version` (in the [Version extension](https://github.com/stac-extensions/version))

The different fields exist to give data providers more flexibility depending on their needs.

In Item Properties:
- `processing:version` is useful if a single version number is available for the metadata or data that users should be able to filter on.
 A popular example for this is the processing baseline in Sentinel missions.
- `processing:software` is used if the software libraries/tools are important to know, but it's not important to filter on them.
 They are mostly informative and important to be complete for reporducibility purposes.
 Thus, the values in the object can not just be version numbers, but also be e.g. tag names, commit hashes or similar.
 For example, you could expose a simplified version of the `Pipfile.lock` (Python) or `package-lock.json` (NodeJS).
 If you need more information, you could also link to such files via the relation type `processing-software`.
- `version` is usually not used in the context of processing and describes the version of the metadata.

### Linking the Items

In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing. They could be used to trace back the processing history of the dataset.
In Items that declare this `processing` extension, it is recommended to add one or more [Links](https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types) with `derived_from` or `via` relationships to the eventual source metadata & data used in the processing.
They could be used to trace back the processing history of the dataset.

### Suggested Processing Levels

Expand DownExpand Up@@ -99,6 +137,7 @@ The following types should be used as applicable `rel` types in the
| derived_from | URL to a STAC Item that was used as input data in the creation of this Item. |
| processing-expression | A processing chain (or script) that describes how the data has been processed. |
| processing-execution | URL to any resource representing the processing execution (e.g. OGC Process API). |
| processing-software | URL to any resource that identifies the software and versions used for processing the data, e.g. a `Pipfile.lock` (Python) or `package-lock.json` (NodeJS). |

## Contributing

Expand Down

10 changes: 4 additions & 6 deletions
examples/collection.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -17,11 +17,9 @@
 ],
 "url": "https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi",
 "processing:lineage": "Generation of Level-1C User Product",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S2 Processing and Archiving Facility",
 "processing:software": {
 "IPF-S2L1C": "02.06"
 }
 "processing:version": "02.06"
 },
 {
 "name": "Processing Corp.",
Expand DownExpand Up@@ -82,8 +80,8 @@
 60
 ],
 "processing:level": [
 "L1C",
 "L2A"
 "L1",
 "L2"
 ]
 },
 "links": [
Expand Down

5 changes: 3 additions & 2 deletions
examples/item.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -29,11 +29,12 @@
 ],
 "sar:product_type": "GRD",
 "processing:lineage": "GRD Post Processing",
 "processing:level": "L1C",
 "processing:level": "L1",
 "processing:facility": "Copernicus S1 Core Ground Segment - DPA",
 "processing:software": {
 "Sentinel-1 IPF": "002.71"
 }
 },
 "processing:datetime": "2016-08-23T00:30:33Z"
 },
 "links": [
 {
Expand Down

56 changes: 32 additions & 24 deletions
json-schema/schema.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -3,6 +3,18 @@
 "$id": "https://stac-extensions.github.io/processing/v1.1.0/schema.json#",
 "title": "Processing Extension",
 "description": "STAC Processing Extension for STAC Items and STAC Collections.",
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 },
 "anyOf": [
 {
 "$comment": "This is the schema for STAC Items.",
Expand DownExpand Up@@ -33,12 +45,7 @@
 "$ref": "#/definitions/fields"
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ]
 }
 },
 {
 "$comment": "This is the schema for STAC Collections.",
Expand DownExpand Up@@ -72,11 +79,6 @@
 }
 }
 },
 "allOf": [
 {
 "$ref": "#/definitions/stac_extensions"
 }
 ],
 "anyOf": [
 {
 "$comment": "Requires at least one provider to contain processing fields.",
Expand DownExpand Up@@ -170,18 +172,6 @@
 ],
 "definitions": {
 "stac_extensions": {
 "type": "object",
 "required": [
 "stac_extensions"
 ],
 "properties": {
 "stac_extensions": {
 "type": "array",
 "contains": {
 "const": "https://stac-extensions.github.io/processing/v1.1.0/schema.json"
 }
 }
 }
 },
 "require_provider_role": {
 "type": "object",
Expand All@@ -206,7 +196,9 @@
 {"type": "object", "required": ["processing:lineage"]},
 {"type": "object", "required": ["processing:level"]},
 {"type": "object", "required": ["processing:facility"]},
 {"type": "object", "required": ["processing:software"]}
 {"type": "object", "required": ["processing:software"]},
 {"type": "object", "required": ["processing:version"]},
 {"type": "object", "required": ["processing:datetime"]}
 ]
 },
 "fields": {
Expand DownExpand Up@@ -257,6 +249,22 @@
 "Copernicus S1 Core Ground Segment - DPA"
 ]
 },
 "processing:version": {
 "title": "Processing Version",
 "type": "string",
 "examples": [
 "0.2.0"
 ]
 },
 "processing:datetime": {
 "title": "Processing Datetime",
 "type": "string",
 "format": "date-time",
 "pattern": "(\\+00:00|Z)$",
 "examples": [
 "2020-01-05T12:34:55Z"
 ]
 },
 "processing:software": {
 "title": "Processing Software Name / version",
 "type": "object",
Expand Down