Re-imagine Registry Handling of Product Versioning

[jordanpadams]

💡 Description

How do can we track in the registry product identifiers and versions when the actual product identifiers/versions according to their metadata do not match up?

Context products are a prime example of this, but also starting to occur for actual products.

Initial proposed solution:

• design a method for users to specify supersedes in the product metadata OR via a product_metadata_supplemental specific to superseding of data sets
  • should this be an LDD?
• create new index to track deprecated LIDs
• create new internal/ops unique identifier (UID) for each product in the registry that is auto-generated at the time of ingestion
• version tracking of the products is either determined upon ingestion or a new index (or graphDB?) is created to manage product versions
• Automate setting and management of of archive status across the archive

General Example

Product A changes its logical identifier from version "1" of the product (lid_x) to version "2" (lid_y), we want to know that these are actually revisions of the same product.

NOTE: The versions may not be in appropriate ascending order, e.g. you could have lid_x::1.0 and lid_y::1.0, but really lid_x::1 == Product A v1 and lid_y::1.0 == Product A v2

Real Example

The logical identifier for the Cosmic Ray Subsystem Instrument context product was changed at some point to meet the new context product LID guidelines.

So the context product with LIDVID urn:nasa:pds:context:instrument:crs.vg1::1.0 was superseded by LIDVID urn:nasa:pds:context:instrument:vg1.crs::1.0

[tloubrieu-jpl]

To manage the superseding lid we need somewhere information saying that a new lid superseding a pre-existing lid.
That could be managed in the pds4 label or in other CSV file (supplemental metadata).

[msbentley]

To understand this better, are we talking cases where the LID has changed for a given product, such that versions alone are not enough to figure out what is the latest?

[jordanpadams]

@msbentley correct. "Deprecated" context products are a good example of this. M2020 cameras just did this as well.

[jordanpadams]

@msbentley just updated the original ticket description with a real-world example

[mdrum]

Is this a question of how to reflect searches and relationships accurately in the API?

The information to determine this should make it into the registry for context products, at least. Inside the labels of the products with the new LID, there's generally an alias list (from https://pds.nasa.gov/data/pds4/context-pds4/instrument/ds1.ids_1.0.xml)

<Alias>
<alternate_title>urn:nasa:pds:context:instrument:ids.ds1</alternate_title>
</Alias>
<!--  deprecated LID  -->
</Alias_List>

This is something I had to add logic to Archive Navigator to check, and replace any references to the older LID with links to the new one.

I would certainly prefer that this be handled at the API level though

[jordanpadams]

@mdrum this is part of it, but it also means we basically need to basically give each product a "registry_identifier" and "vid" separate from the actual product LIDVID since LIDs may be changing.

definitely applies to context products, but also to other products as well. Tracking this info in labels would be great, but also doesn't necessarily apply for historical data, so the registry will need to manage that information outside of the archive metadata.

[jordanpadams]

📆 February status: Deferring this task to B13.0 for time being. Workarounds exist we can discuss with the archives, and we have other tasks in the queue that take higher priority. No impact on B12.1 since this task was added as a nice-to-have for this particular build cycle. Will be added to B13.0

[tloubrieu-jpl]

@jordanpadams can we table that for this sprint ? I don't feel like we will be able to finalize what Eugene started that soon.

[jordanpadams]

📆 June status: 75% complete. Awaiting pull request review and merge.

[jordanpadams]

📆 July status: Still 75% complete. Awaiting pull request review and merge. Implementation in API has been tabled until after NASA-PDS/pds-api#180

[jordanpadams]

NASA-PDS/registry-mgr#46 was merged