Support for domain specific metadata

[ochiengolanga]

The BBC got a number of data fields that need to be supported in order to deliver archive workflows. Specifically, BBC Archives need to be able to add and update key/value metadata that describes the programme/series/episode/genre/admin details.

These details may be available in embedded metadata on import, but generally speaking, they’re added as part of the curation process that happens when the images are uploaded by archivists, or are curated on delivery from the photo publicity team.

• Programme Title
• Programme Number
• Programme Maker
• Previous Titles
• Channels
• Season
• Series Number
• Number of Episodes
• Programme Genre
• Commissioning Department

The fields need to be editable, and be able to be used in search terms and filters on searches.

That ability to support domain-specific schemas of editable/filterable metadata is a generally useful feature for the Grid and would be useful for other adopters of the system.

Would welcome feedback from the Guardian as to whether this is a generally useful feature that should be implemented in the core functionality of the Grid.

[ochiengolanga]

Proposed approach

Add DomainMetadata trait to ImageMetadata model and use subtyping to support domain metadata models

Introduce a DomainMetadata trait in the ImageMetadata core model and allow adopters to subtype this trait and define their own domain specific metadata models with their respective fields. (Idea borrowed from UsageRights implementation)

Implementation

Data

Add a DomainMetadata trait field in ImageMetadata.scala and through subtyping domain specific metadata models can be defined and used as a field in the ImageMetadata model.

Sample code:

trait DomainMetadata {}

trait DomainMetadataSpec {
	val name: String
	val description: String
	...
}

final case class BBCArchives(programmeTitle: String, programmeNumber: String, ...) extends DomainMetadata

object BBCArchives extends DomainMetadataSpec {
	val name = "BBC Archives"
	val description = "BBC Archives domain metadata model for archive workflows"
	...
}

final case class BBCRadio(fieldA: String, ...) extends DomainMetadata

object BBCRadio extends DomainMetadataSpec {
	val name = "BBC Radio"
	val description = "BBC Radio domain metadata model for radio pic team workflows"
	...
}

case class ImageMetadata(
	...,
	domainMetadata: Option[DomainMetadata] = None # Can also be a list of DomainMetadata models
)

case class Image(
	...,
	metadata: ImageMetadata
)

Backend

The list of applicable / usable domain specific metadata models can be defined in configuration (preferably metadata-editor service) as a list of full class names and reflection can be used to validate the existence of the classes or case objects.

Introduce a GET HTTP API endpoint that can receive requests to provide a list of applicable / usable domain specific metadata models and their respective fields.

UI

Kahuna (UI) will make a HTTP request to an API (preferably metadata-editor service) to GET a list of applicable / usable domain specific metadata models and their respective fields to render HTML form elements on the image metadata form.

Concerns

Is it CRUD friendly?

Read - Field aliases can be used to display these fields under “Additional information” section and allows the fields to be used to search e.g. imageMetadata.domainMetadata.programmeTitle = Random programme title

Create, Update, Delete - Values can be defined using a single PUT HTTP API endpoint in the backend (preferably metadata-editor service) to do updates of the domain specific metadata field. Updates should be handled just like any other metadata update by recording edits to DynamoDB and updating the updated metadata in Elasticsearch.

Sample JSON update payload:

{
  "data": {
    "description": "pexels-alana-sousa-3409498",
    "byline": "hey",
    "domainMetadata": {
		"programmeTitle": "Updated random programme title"
	}
  }
}

Support for rich data types

Serializable and deserializable scala data types can be used in domain metadata model shape.

Validation

Both client-side and server-side validation can be implemented to ensure compliance of the user input based on different metadata fields belonging to domain specific metadata models.

[tjsilver]

@andrew-nowak and I have taken a look at this and it looks complex but feasible. We don't anticipate any issues, but please keep us in the loop as you work on it. Good luck!

[paperboyo]

Done in (mainly) #3496.