Proposal for JSON representing USFM/USX Contents

[kavitharaju]

A JSON representation for USFM/USX Contents

By Kavitha Raju, on behalf of the working group consisting of

Joel Mathew, Mark Howe and Kavitha Raju

Motivation

As USFM format, which is a text markup, and USX, which is XML, are being extensively used, the programming world in general is moving towards newer formats, with JSON (JavaScript Object Notation) being a prominent one among them. It becomes necessary to facilicate the representation of these content in such a format to enable

• Faster and unambigious parsing
• Cross-platform support and easy data exchange between tools and technologies
• Leverage the advancements in data processing

The Design Principles

The key principles we have tried to adhere to, in designing the JSON structure, are the following

1. Keep the format close to the existing USX structure.
   • The chapters and verses are treated as milestones, just marking the start of a chapter or verse, but not containing any text or other contents within them. Chapter elements are valid at the outer level(same level as id and book headers, while verse elements, like character elements, are valid within paragraph like enclosures, at same level as texts.
   • The node types in USX(para, char, book, chapter, verse, ms etc) and the marker names in USFM (which became the style attribute in USX) are together used to define the element type in the JSON. {"type" : "para:p"}, {"type": "verse:v"}, etc.
   • Each attribute in a USX node becomes a key in the corresponding JSON element.

       {
         "type": "book:id",
         "code": "GEN"
       },
       {
         "type": "chapter:c",
         "number": "1",
         "sid": "GEN 1"
       }
   

   • The text content of the USX, becomes the value of a text field for those elements where nested elements(including character markers) are not possible.

       {
         "type": "para:ide",
         "text": "UTF-8"
       },
       {
         "type": "para:usfm",
         "text": "3.0"
       }
   

   • In all other elements where nested contents are possible, a content field is provided as an array.

       {
         "type": "para:is",
         "content": [
           "Introduction"
           ]
       },
       {
         "type": "para:ip",
         "content": [
           {
             "type": "char:bk",
             "content": [
               "The Gospel according to Mark"
                ]
           },
           " begins with the statement..."
         ]
       }
   

2. Do not tie the structure to any particular usecase , but provide a basic representation of the USFM contents, leaving space for customization for different needs.
3. Keep it light, not adding extra information or nesting, than the base representations contain.

Special Treatments

Though in general we are proposing a direct JSON representation of the USX contents, the following exceptions have been made in that transfer.

1. Excluded the verse end and chapter end USX nodes, keeping just their start milestone like it is in USFM. This is done to facilitate easy editing of those contents when using this JSON as the underlying format for an editor interface.
2. ca, cp, va and vp are always represented as separate elements and not as altnumber and pubnumber, because
   these may occur not just at the chapter/verse start but also in between their contents and then their representation is inconsistent in USX.

Samples

Samples of this Proposed JSON are available here

Open Questions

1. Name of the JSON: We called it JOUST for Javascript Object notation for USFM Syntax Trees.
2. Versioning: Start from 1.0.0 or follow same version number as USFM/USX version?

[RobH123]

Yeah, I think this might be one of the most sensible JSON representations of USFM/USX that I've seen. The simplicity (or lightness as you call it) is the key. Congrats.

(I came to a similar conclusion with my internal Python format -- USFM seems well represented as list/array type objects which can then be sequentially processed. As soon as you start to use lots of dictionaries/maps, you get messed up by the overlapping hierarchies.)

Didn't like the proposed name at first glance (felt it should have bible or scripture in the name), but on second thoughts, it clearly signifies its design philosophy.

So a big thumbs up at first glance.

P.S. I did two other things that you might possibly consider: 1/ I disambiguated cl fields which have two quite distinct uses in USFM (depending on where they occur--I gave one use case a modified marker), and 2/ I copied (not moved) chapter numbers into the place where you normally want to print them (giving them a different c# marker/name). If these two things are done when the JSON is formed, it would make sequential processing of your lists a lot more straight-forward later on.

[kavitharaju]

Thank you 🙂

[kavitharaju]

Shall take your suggestions to the group and see what we can do.

[mhosken]

I like the use of "tag:@Style". Very succinct.

This will involve someone in categorising markers as to whether they can take marker content or not. It also means that any readers (or writers) will have to have such an awareness or 'hope for the best'. It may turn out to be easier to pay the byte cost and use "content": ["single value"] over "text": "single value". But that's for you guys to decide.

The current USFM/USX grammar distinguishes the different use cases of things like \ca and \xt and can do the right thing with them. I suggest that there is no need to resolve this issue in a different way in JSON. If converting between different formats, if JSON uses a special way of handling these, then special code will need to be written.

As to a name, I quite like USJ (Unified Scripture JSON).

[kavitharaju]

use "content": ["single value"] over "text": "single value".

This is something I had battled over in my mind. Using content everywhere would give users consistency. They will always know where to look for the contents. I am totally willing to re-consider it and shall take it back to the group. On the other hand using "text" and "content" gives us a way to let users know which are the objects where they can expect multiple items and which are not.

[kavitharaju]

The current USFM/USX grammar distinguishes the different use cases of things like \ca and \xt and can do the right thing with them.

I had noticed that the new grammar expects the ca, cp, va vp markers to occur immediately after c and v and also expects a strict order for them. But saw the USX samples in test suite a bit confusing. Would like to learn more about this.

[kavitharaju]

USJ (Unified Scripture JSON)

Up voting this. 👍 . Clearly says that it goes along with USX

[joelthe1]

USJ sounds good to me! 👍

[jwickberg]

A wasn't sure about attributes. With USX, the attributes are XML attributes of the element, so there is an easy way to get all of them. The JSON format currently requires you to filter out the some fields and what you have left are the attributes.

Maybe an "attributes": {"attribute1": "value1", "attribute2": "value2" } approach would be clearer and easier to use.

[kavitharaju]

That would be an additional hierarchy in the structure. But will be a way to separate out certain fields easily.

Also, because USX treats several things in the same level in its attribute list, it would pose the question, which all attributes?
Only the attributes as per USFM, like "lemma", "gloss", "link-href" etc, or
all the attributes in the USX, including book code, chapter number, verse number, altnumber, pubnumber, category etc?

[mvahowe]

Personally, The JSON format currently requires you to filter out the some fields doesn't bother me much because, in reality, anyone working with this JSON is going to write a library or borrow someone else's, and that kind of filtering is literally one line of code in most modern languages. That said, we set out to design JSON that was basically lexed USFM, with the expectation that this JSON would be turned into something that was actually useful. So, on that basis, making things explicit is almost the only requirement, so we should probably store attributes explicitly.

[jwickberg]

I'm fine with whichever way you decide to go - just thought I would mention this as an idea.

[Omikhleia]

Quick remark: idempotent format conversion should be considered, i.e. the ability of full round-trip (USFM -> USJson -> USFM) without loss of information and ideally semantically identical.

[Omikhleia]

... the programming world in general is moving towards newer formats ...

By the way, this is a real bad rationale for any format proposal. (Conversion, if needed, from whatever input format into any internal format for processing should be obvious for any real programmer and developer). The audience should not be targeted on hype and developers -- rather on having well defined-format(s) for interchange and archiving.

[jonathanrobie]

Quick remark: idempotent format conversion should be considered, i.e. the ability of full round-trip (USFM -> USJson -> USFM) without loss of information and ideally semantically identical.

Absolutely. USFM, USX, and the JSON format should have the same data model. In the current draft format for USFM and USX, they share a common data model, which is based on the USX representation, and they are just different serializations of the same data model. The JSON format should do the same.

[jonathanrobie]

... the programming world in general is moving towards newer formats ...

By the way, this is a real bad rationale for any format proposal. (Conversion, if needed, from whatever input format into any internal format for processing should be obvious for any real programmer and developer). The audience should not be targeted on hype and developers -- rather on having well defined-format(s) for interchange and archiving.

I think it's clear that there is a group of programmers who want to work with Scripture but do not know XML or USFM. As long as our formats are different serializations of the same data format, I think we can support them.

I don't think that "the programming world in general" is storing structured documents as JSON. I don't think JSON will replace HTML, and I don't think of the JSON format as a replacement for USX. So I agree that this sentence implies things that might not be true.

Regardless, if all three formats exist, with the same data model, we can see how this pans out.

[irahopkinson]

We are using USJ in Platform.Bible (despite it not being released yet!) internally to represent Scripture. I'm working on the editor which takes in USJ and have found something I think is missing from the MarkerObject. Is this the right place to raise this or is there a better place?

I've found that eid is missing. It is needed for ending milestones. Unlike chapter or verse markers in USJ that are more like milestones, actual milestones typically have a separate starting and ending milestone marker to allow them to be disambiguated since milesstones can be nested or overlapping.

[joelthe1]

Thanks for sharing. We will be taking a look at this issue in the next few days and respond here.

[irahopkinson]

Thanks. I suspect there is more needed than this but this is the only concrete one I have found so far.