RDFe – expression-based mapping of XML documents to RDF triples

Consider an XML document describing drugs (contents taken from drugbank [2]):

<drugs xmlns="http://www.drugbank.ca">
  <drug type="biotech" created="2005-06-13" updated="2018-07-02">
    <drugbank-id primary="true">DB00001</drugbank-id>
    <drugbank-id>BTD00024</drugbank-id>
    <drugbank-id>BIOD00024</drugbank-id>
    <name>Lepirudin</name>
    <!-- more content here -->
    <pathways>
      <pathway>
        <!-- more content here -->
        <enzymes>
          <uniprot-id>P00734</uniprot-id>
          <uniprot-id>P00748</uniprot-id>
          <uniprot-id>P02452</uniprot-id>
          <!-- more content follows -->
        </enzymes>
      </pathway>
    </pathways>
    <!-- more content here -->
  </drug>
  <!-- more drugs here -->
</drugs>

We want to map parts of these descriptions to an RDF representation. First goals:

The details are outlined in the table below. Within XPath expressions, variable $drug references the XML element representing the resource.

This model is easily translated into an RDFe document, also called a semantic map:

<re:semanticMap iri="http://example.com/semap/drugbank/"
                targetNamespace="http://www.drugbank.ca"
                targetName="drugs"
                xmlns:re="http://www.rdfe.org/ns/model"
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                xmlns:db="http://www.drugbank.ca">
    
  <re:namespace iri="http://example.com/resource/drug/" prefix="drug"/>
  <re:namespace iri="http://example.com/ontology/drugbank/" prefix="ont"/>
  <re:namespace iri="http://www.w3.org/2000/01/rdf-schema#" prefix="rdfs"/>    
  <re:namespace iri="http://bio2rdf.org/uniprot:" prefix="uniprot"/>
    
  <re:resource modelID="drug" 
               assertedTargetNodes="/db:drugs/db:drug" 
               targetNodeNamespace="http://www.drugbank.ca"
               targetNodeName="drug"
               iri="db:drugbank-id[@primary = 'true']/concat('drug:', .)" 
               type="ont:drug">
    <re:property iri="rdfs:label" 
                 value="db:name" 
                 type="xs:string"/>        
    <re:property iri="ont:updated" 
                 value="@updated" 
                 type="xs:date"/>
    <re:property iri="ont:drugbank-id" 
                 value="db:drugbank-id[@primary = 'true']" 
                 type="xs:string"/>
    <re:property iri="ont:drugbank-alt-id" 
                 value="db:drugbank-id[not(@primary = 'true')]"  
                 type="xs:string"/>
    <re:property iri="ont:enzyme" 
                 value=".//db:enzymes/db:uniprot-id/concat('uniprot:', .)" 
                 type="#iri"/>
  </re:resource>
    
</re:semanticMap>

The triples are generated by an RDFe processor, to which we pass the XML document and the semantic map. Command line invocation:

  shax "rdfe?dox=drugs.xml,semap=drugbank.rdfe.xml"

The result is a set of RDF triples in Turtle [7] syntax:

@prefix drug: <http://example.com/resource/drug/> .
@prefix ont: <http://example.com/ontology/drugbank/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix uniprot: <http://bio2rdf.org/uniprot:> .

drug:DB00001 
        rdf:type            ont:drug ;
        rdfs:label          "Lepirudin" ;
        ont:updated         "2018-07-02"^^xs:date ;
        ont:drugbank-id     "DB00001" ;
        ont:drugbank-alt-id "BTD00024" ;
        ont:drugbank-alt-id "BIOD00024" ;
        ont:enzyme          uniprot:P00734 ;
        ont:enzyme          uniprot:P00748 ;
        ont:enzyme          uniprot:P02452 ;
             …
drug:DB00002 
        rdf:type ont:drug ; 
             …
drug:DB00003 
        rdf:type ont:drug ;
             …
…

Some explanations should enable a basic understanding of how the semantic map controls the output. The basic building block of a semantic map is a resource model. It defines how to construct the triples describing a resource represented by an XML node:

<re:resource modelID="drug" 
             assertedTargetNodes="/db:drugs/db:drug" 
             iri="db:drugbank-id[@primary eq 'true']/concat('drug:', .)" 
             type="ont:drug">
  <re:property iri="rdfs:label" 
    value="db:name" 
    type="xs:string"/>        
  <!-- more property models here -->
</re:resource> 

The @iri attribute on <resource> provides an XPath expression yielding the resource IRI. The expression is evaluated in the context of the XML node representing the resource. Note how the expression language XPath is used in order to describe the IRI as a concatenation of a literal prefix and a data-dependent suffix. Every node returned by the expression in @assertedTargetNodes, evaluated in the context of the input document, is mapped to a resource description as specified by this resource model element.

Each <property> child element adds to the resource model a property model. It describes how to construct triples with a particular property IRI. The property IRI is given by @iri, and the property values are obtained by evaluating the expression in @value, using the node representing the resource as context node. (In our example, the value expressions are evaluated in the context of a <drug> element.) As the examples show, the XPath language may be used freely, for example combining navigation with other operations like concatenation. The datatype of the property values is specified by the @type attribute on <property>. The special value #iri signals that the value is an IRI, rather than a typed literal. Another special value, #resource, will be explained in the following section.

Our drug document references articles:

<drugs xmlns="http://www.drugbank.ca">
  <drug type="biotech" created="2005-06-13" updated="2018-07-02">
    <drugbank-id primary="true">DB00001</drugbank-id>
   <!-- more content here -->
    <general-references>
      <articles>
        <article>
          <pubmed-id>16244762</pubmed-id>
          <citation>
            Smythe MA, Stephens JL, Koerber JM, Mattson JC: A c…</citation>
        </article>
        <!-- more articles here -->
      </articles>
    </general-references>
   <!-- more content here -->
</drugs>

RDF is about connecting resources, and therefore our RDF data will be more valuable if the description of a drug references article IRIs which give access to article resource descriptions - rather than including properties with literal values which represent properties of the article in question, like its title and authors.

Assume we have access to a document describing articles:

<articles>
  <article>
    <pubmed-id>16244762</pubmed-id>
    <url>https://doi.org/10.1177/107602960501100403</url>
    <doi>10.1177/107602960501100403</doi>
    <authors>
      <author>Smythe MA</author>
      <author>Stephens JL</author>
      <author>Koerber JM</author>
      <author>Mattson JC</author>            
    </authors>
    <title>A comparison of lepirudin and argatroban outcomes</title>
    <keywords>
      <keyword>Argatroban</keyword>
      <keyword>Lepirudin</keyword>
      <keyword>Direct thrombin inhibitors</keyword>
    </keywords>
    <citation>Smythe MA, Stephens JL, Koerber JM, Mattson JC: A ...</citation>
    <abstract> Although both argatroban and lepirudin are used ...</abstract>
  </article>
  <!—more articles here -->
</articles>

We write a second semantic map for this document about articles:

<re:semanticMap  iri="http://example.com/semap/articles/"
                 targetNamespace="" targetName="articles" …>
  <re:namespace iri="http://example.com/resource/article/" prefix="art"/>
  <!-- more namespace descriptors here -->
  <re:resource modelID="article" iri="pubmed-id/concat('art:', .)"
               targetNodeNamespace=""
               targetNodeName="article"
               type="ont:article">        
   <re:property iri="ont:doi" value="doi" type="xs:string"/>        
   <re:property iri="ont:url" value="url" type="xs:string"/>
   <re:property iri="ont:author" value=".//author" list="true" type="xs:string"/>
   <re:property iri="ont:title" value="title" type="xs:string"/>        
   <re:property iri="ont:keyword" value="keywords/keyword" type="xs:string"/>
   <re:property iri="ont:abstract" value="abstract" type="xs:string"/>
   <re:property iri="ont:citation" value="citation" type="xs:string"/>
  </re:resource>
</re:semanticMap/>

and we extend the resource model of a drug by a property referencing the article resource, relying on its XML representation provided by an <article> element:

<re:property iri="ont:ref-article" 
  value="for $id in .//db:article/db:pubmed-id return 
         doc('/ress/drugbank/articles.xml')//article[pubmed-id eq $id]" 
  type="#resource"/>

The value expression fetches the values of <pubmed-id> children of <article> elements contained by the <drug> element, and it uses these values in order to navigate to the corresponding <article> element in a different document. This document need not be provided by the initial input – documents can be discovered during processing. While the items obtained from the value expression are <article> elements, the triple objects must be article IRIs giving access to article resource descriptions. Therefore two things must be accomplished: first, the output must include triples describing the referenced articles; second, the ont:ref-article property of a drug must have an object which is the article IRI used as the subject of triples describing this article. The article IRI, as well as the triples describing the article are obtained by applying the article resource model to the article element. All this is accomplished by the RDFe processor whenever it detects the property type #resource. Our output is extended accordingly:

drug:DB00001 a ont:drug ;
   rdfs:label        "Lepirudin" ;
   …
   ont:ref-article   art:16244762 ;
        …
art:16244762 a ont:article ;
   ont:abstract  "Although both argatroban and lepirudin are used for ..." ;
   ont:author    "Stephens JL" , "Koerber JM" , "Mattson JC" , "Smythe MA" ;
   ont:citation  "Smythe MA, Stephens JL, Koerber JM, Mattson JC: A com … " ;
   ont:doi       "10.1177/107602960501100403" ;
   ont:keyword   "Argatroban" , "Lepirudin" , "Direct thrombin inhibitors" ;
   ont:title     "A comparison of lepirudin and argatroban outcomes" ;
   ont:url       "https://doi.org/10.1177/107602960501100403" .

The property model which we just added to the resource model for drugs contains a “difficult” value expression – an expression which is challenging to write, to read and to maintain:

  for $id in .//db:article/db:pubmed-id return 
  doc('/products/drugbank/articles.xml')//article[pubmed-id eq $id]"

We can simplify the expression by defining a dynamic context and referencing a context variable. A <context> element represents the constructor of a dynamic context:

<re:semanticMap  iri="http://example.com/semap/drugbank/" …>
  ...
  <re:context>
    <re:var name="articlesURI" value="'/products/drugbank/articles.xml'"/>
    <re:var name="articlesDoc" value="doc($articlesURI)"/>
  </re:context>
      …
</re:semanticMap>

The values of context variables are specified by XPath expressions. Their evaluation context is the root element of an input document, so that variable values may reflect document contents. A context constructor is evaluated once for each input document. The context variables are available in any expression within the semantic map containing the context constructor (excepting expressions in preceding siblings of the <var> element defining the variable). Now we can simplify our expression to

  for $id in .//db:article/db:pubmed-id return 
  $articlesDoc//article[pubmed-id eq $id]

As a context constructor may also define functions, we may further simplify the value expression by turning the navigation to the appropriate <article> element into a function. The function is defined by a <fun> child element of <context>. We define a function with a single formal parameter, which is a pubmed ID:

<re:context>
  <re:var name="articlesURI" value="'/products/drugbank/articles.xml'"/>
  <re:var name="articlesDoc" value="doc($articlesURI)"/>
  <re:fun name="getArticleElem" params="id"
          code="$articlesDoc//article[pubmed-id eq $id]"/>
  </re:function>            
</re:context>

Expressions in this semantic map can reference the function by the name getArticleElem. A new version of the value expression is this:

  .//db:article/db:pubmed-id/$getArticleElem(.)

For each input document a distinct instance of the context is constructed, using the document root as context node. This means that the context may reflect the contents of the input document. The following example demonstrates the possibility: in order to avoid repeated navigation to the <article> elements, we introduce a dictionary which maps all Pubmed IDs used in the input document to <article> elements:

  <re:var name="articleElemDict" 
          value="map:merge(distinct-values(//db:article/db:pubmed-id)
                           ! map:entry(., $getArticleElem(.)))"/>

An updated version of the value expression takes advantage of the dictionary:

  .//db:article/db:pubmed-id/$articleElemDict(.)

The dictionary contains only those Pubmed IDs which are actually used in a particular input document. For each input document, a distinct instance of the dictionary is constructed, which is bound to the context variable $articleElemDict whenever data from that document are evaluated.

A semantic extension is a set of one or more semantic maps, together defining a mapping of XML documents to a set of RDF triples. A semantic extension comprises all semantic maps explicitly provided as input for an instance of RDFe processing, as well as all maps directly or indirectly imported by these (see below).

A semantic map is a specification how to map a class of XML documents (defined in terms of target document constraints) to a set of RDF triples. It is represented by a <semanticMap> element and comprises the components summarized below.

A semantic map IRI identifies a semantic map unambiguously. The map IRI should be independent of the document URI.

The target document constraint is a set of conditions met by any XML document to which the semantic map may be applied. The constraint enables a decision whether resource models from the semantic map can be used in order to map nodes from a given XML document to RDF resource descriptions. A target document assertion is an XPath expression, to be evaluated in the context of a document root. A typical use of target document assertions is a check of the API or schema version indicated by an attribute of the input document.

A semantic map may import other semantic maps. Import is transitive, so that any map reachable through a chain of imports is treated as imported. Imported maps are added to the semantic extension, and no distinction is made between imported maps and those which have been explicitly supplied as input.

RDF namespace bindings define prefixes used in the output for representing IRI values in compact form. Note that they are not used for resolving namespace prefixes used in XML names and XPath expressions. During evaluation, XML prefixes are always resolved according to the in-scope namespace bindings established by namespace declarations (xmlns).

Context constructor and resource models are described in subsequent sections.

A resource model is a set of rules how to construct triples describing a resource which is viewed as represented by a given XML node. A resource model is represented by a <resource> element and comprises the components summarized below.

The resource model ID is used for purposes of cross reference. A resource model has an implicit resource model IRI obtained by appending the resource model ID to the semantic map IRI (with a hash character (“#”) inserted in between if the semantic map IRI does not end with “/” or “#”).

The resource IRI expression yields the IRI of the resource. The expression is evaluated using as context item the XML node used as target of the resource model.

A target node assertion is an expression to be evaluated in the context of each input document passed to an instance of RDFe processing. The expression yields a sequence of nodes which MUST be mapped to RDF descriptions. Note that the processing result is not limited to these resource descriptions, as further descriptions may be triggered as explained in the section called “Linking resources”.

A target node constraint is a set of conditions which is evaluated when selecting the resource model which is appropriate for a given XML node. It is used in particular when a property model treats XML nodes returned by a value expression as representations of an RDF description (for details see the section called “Linking resources”).

Resource type IRIs identify the RDF types of the resource (rdf:type property values). The types are specified as literal IRI values.

Property models are explained in the following section.

A property model is represented by a <property> child element of a <resource> element. The following table summarizes the major model components.

The property IRI defines the IRI of the property. It is specified as a literal value.

The object value expression yields XDM items [11] which are mapped to RDF terms in accordance with the settings of the property model, e.g. the object type. For each term a triple is constructed, using the term as object, a subject IRI obtained from the IRI expression of the containing resource model, and a property IRI as specified.

The object type controls the mapping of the XDM items obtained from the object value expression to RDF terms used as triple objects. The object type can be an XSD data type, the token #iri denoting a resource IRI, or the token #resource. The latter token signals that the triple object is the subject IRI used by the resource description obtained for the value item, which must be a node. The resource description is the result of applying to the value node an appropriate resource model, which is either explicitly specified (@objectModelID) or determined by matching the node against the target node constraints of the available resource models.

The language tag is used to turn the object value into a language-tagged string.

The object resource model is evaluated in conjunction with object type #resource. It identifies a resource model to be used when mapping value nodes yielded by the object value expression to resource descriptions.

The RDF list flag indicates whether or not the RDF terms obtained from the object value expression are arranged as an RDF list (default: no).

The reverse flag can indicate that the items obtained from the object value expression represent the subjects, rather than objects, of the triples to be constructed, in which case the target node of the containing resource model becomes the triple object.

Conditional settings is a container for settings (e.g. property IRI or object type IRI) applied only to those value items which meet a condition. The condition is expressed by an XPath expression which references the value item as an additional context variable (rdfe:value).

Using RDFe, the construction of RDF triples is based on the evaluation of XPath expressions. Evaluation can be supported by an evaluation context consisting of variables and functions accessible within the expression. The context is obtained from a context constructor represented by a <context> element. A distinct instance of the context is constructed for each XML document containing a node which is used as context node by an expression from the semantic map defining the context. The context constructor is a collection of variable and function constructors. Variable constructors associate a name with an XQuery expression providing the value. Function constructors associate a name with an XQuery function defined in terms of parameter names, return value type and an expression providing the function value. As the expressions used by the variable and function constructors are evaluated in the context of the root element of the document in question, variable values as well as function behaviour may reflect the contents of the document. Variable values may have any type defined by the XDM data model, version 3.1 [11] (sequences of items which may be atom, node, map, array or function). Context functions are called within expressions like normal functions, yet provide behaviour defined by the semantic map and possibly dependent on document contents.

Processing input is

Processing output is a set of RDF triples, usually designed to express semantic content of the XML documents.

The set of contributing semantic maps consists of the set explicitly supplied, as well as all semantic maps directly or indirectly imported by them.

The set of contributing XML documents is not limited to the initial input documents, as expressions used to construct triples my access other documents by derefencing URIs found in documents or semantic maps. This is an example of navigation into a document which may not have been part of the initial set of input documents:

 <re:property iri="ont:country" type="#resource" 
              value="country/@href/doc(.)//country"/>

RDFe thus supports a linked data view.

Understanding the processing of semantic maps is facilitated by the auxiliary concepts of a “hybrid triple” and a “preliminary resource description”. When a property model uses the type specification #resource, the nodes obtained from the object value expression of the property model are viewed as XML nodes representing resources, and the triple objects are the IRIs of these resources. The resource is identified by the combined identities of XML node and resource model to be used in order to map the node to a resource description. When this resource has already been described in an earlier phase of the evaluation, the IRI is available and the triple can be constructed. If the resource description has not yet been created, the IRI is still unknown and the triple cannot yet be constructed. In this situation, a hybrid triple is constructed, using the pair of XML node and resource model ID as object. A hybrid triple is a preliminary representation of the triple eventually to be constructed. A resource description is called preliminary or final, dependent on whether or not it contains hybrid triples. A preliminary description is turned into a final description by creating for each hybrid triple a resource description and replacing the hybrid triple object by the subject IRI used by that description. The resource description created for the hybrid triple object may itself contain hybrid triples, but in any case it provides the IRI required to finalize the hybrid triple currently processed. If the new resource description is preliminary, it will be finalized in the same way, by creating for each hybrid triple yet another resource description which also provides the required IRI. In general, the finalization of preliminary resource descriptions is a recursive processing which ends when any new resource descriptions are final.

The scope of processing is controlled by the asserted resource descriptions, the set of resource descriptions which MUST be constructed, given a set of semantic maps and an initial set of XML documents. Such a description is identified by an XML node representing the resource and a resource model ID identifying the model to be used for mapping the node to an RDF description. (Note that for a single XML node more than one mapping may be defined, that is, more than one resource model may accept the same XML node as a target.) The asserted target nodes of a resource model are the XML nodes to which the resource model must be applied in order to create all asserted resource descriptions involving this resource model.

Any additional resource descriptions are only constructed if they are required in order to construct an asserted resource description. An additional resource description is required if without this description another description (asserted or itself additional) would be preliminary, that is, contain hybrid triples. As the discovery of required resource descriptions may entail the discovery of further required resource descriptions, the discovery process is recursive, as explained in the section called “Hybrid triples and preliminary resource description”.

The asserted target nodes of a resource model are determined by the target node assertion of the resource model, an expression evaluated in the context of each initial XML document. Note that the target node assertion is not applied to XML documents which do not belong to the initial set of XML documents. Such additional documents contribute only additional resource descriptions, no asserted resource descriptions. Initial documents, on the other hand, may contribute asserted and/or additional descriptions.

The processing of semantic maps can now be described as a sequence of steps:

For a formal definition of the processing see Appendix A, Processing semantic maps - formal definition.

Minimal conformance requires a processing as described in this paper. It includes support for XPath 3.1 expressions in any place of a semantic map where an XPath expression is expected:

If an implementation provides the XQuery Expressions Feature, it must support XQuery 3.1 [12] expressions in any place of a semantic map where an XPath expression is expected.

An implementation may support implementation-defined XPath extension functions. These may in particular enable the parsing of non-XML resources into XDM node trees and thus support the RDFe-defined mapping of non-XML resources to RDF triples.