Controlled Vocabularies

Licence controlled vocabularies

• Be published under an open licence.
• Be operated and/or maintained by an institution of the European Union, by a recognised standards organisation or another trusted organisation.
• Be properly documented.
• Have labels in multiple languages, ideally in all official languages of the European Union.
• Have terms that are identified by URIs with each URI resolving to documentation about the term.
• Have associated persistence and versioning policies.
• Provide a human readable representation.
• Provide a machine readable representation.
• Include in the machine readable representation a mapping to the EU Vocabularies Licences [[NAL-Licence]].

Mapping the HVD IR to DCAT-AP

Alignment of terminology

The HVD implementation regulation uses the terms Dataset, Bulk Download and API.

In the context of DCAT-AP, a HVD Dataset is mapped on a Dataset, Bulk Download on a Distribution and API on a Data Service. To be conformant with the use of DCAT-AP in the context of the HVD IR, this mapping MUST be followed.

To make the text easier to read, with a HVD Dataset we mean a Dataset in scope of the HVD implementing regulation. The same pattern is applied to other entities.

Dataset Series

As the HVD IR relies on strengthening the existing data sharing practices in the different domains, DCAT-AP HVD follows the same approach. It means that in case that according to the domain a Dataset Series would be considered a HVD Dataset it should be identified in the same approach as for HVD Datasets. As a Dataset Series is often viewed as a Dataset also the HVD categorisation is imposed. More on this identification is found in the next section.

In this annex, no further advises or guidelines are provided to determine whether a dataset should be considered or represented as a Dataset Series. That is not provided by the HVD IR and therefore these advises or guidelines are to be discussed within the context of the generic DCAT-AP.

In scope of HVD IR

A Dataset is a HVD Dataset if and only if a MS has included it in its reporting. The HVD IR defines High-Value Datasets. It may be possible that the same definition applies to multiple entities. In that case, a Member State should select the most appropriate one, according to the rules in the regulation. If the Member State decides to include multiple entities in the reporting, the requirements set out in the HVD IR will apply to all these entities. Also, if a Member State decides to include a dataset in the HVD reporting for which inclusion is not mandatory, then the requirements of the HVD IR will apply. The report is an engagement of the Member State to the European data community to sustain those datasets.

If a re-user discovers a dataset that seems to be in scope of the HVD IR, then the responsible MS should be able to provide an explanation why it is not included in the reporting. One response to this question could be by providing the relevant HVD Dataset corresponding to that dataset.

It is important to note that the identification of a dataset to be a HVD Dataset is an action that has to be approved by its Member State representative responsible for the implementation of the HVD IR. The official reporting and the metadata shared to the public SHOULD coincide. Member States therefore must coordinate their activities to avoid unintended deviations. Consequently, the application of DCAT-AP HVD is only necessary if the intention is to comply with the HVD IR and DCAT-AP metadata is being exchanged. As stated in the introduction, the use of DCAT-AP HVD is not obliged by the HVD IR, but in the case of exchanging DCAT-AP metadata of HVD Datasets this specification should be followed.

Denoting a HVD Dataset

Special cases

When a Dataset is within scope of HVD, it is not mandatory that all distributions are within scope of HVD. Existing metadata remains valid. Our recommendations ensure that existing metadata (specified in DCAT-AP or other frameworks like INSPIRE) remains valid. Becoming a Dataset in scope of HVD is an additional operation.

When a Data Service offers access to multiple datasets and this Data Service fulfils the HVD requirements (e.g. the HVD API for that dataset) for a HVD then the HVD requirements apply only to that HVD. It is common that the same API service endpoint (denoted by a dcat:DataService) provides access to multiple datasets. As such, it is to be expected that only some of the datasets are within scope of HVD. Like for Distributions, the HVD does not enforce that all Datasets associated with a Data Service must be in scope of HVD. Nevertheless, it must be noted that the HVD requirements on a Data Service might indirectly impact the other datasets that are available through the same data service, because a Data Service will share the operational and service level requirements for all its associated datasets.

HVD data category

Identifiers

In general, the requirements of the HVD IR are satisfied when the best practices of DCAT-AP on identifiers are followed. According to HVD IR the identifiers provided in the report should be an online reference to the metadata.

In short these are:

• Identifiers for the main entities (Catalogue, Dataset, Data Service and Distribution) are managed URIs.
• These URIs are dereferenceable, i.e. users are redirected to the metadata describing the entity.
• URIs are well defined, i.e. follow the 10 principles for good URIs.
• Data portals and harvesters implement the identifier guidelines in order not to lose information of alternative identifiers.
• When the metadata in DCAT-AP originates from other metadata (e.g. INSPIRE) then a cross-reference to that original metadata is included. This can be done by using the same identifiers, but also using another relation like the one expressed using dct:source.

Persistent identifiers

The HVD IR requires as part of the reporting requirements (article 5.3), that Licensing Conditions and APIs have persistent links.

Persistence means that, for these entities, Member States take the responsibility to maintain the real world resource indefinitely and additionally reduce the accessibility challenge by maintaining the same name for that real world resource indefinitely. Thus for the entities that MSs include in the reporting and for which the reporting requires a persistent link, a MS makes a persistent commitment.

As DCAT-AP is a Semantic Web data specification, persistence is associated with the use persistent URIs (PURIs) for the metadata descriptions. A general advice for DCAT-AP implementers is to use PURIs for all entities, but mostly for Datasets and Data Services. The practice, though, shows that this is not universally applied. To reduce this gap between the intention and the practice DCAT-AP has proposed a number of guidelines on identifiers [[IdentifierGuidelines]]. Implementers of the HVD IR are advised to read these guidelines to understand how identity might or might not be preserved from one data portal to another, and take the appropriate actions.

In article 5.3 of the HVD IR, the broad term Licensing Conditions is used, while in the other parts of the regulation the term Licence is used. DCAT-AP provides several means to express legal information, notably the properties licence (dct:license) and rights (dct:rights). This may lead to questions whether rights are included by the reporting requirement. As the final objective is to provide a trusted legal statement, it is considered that the requirement for a persistent link applies to rights too.

The reporting requirement for a persistent link for APIs is ambiguous from the perspective of DCAT-AP. In DCAT-AP, there is the identifier for the Data Service, i.e. the description about the API, and the property endpoint URL, which is the technical endpoint via which the data exchange will happen. The impact that the persistency requirement has on each is different and requires special attention from the HVD Dataset publisher. As the HVD IR does not specify precisely which case it covers, both are considered in scope. That means that a Data Service has a PURI and that its endpoint URL is persistent.

DCAT-AP does not impose persistent identification of an endpoint URL. It, however, expects a life-cycle management of the API through metadata. For that, DCAT-AP recommends to follow the DCAT guidelines on Resource life-cycle.

For example, consider an API which is at the end of its lifecycle. According to DCAT(-AP), the PURI of the Data Service could get the status ‘deprecated’ and the endpoint URL could be made void when it is taken offline. Any data portal user would understand that this Data Service should not to used anymore. If the metadata is augmented with the information about the successor of the Data Service, the data portal owner can be guided to the new Data Service.

The impact for a user of the endpoint URL is higher: systems might get broken when the endpoint URL is taken offline. This situation is the result of a shared responsibility: either the publishers did not apply a decent life cycle management, or the users did not inform the publisher about their critical dependency. Because of this, even if the API gives open access to open data, users that are dependent on the API are advised to inform the publisher about their existence. But also, publishers must improve their life cycle management for these APIs so that re-users get the right information and can take the change of the endpoint of the API into account in their roadmaps.

The enforcement of a persistent link for the endpoint URL will reduce the occurrence of such cases, but it will not make them disappear. And this enforcement imposes additional care on the Data Services (APIs) by HVD publishers. When an API is moved to a new platform (e.g. from a local API gateway to an organisation-wide one), the original endpoint URL must be maintained, and also the metadata management must be maintained.

In summary, the recommendation is to have persistency for both aspects of the API: its metadata identifier as its endpoint URL.

Legal information

The HVD IR requires a high level of metadata quality for legal information. The information should be provided in machine and human readable format, using a persistent link. Furthermore, it should be possible to investigate whether the legal conditions are equal or more permissive than the reference CC-BY 4.0.

Despite these strong requirements, the HVD IR does not alter the recommendations and practice of expressing legal information in DCAT-AP. The HVD requirements do extend or precise how the legal information technically should be provided. In the DCAT-AP legal information corresponds to licences and as well to rights expressions. In currently allowed practice, licence information may thus be supplied by a collection of rights statements, in cases that national legislation does not allow to provide a licence document. This is compatible with the HVD IR, and in that case, the HVD requirements will also apply to the rights statements. HVD IR also does not force to adapt the current DCAT-AP principle to indicate the legal information at the most precise level in the metadata description: i.e., Data Service and Distribution, therefore this principle is maintained. The latter has also the consequence that a Distribution or Data Service must be supplied for sharing the legal information.

Catalogue owners are advised to assess the legal information provided by the publishers according to flows in the figures below. For instance, if a publisher provides licence information referring to a licence document made online accessible by the publisher itself, then the publisher of that information must implement the HVD quality requirements for licence documents. The decision trees in the figures allow to assess whether or not additional effort has to be performed.

In the reporting requirements of the HVD IR, the notion terms of use is used. It has been agreed, by the Working Group for DCAT-AP HVD, that providing terms of use information is the same as providing legal information for a Data Service.

Assessment support for licences

• use properties that express a strong equality: skos:exactMatch or owl:sameAs
• use properties that express a permissiveness relation: skos:broadMatch or skos:narrowMatch

Using the SKOS matching properties requires to agree on a direction to indicate "more permissive than". In accordance to the hierarchy relationship of SKOS in which the most generic concept (the concept higher in the hierarchy, i.e. closer to the top-concept) is more general than the leaf concept the recommendation is to have the most permissive licences at the top and the most restrictive at the bottom.

From the definition of the SKOS relations and the definition of the corresponding SKOS mapping relations, a MS specific licence will be expressed as follows as more permissive than CC-BY-4.0.

Contact Point

The HVD IR request a contact point for APIs.

This requirement is implemented as the following recommendation. A contact point is mandatory for HVD Data Services and recommended for HVD Datasets either in the form of a (persistent) email address or a link to a contact form on a webpage, e.g. to contact a service desk.

Specific data requirements

The HVD implementation regulation describes, in its Annex, precisely the data elements that should be provided for a HVD Dataset. A HVD Dataset must conform to the rules defined in the HVD IR.

It is recommended to provide a reference to a public document (for instance: data standards) that describes the internals of the Dataset (or Distribution) using the property conforms to. This ensures that the information is made publicly accessible for re-users. It can be used by experts to verify if the Dataset matches the HVD requirements.

An alternative approach is the use of a self declaration of conformance. In this case the publisher of the HVD Dataset declares itself that it conforms to all data technical details the HVD IR imposes. The INSPIRE community has used this approach before. As the assessment of the validity of such self-declaration is a domain specific activity requiring expert knowledge, DCAT-AP HVD does not proposes a general self declaration statement. But leaves the design of such statement to the data experts of the HVD categories to provide an trustable approach.

Connecting Data Services with Datasets

• Direct: The Data Service refers with the property serves dataset to the Dataset.
• Indirect: The Data Service is referred by the property access service from a Distribution to a Dataset.

In case of a HVD reporting the indirect approach will be transformed into the direct approach. This is a natural consequence from the HVD IR wording: the dataset shall be available for reuse through APIs. Technically it corresponds to the following closure: the APIs for a HVD Dataset are the Data Servies that are directly or indirectly connected with the HVD Dataset.

Data publishers that solely use the indirect approach might face a undesired side effect of their choice: namely that they must encode a HVD Distribution even that is not required nor necessary. That is a consequence of the local publishers agreements but it might have policy impacts: namely also that superfluous Distribution becomes subject to the HVD IR.

Having two routes to declare the presence of a API makes the expression of the obligation as a single cardinality in the specification non trivial. To ensure that at least an API is present for a HVD Dataset the SHACL validation rules have been extended.

A similar request, although less visible, is the possibility of downloading the data in bulk. The Annex of the HVD IR requests frequently this need. Despite it is not obligatory in all cases, it is a strong recommendation. In the formal model such strong recommendations are hard to implement. Adding a minimum cardinality of 1 would enforce the presence, which is too strict. But relaxing the minimum cardinality may be an opening to not provide this information. Reviewers have pointed out this challenge. Therefore the following approach has been applied in this specification to ensure that publishers consider this need. The formal constraints have been relaxed, the usage qualification has been set to recommendation, and additional usage notes including this explanation have been added. In addition the SHACL validation rules have been extended with a check to validate if each HVD dataset has a HVD Distribution. This check has severity Violation, to force publishers to reconsider in case of missing distributions their provided information.

Forcing the presence of at least one distribution or data service for a Dataset, is also implicitly the result of the need for providing legal information. As DCAT-AP mandates this to be expressed at the level of Distributions or Data Services, a Dataset will have no legal information associated if none of them is present. This supports the case for a strong validation check.

Reporting

Example

Datasets in scope of HVD

Consider that a dataset "The population of bees" is within scope of the HVD while another dataset "The population of wasps" is not. Both datasets however, are in scope of the INSPIRE directive. The dataset "The population of bees" is also tagged with the appropriate HVD Category. Observe that the solely the most granular concept "Nature preservation and biodiversity" is provided. The generic top-level category can be derived from it.

Both datasets are published by the Environment Agency of the EU Memberstate ExampleMS using a persistent identifier.

The datasets are published on the EU Memberstates national data portal https://dataportal.exampleMS.gov. This portal provides another identifier to the datasets. Because that new identifier is not the master identifier, the portal avoids this by sharing this identifier in its published DCAT-AP catalogue by listing it as an additional identifier.

Bulk downloads for HVD Datasets

The datasets are downloadable in various formats and level of detail. In our example, the data is available in two formats: RDF and ESRI shapefile format. According to the HVD IR the datasets must minimally be available in bulk download with the granularity of 50 square kilometres and with a bi-yearly update frequency. It must also be available in an open format for geospatial data. Based on these requirements, the publisher of the dataset decides to indicate that the shape-based distribution is a HVD bulk download.

The HVD IR also specifies that the dataset should at least provide information about the number of bees, the calculation method, the amount of honey being harvested and the number of beekeepers active in the area. The publisher describes the data semantically using an application profile, and provides detailed data schema documentation for each distribution.

The HVD Dataset is accessible via an API

According to the HVD IR, the "bee population" dataset must be made available via an API. The dataset publisher has an API platform deployed, via which data users have access to realtime data. This API platform supports all datasets of the publisher.

Because the API platform is provided as the API for the "bee population" dataset, the HVD implementing regulation requirements apply. This means that the endpoint URL must be persistent. The publisher should perform maximal effort to keep the endpoint URL stable. For instance, deploying a new API platform or changing organisation names should not impact the endpoint URL.

To provide information about the use of the API platform, the publisher provides OpenAPI technical documentation and an SLA to document the quality of service.

To address any questions by the users the publisher operates a service desk.

Expressing legal conditions

The Member State (MS) imposes, via its legislation, to use national data licences for public bodies. Therefore, the dataset publisher is required to use one of them. As support to the community, the MS has published the data licences as a SKOS taxonomy, using persistent URIs. For the "wasp population" dataset, a restrictive licence is chosen because the data is based on information that has commercial rights including fees. The "bee population" dataset is shared with a very permissive licence.

In order to support the assessment of the used licences, the MS maps the licences to the NAL Licences [[NAL-Licence]].

The HVD IR requires the licence for the Bee population dataset is at least as permissive as CC-BY 4.0. Since the Bee population licence is https://data.exampleMS.gov/resource/FreeAndOpen and it is an exact match with http://publications.europa.eu/resource/authority/licence/CC0, and this CC0 licence is more permissive than CC-BY-4.0, the HVD requirement is met.

Because in producing the RDF representation additional provenance information is included that is sensitive, the publisher changes the licence for that distribution to a more restrictive one.

Although this restricted licence https://data.exampleMS.gov/resource/NoCommercialUseWithFees does not meet the HVD requirements for the "bee population" dataset, the "bee population" dataset is still conformant to the HVD implementing regulation as the RDF distribution was not within scope of the HVD. The same reasoning holds for the "wasp population" dataset. This illustrates the flexibility the DCAT-AP HVD specification offers to address complex and rare scenarios data publishers might face.

The Data Service exampleMS:EAMS-APIplatform provides access to both datasets. The legal conditions on the usage of the platform for the "bee population" dataset is a combination of the API platform conditions (e.g. no misuse by triggering DDOS activities, no sharing of access tokens to third parties, etc. ) and the dataset conditions. The API request https://orgea.exampleMS.gov/api/v2/beepopulation/ has thus different conditions than https://orgea.exampleMS.gov/api/v2/wasppopulation/. Therefore, the nature of the licence document, associated with a Data Service, is usually more oriented to the use of the API platform rather than to the use of the data it provides access too.

In the example the 'Terms of Use' for the API platform are mentioned as the license. In addition, the API platform can also indicate the SLA it offers.

Reporting

The MS reports its HVD conformance status by providing a catalogue containing all metadata in scope of HVD. To facilitate the conformance assessment, it will only include the Datasets, Dataset Series, Data Services and Distributions that are in scope of HVD. The catalogue will also contain any additional supportive information such as ContactPoints, Agents and the mapping for the licences to the EU Licences.

To reduce the risk of misinterpretation, the Catalogue Resource connecting properties such as dcat:servesDataset and dcat:distribution should be inspected to not refer to Catalogued Resources outside the scope of HVD. In the example below, the reference to the RDF distribution for the "bee population" and the "wasp population" dataset are removed from the reporting catalogue. During the collection stage of the common reporting approach a sparql query will perform this scoping.

Based on this catalogue the MS can be audited for its conformance. During the assessment it might occur that the supplied information is not sufficient, and that the assessment must follow the references outside the supplied catalogue. E.g., when assessing the permissiveness of the licences the details of the referenced EU Licence must be consulted. Crossing these boundaries is a regular occurrence and it can be done during the assessment without impacting the results when the supplied data is based on persistent identifiers (PURIs).

The use of dereferenceable persistent identifiers could also lead to another agreement to supply a more condensed representation of the reporting catalogue. Under the condition that all catalogued resources in scope of HVD are in the Portal for European Data [[DEU]], the reporting by an MS could be reduced to sharing the minimal information how to collect these resources. In the common reporting approach the set of harvested MS catalogues containing HVD information is requested. With that all the necessary information for the reporting is extracted from the DEU.

Validation

• base.ttl: all base constraints expressed in this document.
• ranges.ttl: all the class range constraints expressed in this document.
• usagenotes.ttl: additional constraints that are expressed as usagenotes or recommendations. Including controlled vocabularies, and the presence of an API or a bulk download for a dataset.