Implementation report: publishing a dcat-ap-feed for BOSA (Federal Public Service for Policy and Support) Belgium

Living Standard,

This version:
https://semiceu.github.io/LDES-implementation-reports/bosa-dcat-ap-feed/index.html
Issue Tracking:
GitHub
Editors:
- Julian Rojas Melendez

CC0 To the extent possible under law, the editors have waived all copyright and related or neighboring rights to this work. In addition, as of 16 June 2026, the editors have made this specification available under the Open Web Foundation Agreement Version 1.0, which is available at https://www.openwebfoundation.org/the-agreements/the-owf-1-0-agreements-granted-claims/owfa-1-0. Parts of this work may be from another specification document. If so, those parts are instead covered by the license of that specification document.

Abstract

This implementation report describes how the Federal Public Service for Policy and Support (BOSA) Belgium publishes data resource updates following the DCAT-AP-feeds specification which relies on LDES and the Activity Streams v2.0 vocabulary. The feed runs and is published both at BOSA’s premises and by relying on a GitHub action, which stores the static files of the feed in a public GitHub repository. This also reports on the technical components developed to integrate with the EU Publications Office platform (Piveau) to enable the ingestion and synchronization of DCAT-AP-feeds.

1. Publishing changes about the Belgian federal data catalog as a DCAT-AP-Feed

This section describes how BOSA publishes changes from the Belgian federal DCAT-AP catalog as a DCAT-AP-Feed [DCAT-AP-FEEDS], which is a specific type of Linked Data Event Stream (LDES) [LDES] using Activity Streams entities [activitystreams-vocabulary].

The source code implementation of the pipeline is maintained at rdf-connect/belgium-dcat-ap-feed. The pipeline is implemented with (RDF-Connect) [RDF_CONNECT] and transforms the Belgian DCAT-AP catalog dump into an append-only feed of Activity Streams events.

This implementation is currently deployed in two publication contexts:

  1. BOSA premises deployment: a live feed is published at mqa.data.gov.be/ldes. At the time of writing, this deployment does not yet include the latest updates from the upstream pipeline source code. Coordination with BOSA is ongoing to align their deployment with the newest implementation state.

  2. GitHub Pages deployment: a live feed is also published at rdf-connect.github.io/belgium-dcat-ap-feed/index.trig. This feed is generated and published automatically by a GitHub Action that runs the RDF-Connect pipeline and commits the resulting static feed artifacts, which are then published via GitHub Pages.

1.1. Used prefixes

The following prefixes are used throughout this document:

as: https://www.w3.org/ns/activitystreams#
dct: http://purl.org/dc/terms/
dcat: http://www.w3.org/ns/dcat#
ldes: https://w3id.org/ldes#
tree: https://w3id.org/tree#
rdf: http://www.w3.org/1999/02/22-rdf-syntax-ns#
xsd: http://www.w3.org/2001/XMLSchema#
geodcat: http://data.europa.eu/930/

1.2. General description of the BOSA DCAT-AP feed

The BOSA DCAT-AP feed is an append-only event stream represented as an RDF resource typed ldes:EventStream that is given an IRI. The ldes:EventStream has the ldes:timestampPath property set to as:published, which configures the property used in as:Activity to point to the entity that is the direct object of the activity being altered.

The feed relies on RDF Named Graphs. The static documents are published using application/trig as MIME type.

The BOSA ldes:EventStream definition 
<https://mqa.data.gov.be/ldes/index.trig> a ldes:EventStream, dcat:Dataset;
    ldes:timestampPath as:published;
    dct:title "DCAT-AP Feed of Data.gov.be"@en;
    tree:shape <https://semiceu.github.io/LDES-DCAT-AP-feeds/shape.ttl#ActivityShape>;
    tree:view <https://mqa.data.gov.be/ldes/https_3A_2F_2Fw3id.org_2Fsds_23Stream/index.trig>.

1.3. Activities

The pipeline generates ActivityStreams events (Create, Update, Delete) for changed resources. These activities use the property as:object with the IRI of the DCAT-AP entity. They also use the as:published property with an xsd:dateTime datatype. The activity is identified using an IRI, and the payload that corresponds to the DCAT-AP entity is provided within a named graph matching the activity IRI.

An activity in the BOSA DCAT-AP feed in TriG 
<#ff209fce82bd66a1f46376d704e3ce8c> rdf:type as:Create ;
    as:object <http://data.gov.be/.well-known/genid/dataset/bmdc/bmdc.be:dataset:561> ;
    as:published "2026-06-15T15:31:10.072Z"^^xsd:dateTime .

<#ff209fce82bd66a1f46376d704e3ce8c> {
    <http://data.gov.be/.well-known/genid/dataset/bmdc/bmdc.be:dataset:561> a dcat:Dataset ;
        geodcat:custodian <http://data.gov.be/.well-known/genid/org/d135a1c37e6fb9677908dd2355d646b467d50497> ;
        geodcat:referenceSystem <http://www.opengis.net/def/crs/EPSG/0/4258> ;
        geodcat:resourceType <http://inspire.ec.europa.eu/metadata-codelist/ResourceType/dataset> ;
        geodcat:topicCategory <http://inspire.ec.europa.eu/metadata-codelist/TopicCategory/oceans> ;
        dct:created "2014-04-18T14:23:19Z"^^xsd:dateTime ;
        dct:description "Cet ensemble de données se compose des sources suivantes: (1) G. Billen (1976). Evaluation of nitrifying activity in sediments by darc 14C-bicarbonate incorporation (1976).Water Research Vol. 10, pp. 51-57 (2) 1977. Stalen Noordzee : kruisvaart 1 maart 1977 en 2 mei 1977. Bepaling van nitraat, nitriet, ammoniac, fosfaat, silicium. (3) 1978. Analyse van nitraat, nitriet, fosfaat, silicium: Kruisvaarten 8-16, 1978. (4) D. Van Gansbeke, A. Braeckman, M. De Keere (1980). Nutrientenanalyse van de monitoring van de kustzone door de automatische analyseketen (A11-systeem).Benthos/1980:1 (5) Baeyens, Mommaerts, Goeuens, Dehairs, Dedeurwaerder, Decadt (1984). Schémas dynamiques de l'azote dissous dans la baie sud de la mer du Nord.Sciences estuariennes, côtières et du plateau (1984) 18, 499-510 (6) PMPZ-DBII CRA-OC Nutriments. (7) Martine Somville (s.d.). Dosage de la nitrate réductase en Mer du Nord et dans la partie aval de l'estuare de l'Escaut. (8) FLEX 1976: Analyse van nitraat, nitriet, fosfaat, silicium. (9) J.P. Mommaerts, W. Baeyens, G. Decadt (s.d.). Groupe « Matières organiques » : Rapport de synthèse 1977-1978. Synthese des recherches sur les nutriments.OM/1977-1978: 02 (10) Kruisvaarten 1977 (1-7): bepaling van NO2-, NO3-, NH3, PO4---, silicium. (11) Martine Somville (s.d.). Nitrification et dénitrification dans l'estuaire de l'escaut.OM/1978 20 (12) A. Vandenhoudt (s.d.). Analyses des éléments nutritifs du coq van het de l'enquête van de maandelijkse in de zuidelijke noordzee, 1979.Survey/1980:01 (13) A. Vandenhoudt (s.d.). Nutrient analyses van de maandelijkse survey's van het rooster in de zuidelijke noordzee, 1980.Survey/1981:01 (14) A. Vandenhoudt (s.d.). Analyses nutritionnelles du coq van het de l'enquête van de maandelijkse in de zuidelijke noordzee, 1981.Survey/1982:01 (15) A. Gaspar, J.H. Hecq, M. Licot (s.d.). Résultats des dosages de l'activité enzymatique nutritionnelle du zooplancton (activité amylasique).SURVEY/1981 : 07 (16) J. P. Mommaerts, I. Elskens (s.d.). Seizoen variatie en ruimtelijke verspreiding van de nutrienten in de zuidelijke Noordzee.Enquête 1978: 02 (17) Decadt, Goeyens (s.d.). Seizoenvariatie en ruimtelijke verspreiding van de nutrienten in de Zuidelijke Noordzee.Enquête 78.07"@fr-t-en;
        dct:identifier "doi:10.24417/bmdc.be:dataset:561" ;
        dct:issued "2026-05-13T15:12:08.556Z"^^xsd:dateTime ;
        dct:language <http://publications.europa.eu/resource/authority/language/ENG> ,
            <http://publications.europa.eu/resource/authority/language/FRE> ,
            <http://publications.europa.eu/resource/authority/language/NLD> ;
        dct:modified "2019-05-02T15:44:08Z"^^xsd:dateTime ;
        dct:publisher <https://org.belgif.be/id/CbeRegisteredEntity/0353070496> ;
        dct:rightsHolder <https://org.belgif.be/id/CbeRegisteredEntity/0353070496> ;
        dct:spatial <http://marineregions.org/mrgid/22253> , <http://marineregions.org/mrgid/2350> ;
        dct:temporal [
            a dct:PeriodOfTime ;
            dcat:endDate "1982-01-31"^^xsd:date ;
            dcat:startDate "1977-01-01"^^xsd:date
        ] ;
        dct:title "CRA : Nutriments dans la baie sud de la mer du Nord, par VUB, ULB et MUMM (1976-1982)"@fr-t-en 
        ...
        .
}

1.4. Bucketization and state tracking

Changes are tracked by comparing the current catalog dump with the previous state stored in a local LevelDB instance (pipeline/leveldb/). The generated activities are then bucketized using a tree:TimebasedFragmentation strategy with the following parameters:

Bucketization state is persisted in pipeline/feed-state/bucketize-state.json, allowing incremental execution and reproducible feed generation across runs.

1.5. Traversing the search tree

The feed MUST follow the chronological search tree from the Server Primer [LDES-SERVER-PRIMER]. Consequently, LDES clients SHOULD traverse the feed using the chronological ascending order mode specified in the LDES specification [LDES].

2. Implementation report

2.1. Architecture

The feed generation is implemented as an RDF-Connect [RDF_CONNECT] pipeline defined in pipeline/github-pipeline.ttl in the source repository. The current implementation composes six processors in sequence:

  1. **rdfc:GlobRead (<globReader>)** reads the compressed DCAT data dump ../all/datagovbe_edp.xml.gz.

  2. **rdfc:GunzipFile (<gunzipper>)** decompresses the RDF/XML dump.

  3. **rdfc:DumpsToFeed (<asConverter>)** converts the dump to Activity Streams events based on the stored state from previous executions.

  4. **rdfc:Sdsify (<sdsifier>)** annotates members with SDS metadata which allows for stream-based fragmentation and publishing as a LDES.

  5. **rdfc:Bucketize (<bucketizer>)** applies time-based fragmentation and writes state to ./feed-state/bucketize-state.json.

  6. **rdfc:LdesDiskWriter (<ldesDiskWriter>)** writes the resulting feed fragments into ../docs/.

The data and metadata channels are explicitly wired between processors (<compressed_data>, <decompressed_data>, <as_stream>, <sds_stream>, <sds_metadata>, <bucketized_stream>, <bucketized_metadata>), which ensures deterministic ordering from source dump ingestion to static feed publication.

For the GitHub Pages deployment, this pipeline is executed by .github/workflows/create-feed.yml on each push to master and on the cron schedule 0 0 */5 * * (every 5 days at midnight UTC).

[Belgian DCAT-AP RDF/XML dump: all/datagovbe_edp.xml.gz]
         ↓
  [GlobRead]
         ↓
  [GunzipFile]
     ↓
 [DumpsToFeed Processor] → [LevelDB State]
         ↓
     [Sdsify Processor]
         ↓
 [Bucketize Processor] → [Feed State JSON]
         ↓
  [LdesDiskWriter Processor]
         ↓
 [docs/ directory] → [GitHub Pages / BOSA deployment]

2.2. Setup

2.2.1. Prerequisites

2.2.2. Local development

cd pipeline
npm install
npx rdfc github-pipeline.ttl

2.3. Publisher implementations

2.4. Additional Resources

2.5. Support

For issues specific to:

3. Consuming DCAT-AP-Feeds: integration with Piveau at the EU Publications Office

This section reports on the consumption side of the DCAT-AP-Feed ecosystem. Specifically, it describes the technical work carried out to enable the EU Publications Office (OP) to automatically ingest DCAT-AP-feeds into their internal catalog management platform.

3.1. Context: Piveau at the EU Publications Office

The OP manages DCAT-AP catalogs submitted by EU member states through an open-source platform called Piveau. Piveau is an infrastructure for managing and harvesting open data portal catalogs, and it relies on Piveau Consus as its pipeline orchestration system. Piveau Consus is a segment-based pipeline engine that chains individual import and export processors to transform and route datasets between sources and sinks in an ETL (Extract, Transform, Load) process.

In this pilot, we extended Piveau to support DCAT-AP-feed consumption. The adoption of DCAT-AP-feeds offers a complementary pathway: instead of periodically re-fetching entire catalogs, a feed-based consumer can subscribe to an append-only event stream and incrementally synchronize its internal state, processing only the changes since the last run.

3.2. The piveau-consus-importing-ldes plugin

To enable DCAT-AP-feed consumption within Piveau, a new Piveau Consus segment was developed and made available as open-source at rdf-connect/piveau-consus-importing-ldes. This segment acts as a drop-in LDES importer for Piveau Consus pipelines, exposing an HTTP endpoint that Piveau Consus invokes to start feed traversal.

3.2.1. Architecture

The plugin is implemented as an RDF-Connect [RDF_CONNECT] pipeline (pipeline.ttl) composed of two processors:

  1. **rdfc:HTTPConsumer (<httpInput>)** exposes an HTTP server (default port 3000) that accepts PUT requests with application/json payloads following the Piveau Consus segment wire format. When a request is received, it writes the decoded descriptor to an output channel for the next processor. The host and port are configurable via the LDES_PORT environment variable.

  2. **rdfc:Piveau (<piveau>)** reads the descriptor from the channel, initializes an LDES client against the configured endpoint URL, streams each RDF entity from the source, converts the RDF quads to N-Quads serialization using N3.js, and forwards each entity to the next segment in the Piveau Consus pipeline chain via HTTP PUT. The processor implements retry logic and continues processing after individual entity errors.

Consus Control Plane
        ↓
    [Descriptor with LDES config]
        ↓
   HTTP PUT to /pipe
        ↓
   HTTPConsumer (listens)
        ↓
   Piveau Processor
        ├─ Connect to LDES endpoint
        ├─ Stream RDF entities
        └─ Convert to N-Quads
        ↓
   Send to next Consus segment
        ↓
   [Next segment processes metadata]

3.2.2. Configuration

Piveau Consus invokes the segment by sending a PUT request to http://<host>:<port>/pipe with a JSON body following the Piveau Consus pipeline message format. The config block of the segment body accepts the following LDES-specific options:

url
Required. The URL of the LDES feed entrypoint (e.g., the DCAT-AP-feed root document).
catalogue
Identifier of the target catalogue in Piveau into which the resources will be imported.
statePath
Path to a local JSON file used to persist the client’s replication state across runs. When provided, the client resumes from the last synchronized position, enabling incremental ingestion.
ordered
Traversal order of the LDES event stream. Accepted values: ascending, descending, or none (default). For DCAT-AP-feeds that use ldes:timestampPath, ascending is the recommended setting.
fetchConfig
Optional HTTP client configuration object. Allows tuning of retry behaviour, concurrency limits, and other transport-level settings used when fetching feed fragments.
Example Piveau Consus pipeline message sent to piveau-consus-importing-ldes 
{
  "header": {
    "id": "e106e79b-ad9a-4c9c-ade7-b49c58ff53fb",
    "runId": "08501a2a-4245-4804-8e1a-9303b49f6277",
    "startTime": "2024-03-21T12:00:00",
    "name": "ldes-source",
    "title": "Harvester - LDES source",
    "version": "2.0.0",
    "context": "BEL",
    "transport": "payload"
  },
  "body": {
    "segments": [
      {
        "header": {
          "name": "piveau-consus-importing-ldes",
          "segmentNumber": 1,
          "title": "Importing LDES",
          "processed": false
        },
        "body": {
          "endpoint": {
            "address": "http://localhost:3000/",
            "method": "PUT",
            "protocol": "http"
          },
          "config": {
            "url": "https://rdf-connect.github.io/belgium-dcat-ap-feed/index.trig",
            "catalogue": "bosa-dcat-ap",
            "statePath": "/tmp/bosa-feed-state",
            "ordered": "ascending"
          }
        }
      },
      {
        "header": {
          "name": "piveau-consus-exporting-hub",
          "segmentNumber": 2,
          "title": "Exporting hub",
          "processed": false
        },
        "body": {
          "endpoint": {
            "address": "http://piveau-consus-exporting-hub/pipe",
            "method": "PUT",
            "protocol": "http"
          }
        }
      }
    ]
  }
}

3.2.3. Deployment

Locally:

npm install
npm run build
npx rdfc pipeline.ttl

Docker:

docker build -t piveau-consus-importing-ldes .
docker run --publish 3000:3000 piveau-consus-importing-ldes
# To use a custom port:
docker run --publish 3001:3001 -e LDES_PORT=3001 piveau-consus-importing-ldes

The TypeScript sources are in src/ and compile to lib/. The plugin exports the Piveau and HTTPConsumer classes that implement the RDF-Connect Processor interface.

3.2.4. RDF processing, state management and error handling

RDF processing: The Piveau processor uses the ldes-client library to stream RDF events from LDES-compliant endpoints and N3.js to convert between RDF quad formats. All quads are serialized as N-Quads for transport through the Consus pipeline. Entities are processed individually via quad streaming, making the plugin suitable for large LDES sources.

State management: When statePath is configured, the plugin persists a LevelDB-based state at the given path. On subsequent runs the LDES client resumes from the last known position, preventing duplicate harvesting of unchanged records and enabling incremental ingestion.

Error handling: The plugin includes HTTP retry logic for transient failures and graceful per-entity error logging. Processing continues after individual entity errors, and connection pooling with configurable concurrency (via fetchConfig) prevents resource exhaustion on large feeds.

3.3. Additional Resources

Conformance

Conformance requirements are expressed with a combination of descriptive assertions and RFC 2119 terminology. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC 2119. However, for readability, these words do not appear in all uppercase letters in this specification.

All of the text of this specification is normative except sections explicitly marked as non-normative, examples, and notes. [RFC2119]

Examples in this specification are introduced with the words “for example” or are set apart from the normative text with class="example", like this:

This is an example of an informative example.

Informative notes begin with the word “Note” and are set apart from the normative text with class="note", like this:

Note, this is an informative note.

References

Normative References

[ACTIVITYSTREAMS-VOCABULARY]
James Snell; Evan Prodromou. Activity Vocabulary. URL: https://w3c.github.io/activitystreams/vocabulary/
[DCAT-AP-FEEDS]
SEMIC. DCAT-AP Feeds Specification. URL: https://semiceu.github.io/LDES-DCAT-AP-feeds/
[LDES]
Pieter Colpaert. Linked Data Event Streams. 2025-10-07. URL: https://semiceu.github.io/LinkedDataEventStreams/
[RDF_CONNECT]
Arthur Vercruysse; Ieben Smessaert; Julian Rojas Melendez. RDF-Connect. URL: https://rdf-connect.github.io/
[RFC2119]
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: https://datatracker.ietf.org/doc/html/rfc2119

Non-Normative References

[LDES-SERVER-PRIMER]
Pieter Colpaert. LDES Server Primer. 2025-10-07. URL: https://semiceu.github.io/LinkedDataEventStreams/server-primer