CharmingGroot

Building an RML Engine in Rust, Ep. 1

architecture11 min한국어

Also available in Korean: Rust로 RML 엔진 만들기 ep1

This is a design decision record. The PoC has not been run yet; this post is the plan and its rationale. Numbers I measured myself come from a local MacBook (macOS, Apple Silicon). Performance numbers for other engines are the ones their sources published.

Background - one adapter per SIEM

I am building a security operations automation product. It receives detection events from several SIEMs, judges whether they are threats, and blocks confirmed threats on the device before a human looks at them (pre-blocking).

In the current structure, every SIEM we add comes with its own adapter: collection code, field conversion rules, and join code that ties the data to everything else, all written per SIEM.

One SIEM calls an IP src_ip, another calls it sourceAddress, a third calls it attacker_ip, so the code that mixes them grows with every combination of sources.

The more judgment we hand to the machine, as with pre-blocking, the more it becomes a risk in itself that you cannot see at a glance which data was linked to which, and how.

So we decided to bring in an ontology instead of adapters.
The goal is that adding a new SIEM means adding normalization rules and a mapping document, not adapter code, and that which field fed which decision stays written down.

Adopting an ontology involves concept design, data normalization and management, versioning, governance, and response automation. This devlog covers only one piece: the engine that brings data in according to the concepts.

Goal

Build an engine in Rust (RMLrs) that executes RML mappings record by record, lightly, and exactly as the specification says. The aim is not to add features existing engines lack, but to run the same mapping to the same result, faster and lighter.

AxisTargetCriterion
Execution modelRecord-level streaming, light enough to stay residentSee “Benchmark targets” below
Spec conformanceAt least as good as the best current engine (CARML)Official RML tests: Core 76/76, IO ≥ 61/73, FNML ≥ 18/20, CC 35/35, LV 41/41, plus SQL 77/77, which CARML does not cover
ReachSame result from one binary, from Python, and in the browserPython wheel and WASM right after Core
LicenseUsable inside a commercial productApache-2.0
EcosystemListed in the standard ecosystemOfficial RML implementation report, awesome-kgc-tools

Benchmark targets - at least the mainstream engines

Performance is measured the way the RML community measures it in its official challenge. The 2026 KGCW challenge performance track gave every participating engine a VM with the same spec (Intel Xeon Gold 5220 2.2GHz, 16GiB RAM, 120GiB disk), ran each engine inside Docker with a harness adapted from the official KROWN benchmark, and sampled execution time, RAM, and CPU at fixed intervals. There are six scenarios.

KROWN scenarioWhat varies
Data-sizeNumber of records, properties per record
JoinsJoin cardinality, share of data that joins
MappingsNumber of TriplesMaps, predicate-object maps per TriplesMap
Named GraphsNumber and position of graph maps (subject map, predicate-object map)
Duplicates-emptyShare of duplicates, share of empty values
Collections and ContainersRML-CC aggregation

Only CARML (with two backends) and SDM-RDFizer entered the 2026 performance track. From the report, these are the facts worth using as a baseline.

  • CARML beat SDM-RDFizer on time and CPU in three of four scenarios; SDM-RDFizer was fastest on Named Graphs.
  • CARML’s peak resident memory was about 3.0 to 3.9GB, nearly independent of input size (including the 10-million-row Data-size case, with a 1GB JVM heap setting).
  • SDM-RDFizer remembers every triple it has produced in order to remove duplicates, so its memory grows with the input.
  • The report published results only as relative rankings between engines. Absolute times were not published, so a comparison means running the same harness yourself.

The C++ engine FlexRML published absolute numbers in its README, measured on the same KROWN scenarios and on GTFS-Madrid-Bench (Ryzen 5 7500F, 6 cores, average of 3 runs). For example, it produced 39.6 million triples for GTFS 100_csv in 5.7 seconds with 875MiB peak memory. It has no official conformance results, though.

From this, the performance targets for RMLrs are:

TargetReference engineHow it is measured
Better time, CPU, and RAM than CARML and SDM-RDFizer in all six KROWN scenariosEngines in the official performance trackSame KROWN harness, same machine spec, same number of runs as the challenge
Peak memory does not grow with input size (flat even at 10 million rows)SDM-RDFizer (grows with input), CARML (fixed at about 3 to 4GB)KROWN Data-size, Duplicates-empty
Faster and lighter than Morph-KGCThe most used Python engineKROWN ships a Morph-KGC run configuration, so it can be compared under the same conditions
At or below FlexRML’s numbers on GTFS-Madrid-BenchThe fastest native engineRerun side by side with FlexRML on the same machine
Resident cost: instant start, idle memory in the tens of MB or lessMorph-KGC import at about 130MB, JVM enginesStart time and idle RSS on the same machine

These numbers are targets. Whether they are met will be checked with published measurements during the performance work.

Conformance rules - what counts as “conformant”

  • The reference is the official test cases of the KG-Construct Community Group. Tests for each module (RML-Core, IO, FNML, CC, LV, star) run as published. No copies with mappings or expected outputs edited to suit the engine.
  • Pass or fail is decided by graph isomorphism. The set of generated triples must equal the expected output (N-Quads). A test that expects an error passes only if that error occurs. An error caused by “not implemented” does not count as a pass.
  • Results are reported in the official implementation report format (testcase_id,status,notes, with status passed, failed, or inapplicable). Cases whose expected output is officially under review as wrong (for example, some RML-IO cases) are marked separately, as the challenge does.
  • The spec’s notation is accepted as written. That includes the changes the official tests made in 2024-2025: source paths (RelativePathSource), JSONPath (RFC 9535 notation $.), and natural datatypes.

Before building the engine, this episode plans how to check, with tools that exist today, whether the structure the engine is meant for actually works.

Question 1

Is RML alone enough?

At first I wanted RML to do everything. RML is a standard for declaring “this field in the source is that concept” in a mapping document, and identifiers (IRIs) built from the same template become the same entity, so sources mix without join code (조인 코드 없이 데이터를 섞는 법, RML, in Korean).

But with RML alone, every SIEM adds one more mapping. The adapter code has just turned into a mapping; the “one per source” structure is unchanged.

So I read the code of the log pipeline Vector. Vector takes events one at a time, renames fields and converts values with VRL, and hands them to a database or Kafka. It aligns each source’s fields to common fields; it does not link records from different sources as the same entity.

The two have different purposes. Vector aligns each source’s fields to common fields; RML connects records to concepts. Give each its own job.

SIEM A ─┐
SIEM B ─┼─▶ [Vector] per-source fields → common fields   ← all source differences absorbed here
SIEM C ─┘     (VRL, one event at a time, stateless)
              │ common-field records
              ▼
        [RML engine] common fields → entity/relation graph   ← one mapping, for the common schema
              │ (link the same IP/host by identifier; identity rules such as tenant and time live here)
              ▼
        graph store → verdict · pre-blocking

Decision: normalization in Vector, entity linking in RML. Split this way, the RML mapping sees only the common schema, regardless of how many SIEMs there are. The mapping stays small and stable, and the identity rules sit in one place. When sources grow, only Vector’s VRL rules grow.

Question 2

Does a DataFrame have to stay resident?

Morph-KGC, the most used Python RML engine, runs on pandas DataFrames. It is designed for batch work: collect the data, then process it all at once. I measured how much that costs inside a service that runs continuously in a container.

Measurement (local MacBook)MemoryTime
Empty Python16MB0.01s
import morph_kgcabout 130MB1.4s
Column operation on 1M rows, pandas301MB0.8s
Same task, Polars184MB0.2s

Polars cuts processing memory and time a lot, but as long as it runs from Python, the resident weight barely drops. Split the mapping apart, though, and field matching and identifier generation are stateless work that finishes inside a single record. Only joins and deduplication need state, and a lookup table or a database can handle those. I judged the DataFrame to be an implementation choice of the engine rather than a requirement, and left it out of the always-on path.

PoC plan - check with existing tools first

The PoC has one goal: check whether data from two SIEMs lands on the same IP node through a single mapping for the common schema. Nothing else is in scope.

Setup

StepToolRole
1Synthetic dataThree SIEMs with different field names. Start with two; the third checks that “adding one leaves the mapping unchanged” (no internal data)
2VectorConvert each SIEM to common fields and load them into a Postgres events table
3One mappingCommon schema (events) → detection, IP, host concepts. The identity rule (tenant) is part of the template
4-AOntop (virtual graph)Exposes the graph over Postgres at query time via R2RML. No copy
4-BMorph-KGC (batch materialization)Builds the graph from the same mapping and loads it into a store. Kept as a separate batch job to avoid resident cost
5SPARQL query“Group detections that different SIEMs reported with the same source IP”

Success criteria

  1. Detections from two SIEMs land on the same IP node.
  2. When one more SIEM is added, only the Vector rules change; the mapping stays the same.
  3. The same internal IP in different tenants stays as different nodes (identity rule check).
  4. 4-A and 4-B return the same answer to the same query (both engines read the same mapping the same way).

Also measured: resident memory, memory while processing, start time, and the time until one detection shows up in the graph (query time for 4-A, batch interval for 4-B).

Why build RMLrs

To be honest, this structure works with existing RML engines. If the PoC succeeds, an engine that passes all the official tests, such as CARML or SDM-RDFizer, is enough to run the product.

Building RMLrs, an RML implementation in Rust, is a personal ambition. There are two things I want to learn properly.

  • I want to internalize RML. Writing mappings is different from implementing, end to end, how a mapping becomes triples. By turning the spec into code line by line, covering identifier generation, missing values, natural datatypes, and joins, I want to understand ontology data normalization from the ground up.
  • I want to learn Rust. I want to work on the problems Rust is good at, such as parsers, streaming, and memory-frugal joins, in a project of real size.

It starts as ambition, but the result should be useful. Measuring the engines also showed a gap. Real-time pre-blocking needs detections to reach the graph as they arrive, and each existing engine has something in the way.

Vector ─(Kafka · HTTP, common-field JSON)─▶ [record-level RML engine] ─▶ incremental graph update ─▶ verdict · pre-blocking
                                              light and resident, same single mapping
What I wantWhyExisting engines
Record-level streamingPre-blocking is faster when detections enter the graph immediatelyMorph-KGC and SDM-RDFizer are batch; RMLStreamer (Flink) has stalled since 2024
Light resident footprintAlways-on container servicePython engines take about 130MB just to import; JVM engines (CARML, RMLMapper, BURP) are heavy too
Spec conformanceSo the mapping document works regardless of engineMorph-KGC 2.10.0 passes 1 of 61 official RML-Core output tests with the spec’s notation (my measurement). Fully conformant: SDM-RDFizer, CARML (unreleased branch), BURP-Error
Commercial-friendly licenseTo ship it in a productFlexRML (C++) is light but AGPL-3.0; rml-mapper-lib (Rust) states no license and has no conformance results

No engine has all four yet. So RMLrs targets these four rather than new features. The principles:

  • Pure: the engine only does mapping + records → triples. Collection, storage, and verdicts belong to the stages before and after it.
  • Spec first: pass the official tests as published.
  • Apache-2.0: it must be usable inside a product.
  • Prove it by measurement: speak with official conformance tests and benchmark numbers.

Because the mapping document is a standard, the single mapping from the PoC keeps working when only the engine is swapped for RMLrs.
Prove the structure with existing engines first, then verify my engine with the same mapping.

Summary

  1. Split normalization (Vector) from entity linking (RML). One mapping for the common schema.
  2. The PoC is Vector → Postgres → one mapping → Ontop (virtual) / Morph-KGC (batch).
  3. Success criteria: same IP node, mapping unchanged when a SIEM is added, identity rule works, same answer from both engines.
  4. The structure works with existing engines, but I am building RMLrs to internalize RML and learn Rust. The target is record-level, light, spec-conformant, and commercially licensed, all at once.
  5. The PoC mapping stays the same when the engine changes. That is why I chose RML.

References