CharmingGroot

Building an RML Engine in Rust, Ep. 2

architecture4 min한국어

Also available in Korean: Rust로 RML 엔진 만들기 ep2

Summary

  • All 76/76 official RML-Core tests pass (JSON source, mapping parser, engine, N-Quads output)
  • Streaming execution and 128-bit hash-based duplicate removal: memory for 1M rows went from 3,747 MB to 275 MB
  • Python wheel and WebAssembly package
  • SQL sources: PostgreSQL, MySQL, SQL Server
  • RML-IO: logical targets, compression (gzip, zip, tar.gz, tar.xz), XML (XPath)
  • Benchmarked against Morph-KGC with the KROWN execution framework
  • Official tests passed: 89 → 214 out of 365

What was built

Mapping mapping.ttl Data JSON·CSV·SQL·XML parser Turtle → RML model source one record at a time engine triples · joins · dedup writer N-Quads · logical targets file · stdout Python · browser 4 compressions, streamed
The parser reads the mapping, the source reads the data, and the engine builds triples per record and hands them straight to the writer.

The official tests are the test cases published by KG-Construct, run as-is without changes.
Results are written in the same format as the official implementation report, and if any test that used to pass starts failing, CI blocks the commit.

Notable issue 1

The first run used over three times the memory of Morph-KGC

Running 1M CSV rows (6M outputs) side by side with Morph-KGC, the time was similar but memory use was more than three times higher.
The engine loaded the whole source into memory and held all 6M outputs to remove duplicate statements.

AS-IS Whole CSV in memory Collect all 6M outputs sorted set for dedup Write at once 3,747MBpeak memory TO-BE CSV line by line Write as soon as built Keep only hashes skip if hash already seen 275MBpeak memory
What changed: read the CSV line by line, write each statement as soon as it is built, keep only hash values for duplicate checks.

A hash value reduces a statement to a fixed-length number (here 128 bits, 16 bytes). Only this value is stored instead of the statement, and when the same value shows up again, the statement is treated as already written and skipped.

1M rowsTimePeak memory
RMLrs AS-IS10.98 s3,747 MB
Morph-KGC 2.10.09.53 s1,165 MB
RMLrs TO-BE (streaming + faster output formatting)5.70 s275 MB

Median of three runs, measured directly on a MacBook (Apple M4, 16 GB).

Notable issue 2

A faster hash nearly dropped 100,000 outputs

The profiler showed the hash computation (SipHash) standing out, so I swapped in a faster hash I wrote myself.
It got down to 5.0 seconds, but the hash of the output file no longer matched the previous one.
Comparing against Morph-KGC’s output showed that IRIs differing in a single digit got the same hash value, and about 99,000 statements had been wrongly dropped as duplicates.

AS-IS · home-made fast hash …/person/100008 …/person/100009 hash a1f3… the second is "already seen" → dropped (about 99,000) TO-BE · back to SipHash …/person/100008 …/person/100009 hash 7c02… hash e95b… both kept
What changed: back to the proven SipHash so different statements get different hash values. The hash values shown are examples.

I went back to the proven SipHash and added a test that feeds 300,000 IRIs differing in one digit.
I also checked that this test actually fails with the bad hash (only 280,000 of 300,000 remain).
Looking only at time and memory, I would have committed it as it was.

Measuring with the KROWN execution framework

I plugged RMLrs into KROWN’s execution framework (exectool) and measured it against Morph-KGC under the same conditions.
The framework runs each engine in Docker and records time, CPU, and RAM.

Because the test machine has only 8 GB of memory, Morph-KGC was run with its process count reduced to 1 and 4.

Time (s) RMLrs 19.9 Morph-KGC ×1 59.7 Morph-KGC ×4 40.3 CPU time (s) RMLrs 17.4 Morph-KGC ×1 55.6 Morph-KGC ×4 109.3 RAM increase (MB) RMLrs 358 Morph-KGC ×1 1,412 Morph-KGC ×4 2,956
KROWN raw scenario, 1M rows (20 columns), median of three runs. ×1 and ×4 are Morph-KGC's process count.
RowsEngine and settingTimeCPU timeRAM increase
100KRMLrs2.44 s2.3 s3-20 MB
100KMorph-KGC, 1 process7.23 s7.3 s184-328 MB
100KMorph-KGC, 4 processes5.41 s9.8 s474-586 MB
1MRMLrs19.88 s17.4 s358 MB
1MMorph-KGC, 1 process59.74 s55.6 s1,412 MB
1MMorph-KGC, 4 processes40.29 s109.3 s2,956 MB

At 1M rows, RMLrs was 2-3 times faster than Morph-KGC, with a RAM increase of one quarter to one eighth.
In every run, the two engines’ outputs were identical after sorting.

Measurements are medians of three runs in the Mac’s Docker VM (10 CPUs, 8 GB).
Time includes container startup, and RAM is the increase in the whole VM’s usage.
I plan to measure again on a machine with higher specs.

Summary of this round

214 of the 365 official tests pass.
Streaming and hash-based duplicate removal brought memory for 1M rows from 3.7 GB to 275 MB, and after changing a hash or a key, I check again that the output is unchanged and rerun the benchmark.

Next

  • Remaining KROWN scenarios (joins, databases) and comparisons with other engines (CARML, SDM-RDFizer)
  • Extension modules: FNML, CC, LV, star
  • Memory: the hash set when outputs are very large, and XML memory that grows with document size

References