CharmingGroot

Security Operations Automation Agent Retro, Week 17 - Fixing the Triage Worker Bottleneck

architecture8 min한국어

Also available in Korean: 보안관제자동화 에이전트 회고 17주차 - triage 워커 병목 개선

Background

Triage started falling behind in the dev environment.
The system collects security alerts, groups them into incidents, and uses an LLM to decide whether each incident is a true positive (TP) or a false positive (FP).
That process is triage, and its result is the verdict.

An incident is a group of related alerts. Triage decides whether an incident is a real threat and sets the response order; the verdict is the outcome, TP or FP.

alert Alert grouping into incidents Triage queue lowest ID first Triage worker N per batch LLM 2 calls Verdict table Actions writes result
The triage worker takes incidents a few at a time (a batch) and sends them to the LLM. The highlighted parts are what this post covers.

When one new customer was connected, alerts that had been held until the connection were reprocessed, and thousands of incidents appeared at once.
This post calls those incidents the backlog.
New incidents from other customers then piled up without being triaged.

There turned out to be three bottlenecks.
The third only showed up after the queue and threads were fixed.

1 Triage queue · one queue … New incidents Backlog 2 One scheduler thread Triage worker (minutes) Alert grouping Collection … 14 more 3 Inside a triage batch 17 s 17 s 17 s one by one, waiting on LLM The backlog holds the lowest IDs, new incidents wait behind it Virtual-thread option: 17 jobs share one thread A bigger batch size still gives ~3 per minute
Three bottlenecks, covered in Changes 1 to 3.

For 1, the queue was split into a default queue and a backlog queue.
For 2, scheduling was split into 8 shared threads plus a dedicated thread per triage worker.
For 3, only minimal parallelism was applied for now: several items in a batch run at the same time.
The goal of this round was to fix the queue and thread structure, not to push throughput.
Throughput work comes in the next round.

Change 1

Splitting the queue

The triage worker takes incidents without a verdict, lowest ID first, and processes a few at a time as a batch.
Oldest first keeps old incidents from being pushed back forever, so it is the right rule on a normal day.
But when thousands of backlog incidents appeared at once they held the lowest IDs, and new incidents had to wait behind them.
Throughput was about 3 per minute, so clearing the backlog at that rate would have taken more than a day and a half.

AS-IS … Triage worker New incidents Thousands of backlog incidents hold the lowest IDs takes from the front TO-BE Default queue Valve limit · pause Existing worker Backlog queue Valve limit · pause New worker Verdict table shared
What changed: the backlog moved to its own queue.

A column records which queue each incident belongs to; the existing worker takes only the default queue and a new worker takes only the backlog queue.
Both workers write their results to the same verdict table.
Every downstream step reads the verdict table, so the only thing that differs is which worker picks the incident up.
Each queue got a valve, so concurrency and pause can be changed from the UI.
Pausing the default queue stops live triage, so it goes through a confirmation dialog.

Retries of failed triage were not split into their own queue.
A retry is not a separate worker; each worker picks up its own queue's failed items first in every batch.
Moving retries to a third queue would lose track of which queue they came from, and backlog failures would block live triage again.

Change 2

Splitting the scheduler threads

After deploying the new worker and counting triages per minute, the existing worker did zero while the new one was working, and the other way around.
Each queue had its own lock, so a thread problem seemed likely.
It turned out that 17 scheduled jobs were sharing one thread.
While a triage batch held that thread for about 5 minutes, the alert grouping job, due every 60 seconds, ran only every 150 to 172 seconds.
Each stretched interval ended within a second of a triage batch or a collection run finishing.

The first guess was that the thread count setting (spring.task.scheduling.pool.size) defaults to 1.
That was only half right.
With the virtual-thread option on (spring.threads.virtual.enabled=true), Spring Boot creates a SimpleAsyncTaskScheduler instead of a thread pool.
This scheduler has a single scheduling thread, and fixedDelay tasks run one after another on that thread.
The thread count setting does not apply to this scheduler, so raising it changed nothing.

AS-IS 1 thread shared by 17 scheduled jobs 0 min 1 min 2 min 3 min 4 min 5 min 6 min Triage batch holds the thread for ~5 min Alert grouping misses its slots, then runs in a burst after the batch TO-BE 8 shared threads + dedicated triage threads Triage batch runs on a dedicated thread Alert grouping runs on time every 60 s
What changed: triage batches run on dedicated threads, other scheduled jobs on 8 shared threads.

The scheduling thread pool is now defined in code, with 8 shared threads.
The number was chosen from the production server’s specs.
When a scheduler bean exists, Spring Boot does not create its default one, so every @Scheduled job lands in this pool.

@Bean
fun taskScheduler(): ThreadPoolTaskScheduler =
    ThreadPoolTaskScheduler().apply {
        poolSize = 8
        setThreadNamePrefix("scheduled-")
    }

The triage workers are not in this pool.
A triage batch takes minutes and would hold a shared thread for a long time, so each worker creates its own single-thread scheduler and schedules itself.

Registering that dedicated scheduler as a bean would make Spring treat it as the shared scheduler, so it is not registered as a bean.

Change 3

In-batch parallelism

Real throughput work comes in the next round; this time only enough was applied to clear the bottleneck left after the queue and threads were split.

Even after the threads were split, the backlog queue ran at about 2.6 per minute.
Each queue runs one batch at a time under a single lock, and inside a batch items were still triaged one at a time.
One triage calls the LLM twice (verdict, rationale) and takes about 17 seconds.
Actual computation is a little over a second; the rest is waiting for the LLM to respond.
One at a time, the number finished per minute is set by how long one item takes.
At around 17 seconds an item, the ceiling is a little over 3 per minute, so raising the batch size from 5 to 10 did not change throughput.

The easiest fix is to add threads and run triage in parallel without limits.
But the LLM server has limits on concurrent calls and calls per minute, and rejects requests over the limit with 429 (Too Many Requests).
If concurrent calls grow with the thread count, the limit can make triage fail all at once.
So the batch stays, and inside it each queue runs only up to a fixed number at a time.

AS-IS 0 s 30 s 60 s 90 s 120 s 150 s One at a time 17 s Slow, 40 s 17 s 17 s Polling interval 60 s TO-BE 2 at a time 17 s 17 s 17 s 17 s 17 s 17 s Slow, 40 s 17 s 17 s 17 s Polling 5 s Batch deadline 90 s
What changed: two items run at once inside a batch, and the polling interval dropped from 60 to 5 seconds.
  • The batch size stays the same, and up to the concurrency limit items run together. A new item starts as soon as one finishes, so one slow item does not hold up the rest.
  • An item that runs past 60 seconds is cut off and sent to the retry list.
  • After 90 seconds a batch starts no new items and only finishes the ones in progress. Items that did not start are not failures; they move to the next batch.
  • Starting values are 3 for the default queue and 2 for the backlog queue. The valve now controls concurrency instead of batch size.

The polling interval had to change too.
The next batch starts 60 seconds after the previous one ends (fixedDelay).
Keeping that interval and running 3 at a time gives a 20-second batch plus a 60-second wait, about 13 items in 6 minutes, barely better than one at a time.
So when items remain in the queue the polling interval drops to 5 seconds, and it stays at 60 seconds only when the queue is empty.
That gives about 43 items in the same 6 minutes, about 7 per minute.

Change 4

Why not slots

Another option was to drop batches entirely and always keep N triages in flight.
The worker holds N slots, and as soon as a slot frees up it takes the next item from the queue.
With no batch boundaries, a slow item ties up only its own slot while the others keep going.

Adopted In-batch parallelism 2 at a time 17 s 17 s 17 s Slow, 60 s 17 s 17 s 17 s 17 s 17 s 17 s idle at batch end next batch Considered Slot approach 2 slots 17 s 17 s 17 s 17 s 17 s 17 s Slow, 60 s 17 s 17 s no idle time needs in-flight tracking
In-batch parallelism leaves idle time at the end of a batch; slots avoid it but need more machinery.

On throughput alone, slots are a little better.
With in-batch parallelism the other slots sit idle at the end of a batch while one slow item finishes, and that wait can stretch to the per-item limit of 60 seconds.

The slot approach, however, needs far more changes.

In-batch parallelism (adopted)Slots
ThroughputScales with concurrency, small loss at batch endScales with concurrency, almost no loss
Slow itemsThe rest wait at batch endOccupy only their own slot
Picking from the queueUnchanged (one lock per queue)In-progress marker per incident + FOR UPDATE SKIP LOCKED
Multiple podsOnly the pod holding the lock runsSlots per pod, in-progress markers prevent duplicates
When a pod diesThe next batch picks up items without a verdictAnother pod takes over after the expiry time
Emergency stop, batch recordsUnchangedBatch-level decisions redesigned around time windows

Without batches, decisions such as counting consecutive failures for an emergency stop, or “every item in this batch failed”, would all have to be rebuilt around time windows.
The only gain is the idle time at batch end, and the cost is touching everything from how items are picked to failure handling.

The biggest reason was the schedule. (could have used another pair of hands..)
The deadline was fixed and there was too much to change, so making that change before the deadline was too risky.
So the current one-lock-per-queue structure stays, and parallelism happens only inside a batch.
If measurements show batches often stretching to the time limit because of slow items, slots will be revisited.

Overlapping batches were ruled out too.
Incidents still being processed by one batch have no verdict yet, so the next batch picks them up again, and concurrent calls multiply with each overlapping batch and can exceed the limit.

Results

Triages per minutehigher is better Before · one at a time 2.6 After · two at a time 5.5 ~2.1× Time to clear 5,956 backlog incidentslower is better Before · one at a time ~38 h After · two at a time ~18 h under half
Backlog queue. One batch measured at 4 at a time ran at 10.5 per minute.

The comparison used the backlog queue with more than 5,000 items waiting, with the batch size fixed at 10, over windows before and after the deploy.

MetricBeforeAfter
Backlog triages per 10 minutes21–3049–63 (avg. ~55)
Time to clear 5,956 backlog incidents (from throughput)~38 h~18 h
One LLM call, p50 / p9015.8 s / 19.4 s16.6 s / 21.1 s
LLM calls per minute6.012.2
Rate-limit errors (429)00
Collection interval, top 5%100–102 s69 s
Collection intervals over 90 s47–5024–27
Collection intervals over 120 s28–2922–26
Scheduling threads1, shared by 17 jobs8 shared + 2 for triage

Throughput work is planned as the next task in the remaining development period.

References