Security Operations Automation Agent Retro, Week 14 - Bounded RSI

Also available in Korean: 보안관제자동화 에이전트 회고 14주차 - Bounded RSI

The threat indicators a detection system references come from two sources. One is externally verified: indicators an outside organization or community has already confirmed as malicious and registered. The other is self-generated: indicators the system produces on its own, reasoning from its own detection and verdict history that “this has been observed repeatedly and confirmed as an attack, so treat it as malicious going forward.”

This distinction matters because a self-generated indicator is built on the system’s own past judgment. An externally verified indicator was independently confirmed by a third party, but a self-generated one amounts to the system telling itself, “I judged it this way before, so I’ll judge it this way again.” Left uncontrolled, this self-referential structure lets one wrong call become the basis for the next, which becomes the basis for the one after that, accumulating unverified confidence with no external check.

The principle came first, set at the governance level

When we designed the self-improvement loop, the risk was obvious even without numbers. So we set the principle up front: separate candidate generation from actual deployment, require human approval before anything takes effect, and keep versioned history so changes can be rolled back. At that point there was no measurement showing “this is exactly how dangerous it is.” It was a principle established on governance grounds alone.

Whether the principle was right, we only confirmed about a month later, by accident, while auditing a path where it hadn’t yet been fully enforced.

What we measured

We ran two queries to check how much self-generated indicators were actually influencing decisions.

Q1: number of active self-generated indicators
Q2: number of decisions influenced by a self-generated indicator alone (no corroborating external indicator)

Placed next to the same queries from three weeks earlier:

Metric	3 weeks ago	Now	Multiplier
Q1: active self-generated indicators	32	86	2.7x
Q2: decisions influenced by a self-only match	1	135	135x

Q1 and Q2 don’t scale together. The indicator count grew less than 3x, but the number of times those indicators actually factored into a decision grew by two orders of magnitude.

The cause: provenance information disappears right before scoring

An indicator carries provenance (externally verified / self-generated) at the point it’s stored. It passes through three stages between lookup and scoring, and this information drops out at the second stage.

storage lookup (provenance present)
  -> lookup layer: reads only active status, not provenance
    -> object mapped for scoring: no field exists to carry provenance
      -> strength rating: computed from the remaining conditions, provenance unknown

The strength-rating rule is roughly: if an indicator is recent, high-confidence, and has a confirmed attack type, it gets the top tier. If it’s recent, or corroborated by multiple independent sources, it gets the next tier down. A self-generated indicator comes from a single source, so it never satisfies “corroborated by multiple sources” - but “recent” alone is enough to earn that same next-tier rating. Since no condition anywhere distinguishes “externally verified” from “self-generated,” you can’t tell the two apart just by looking at the resulting rating. As far as the scoring logic is concerned, they’re fully equivalent.

Why the multipliers diverged

The logic that generates self-generated indicators already excludes anything already registered externally when picking candidates. So the raw indicator count grows roughly linearly over time. We didn’t directly measure why decision-influence grew faster than that, so here’s an estimate: total influence is approximately (indicator count) x (per-indicator probability of factoring into a scoring decision), and since the strength rating carries no provenance distinction, that per-indicator probability should stay at the same level as an externally verified indicator’s. If so, total influence grows faster than the indicator count itself. Disproving this estimate would require measuring the per-indicator probability separately at each point in time, which we didn’t do this round.

All 135 cases were self-only matches with zero corroborating external indicator. We simulated two ways of capping this.

Option	Rule	Decisions affected	Dropped below promote	Dropped below critical
Provenance-based cap	Any self-only match with no external verification gets its strength rating fixed at the floor, regardless of grade	135	26	8
Grade-based cap	Only confirmed-grade indicators keep the higher rating; pending-verification grade gets fixed at the floor	6	6	0

The grade-based cap was nearly ineffective. Only 6 indicators were even at the pending-verification grade, and every one of those had already been human-approved, so they already carried real confidence. The provenance-based cap is the one that actually does something - but it also downgrades confirmed-grade indicators that were already judged true positives or had already triggered a real response. What’s left is a tradeoff between the accuracy of already-verified judgments and blocking self-reference.

The principle and the evidence arrived at different times

This case is not the trigger that led us to put boundaries on the self-improvement loop. The principle predates this measurement. What this measurement showed is two things. First, the risk the principle anticipated is real, at a scale you can actually measure (135x). Second, the principle hasn’t been enforced uniformly across every implementation path (the missing provenance field in the scoring path). Establishing a design principle and actually carrying it through every implementation path are two different jobs.

Judgment criteria

If you’re designing a self-improvement loop, you don’t need to wait for numbers first. A structure where the system’s own output becomes the input to its own future judgment, without external verification, deserves a governance-level boundary before it becomes a problem in practice, not after. What this case shows is only after-the-fact evidence that the boundary wasn’t overcaution.

If a loop like this is already running, don’t treat “we established a principle” as grounds for reassurance on its own. Whether the principle is actually reflected in every implementation path has to be checked separately. In particular, measure the scale metric (indicator count) and the decision-influence frequency separately. If their growth rates diverge, that gap is exactly where the boundary is missing.