Using rigorous statistical tests to catch results drifting from theoretical expectation
How binomial hit-rate tests, RTP Z-tests and outlier filtering combine into real-time anomaly detection with tiered alerts when results drift from theory.
A gaming system faces risk along several dimensions: mathematical risk (extreme statistical deviation), system risk (misconfiguration, program defects), and behavioral risk (abnormal betting patterns). Statistical anomaly detection applies rigorous statistical testing to monitor in real time whether a game's actual performance deviates from theoretical expectation, so that anomalies can be detected and escalated at an early stage.
What makes this hard is that a gaming product is supposed to exhibit enormous variance. In a well-designed slot machine, several hundred consecutive losing spins in the short term, or a sudden payout of several thousand times the bet, fall entirely within the expectations of the mathematical model. This means "anomaly" cannot be judged by intuition or by a fixed threshold: firing an alert simply because RTP looks high on a given day yields nothing but a screen full of noise.
Put differently, the core problem risk control has to solve is: how to separate signal from data that is full of noise. That is exactly the question statistical hypothesis testing was designed to answer, and it is why we built the entire detection logic on a hypothesis-testing framework.
All detection logic revolves around a single statistical framework, and understanding that framework is a prerequisite for understanding every design trade-off that follows.
From this follow two classes of error that necessarily exist, and the trade-off between them runs through the whole risk control design:
The crux is this: reducing false positives necessarily raises false negatives, and vice versa. This is a hard statistical trade-off that cannot be optimized away. The only way to improve both at once is to increase the sample size, that is, to raise the power of the test, meaning "the probability of successfully detecting a problem when the problem genuinely exists." This also explains why, when designing a monitoring mechanism, planning the sample size is more fundamental than tuning the thresholds.
For each bet type and probability table label, we continuously track whether the actual win rate deviates from the theoretical value in a statistically significant way. Using the confidence interval of the binomial distribution, we set a significance level of α = 1/10000 (an extremely strict threshold), so that an alert fires only when something is genuinely anomalous:
H₀: actual win rate = theoretical win rate
H₁: actual win rate ≠ theoretical win rate
if p-value < α (0.0001), flag as anomalous
An extremely strict significance level keeps the false positive rate low: an alert fires only when the statistical evidence is overwhelming, avoiding interference from normal random fluctuation.
We chose the binomial test as the first line of defense because "won or not" naturally satisfies the three premises of the binomial distribution:
All three premises hold in a normally functioning game. Conversely, when the test keeps failing, what gets refuted may be not only the probability value but independence itself: for instance, some state that was not correctly reset, causing the previous round's outcome to influence the next. Problems of this kind are completely invisible at the level of a single round, yet leave a clear trace at the level of the distribution.
Sample size is what determines whether a binomial test can stand at all. The rule of thumb requires that both the expected number of successes and the expected number of failures reach a certain scale (a common threshold is no fewer than 10 each) for the normal approximation to be accurate enough. This has a direct corollary: low-probability events require an enormous sample size to test. An ordinary line with a 30% win rate may be assessable after a few thousand rounds, but a special feature with a trigger rate of one in ten thousand may need millions of rounds to accumulate a statistically meaningful sample. Our approach is therefore to set observation thresholds in tiers: items whose sample size has not met the requirement are not tested at all, rather than forcing a test on an insufficient sample and arriving at an unreliable conclusion. Silence is safer than a wrong conclusion.
We run a Z-test on cumulative RTP to verify whether the actual RTP deviates significantly from the locked theoretical RTP:
Z = (actual RTP - theoretical RTP) / SE(RTP)
if |Z| > Zα/2, RTP deviates significantly
When the tuning configuration provides enough information (an explicit locked theoretical RTP value exists), we use the exact theoretical value as the baseline; when the configuration is insufficient, we use the theoretical maximum RTP as a conservative baseline to avoid false negatives.
The division of labor between the Z-test and the binomial test is that the win rate test looks at frequency, while the RTP test looks at monetary scale. The two can disagree, and the disagreement itself is valuable diagnostic information:
The most common mistake when reading a Z value is overlooking that the standard error shrinks as the sample size grows. This means that with a very large sample, a practically negligible deviation (say, an RTP off by 0.05 percentage points) can still produce a large Z value and trigger an alert. Statistically significant does not mean materially important. The correct approach is therefore to watch two quantities at once: the Z value judges "whether this deviation is real," while the effect size (how many percentage points it actually deviates by) judges "whether this deviation is worth acting on." Only when both hold does it constitute a signal that warrants action.
Another premise to be careful about is that the standard error of RTP cannot simply borrow the standard error formula for win rate. The distribution of payout amounts is heavily right-skewed (a great many zeros and small payouts, plus a very small number of enormous ones), and its variance is contributed mainly by the large wins in the tail. The standard error of RTP must therefore be derived from the actual payout distribution rather than by assuming a normal distribution. Ignoring this systematically underestimates the standard error and, in turn, greatly overstates the severity of an alert.
If the initial win rate test finds an anomaly, the system performs a second round of analysis, recomputing RTP after filtering out extreme payouts above PR95:
This two-stage analysis effectively distinguishes "normal fluctuation with good luck" from "a genuine system problem," substantially reducing alert fatigue for the technical team.
The legitimate justification for filtering outliers is that a handful of extreme values can dominate the overall statistic. In a high-volatility mathematical model, a single payout of several thousand times the bet can account for a substantial share of the period's cumulative payout, causing the mean to swing violently without reflecting the overall health of the system. Recomputing after filtering amounts to asking a more precise question: "With those few extreme events excluded, does the overwhelming majority of remaining rounds still look normal?"
But the technique carries clear risks of its own, and they must be clearly understood:
We therefore position filtering as an auxiliary diagnostic tool, not a basis for judgment. Both sets of results, before and after filtering, are recorded in full so that later analysis can see "what the filtering changed," and that difference is often more diagnostically valuable than either set of numbers on its own.
A statistical test has to be defined over some range of data, and the choice of time window directly determines what kinds of problems can be detected. This is one of the most underestimated decisions in risk control design.
Since no single window can satisfy every need at once, the practical solution is to run multiple windows in parallel: the short window provides immediacy, the long window provides sensitivity, cumulative statistics provide long-run convergence verification, and the three conclusions are cross-checked. When the short window alerts and the long window is normal, this usually points to a change that just happened; when the long window alerts and the short window is normal, it is more likely an old problem that has already been corrected but has not yet slid out of the window.
There is one more implementation detail worth noting about sliding windows: window boundaries create alert flapping. When an extreme data point sits right at the edge of the window, its entry and exit make the test result oscillate back and forth between normal and anomalous. The solution is to set asymmetric thresholds for raising and clearing an alert: raising requires stronger evidence, while clearing requires several consecutive periods of normality, thereby avoiding repeated notifications for the same problem.
Detection results need to reach the technical team in real time, and the alerting channel design includes:
The real purpose of alert tiering is to avoid the "cry wolf" effect. A monitoring system that frequently produces false positives is more dangerous than no monitoring at all, because the team gradually learns to ignore it, and when a genuinely serious problem occurs, that alert drowns in identically formatted noise with nobody looking at it. Our tiering principle is therefore that each tier corresponds to a clearly defined expected action; if nobody knows what to do upon receiving an alert, that alert should not exist.
Several suppression mechanisms accompany the tiering: deduplication ensures the same root cause is reported only once before it clears; aggregation merges multiple related alerts within the same period into a single summary, so that one common cause does not spawn dozens of separate messages; and escalation automatically raises the tier of a low-tier alert that has gone unhandled for a long time, preventing a problem from being shelved indefinitely just because its tier was low.
Finally we return to that unavoidable fundamental trade-off. As noted earlier, an extremely strict significance level suppresses false positives, but the price of that choice has to be honestly acknowledged: it simultaneously raises the probability of false negatives, meaning that real problems of smaller magnitude need more accumulated samples before they are found.
We accept that price, and the reason lies in the asymmetric cost structure of the two. The cost of a false positive is immediate and cumulative: every false positive consumes the team's attention and slowly erodes the credibility of the whole monitoring system, and once that erosion sets in it is very hard to reverse. The cost of a false negative, by contrast, is partly absorbed by other mechanisms: the multi-window design means a small deviation will eventually accumulate in the long window until it is detectable; a complete audit trail keeps retrospective investigation always feasible; and pre-launch mathematical validation together with deterministic testing intercepts most misconfigurations at the source.
In other words, statistical detection is not the only line of defense, so it does not have to carry all of the false negative risk alone. Precisely because it sits within a multi-layer defensive system, we have the latitude to tune it toward the high-precision end, so that every alert it does emit deserves to be taken seriously. If it were the only line of defense, we would have to loosen the threshold and accept the noise that comes with it; a system like that is, in the end, usually one nobody looks at.