TL;DR

• Symptom: After a RASP/WAF agent was rolled out, customers lost buttons, dropdowns and styles in a legacy web app.
• Cause: A default Content-Security-Policy blocked third-party assets. Our fix then kept disappearing because a hidden cron script restored an old Apache config.
• Lesson: crontab -l shows one user's crontab, not the system. Audit /etc/cron.*, /var/spool/cron/ and systemd timers, and inventory what you bake into base images.

Incident Overview & Impact

• Scale: 250+ dedicated EC2 instances running an un-migrated legacy Java monolith, one instance per customer environment.
• Impact: Intermittent UI failures, dead dropdowns and missing third-party assets for customers using the legacy portal during a migration period.
• My role: On-call engineer. I led triage, coordinated the fix across environments, verified browser-level behavior, and coordinated the fleet-wide remediation.

The Setup: The Monolith and the Security Injection

The application was a legacy Java monolith made up of four core services bundled together. Each service powered a distinct part of the same product, with its own login flow and routing.

We were in the middle of a multi-year modernization, so deep refactoring was paused. The goal was to keep this environment stable and treat it as-is.

To improve our security posture, we rolled out a Web Application Firewall (WAF) and Runtime Application Self-Protection (RASP) solution, Rapid7 tCell. The agent was injected directly into the Java application's runtime.

Shortly after deployment, tickets started arriving: customers reported that core parts of the web application were behaving abnormally. Buttons were missing, dropdowns were dead, and UI assets were vanishing.

Phase 1: The RASP & CSP Culprit

We opened an incident and went through the logs and browser dev tools. The root cause of the UI breakage was a classic modern web conflict:

1. The architecture: A browser requested the web app. The request passed through Apache, hit the Java application (where the tCell agent ran inside the JVM), and generated the response.
2. The strict security header: Out of the box, the agent attached a restrictive Content-Security-Policy (CSP) header, along the lines of default-src 'self'. This tells the browser to load scripts, styles and assets only from the exact same origin.
3. The third-party CDNs: The legacy app relied on JavaScript frameworks, font icons and styling components hosted on external CDNs and separate subdomains.
4. The browser interdiction: Because those hosts were not allowed in the tCell CSP configuration, the browser blocked them. Visual components disappeared silently.

The Fix

To solve this without turning security off, we did the following:

• Added the trusted external asset sources to the tCell CSP configuration in the tCell dashboard (for example *.example.com and fonts.googleapis.com, shown here as illustrative values).
• Updated Apache's proxy configuration (httpd.conf) so the security headers generated by the Java agent passed through to the browser via ProxyPassReverse and were not overwritten or stripped by hardcoded Apache rules.
• Temporarily switched tCell to Report-Only mode to catch any remaining missing-domain violations in the logs without breaking the production UI. After validating the policy in our staging environment, we moved back to enforcement mode.

We applied the fixes to the primary environments, restarted Apache, verified that everything looked healthy, and closed the incident.

Or so we thought.

Phase 2: The Mystery of the Reverting Configuration

About 20 minutes later, the alarms started again. The same UI failures were back.

We returned to the servers and found that the Apache configuration changes we had just made had been reverted to their original state.

We did not have continuous configuration-drift correction (such as an always-on Ansible or Chef run) on these legacy nodes. Changes were pushed through Ansible Tower during deployments, and nothing was supposed to overwrite configuration files on a timer.

I went through the standard checklist:

• Did a deployment pipeline or automation job run? No.
• Was anyone else logged in or running scripts? No.
• Do the user crontabs show anything (crontab -l)? Completely empty.
• Is there unusual sudo activity in /var/log/auth.log? Nothing out of the ordinary.

The configuration was reverting like clockwork, yet no standard log or user crontab showed who or what was doing it. We were chasing a ghost.

Phase 3: Unmasking the Ghost in the System Cron Directories

Since the user-level checks were empty, we widened the search to system-level scheduling and used our runtime audit tool, Twistlock, which records commands executed with elevated privileges. These were the discovery commands:

# Check all cron directories and system cron tables
ls -la /etc/cron.*

# Search recursively across cron configuration for Apache references
grep -r "httpd" /etc/cron*

# Inspect systemd timers that might trigger background jobs
systemctl list-timers --all

The directory listing exposed the culprit: a script in /etc/cron.hourly/, completely invisible to crontab -l. The audit trail of root-executed commands lined up with it, showing Apache being restarted at the same moments the configuration reverted.

A redacted version of the script's logic:

#!/bin/bash
# Legacy Apache "safety net": restore the baseline config if it drifted
if ! cmp -s /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf.bak; then
  cp -f /etc/httpd/conf/httpd.conf.bak /etc/httpd/conf/httpd.conf
  systemctl restart httpd
fi

Why the timing looked odd

If the script runs hourly, why did the config revert about 20 minutes after our fix? Scripts in /etc/cron.hourly/ are not triggered on the hour. A system cron entry (commonly /etc/cron.d/0hourly or /etc/crontab) runs run-parts over the directory at a fixed minute offset. On many RHEL-style systems that is a minute or so past the hour. Our fix landed roughly 40 minutes into the hourly cycle, so the next run came about 20 minutes later.

Why it stayed invisible

The script only acted when the live config differed from its stored baseline. For months the two files matched, so the script quietly did nothing. It stayed unnoticed until the first time anyone legitimately changed Apache, which was our CSP header work.

Why Was It There? (A Blameless Perspective)

Long before our team arrived, engineers had built a Golden AMI used to spin up new customer environments. It is easy to see the intent. Someone wanted a guarantee that if a developer or sysadmin accidentally corrupted an Apache configuration file, the system would self-heal within the hour and prevent extended downtime.

Because configuration changes were rare on this legacy product, the script sat unnoticed. It was a landmine waiting for the day we needed to change the web server rules.

Phase 4: Remediation at Scale

Finding the script on one server was a victory. The real challenge was scale: the same Golden AMI artifact was baked into every one of our 250+ legacy instances, each running a customer's dedicated environment.

I coordinated the fleet-wide remediation:

1. Stop the revert: We removed the script from /etc/cron.hourly/ across the fleet using automated SSH and Ansible orchestration.
2. Re-apply the fixes: We re-applied the tCell CSP configuration and the Apache header-passing rules.
3. Verify: We confirmed that configuration drift had stopped, the audit logs were clean, and the tCell dashboard showed no unexpected CSP violations.

Beyond crontab -l

Relying only on crontab -l gives a false sense of security, because it shows only the current user's crontab. When you chase recurring, unexplained behavior on a Linux host, widen the search:

• /var/spool/cron/: per-user cron files on disk
• /etc/crontab and /etc/cron.d/: system-wide cron tables
• /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/, /etc/cron.monthly/: scripts run on a schedule regardless of user crontabs
• systemctl list-timers --all: systemd timer units, which many modern distributions use instead of cron

A quick one-shot check:

ls -la /etc/cron.* /var/spool/cron/ 2>/dev/null
grep -r "httpd" /etc/cron* /var/spool/cron/ 2>/dev/null
systemctl list-timers --all

Preventing the Next Ghost

An incident like this is cheap to prevent once you know what to look for:

1. Declare Apache configuration as code. Use Ansible or Chef so any local, unreviewed change to the config is visible and reversible.
2. Inventory what goes into Golden AMIs. Review every script, cron job and background binary baked into a base image before promoting it, and remove this safety net from the image itself so new environments do not inherit it.
3. Detect configuration drift. Run periodic scans that alert when a production config file diverges from its version-controlled baseline.
4. Monitor critical directories. Use file integrity monitoring or auditd rules (or Falco/Twistlock) on /etc/ and /etc/cron.* to alert on unexpected file creation or modification.

Closing Thoughts

Legacy systems will keep throwing curveballs. Tracking down a hidden cron script across 250 EC2 instances taught me to widen the search whenever the obvious tools come back empty, and to treat everything baked into a base image as code someone has to own.

Across 4 Jenkins instances and 3000+ jobs, we turned an unreadable pile of failures into a ranked, routable list of causes: unclassified failures fell from about 40% to 10-15%, 789 chronically failing jobs were identified, and peak-time load dropped 36%.

TL;DR

• Problem: 4 Jenkins instances and 3000+ jobs, with no visibility into why jobs fail or which causes to fix first.
• Approach: failure analysis outside Jenkins (Logstash, Elasticsearch, Grafana), regex rules maintained through pull requests, an ML classifier for the long tail, and a weekly report written by a Claude agent through a read-only Grafana MCP server.
• Results: unclassified failures down from about 40% to 10-15%, 789 chronically failing jobs identified, and peak-time load down 36%.

My role. I proposed this project and built the pipeline, dashboards, classifier and reporting agent described below.

The problem: failures everywhere, visibility nowhere

Our CI estate is four Jenkins instances, each serving a different purpose, with more than 3000 jobs. About 600 of them run every day: maintenance, upgrades, integration tests per agent, per OS flavour and per supported version, promotions and acceptance tests.

The topology makes failures hard to explain. Controllers run in a data center, workers are provisioned on a GKE cluster, and test jobs request temporary VMs from vmpooler, which can land in the data center, AWS or GCP. Packages live in JFrog Artifactory, and other services such as ABS sit in the path. A single red build can come from:

• a network problem between any two of those pieces
• a service being down (vmpooler, ABS, Artifactory)
• an OS-specific issue
• a job configuration error
• a faulty build agent
• memory exhaustion

Nobody could say which of these caused the most failures, so triage was driven by whoever shouted loudest. Failures also cascade. A job that fails after provisioning leaves its VMs unused but still holding capacity, in some cases 300+ at a time, and automatic retries multiplied the damage. We needed to see why jobs fail before we could decide what to fix first.

Starting point: the Build Failure Analyzer plugin and its limit

We began with the Build Failure Analyzer (BFA) plugin. You maintain a knowledge base of regex patterns, and BFA matches failed builds against it and shows the cause on the build page.

It worked, but it does all of its matching on the controller. That is the wrong place for it. When a major issue makes pipelines fail in bulk, every failed build triggers regex matching against dozens of patterns, adding load to a controller that is already struggling. A tool meant to diagnose an incident could make the incident worse.

So we set one rule: analysis happens outside Jenkins.

BFA had a second problem. Anyone could add patterns, and over time developers piled up many overlapping regexes with no review. We wanted the knowledge base to stay clean as it grew.

Architecture

Jenkins only ships console logs. Logstash cleans them, tags known failures with regex and asks the classifier about the rest. Elasticsearch stores the result for Grafana dashboards, and a Claude agent reads the same data through Grafana's read-only MCP server to write the weekly report.

Classification: regex first, ML for the long tail

Regex alone has a ceiling: if no pattern matches, the failure is "unclassified", and about 40% of our failures landed there. We also wanted broader categories than individual patterns give, because different teams care about different failures:

• DevOps wants infrastructure failures.
• IT wants SSL and network problems.
• Developers want package and internal failures.
• Job owners want to see job configuration issues.

Building the training data. We first wrote regex rules by hand and collected real failure logs labelled by them. We then used AI to propose failure reasons and categories for logs the rules missed, and a human reviewed these labels before any training. Low-confidence labels were filtered out so noisy examples did not teach the model the wrong thing.

The model. A deliberately simple, CPU-friendly pipeline that can run inside a log pipeline:

• Logs are cleaned first: ANSI codes stripped, UUIDs and long hex strings redacted, and timestamps, paths, URLs, IPs, versions, hostnames and exit codes normalized so the model learns patterns, not one-off values.
• Features are TF-IDF over word unigrams and bigrams (up to 5,000 features), plus binary keyword flags for known signals such as OOM, disk full, network, permissions, test failures, dependencies, containers and databases.
• A scikit-learn Logistic Regression with balanced class weights predicts one of six categories: Infrastructure Failure, Job Failure, Memory Issue, Shell script exited with failure, Test Failure and Workspace/Env Failure.
• Classes are capped during balancing so common failures do not drown out rare ones.

Serving. The model sits behind a small FastAPI service. Logstash sends each failed build's log to it and attaches the predicted category, confidence and processing time to the event before it reaches Elasticsearch. Fixed rules for unambiguous signals such as OOM, disk full and permission errors run before the model.

Rollout. The classifier ran in a staging environment for a period of testing and verification before we promoted it to production.

Making it easy for developers to contribute

Asking a developer who has never used Logstash to edit a Logstash config is a nightmare, and we also wanted version control over every change. All server configuration is managed through Puppet, so we put the regex knowledge base behind a pull request flow:

1. A developer proposes a pattern for a failure they keep seeing.
2. A GitHub Action checks whether an existing regex already covers it.
3. If it is covered, the PR is rejected as a duplicate. If not, the pattern is added.
4. Puppet rolls the updated rules out to the Logstash servers.

Every rule has an author, a review and a history, and the regex sprawl that hurt us with BFA is contained by design.

The AI agent: weekly reports instead of ever-longer retention

Once the dashboards existed, people asked for longer retention. Keeping more raw console logs costs a lot of storage, so we looked for another way to preserve long-term insight.

We built an agent on Claude that produces a weekly report automatically. It highlights which failure categories rose, which Jenkins instances are healthy, and which incidents mattered most. The report goes to a Slack channel for the team and to SharePoint as a permanent reference. The weekly reports now carry the long-term trend, while detailed logs stay at 60 days.

How the agent gets its data. The agent uses the Grafana MCP (Model Context Protocol) server, so it reads the same enriched Elasticsearch data our dashboards show, through Grafana's existing data source, instead of scraping logs.

Read-only by design. The MCP server runs in read-only mode, so the agent can query and summarize but cannot change dashboards or data. Access uses a read-only Grafana token supplied through an environment variable, so no credentials live in code and the agent never holds Elasticsearch credentials.

Why Grafana MCP, not Elasticsearch directly. Grafana is already the single place where our data sources meet. If we want the weekly report to include vmpooler uptime or ABS VM capacity problems, we add those as Grafana data sources and the agent can query them through the same MCP server and the same read-only token. Reports can grow with new signals without writing a new integration each time, and the capacity and outage data are exactly what we need to explain failures such as VMs being unavailable.

Incident awareness. Failures carry metadata that links them, so a burst of hundreds of failures from one vmpooler outage can be recognized as one incident, not hundreds of separate problems.

Results

Test failures leading the list is expected, since that is what CI exists to catch. The other two are more actionable: setup failures cluster around new product versions, and capacity shortages point to a platform limit, not a bug in any job.

Scheduling. The data showed many teams' scheduled jobs requesting VMs at the same moment. Spreading them across time slots cut peak-time load by 36%.

What it did not fix. Many jobs are long-running, so failure rates stayed a problem and wasted VMs were reduced, not eliminated. What we gained was knowing which jobs fail continuously while also consuming VMs, which let us stop the biggest repeat offenders.

Lessons and what is next

The system gave us visibility, and it also showed where it can go further:

• Rank causes by cost, not count. Ingest vmpooler data so failures can be ranked by VM-hours wasted, which turns "300+ wasted VMs" into a measured metric.
• Report incidents and rates. Lead the weekly report with grouped incidents, and compare failure rates per instance instead of raw counts, since more builds can mean more failures without anything getting worse.
• Keep weekly aggregates as data. Store each week's rollup in a small structured index so quarter-over-quarter questions need no raw logs.
• Make uncertainty explicit. Add a calibrated confidence threshold so low-confidence predictions are reported as "uncertain" and routed to review, and use those logs to grow the regex rules and training set.
• Evaluate harder. Hold out a human-verified set, split train and test by job or time so repeat failures from one incident do not leak across, and monitor class share and confidence for drift.
• Stop waste at the source. Pre-flight health checks on vmpooler and Artifactory before provisioning, retry caps with a circuit breaker, and guaranteed VM release in post-build steps with a short lease TTL as a backstop.
• Harden the pipeline. Persistent queues in Logstash and an alert when an instance stops sending logs, so silence is never mistaken for success.

Closing thoughts

The biggest win was not any single model or tool. It was being able to see failures by cause. Once we could, the next steps were practical: spread out scheduled jobs that competed for VMs, stop chronically failing jobs from consuming capacity, and point each failure at the team that can fix it.

Three choices carried the project: keep analysis off the system that is already failing, combine regex precision with ML coverage behind a reviewable pull request flow, and let a weekly report carry long-term trends instead of storing ever more raw logs.

If you run CI at scale and have tackled failure analysis a different way, I would like to hear about it in the comments.

Tech stack

Jenkins (4 instances, GKE agents) · Logstash with the Jenkins Logstash plugin and Grok · Elasticsearch (daily indices, 60-day retention) · Grafana Cloud with Private Data Source Connect · Python, scikit-learn and FastAPI · Claude with the Grafana MCP server (read-only) · Puppet and GitHub Actions · Slack and SharePoint · vmpooler, Artifactory