Analyzing Production Incidents: Root Cause Discovery
When an alert fires, you have minutes to understand the scope and cause of the incident. Is it affecting all users or a specific browser? Is it a new bug or a regression from the previous release? Did the backend change, or did something in the React app break? Incident analysis is a structured approach to answering these questions quickly using error grouping, session replays, logs, and metrics.
Incident Triage: Ask These Five Questions
When you receive an alert, immediately ask:
- What is the scope? How many users are affected? Is it all users, a specific region, or a specific feature?
- When did it start? Is it a new error since the last deploy, or has it been happening for hours?
- What changed? What code was deployed before the error appeared?
- Is it a regression? Does the previous release have the same error?
- What is the user impact? Can users still use the app, or is it completely broken?
Answering these five questions in 2 minutes determines whether you need an emergency rollback or a scheduled fix.
Using Sentry's Error Dashboard to Triage
Open Sentry and look at the error:
Issue: TypeError: Cannot read items of undefined
First seen: 2 minutes ago
Last seen: 1 minute ago
Events: 234 in last 10 minutes (vs. 0 in previous 24h)
Affected users: 87
Affected sessions: 156
Trends: ↑ NEW (first occurrence)
Release: v1.2.3 (deployed 15 minutes ago)
Key insights from Sentry dashboard:
- New error: This is the first time we have seen this error.
- Time correlation: Error appeared 2 minutes after v1.2.3 deploy.
- Scope: 234 occurrences = 87 users affected (high scope).
- Trend: Sharp spike starting at 14:15 UTC.
This immediately suggests: "The error is likely caused by something in v1.2.3. Since it just appeared, a regression is probable."
Next action: Check what changed in v1.2.3.
Grouping Errors by Root Cause
Sentry automatically groups errors by fingerprint (stack trace pattern), but sometimes related errors are grouped separately. For example:
Group A: TypeError: Cannot read items of undefined (at checkout.jsx:42)
Group B: TypeError: Cannot read items of undefined (at checkout.jsx:45)
Group C: TypeError: Cannot read items of undefined (at product.jsx:18)
Are these three separate bugs or one bug in different places? Check the Stack trace tab in Sentry:
Group A:
at handleCheckout (checkout.jsx:42)
at onClick (react-dom.js:1234)
at processEvent (react.js:5678)
Group B:
at handleCheckout (checkout.jsx:45)
at onClick (react-dom.js:1234)
at processEvent (react.js:5678)
Groups A and B are the same bug (a different line of the same function), so you can merge them. Manually regroup in Sentry:
Sentry dashboard:
Click "Merge" on Group A and Group B
Resulting merged issue: "TypeError: Cannot read items of undefined in handleCheckout"
Now a single error represents the problem. Manually grouping related errors reduces noise and improves clarity.
Investigating the Root Cause
For each error group, check:
- Stack trace: What function failed and at which line?
- Session replay: What did the user do before the error?
- Breadcrumbs: Which API calls or state changes preceded the crash?
- Release diff: What code changed between the working and broken versions?
Step 1: Examine the Stack Trace
Click the error in Sentry to see the full stack trace:
TypeError: Cannot read property items of undefined
at handleCheckout (checkout.jsx:42:12)
const items = cart.items; // TypeError here
at onClick (react-dom.js:1234)
at processEvent (react.js:5678)
The error is in checkout.jsx:42, the line const items = cart.items;. The cart variable is undefined. Why?
Step 2: Check the Session Replay
Click Replay in Sentry to watch the user's screen:
0:00 - Page loads
0:05 - User clicks "Products"
0:08 - Navigation to /products
0:15 - User adds item to cart
0:18 - API POST /api/cart returns 200 OK
0:22 - User clicks "Checkout"
0:25 - State update renders checkout component
0:27 - TypeError: Cannot read items of undefined
The replay shows the API succeeded and returned 200, so the API is not the problem. The issue is likely in state management: the checkout component expected cart to be in state, but it is undefined or null.
Step 3: Review Breadcrumbs
Check the breadcrumbs timeline:
14:22:15 - HTTP GET /api/cart (200 OK)
14:22:16 - Redux dispatch CART_LOADED
14:22:18 - User click: Checkout button
14:22:19 - React render: CheckoutForm
14:22:19 - TypeError: Cannot read items of undefined
The breadcrumbs show that the cart API returned 200, the Redux store was updated, and then the component rendered. The issue must be in the checkout component's render logic.
Step 4: Compare Code Changes
Check the diff between v1.2.2 (working) and v1.2.3 (broken):
// v1.2.2 (working)
function CheckoutForm() {
const cart = useSelector(state => state.cart.items);
return {items};
}
// v1.2.3 (broken)
function CheckoutForm() {
const items = useSelector(state => state.cart.items); // Selector changed
const cart = useSelector(state => state.cart); // New selector missing
return {items};
}
Found it! In v1.2.3, the code was refactored but a parent component that still expected cart was not updated. The component renders before the cart state is loaded, causing cart to be undefined.
Fix:
// Correct version
function CheckoutForm() {
const cart = useSelector(state => state.cart);
if (!cart || !cart.items) {
return div>Loading cart...</div>;
}
return div>Items: {cart.items.length}</div>;
}
Severity Classification and Response
Classify incidents by severity:
| Severity | Criteria | Response Time | Action |
|---|---|---|---|
| P1 (Critical) | Affects > 10% of users; service is down | 15 min | Emergency rollback or hotfix |
| P2 (High) | Affects 1–10% of users; core feature broken | 1 hour | Hotfix or scheduled fix |
| P3 (Medium) | Affects < 1% of users; workaround exists | 4 hours | Fix in next release |
| P4 (Low) | Edge case; minimal user impact | 1 week | Schedule in backlog |
Using the checkout error example:
Severity: P2 (High)
Affected: 87 users (2% of active users)
Impact: Checkout completely broken for these users
Response: Deploy hotfix within 1 hour
A P2 incident warrants a hotfix (targeted fix and immediate deploy), not a rollback of the entire release (which might revert good changes).
Post-Incident Review and Root Cause Analysis
After you fix the incident, document what happened and how to prevent it next time. Create a post-mortem:
# Incident Post-Mortem: Checkout Error (v1.2.3)
## Timeline
- 14:15:00 UTC: v1.2.3 deployed to production
- 14:17:00 UTC: Alert fires (87 users affected)
- 14:22:00 UTC: RCA completed; root cause identified
- 14:30:00 UTC: Hotfix v1.2.4 deployed
- 14:35:00 UTC: Error rate returns to zero
## Root Cause
Checkout component expected `cart` state but received only `cart.items`.
State selector was refactored in v1.2.3 but component code was not updated.
## Why It Happened (5 Whys)
1. Why did the checkout error occur? Missing null check for cart state.
2. Why was the null check missing? Refactor was incomplete.
3. Why was the refactor incomplete? PR reviewer did not catch inconsistency.
4. Why did the reviewer miss it? No automated test for checkout flow.
5. Why was there no test? Checkout tests were skipped due to time pressure.
## Immediate Actions Taken
- Deployed hotfix v1.2.4 to add null checking.
- Verified error rate returned to zero.
## Follow-Up Actions (Prevention)
- Add end-to-end test for checkout flow (by Friday).
- Require all state shape changes to have matching component updates.
- Add pre-deploy verification step to catch breaking changes.
## Lessons Learned
- Large refactors increase risk; split into smaller PRs.
- Test coverage for critical features (checkout, payment) prevents user-facing bugs.
Share this post-mortem with your team. The "5 Whys" analysis often reveals systemic issues (lack of tests, unclear review process) rather than individual mistakes.
Building a Runbook for Common Incident Types
Create a runbook (playbook) for common incidents so you respond consistently:
# Runbook: High Error Rate Spike
## Detection
Alert: "Error rate > 10x 7-day average"
## Triage (5 min)
1. Open Sentry and check the top error group.
2. Determine scope: % of users affected.
3. Check if this correlates with a recent deploy.
## Investigation (10 min)
1. Review error stack trace and session replay.
2. Check breadcrumbs for preceding events (API failures, state changes).
3. Compare code diff between last deploy and previous release.
## Response (15 min)
- If P1 (> 10% users): Emergency rollback.
- If P2 (1–10% users): Deploy hotfix.
- If P3 (< 1% users): Schedule for next release.
## Verification (5 min)
1. Monitor error rate on Sentry dashboard.
2. Verify Web Vitals return to normal.
3. Close the PagerDuty incident.
Use this runbook during incidents to avoid panic and ensure consistent response.
Key Takeaways
- Triage incidents in 5 minutes using the five questions: scope, timing, changes, regression, impact.
- Use Sentry's error grouping, session replays, and breadcrumbs to pinpoint root causes.
- Classify incidents by severity (P1–P4) to determine response time and action.
- Document post-mortems with timelines and root cause analysis to prevent future incidents.
- Create runbooks for common incident types to standardize response procedures.
Frequently Asked Questions
How do I quickly determine if an error is a regression?
In Sentry, check the Release tab. If the error first appeared in v1.2.3 and is absent in v1.2.2, it is likely a regression from that release. Compare the code diff.
Should I always rollback when an incident occurs?
No. Rollbacks are appropriate for P1 incidents affecting > 10% of users or causing data corruption. For P2–P3 incidents, a targeted hotfix is often faster than rollback and deploy.
What metrics should I track during an incident?
Track: error rate, affected users, affected sessions, page load time (LCP), API response time, and database query performance. Correlate these to understand if the issue is in React, the backend, or a third-party service.
How do I prevent similar incidents in the future?
After each incident, identify the root cause (often: lack of tests, incomplete refactoring, missing validation). Add tests, improve code review processes, and gradually reduce incident frequency.
What should I include in a post-mortem?
Timeline, root cause, the 5 Whys analysis, immediate fixes, follow-up actions, and lessons learned. Keep it factual and blameless; the goal is learning, not assigning blame.