Feature Flag Kill Switches: Rollback Without Redeployment

A kill switch is a feature flag that lets you disable a broken feature in seconds without redeploy. An ops toggle is always on, unused until an incident forces you to flip it off. Kill switches are essential for production systems: when a feature is causing database overload, payment failures, or user data corruption, you need a way to stop it instantly. This article covers designing kill switches, setting up on-call alerting, and incident response workflows.

In 2020, a new analytics library leaked user PII by logging request parameters to a third-party service. The issue was caught 45 minutes after deployment when security found the logs. Without a kill switch, we'd have had to rebuild and redeploy (30+ minutes), leaving the PII exposed. With a kill switch, we flipped the flag off in 10 seconds. Incident time: 11 minutes. Without it: 45+ minutes. That's the value of ops toggles.

Designing Effective Kill Switches

Not every feature needs a kill switch. Save kill switches for high-risk code: payments, auth, data processing, external API calls, and new algorithms. Don't add a kill switch for moving a button or changing copy.

Kill switch design:

1. Name clearly: Use the ops_ prefix (e.g., ops_payment_processor_new).
2. Always ON by default: The kill switch should be ON (feature enabled) during normal operation. Flipping OFF disables the feature.
3. Immediate effect: Don't require a page reload or session restart. When the flag is flipped off, the next request should respect the new value.
4. Fallback behavior: When the flag is OFF, gracefully degrade or use the legacy code path. Don't crash.
5. Alert on changes: Log and alert when a kill switch is flipped. This notifies the on-call team that someone is fighting an incident.

Example kill switch implementation:

// paymentApi.js
import { useFeatureFlag } from './FlagContext';

export function usePaymentProcessor() {
  const useNewProcessor = useFeatureFlag('ops_payment_processor_new', true);

  return {
    processPayment: async (paymentDetails) => {
      if (useNewProcessor) {
        return processPaymentV2(paymentDetails);
      } else {
        console.warn('Payment V2 disabled. Falling back to legacy processor.');
        return processPaymentLegacy(paymentDetails);
      }
    },
  };
}

async function processPaymentV2(details) {
  // New payment processor (potentially risky)
  const response = await fetch('https://api.newpaymentprovider.com/charge', {
    method: 'POST',
    body: JSON.stringify(details),
  });
  if (!response.ok) throw new Error('Payment failed');
  return response.json();
}

async function processPaymentLegacy(details) {
  // Proven legacy processor
  const response = await fetch('/api/payment/legacy', {
    method: 'POST',
    body: JSON.stringify(details),
  });
  if (!response.ok) throw new Error('Payment failed');
  return response.json();
}

When the on-call engineer disables ops_payment_processor_new, all new payment requests use the legacy processor. No redeploy, no downtime.

Immediate Flag Updates: Server-Sent Events or Polling

Feature flags are typically cached on app startup. But if a flag is changed on the server, the React app might not know for several minutes (if the cache TTL is 5 minutes, some apps will have stale flags for up to 5 minutes).

For kill switches, you need faster updates. Options:

Option 1: Polling (Simple)

Poll for flag updates every 10–30 seconds. Low overhead, high latency.

// FlagContext.js - Add polling for kill switches
useEffect(() => {
  const interval = setInterval(() => {
    // Only refetch if any ops toggles are active
    const hasOpsToggles = Object.keys(state.flags).some((k) =>
      k.startsWith('ops_')
    );
    if (hasOpsToggles) {
      refetch(); // Refresh flags
    }
  }, 30000); // Poll every 30 seconds

  return () => clearInterval(interval);
}, [state.flags, refetch]);

Downside: up to 30 seconds of stale flags before the update is reflected.

Option 2: Server-Sent Events (Better)

Server sends flag updates in real-time via SSE. React app listens and updates flags immediately.

// FlagContext.js - SSE listener
useEffect(() => {
  const eventSource = new EventSource(
    `/api/flags/subscribe?userId=${userId}`
  );

  eventSource.addEventListener('flag_changed', (event) => {
    const { flagName, value } = JSON.parse(event.data);
    dispatch({
      type: 'FLAG_UPDATED',
      payload: { flagName, value },
    });
  });

  return () => eventSource.close();
}, [userId, dispatch]);

// Update reducer to handle FLAG_UPDATED
function flagReducer(state, action) {
  switch (action.type) {
    // ... existing cases ...
    case 'FLAG_UPDATED':
      return {
        ...state,
        flags: {
          ...state.flags,
          [action.payload.flagName]: action.payload.value,
        },
      };
    default:
      return state;
  }
}

Backend pseudocode:

// Backend: send SSE updates to connected clients
app.get('/api/flags/subscribe', (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Access-Control-Allow-Origin', '*');

  const clientId = req.user.id;
  res.write(': keep-alive\n\n');

  // Subscribe to flag changes
  flagService.on('change', (flagName, value) => {
    // Send update to this client (or broadcast to all)
    res.write(`event: flag_changed\ndata: ${JSON.stringify({
      flagName,
      value,
    })}\n\n`);
  });

  req.on('close', () => {
    // Client disconnected
    flagService.off('change'); // Unsubscribe
  });
});

Upside: Flags update in real-time, within milliseconds. Downside: Requires backend SSE support; slightly more complex.

For most apps, polling every 30 seconds is a good balance: simple to implement, low overhead, and fast enough for kill switches.

On-Call Alerting and Incident Response

When a kill switch is flipped, alert the team immediately. This helps people understand what's happening during an incident.

Alert rules:

1. Log every flag change with timestamp, flag name, old value, new value, and who changed it.
2. Alert on ops-toggle changes (e.g., an ops toggle being turned off is worth a page).
3. Correlate to errors: If errors spike and an ops toggle was flipped 1 minute earlier, that's your suspect.

Example Datadog alert:

# Alert when ops_* flag is changed from true to false
IF metric('flag.change') is true
  AND tag('flag_name') matches 'ops_.*'
  AND tag('value_before') is true
  AND tag('value_after') is false
THEN notify @pagerduty-oncall

Incident workflow:

1. Error spike detected in monitoring (automated alert).
2. On-call engineer pages in.
3. Engineer checks kill-switch logs: "ops_payment_v2 flipped OFF 2 minutes ago."
4. Engineer checks error rate: errors peaked right after the toggle.
5. Conclusion: ops_payment_v2 is likely the cause.
6. Action: investigate the error in logs, or flip the toggle back ON if safe.

Testing Kill Switches

Kill switches should be tested regularly. Once a quarter, simulate an incident:

1. Flip a kill switch OFF.
2. Verify that the feature gracefully degrades (legacy code path used, no crashes).
3. Verify that errors are logged and monitored.
4. Flip the switch back ON.
5. Verify that the feature works again.

Example test:

// __tests__/payment.test.js
import { render, screen } from '@testing-library/react';
import { FlagProvider } from '../FlagContext';
import { Checkout } from '../Checkout';

test('payment gracefully degrades when ops_payment_v2 is disabled', () => {
  const flagsMock = {
    ops_payment_processor_new: false, // Simulate kill switch OFF
  };

  render(
    <FlagProvider initialFlags={flagsMock}>
      <Checkout />
    </FlagProvider>
  );

  // Verify legacy processor UI is shown
  expect(screen.getByText(/legacy payment/i)).toBeInTheDocument();
});

Kill Switch Best Practices

Do:

• Use kill switches for high-risk features (payments, auth, data processing).
• Name kill switches with ops_ prefix.
• Implement graceful fallback (don't crash when the flag is OFF).
• Refresh flags frequently (poll every 30 sec or use SSE).
• Alert on ops-toggle changes and log each change.
• Test kill switches quarterly with incident simulations.

Don't:

• Add kill switches for trivial changes (CSS tweaks).
• Leave a kill switch in code for > 3 months; document its purpose or remove it.
• Flip a kill switch and forget to investigate the issue; always post-mortem.

Key Takeaways

• Kill switches (ops toggles) let you instantly disable broken features without redeploy.
• Design kill switches to always be ON by default, with a graceful fallback when turned OFF.
• Use polling (every 30 sec) or SSE for real-time flag updates.
• Alert on ops-toggle changes and correlate them to error spikes.
• Test kill switches quarterly via incident simulations.

Frequently Asked Questions

How is a kill switch different from a feature flag?

A feature flag controls the initial rollout (0% → 100%). A kill switch is a long-lived ops toggle, used only in emergencies. Technically, they're the same mechanism, but kill switches have different naming, alerting, and testing practices.

What if the flag service itself goes down?

If your config service is unavailable, the app will have cached flags from the last fetch. For kill switches, this is risky: you can't disable a broken feature if the flag service is down. Mitigation: keep the config service highly available (use a CDN, edge cache, or replicate to edge locations). Use a short cache TTL for ops toggles (e.g., 30 sec) so even if the service recovers, the app quickly picks up the new value.

Can I use environment variables as kill switches?

No. Environment variables are set at build time and require a redeploy to change. Kill switches must be changeable at runtime without a redeploy. Use a config service (API, Redis, or a flag management platform).

Should I keep kill switches in code after they're no longer needed?

For frequently-used ops toggles (payment processor, auth), keep them indefinitely—they're your emergency escape hatch. For experimental kill switches, delete them after 3 months if they're never used. Document in the code why a kill switch exists.

How do I ensure users don't see partial state when a kill switch is flipped?

If a kill switch is flipped while a user is mid-checkout, they might see the old UI but the new backend (or vice versa). Mitigation: track which variant the user is in at the start of a transaction (e.g., transaction.variant = 'v2'). Complete the transaction with that variant, even if the flag changes mid-way. This ensures consistency.

Further Reading

• Incident Response Best Practices (Google SRE Book)
• Feature Flags in Production (O'Reilly)
• Kill Switches and Circuit Breakers (Martin Fowler)
• Observability for Kill Switches (New Relic)