Monitoring That Catches Failures Before Customers (Prometheus + Grafana)

The Monitoring Crisis

Your app goes down. You're asleep. A customer tweets about it. Your support team wakes you up. You've been down for 20 minutes.

This happens because you don't have monitoring. You ship code, hope it works, and only react when it breaks.

Real monitoring changes this: Your alert goes off. You fix it. Customer never knows.

Monitoring vs. Observability (The Distinction)

Monitoring: You know when something is wrong. ("API response time just doubled")

Observability: You know what went wrong and why. ("Response time doubled because database connection pool is exhausted because we have 1,000 concurrent users")

For early stage, monitoring is enough. You can fix the immediate problem, then debug deeper if needed.

What to Monitor (The Essential 7)

1. Request Latency (Response Time)

• What: How long does each request take?
• Target: p50 < 100ms, p99 < 500ms (adjust for your app)
• Alert: If p99 > 1 second, wake me up

API Response Time:
  p50: 45ms
  p99: 120ms
  max: 450ms

2. Error Rate

• What: What % of requests are failing?
• Target: < 0.1%
• Alert: If > 1%, wake me up

3. Throughput (Requests Per Second)

• What: How many requests is your app handling?
• Target: Know your baseline; alert if it drops (weird) or spikes
• Alert: If spike suggests DoS or bug

4. Memory Usage

• What: Is your app leaking memory?
• Target: Stable (not growing over time)
• Alert: If > 80% of available, wake me up

5. CPU Usage

• What: Is your server overloaded?
• Target: < 70% average
• Alert: If > 85%, consider scaling

6. Database Connections

• What: Are you running out of DB connection pool?
• Target: < 80% of max
• Alert: If > 90%, you'll run out soon

7. Request Queue Depth (If Using Job Queue)

• What: Are jobs piling up?
• Target: Should process faster than new jobs arrive
• Alert: If queue > 1,000, wake me up

Setup (15 Minutes)

Step 1: Run Prometheus (Docker)

# prometheus.yml
global:
  scrape_interval: 15s
 
scrape_configs:
  - job_name: 'nodejs-app'
    static_configs:
      - targets: ['localhost:3000']

docker run -d \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Prometheus now runs on http://localhost:9090

Step 2: Instrument Your Node.js App

npm install prom-client

const express = require('express');
const client = require('prom-client');
 
const app = express();
 
// Create metrics
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.1, 0.5, 1, 2, 5]
});
 
const httpRequestsTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code']
});
 
// Middleware to track requests
app.use((req, res, next) => {
  const start = Date.now();
 
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestDuration
      .labels(req.method, req.route?.path || 'unknown', res.statusCode)
      .observe(duration);
    
    httpRequestsTotal
      .labels(req.method, req.route?.path || 'unknown', res.statusCode)
      .inc();
  });
 
  next();
});
 
// Expose metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});
 
app.listen(3000);

Step 3: Run Grafana

docker run -d \
  -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD=admin \
  grafana/grafana

Grafana runs on http://localhost:3000 (user: admin, password: admin)

Step 4: Connect Grafana to Prometheus

1. In Grafana, go to Configuration → Data Sources
2. Click "Add Data Source"
3. Choose "Prometheus"
4. URL: http://localhost:9090
5. Save

Step 5: Create Dashboard

{
  "dashboard": {
    "title": "API Monitoring",
    "panels": [
      {
        "title": "Request Latency (p99)",
        "targets": [
          {
            "expr": "histogram_quantile(0.99, http_request_duration_seconds_bucket)"
          }
        ]
      },
      {
        "title": "Error Rate",
        "targets": [
          {
            "expr": "rate(http_requests_total{status_code=~'5..'}[5m])"
          }
        ]
      },
      {
        "title": "Requests Per Second",
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])"
          }
        ]
      },
      {
        "title": "Memory Usage",
        "targets": [
          {
            "expr": "process_resident_memory_bytes"
          }
        ]
      }
    ]
  }
}

Or use Grafana's UI to create panels (easier for non-PromQL users)

Alerting (Get Notified When Things Break)

Alert Rule (Prometheus)

# alerts.yml
groups:
  - name: api
    interval: 1m
    rules:
      - alert: HighResponseTime
        expr: histogram_quantile(0.99, http_request_duration_seconds_bucket) > 1
        for: 5m
        annotations:
          summary: "API response time high (p99 > 1 second)"
 
      - alert: HighErrorRate
        expr: rate(http_requests_total{status_code=~"5.."}[5m]) > 0.01
        for: 1m
        annotations:
          summary: "API error rate high (> 1%)"
 
      - alert: HighMemory
        expr: process_resident_memory_bytes > 500000000
        for: 5m
        annotations:
          summary: "Memory usage high (> 500MB)"

Send Alerts to Slack

Install Alertmanager, configure webhook:

# alertmanager.yml
global:
  resolve_timeout: 5m
 
route:
  receiver: 'slack'
 
receivers:
  - name: 'slack'
    slack_configs:
      - api_url: 'YOUR_SLACK_WEBHOOK_URL'
        channel: '#alerts'
        title: 'Alert: {{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.summary }}'

Now when an alert fires, Slack notifies you immediately.

Real Scenario: Production Deployment (100K Users)

You deploy a new feature. 5 minutes later:

🔴 ALERT: HighResponseTime
p99 latency jumped from 120ms to 800ms

You check Grafana dashboard:

API Response Time (p99): 800ms
Requests Per Second: 1,200 (spike from normal 400)
Database Connections: 98 (near max of 100)

Diagnosis: New feature is causing N+1 queries. Database connection pool exhausted.

Action: Revert the feature. Connections drop to normal. Latency back to 120ms.

Without monitoring: Customer reports slowdown in email. Takes 30 minutes to notice. 30,000 users affected.

With monitoring: You know in 2 minutes. Fix in 5. Customer never notices.

Best Practices

Do:

• Alert on outcomes (latency, error rate), not raw metrics (CPU)
• Set reasonable thresholds (know your baseline first)
• Test alerts (make sure Slack integration works)
• Review alerts weekly (are they noisy? Too quiet?)
• Keep dashboards simple (one page, 5-7 graphs max) Don't:
• Alert on everything (noise = ignored alerts)
• Alert on CPU usage (it's a symptom, not the problem)
• Set thresholds without baseline data (measure first, then decide)
• Ignore alerts (they're trying to save your business)
• Run only Prometheus without alerting (just monitoring failures isn't enough)

Common Mistakes

Production Checklist

• ✅ Prometheus running (separate machine/container)
• ✅ Node.js app instrumenting requests (prom-client)
• ✅ Grafana dashboards created (5-7 key metrics)
• ✅ Alerts configured (latency, error rate, memory)
• ✅ Slack integration working (test alert)
• ✅ Baseline collected (know what "normal" looks like)
• ✅ Review cadence set (weekly dashboard review)

Your Competitive Edge

Founders using monitoring:

• Know about problems before customers do
• Fix issues in minutes instead of hours
• Have data to show performance improvements
• Can confidently ship features (know if they broke something)
• Sleep better (alerts wake you, not customer support)

Start this week. Add prom-client to your app. Run Prometheus in Docker. Create one Grafana dashboard. You'll never go back.