Stop fighting fires: a systematic maintenance checklist to keep your production software running smoothly.
Production software is never set-and-forget. Even the most carefully built systems degrade over time: dependencies fall out of date, disk space fills up, response latencies creep upward, and security vulnerabilities emerge. Without a deliberate maintenance routine, teams spend their cycles fighting fires instead of building features. A structured checklist keeps your team proactive, predictable, and prepared. Below is a practical, battle-tested set of practices that any team running production software should adopt and revisit regularly.
Every production system requires reliable monitoring that covers the four golden signals: latency, traffic, errors, and saturation. Define clear thresholds for each — for example, alert when p99 latency exceeds 500 ms for more than five minutes, or when the error rate climbs above 1% of total requests. Avoid alert fatigue by tuning sensitivity: too many false positives desensitise the team; too few and you miss early warning signs.
Set up both infrastructure-level and application-level alerts. Infrastructure alerts track CPU, memory, disk I/O, and network bandwidth. Application alerts watch business-critical endpoints, queue depths, database connection pools, and external API response times. Every alert must have a runbook or a clear description so that any on-call engineer can triage it in minutes, not hours.
Structured logging is non-negotiable. Every service should emit logs in a consistent format — JSON is the industry standard — with fields for request ID, user ID, timestamp, severity, and service name. Centralise these logs in a platform like Elasticsearch, Loki, or a cloud-native solution so that teams can search, filter, and correlate across services without SSH-ing into boxes.
Define a log retention policy based on regulatory and debugging needs. Keep two weeks of detailed logs for active debugging, and archive older logs to cold storage for compliance. Regularly review log volume to prevent runaway costs: log what you actually use in debugging, and drop verbose debug output in production unless temporarily enabled via feature flags.
Backups are only useful if you test restoring them. Schedule automated backups of databases, file stores, and configuration stores — at minimum daily for databases and weekly for configuration. Store backups in a separate region or cloud account to survive a regional outage. Once a quarter, run a full restore drill in a staging environment and measure the recovery time objective (RTO) and recovery point objective (RPO).
Document the disaster recovery playbook step by step: which databases to restore first, how to verify data integrity, and how to redirect traffic to the recovery environment. Without a documented and tested plan, a real disaster will cripple your team’s response time and turn a minor outage into a major incident.
Outdated software is the most common attack vector. Subscribe to security advisories for every runtime, library, and operating system you use. Apply critic hours and high-severity patches within a week. Automate vulnerability scanning in your CI/CD pipeline so that new dependencies are checked before they reach production.
Rotate secrets — database passwords, API keys, TLS certificates — on a regular cadence. Use a secrets manager (like HashiCorp Vault or AWS Secrets Manager) and never store sensitive values in environment variables that can leak in logs. Additionally, perform periodic penetration tests or use a bug bounty programme to uncover blind spots your automated scans miss.
Dependencies accumulate technical debt silently. Set up automated dependency update tools — Dependabot, Renovate, or your language’s equivalent — to open pull requests for minor and patch versions. Review those updates with a focus on changelogs: a change that fixes a memory leak in a library you use may be worth a mid-sprint upgrade.
Schedule a quarterly “dependency grooming” sprint where the team upgrades major versions of critical libraries. For each upgrade, run the full test suite and performance benchmarks to catch regressions early. Keep a spreadsheet or ticket that tracks the upgrade status of major infrastructure packages like the language runtime, web server, and database driver.
Performance regressions rarely announce themselves. Establish baseline metrics for request latency, throughput, and resource usage after every significant deployment. Use continuous profiling tools (like Pyroscope or Datadog Continuous Profiler) to identify hot spots before they cause user-facing slowdowns.
Monitor capacity trends: disk usage growth rate, network bandwidth utilisation, and database connection counts. When utilisation crosses 70% of a resource’s limit, plan a capacity increase. Automate scaling triggers for stateless workloads (e.g., Kubernetes Horizontal Pod Autoscaler) but ensure stateful services like databases have over-provisioned headroom to handle traffic spikes without manual intervention.
A maintenance checklist is incomplete without a clear incident response process. Define severity levels (SEV1–SEV3) with corresponding response times and escalation paths. Equip each on-call team with a dedicated rotation schedule, a primary and secondary contact, and a handover process between shifts.
Post-incident, conduct a blameless post-mortem within three business days. Capture what went wrong, what monitoring gaps existed, and what automation could prevent recurrence. Track action items in your project management tool and assign owners. A team that learns from every incident steadily reduces the rate of repeat failures.
Production systems accumulate digital debris. Schedule automated tasks for log rotation, temporary file cleanup, and database vacuuming or compaction. Monitor disk usage on all nodes and set alerts when it crosses 80% capacity. Stale records in databases — abandoned user accounts, expired sessions — should be purged or archived quarterly.
Review TLS certificate expiration dates monthly and automate renewal where possible (e.g., Let’s Encrypt with cert-manager). For older systems that still rely on manual certificate renewal, add a recurring calendar reminder 30 days before expiry. One missed certificate can take an entire service offline in minutes.
All the checklists in the world are useless if the team doesn’t maintain them. Keep an evergreen runbook that documents the steps for common maintenance tasks: restarting a service, scaling a cluster, adding a new node, or running a database migration. Store the runbook in a central, easily searchable location alongside your monitoring playbooks.
Schedule a triage session once per sprint to update the runbook based on recent incidents or configuration changes. Pair junior engineers with seniors during these sessions to build institutional knowledge. When someone leaves the team, their undocumented tribal knowledge leaves with them — protect your team from that loss.
A checklist only works if you use it. Embed these activities into your regular sprint cycle: weekly alert reviews, monthly security patch windows, quarterly backup drills, and semi-annual dependency upgrades. Use automation as much as possible — infrastructure as code, scheduled jobs, and policy-as-code tools (like Open Policy Agent) — to reduce human error.
Production maintenance is not glamorous, but it is the foundation of reliability. A team that invests in a structured checklist will spend less time in incident war rooms and more time delivering value to users. Start with the practices most relevant to your current pain points, then expand to cover the full list. Your future on-call self will thank you.