A higher education institution in Transilvania was running its central information system on an Oracle database on a Dell PowerEdge R550 server with Dell PERC H755 controller, 10 SAS 558 GB 15k RPM HDDs in RAID10 + 1 hot spare. At 04:00 AM, 6 out of 10 disks went offline simultaneously. We brought it back to production in under 24 hours — without replacing any disk.
Context
The configuration seemed solid on paper:
- Dell PowerEdge R550 server
- Dell PERC H755 hardware RAID controller
- 10 SAS 558 GB @ 15k RPM HDDs in RAID10 array
- 1 hot spare disk (which, logically, doesn't join the array until confirmed failure)
- Oracle database, institution's central information system
Backup ran automatically each night. At 03:00 the backup had been successfully executed on another server. The real problem wasn't lack of backup — it was that no server was ready to run that backup: to receive the restore, start Oracle on it, and take over production in hours.
This made the strategy: restore original hardware, not deploy new server from backup.
The problem
At 04:00, the PERC H755 controller marked 6 of 10 array disks as offline almost simultaneously. The hot spare remained unused — degradation was too massive for a single spare to repair. Volume disappeared from OS, Oracle DB crashed with inconsistent data files.
6 disks reported simultaneously offline in RAID10 (per iDRAC logs) — statistically doesn't exist as real physical failure. That was the first clue: disks weren't dead — they had been logically marked offline by the controller due to a transient error (backplane, SAS expander, firmware glitch).
Diagnosis
PERC H755 log + SMART check on each SAS disk confirmed: all 6 "offline" disks were physically functional, no massive reallocated sectors, no PRE-FAIL errors. The problem was logical state in controller, not dead hardware.
Strategy decision: not deploy new server from 03:00 backup (lengthy — provisioning, Oracle install, restore, validation), but controlled bring disks back into array on existing hardware. Advantage: maximum potential data loss is just 1 hour (between 03:00 backup and 04:00 incident) — regardless of restore approach.
Intervention
Step 1 — Bringing PERC H755 array back online
Functional disks were reintroduced via foreign config import on PERC H755, with attention to order and role of each disk in RAID10 pairs. Array returned in degraded state but readable.
Critical step: any wrong write — any premature rebuild — can permanently destroy parity and data. Worked read-only until integrity confirmed.
Step 2 — Oracle database recovery
With volume readable again, Oracle DB was still inconsistent. Recovery was done based on redo log files, allowing Oracle to bring data files to consistent state and finalize/rollback in-flight transactions.
Result: database started cleanly, recovery used redo logs past 03:00 — so not even the hour between last backup and incident was lost.
Result
- ✅ RAID10 array rebuilt on PERC H755 without disk replacement
- ✅ Oracle DB recovered from redo logs, back in production
- ✅ Critical data loss: zero for committed transactions
- ✅ Total downtime: under 24 hours (vs days for new server provisioning + restore + validation)
- ✅ Extra cost: zero hardware (vs €30-50k for professional recovery + new server)
Lessons learned
- A backup without a server ready to run it isn't a complete DR strategy. Here we got lucky hardware was recoverable. If disks were truly dead, restore would have taken days, not hours.
- "RAID isn't backup." RAID10 + hot spare protects from one disk failure, not from controller failure, human error or ransomware.
- Documented single point of failure. Critical database on single physical server, without hot-standby replication, is a business risk.
Our post-incident recommendation for similar clients: pre-configured warm-standby server (or Hyper-V/Proxmox cluster with VM-level replication) — backup arrives instantly in a ready-to-start environment, RTO drops from hours to minutes.
Have a critical single-point-of-failure server?
We do infrastructure audits asking concrete questions: what happens when the controller fails at 04:00 AM? How long until the app returns? Do you have a server ready to receive the restore? Was the backup tested at restore this week?