Immersion Cooling Pump Failure Creates Hashboard Hotspot

Cut load immediately. Power off every miner submerged in the tank from the PDU or breaker — not the dashboard. With circulation dead, every minute at hash load cooks more silicon. The whole tank's hashrate is worth less than one $400 hashboard you're about to lose. Free, takes 30 seconds, saves the most money of any step on this page.

Verify the pump motor is energised. Voltmeter across the motor terminals (or VFD output) — confirm three-phase or single-phase voltage is actually present at the pump head. Tripped breaker / blown fuse / dropped contactor accounts for a meaningful share of 'dead pump' tickets that are not the pump at all. Free, multimeter only.

Read the VFD / drive panel for active or last-recorded fault codes. Common codes: `OC` (overcurrent — locked rotor / debris jam), `UV` (undervoltage — facility supply), `OL` (overload — bearing drag or fluid viscosity high), `PHL` (phase loss — wiring), `OT` (motor over-temp — duty cycle / cooling). Note the code; it dictates the rest of the diagnostic branch.

Validate the temperature differential top-to-bottom of the tank with a calibrated thermocouple or thermal probe. Healthy single-phase circulation: delta `< 5 °C`. Stratified loop: `> 10 °C`. Dead pump under continued load: `> 20 °C` and climbing. The number tells you how long you've been stratified — useful for triaging which hashboards to inspect first.

Pull the strainer / suction-side filter. With the pump isolated and tank fluid level dropped below the pickup, remove the strainer cup. Inspect for debris: ASIC fasteners, gasket fragments, chip-cap shards, biofilm. A blocked suction strainer drops pump output to a fraction of nameplate — the pump is fine, the strainer is starving it. Clean, reinstall, restart, re-test flow.

Check for a magnetic-drive coupling slip. With facility power off, manually rotate the pump shaft / impeller magnet. It should spin freely and re-couple to the drive magnet on power-up. If the impeller magnet is decoupled (rotor spinning, magnet stationary) you'll hear a distinctive high-frequency rattle. Mag-drive slip is a known failure mode under cold-start high-viscosity conditions and after debris jams.

Failover to the redundant pump if the loop is built with one. Most production single-phase immersion designs ship (or should ship) with an active+standby pair on a manifold with isolation valves. Manually transfer with the valve handles, energise the standby, verify flow returns, then plan repair on the failed unit at leisure. Fifteen minutes if the manifold is built right; if the manifold isn't built right, that's the lesson — see Prevention.

Bleed air from the pump volute and high points of the secondary loop. After any tank top-up, valve isolation, or pump replacement, air pockets at the volute crown will starve the impeller and produce zero flow despite a healthy pump. Crack the volute bleed plug or the highest manifold tee, vent until steady fluid runs. Five minutes, prevents an hour of misdiagnosis.

Test pump shaft bearings before assuming pump-end failure. Energise the bare pump (suction valve closed briefly) and listen — smooth hum = bearings OK. Grinding / metallic wash = bearings shot. On mag-drive pumps the wear sleeve / journal bearing is the most common failure point, replaceable as a kit ($60-$180 USD typical, $80-$240 CAD).

Megger the motor windings. With the motor isolated, measure insulation resistance phase-to-ground and phase-to-phase at `500 VDC` or `1000 VDC` per the motor nameplate. Healthy: `> 10 MΩ`. Suspect: `1-10 MΩ`. Failed (water / fluid ingress / winding short): `< 1 MΩ`. Common failure: dielectric fluid migrated past a failed shaft seal on non-mag-drive designs, shorted the windings.

Replace the pump head / impeller assembly with an OEM-spec equivalent — same flow curve, same fluid compatibility, same connection geometry. Mag-drive over shaft-seal every time on dielectric immersion (no leak path). Materials: 316 stainless or PTFE-lined wetted parts, ceramic / silicon-carbide journal bearing. Upsize for `+25 %` flow margin over current-load demand to absorb future capacity growth and resist clogging-driven derate.

Pull and inspect every hashboard from the top half of the tank. Top-bank position is where the heat accumulated when circulation died — these are the boards most likely cooked. Look for discoloured silicon, lifted BGA solder balls, scorched solder mask, and cap drift. Bench-test before re-immersing. Replacement-or-rework call is cheaper than re-immersing a marginal board and watching it fail in the next stratification event.

Sample the dielectric fluid for contamination. Look for colour shift (clear → yellow / amber / copper-tinted = oxidation or copper leaching), particulate count, and acid number on synthetic fluids per the manufacturer's spec sheet. Engineered Fluids and LiquidStack publish acceptance limits. Out-of-spec fluid degrades pump bearings, attacks ASIC heatsinks, and reduces dielectric strength.

Install a redundant pump pair if the tank doesn't have one. The fix is mechanical: an active pump and a standby pump on a manifold with isolation valves and a check valve, both wired to the VFD with auto-transfer logic on flow loss. Cost is dwarfed by one cooked top-bank in a 30 kW tank. Mining Hacker default — never run a single-pump immersion loop in production.

Add flow alarms and tank-stratification monitoring. Inline turbine or paddle-wheel flow meter on the return line wired to a relay that powers off the PDU on flow loss; thermocouples top and bottom of tank wired to a comparator alarm at `> 8 °C` delta. Mining-rig PLCs / open-source controllers (Sonoff / Shelly / ESP32) all do this for under $200. The alarm is what saves the silicon — not the pump's reliability.

Right-size the secondary heat reject. The most common upstream cause of a chronically-overworked pump is an undersized dry cooler / fluid-to-air heat exchanger forcing the loop to run hotter than design. Hotter fluid = lower viscosity = same flow, but higher tank ΔT and longer time-to-stratify when flow is lost. Dual-coil dry-cooler with N+1 fan banks recommended for production-grade heat reject.

Stop DIY and book D-Central's immersion service when: more than one hashboard from the tank shows thermal damage, the dielectric fluid is out of spec, you've had two pump failures in 12 months (systemic problem, not a single component), or you need engineering review of a tank build. We carry mag-drive replacements, fluid testing relationships, and bench infrastructure for multi-board diagnostics.

Document the failure for warranty and insurance. Photograph the pump-out condition, save the VFD fault history, log the thermocouple delta time-series, and bag any debris found in the strainer. Pump manufacturers and dielectric fluid suppliers will both want documentation for a warranty claim. Include facility-power voltage logs covering the 24 hours before failure — undervoltage events are a common warranty defence.

Recommission the loop end-to-end before re-energising miners. Pump on, bleed, run 30 minutes at no-load, verify flow at meter ≥ design, verify thermocouple delta `< 5 °C` with miners off but fluid circulating. Only then re-energise miners — and ramp from one rack at a time, verifying delta stays under `8 °C` at each step. Cold-start a 30 kW load all at once and you can stratify before the loop even reaches steady state.

Re-confirm prevention scaffolding before walking away: redundant pump online + tested, flow alarm wired to PDU shutoff, thermocouple delta alarm at `8 °C`, dry-cooler at design capacity, strainer service interval logged, dielectric fluid sample queued for next quarter. Without these, you're scheduling the next failure — and you already paid the tuition on this one.