Immersion Cooling Biofouling Clogs Microchannels

Tier 1 — Pull a fluid sample now. Wide-mouth amber glass jar (NOT plastic — plasticisers leach), 250 mL minimum, drawn from the reservoir bottom drain and a second from the suction strainer cup. Cap, label with date / loop / source location, and ship cold to the dielectric vendor's lab. The lab number is what stops the next argument with the insurance adjuster, the fluid vendor, or your warranty processor. Cost: $40-$120 per sample. Always cheaper than the next cooked board.

Tier 1 — Visually inspect the reservoir interior and the strainer cup. With the loop isolated and tank fluid level dropped, pull the strainer cup, photograph it, and look inside the reservoir. Biofilm is white / cream / amber, gel-like or filamentous, often easiest to see on the underside of the reservoir cap or the inside corners. Metallic debris is grey / black grit; biofilm is biological-looking and won't rinse away in dielectric. Document with photos at this stage — they go to the lab and the insurance file.

Tier 1 — Audit the moisture ingress path. Walk every meter of the loop and the reservoir. Look for: open-top top-ups, compressed gaskets, weeping fittings (any wet spot anywhere is suspect), condensation on chilled secondary-loop plumbing, humid pump-room ambient, uncapped jerry cans of dielectric in storage. Make a list. Without fixing the ingress, every downstream Tier 2 / Tier 3 fix is on a 60-180 day countdown to recolonisation.

Tier 1 — Increase strainer service frequency immediately. From whatever your current schedule is (typically quarterly on production loops) to monthly until you have lab numbers in hand. Biofilm sheds particulate; the strainer is your first line of defence against shed material clogging downstream microchannels. Each strainer pull is also a chance to re-photograph and track progression. Free, takes 15 minutes per service call.

Tier 1 — Drop hash load on suspect plates. If per-plate temperature mapping (IR camera or firmware chip temps) shows specific hashboards running their plates +5 °C or more over the rest, pull load on those boards from the dashboard or PDU. Mining at hashrate against a fouled plate is silicon damage on a clock you don't control. Loss of revenue is a fraction of one cooked hashboard. Run reduced load until you've got the fix plan validated.

Tier 2 — Shock-dose a dielectric-compatible bio-inhibitor. BIT (1,2-benzisothiazolin-3-one) and MIT (methylisothiazolinone) chemistries are the industrial-water-treatment families adapted for hydrocarbon dielectrics; quaternary-ammonium chemistries (e.g. didecyldimethylammonium chloride) are an alternative. Dose only what your fluid vendor approves — uncontrolled dosing breaks dielectric strength, attacks gaskets, or precipitates solids. Typical shock dose: 100-500 ppm active for 24-72 hours of recirculation, then re-test fluid. Do this only after Tier 1 sampling — you need a before lab number.

Tier 2 — Run a sterile in-place flush. With miners off and bio-inhibitor doses circulated, drain the loop to maintenance level, refill with fresh dielectric (same fluid family, manufacturer-approved batch), and circulate at full pump rate for 2-4 hours with strainer pulls every 30 minutes. Each strainer pull catches more shed biofilm. Continue flush cycles until strainer pulls return clean and the fluid runs visually clear. Do not return the loop to hash load until lab numbers come back in spec.

Tier 2 — Replace the suction strainer with a finer mesh. From whatever the loop ships with (often 100 mesh / 150 micron) to 200-400 mesh (75-37 micron) for the recovery period. Finer mesh requires more frequent service but catches more biological particulate during recolonisation watch. Revert to standard mesh once the loop has held clean lab numbers for two consecutive sampling intervals.

Tier 2 — Dehydrate the loop. Moisture is the fuel; remove it. Options: nitrogen purge of the reservoir headspace, vacuum-degas of the dielectric (pull < 50 mbar for 2-4 hours on small loops, longer on larger), or pass the fluid through a portable molecular-sieve dryer. Target Karl Fischer water content below the manufacturer's published limit — typically < 100 ppm for hydrocarbon dielectrics, < 50 ppm for the more hygroscopic synthetic esters. This is the step most operators skip; this is the step that determines whether biofouling returns.

Tier 2 — Seal every moisture ingress path identified in Step 3. Replace compressed reservoir gaskets, re-torque or replace weeping flare fittings, install a desiccant breather on the reservoir if it was previously vented to atmosphere, dehumidify the pump room or relocate equipment, and switch to closed-top sealed jerry cans for fluid storage. Closed-top top-up procedure with nitrogen blanket is the production-grade default; that is your target maintenance procedure going forward.

Tier 3 — Pull every cold-plate from the affected hashboards. Disconnect the dielectric loop, blank the inlet/outlet ports on the plate to keep contamination contained, and remove the plate from the chip mount. Inspect the inlet manifold and the channel array under good light or a 5-10x magnifier. Photograph. Biofilm is visible at this scale; clean channels are mirror-shiny copper or aluminium. Plates that show heavy manifold colonisation move to the bench-flush queue.

Tier 3 — Bench-flush each plate independently. Pump fresh dielectric (or, for badly fouled plates, a vendor-approved aqueous detergent followed by deionised water rinse and dehydration) through each plate at design flow until the effluent runs clean. Measure inlet-to-outlet pressure drop against the OEM spec or a known-clean reference plate. Plates that won't return to spec ΔP after flush are mechanically clogged — Tier 4 territory.

Tier 3 — Replace the TIM layer between chip and plate. Whatever was there is contaminated and probably degraded by the fouling event. Use OEM-spec TIM (thermal pad, paste, or PCM) at OEM-spec thickness; non-spec TIM in microchannel cooled systems frequently exceeds the chip-to-plate gap and starves the plate of contact area. Arctic MX-6, Thermal Grizzly Kryonaut, or the OEM-supplied TIM if the cooling kit ships with one — match the OEM spec exactly.

Tier 3 — Replace the dielectric fluid entirely on heavily-affected loops. Don't try to recover badly-contaminated fluid by filtration; the lab numbers usually show degradation products (acid number, dielectric strength loss) that filter media can't remove. Drain, flush the loop with fresh fluid until effluent runs clean and lab-tests in spec, refill with new dielectric, and dispose of the contaminated fluid through a hazardous-waste channel per local regulation. Fluid replacement cost is a fraction of one cooked top-bank hashboard.

Tier 3 — Install closed-top fluid handling and inline moisture monitoring. Replace open-top reservoirs with sealed nitrogen-blanketed reservoirs or desiccant-vented expansion tanks. Plumb in an inline Karl Fischer-equivalent moisture sensor or schedule sample-based water monitoring at quarterly intervals minimum. Production-grade immersion deployments live or die on this discipline; retrofit it before the next failure schedules itself.

Tier 4 — When to stop DIY. Stop when: lab fluid analysis shows microbial load + degradation byproducts + acid number all out of spec simultaneously (systemic loop contamination), more than one cold-plate returned cooked silicon, a brazed-plate heat exchanger is suspected of cross-contamination between dielectric and secondary-loop sides, or you've flushed once and the same plates re-clogged within 90 days. You're now in fluid-engineering territory — book D-Central's immersion service for a tank-level diagnostic, fluid replacement, and recommissioning audit.

Tier 4 — What D-Central does at the bench. Pull the loop, inspect each cold-plate under proper magnification (sometimes microscopy — biofilm thickness measurement at the inlet manifold), bench-flow-test each plate against OEM spec, contract dielectric-lab fluid sampling, recommendation on bio-inhibitor dosing for the loop chemistry, replacement plates from D-Central inventory or vendor channels, dehydration and recommission, and a written recommissioning report you can hand to a fluid vendor for warranty support or to an insurance carrier for claim documentation.

Tier 4 — Ship and stage safely. Drain the loop to a sealed transit reservoir (no open jerry cans on a freight pallet — leak path and ingress path), bag every cold-plate individually in moisture-barrier packaging with desiccant, and ship the affected hashboards in anti-static double-walled boxes with ≥ 5 cm foam padding. Include the lab analysis (or the sample shipping tracking number), your moisture-ingress audit notes, the strainer-pull photo log, and the plate-temperature map. Documentation on intake saves diagnostic hours, which saves you money on the bench bill.