Antminer S19 – Filesystem Full Error

Hard power-cycle at the breaker for 30 seconds then power back on. This wipes the `/tmp` tmpfs (RAM-backed) and bounces anything wedged in userspace. Complete fix if the fill was purely tmpfs scratch; no effect on `/var/log` or `/config` fills because those persist across reboot. After the boot completes, SSH in and run `df -h` to confirm which (if any) partition is still full.

SSH in as root and truncate all logs in place. Run `truncate -s 0 /var/log/messages /var/log/bmminer.log /var/log/kern.log /var/log/cgminer.log /var/log/syslog 2>/dev/null`. Critical: use truncate, not rm — deleted files held open by a running `bmminer` process don't free blocks until restart, while truncation frees them immediately. Confirm with `df -h` right after. Space should drop visibly.

Clear any failed firmware staging via the web UI. Log in, navigate to System → Upgrade. If a previous staged firmware is listed, cancel/abort it. Reboot. This is often enough to remove a 60-200 MB dead image file from `/home` or `/nvdata` that was left behind by a dropped upload. Re-check `df -h` after reboot.

Factory reset from the dashboard: UI → System → Configuration → Reset. This rebuilds the `/config` overlay from defaults — the specific Tier-1 answer to overlay corruption. You will lose pool, network, and OC settings, so write them down or screenshot the UI pages first. After reset, you re-enter those settings on a clean overlay.

Back up pool and network config outside the miner before anything destructive. Screenshot the UI pages, or over SSH run `cat /config/bmminer.conf /config/network 2>/dev/null` and save the output to your workstation. This is insurance for Tier 3 SD-card recovery — without it, rebuilding the miner is a multi-hour 'what was my stratum URL' hunt.

Install or replace the logrotate config. On broken stock S19 builds, `/etc/logrotate.conf` is missing or misconfigured. Create `/etc/logrotate.d/bmminer` with a rule capping `bmminer.log` at 5M, rotating 3 generations, gzip-compressing rotations. Then `echo '0 * * * * /usr/sbin/logrotate /etc/logrotate.conf' >> /etc/crontabs/root && /etc/init.d/cron restart`. Verify after the next hour with `ls -la /var/log/bmminer.log*` — you should see rotated files appearing.

Kill and restart `bmminer` cleanly after truncating logs. `killall -9 bmminer` then `/etc/init.d/bmminer restart`. Watch `tail -f /var/log/bmminer.log` for the boot sequence. If `bmminer` reports `cannot create config file: No space left on device`, space wasn't actually freed — return to Step 2 and double-check `df -h` is showing the drop before restarting.

Hunt and remove zombie files in `/tmp`. `find /tmp -type f -mtime +1 -ls` — anything older than 24 hours in `/tmp` is almost certainly garbage, since tmpfs resets on reboot and survivors indicate a failed cleanup. Delete with `find /tmp -type f -mtime +1 -delete`. Re-check `df -h /tmp` after. This catches failed firmware stagings that landed in the wrong directory.

Verify the `/config` overlay is actually writable. `touch /config/test-write && rm /config/test-write`. If touch fails with `No space left on device` but `df -h /config` shows free space, the overlay block-map is corrupted — proceed to Tier 3 SD recovery or a full factory reset. A working overlay will complete the touch-then-rm without error.

Inspect and clear crash dumps. `ls -la /var/crash /var/spool 2>/dev/null` — kernel oops dumps and core files can pile up on long-running miners. Remove anything older than a week, but back them up to your workstation first (scp) if you want to diagnose the underlying crashes later. This typically recovers 5-40 MB on a miner that's been crashing quietly for months.

SD-card factory-level recovery. Format a 16 GB or smaller Micro SD card as FAT32 (larger cards fail the handshake on older S19 boards). Download the correct stock firmware for your hardware revision from service.bitmain.com/support/download — verify against the silkscreen on the control board. Write with Balena Etcher. Insert, power cycle. Auto-detect takes 5-15 minutes to reflash rootfs and rebuild overlays. Remove card, reboot. Full filesystem reset without touching hashboards.

Fix log rotation rather than reflashing. If you want a firmware that ships correct rotation and surfaces filesystem use as a first-class metric, Braiins OS+, LuxOS and Vnish publish install packages — check your exact model on each vendor's own list. DCENT_OS (per-chip HW%, closed-loop autotune) is the fully open-source option, but it publishes downloadable signed experimental artifacts for exactly two lanes — the Antminer S9 and the S19j Pro on Zynq/Xilinx control boards. Of the models on this page that is the S19j Pro only; the S19 and S19 Pro are named in the project with no public artifact; the S19j, S19 XP, S19k Pro and S19 Hydro have no public DCENT_OS artifact and no install route. No model, S9 included, has a published witnessed install route yet — treat a DCENT_OS flash as a bench exercise on a spare unit, not a step on a producing miner. Status: https://d-central.tech/dcent-os/beta-status/. Source: github.com/DCentralTech/DCENT_OS.

Read eMMC life_time registers on late-rev S19 control boards. `cat /sys/class/mmc_host/mmc0/mmc0:0001/life_time` returns two hex values. 0x01 0x01 = fresh eMMC. 0x0A 0x0A = 90-100% of rated writes consumed. 0x0B = end of life. If you see 0x0A or 0x0B, the eMMC is retiring blocks faster than the filesystem can trim — the 'filesystem full' is actually 'storage dying.' Archive the miner as a space heater or replace the eMMC.

Check ext4 inode exhaustion with `df -i`. A partition can report 100% inode use even when blocks are 40% free — common when autotune writes thousands of 4-byte state files to `/config`. `find /config /var -xdev -type f | wc -l` ranks directories by file count. Cleanup: after backup, `rm -rf /config/autotune-state/*` (adjust path to your firmware) drops inode use dramatically. Re-verify with `df -i`.

Flash a last-known-good stock image and add external monitoring. Pick the best-behaved stock release for your hardware revision — for most S19 / S19 Pro builds that's the late-2023 Bitmain release that fixed the logrotate regression. Flash via SD per Step 11. Then add an external cron-fed monitoring script (Prometheus node_exporter or a simple curl-to-alert) that alerts on `df -h` >80% on any mount, because stock still doesn't self-monitor.

Stop DIY and book a D-Central repair slot when: (a) SD recovery completes but the miner doesn't finish booting or reboots itself within minutes, or (b) eMMC life_time reads 0x0A or worse. Either condition means the underlying storage device is failing — no amount of log truncation or reflashing fixes a dying eMMC. D-Central ASIC Repair: https://d-central.tech/asic-repair/. 5-10 business day turnaround, Canada-wide, US/international welcome.

D-Central bench process for a filesystem-full control board: dump the existing eMMC contents with a programmer (RT809H or equivalent), clone to a graded replacement BGA153 eMMC chip from our stocked inventory, reflash the stock rootfs onto the new chip (or, on an S19j Pro with a Zynq/Xilinx control board — the only S19-family board DCENT_OS publishes an artifact for — DCENT_OS at the customer's request, as an explicitly experimental install), full post-repair boot test with a known-good hashboard attached, 24-hour burn-in at nameplate. If the board has physical damage around the eMMC (burn marks, lifted pads), we swap the whole control board and preserve hashboard calibration bins where possible.

Ship safely. Control-board-only shipments: anti-static bag, pad with ≥3 cm of foam on every side, double-box. Keep the hashboards at home unless we've specifically asked for them — shipping three hashboards costs more than diagnosing the control board alone. Include a note with: observed symptoms, firmware version at time of failure, your contact info, and what Tier 1-3 steps you already tried. That information saves us diagnostic time, which saves you money.