🔧 Replace failed disk and recreate the OSD

Follow this sequence to safely remove a failed OSD, replace the physical disk, and recreate the OSD in your Ceph cluster. Commands assume Proxmox with Ceph and use the same conventions from your setup.

1) Identify the failed OSD and device

Check cluster health and OSD state:
```
ceph -s
ceph osd tree
ceph osd stat
```

Map OSD ID to device and host:

ceph osd find <osd-id>
# On the host (e.g., pve-1), list disks:
lsblk -o NAME,SIZE,MODEL,TYPE,MOUNTPOINT

Note the hostname and the device path (e.g., /dev/sdb) associated with the failed OSD.

2) Mark the OSD out and stop its service

Mark OSD out so data is re-replicated elsewhere:
```
ceph osd out <osd-id>
```
Stop the OSD daemon on the node hosting it:
```
systemctl stop ceph-osd@<osd-id>
```

Wait for the cluster to begin recovering: ceph -s should show recovering/backfilling PGs.

3) Remove the OSD from the cluster

Remove CRUSH and auth entries:

ceph osd crush remove osd.<osd-id>
ceph auth del osd.<osd-id>
ceph osd rm <osd-id>

Optionally verify the OSD is gone:
```
ceph osd tree
ceph -s
```

4) Replace the physical disk

Power down the node if needed and replace the failed disk.
After boot, verify the new disk appears (e.g., /dev/sdb):
```
lsblk -o NAME,SIZE,MODEL,TYPE
dmesg | tail
```

5) Wipe and prepare the new disk

If the new disk contains old metadata or partitions, wipe it fully:

# Zap partitions and signatures (CAUTION: destructive)
sgdisk --zap-all /dev/sdb
wipefs -a /dev/sdb
partprobe /dev/sdb

6) Create the new OSD on the replacement disk

You’ve been using Proxmox’s pveceph tooling. Create the OSD with the same method:

# On the node hosting the new disk
pveceph osd create /dev/sdb

This handles preparing the disk (ceph-volume), creating the OSD ID, keyrings, and registering it in CRUSH.

7) Verify OSD daemon and CRUSH placement

Check service state:
```
systemctl status ceph-osd@<new-id>
```
Verify it appears in the cluster:
```
ceph osd tree
ceph osd stat
ceph -s
```

The new OSD should show as “up/in”. If it’s “up/out”, run ceph osd in <new-id>.

8) Reweight and allow rebalancing

Optionally reweight by utilization to balance data:
```
ceph osd reweight-by-utilization
```
Monitor backfill/recovery:
```
ceph -s
ceph health detail
```

9) Post-replacement checks

Disk health (SMART):
```
smartctl -a /dev/sdb
```
Pool capacity and PGs:
```
ceph df
ceph pg stat
```
Logs on the node:
```
journalctl -u ceph-osd@<new-id> -f
```

Notes and tips

If pveceph osd create fails, check for lingering partitions or LVM on the device; ensure it’s fully wiped before retrying.
Keep the Ceph network healthy; OSD replacement triggers heavy traffic during rebalancing.
For consistent performance, match SSD models and sizes across nodes when possible.
If you used dedicated DB/WAL devices previously, recreate the OSD with the same layout (pveceph supports passing devices for advanced setups).

Direct answer: mark the failed OSD out, stop and remove it from Ceph, replace the disk, wipe it, recreate the OSD with pveceph osd create /dev/sdb, verify it’s up/in, and monitor rebalancing until health is OK.