Mark's random ramblings

Playing around with old server tech with FreeBSD and ZFS

So I've been playing around with a dual-CPU opteron 6386SE as the hardware is comparatively cheap from ex-server farms, and ECC Registered RAM is also pretty cheap second-hand. This machine is currently running 13 jails and 2 bhyve VMs and is stable as a rock. Plus it has IPMI as it is a SuperMicro board.

The machine has a lot of free space allowing me room to expand if needed.

zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
array0  2.72T   507G  2.22T        -         -    41%    18%  1.00x  ONLINE  -
zroot   79.5G  40.0G  39.5G        -         -    40%    50%  1.00x  ONLINE  -

There is 2x2TB HDDs mirrored and 2x1TB HDDs mirrored running with 2x120GB SSDs acting to speed up the ZIL and ARC (the size is overkill but the drives were cheap to buy)

As you can see there's quite a lot of RAM (a lot of it is unused).

sysctl hw.physmem
hw.physmem: 180285796352
last pid: 42622;  load averages:  4.15,  3.55,  3.19  up 1+22:07:26    09:51:06
311 processes: 1 running, 310 sleeping
CPU 0:   0.6% user,  0.0% nice,  2.5% system,  0.1% interrupt, 96.8% idle
CPU 1:   0.9% user,  0.0% nice,  3.8% system,  0.0% interrupt, 95.3% idle
CPU 2:   0.7% user,  0.0% nice,  3.0% system,  0.1% interrupt, 96.3% idle
CPU 3:   1.1% user,  0.0% nice,  4.7% system,  0.0% interrupt, 94.2% idle
CPU 4:   0.8% user,  0.0% nice,  2.7% system,  0.1% interrupt, 96.5% idle
CPU 5:   0.8% user,  0.0% nice,  3.5% system,  0.0% interrupt, 95.6% idle
CPU 6:   0.7% user,  0.0% nice,  3.4% system,  0.1% interrupt, 95.8% idle
CPU 7:   0.8% user,  0.0% nice,  4.4% system,  0.0% interrupt, 94.8% idle
CPU 8:   0.6% user,  0.0% nice,  5.0% system,  0.0% interrupt, 94.4% idle
CPU 9:   0.5% user,  0.0% nice,  4.1% system,  0.0% interrupt, 95.4% idle
CPU 10:  0.6% user,  0.0% nice,  6.0% system,  0.0% interrupt, 93.4% idle
CPU 11:  0.5% user,  0.0% nice,  4.3% system,  0.0% interrupt, 95.2% idle
CPU 12:  0.6% user,  0.0% nice,  5.5% system,  0.0% interrupt, 93.9% idle
CPU 13:  0.6% user,  0.0% nice,  5.3% system,  0.0% interrupt, 94.1% idle
CPU 14:  0.6% user,  0.0% nice,  4.6% system,  0.0% interrupt, 94.8% idle
CPU 15:  0.7% user,  0.0% nice,  6.2% system,  0.0% interrupt, 93.1% idle
CPU 16:  0.9% user,  0.0% nice,  6.1% system,  0.0% interrupt, 93.0% idle
CPU 17:  0.9% user,  0.0% nice,  6.3% system,  0.0% interrupt, 92.7% idle
CPU 18:  0.9% user,  0.0% nice,  6.3% system,  0.0% interrupt, 92.7% idle
CPU 19:  1.0% user,  0.0% nice,  6.5% system,  0.0% interrupt, 92.5% idle
CPU 20:  1.0% user,  0.0% nice,  6.6% system,  0.0% interrupt, 92.4% idle
CPU 21:  1.0% user,  0.0% nice,  6.6% system,  0.0% interrupt, 92.4% idle
CPU 22:  1.0% user,  0.0% nice,  6.6% system,  0.0% interrupt, 92.4% idle
CPU 23:  1.0% user,  0.0% nice,  6.8% system,  0.0% interrupt, 92.2% idle
CPU 24:  0.9% user,  0.0% nice,  6.0% system,  0.0% interrupt, 93.0% idle
CPU 25:  1.0% user,  0.0% nice,  6.2% system,  0.0% interrupt, 92.9% idle
CPU 26:  0.9% user,  0.0% nice,  6.2% system,  0.0% interrupt, 92.8% idle
CPU 27:  0.9% user,  0.0% nice,  6.4% system,  0.0% interrupt, 92.7% idle
CPU 28:  1.0% user,  0.0% nice,  6.4% system,  0.0% interrupt, 92.6% idle
CPU 29:  1.0% user,  0.0% nice,  6.5% system,  0.0% interrupt, 92.5% idle
CPU 30:  1.0% user,  0.0% nice,  6.6% system,  0.0% interrupt, 92.4% idle
CPU 31:  1.0% user,  0.0% nice,  6.6% system,  0.0% interrupt, 92.3% idle
Mem: 7855M Active, 61G Inact, 545M Laundry, 35G Wired, 571M Buf, 59G Free
ARC: 19G Total, 8339M MFU, 3590M MRU, 1454M Anon, 303M Header, 4820M Other
     5726M Compressed, 13G Uncompressed, 2.37:1 Ratio
Swap: 32G Total, 32G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME     CPU COMMAND
31682     88       59  20    0    40G  7225M select  13 169:17 129.88% mysqld

There's two NICs (technically three if you include the IPMI NIC) so I can also have jails listening on different interfaces to improve network speeds. For example, I have Nextcloud running on a NIC so I get max speeds from it without other jails slowing it down.

Overall I'm super happy. Disk I/O can get a bit slow at times (eg. cron time) but I've set up some jitter there to offset it slightly.

There is also a single CPU Opteron 6276 server with 96GB of RAM running as a file server, again with a dual-socket SuperMicro board but it doesn't have a lot of CPU usage, mainly RAM usage for ZFS, but again with 2 SSDs to help with I/O there. Again with multiple NICs and IPMI so I can have, for example, my Steam library being shared on one NIC using Samba inside a Jail, whereas general network I/O (NFS etc) runs on another NIC.

Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International