Feb 22

Installation completed successfully.

with 8x 3TB drives, 2 SSD
But lets start from the beginning…

Delivery & Hardware System Setup

Installation

FreeBSD Installation with ZFS Setup

  • since i am pretty much a FreeBSD noob, i have been googl’ing a lot and came up with a few things to consider.
    • Sector Alignment – New hard-drives, including SSDs, show 512k sectorsize to the Operating system but use 4k blocks in reality. This matters because, today, where you have multiple layers in between (harddisk, lvm, filesystem, database tablespaces etc.) .. if you screw this up, every single i/o call will multiply, reducing performance dramatically and also puts more wearing on your SSDs. So make sure you have formatting etc. in place to use 4k sectors.
    • there are a few tools on FreeBSD to simulate 4k blocks. Also using encryption allows you to set fixed block sizes.
    • Another thing to consider when partitioning, leave some space between the start of the disk and your first partition. I started at 2megabytes. This helps because its a multiply of 512k and maybe if you need to replace a disk a few years later, you have some room in case the new drive has different sectors as your previous hard-drive.

     

        1. I used the latest production release, FreeBSD 9.1 as DVD image. Burned it onto the disk and used the following pages to guide me through the installation.
        2. my /boot/loader.conf
          zfs_load="YES"
          aesni_load="YES" #Xeon has hardware support for AES which will get used by geli
          geom_eli_load="YES"
          ahci_load="YES"
          vesa_load="YES" # we want a better console
          if_lagg_load="YES" # required if you want to bond multiple devices
          loader_logo="beastie" 
          vfs.root.mountfrom="zfs:zroot"
          geli_ada0p3_keyfile0_load="YES"
          geli_ada1p3_keyfile0_load="YES"
          geli_ada0p3_keyfile0_type="ada0p3:geli_keyfile0"
          geli_ada1p3_keyfile0_type="ada1p3:geli_keyfile0"
          geli_ada0p3_keyfile0_name="/boot/encryption.key"
          geli_ada1p3_keyfile0_name="/boot/encryption.key"
          hw.snd.latency=7
          vm.kmem_size_max="64G"
          vm.kmem_size="48G"
          # zfs tuning i picked up from all the pages everywhere
          vfs.zfs.prefetch_disable="1"
          vfs.zfs.txg.timeout="120" # i believe this is the timer when to actually write to disk. Beware 120seconds loss of data if you dont have a USV.
          vfs.zfs.txg.synctime_ms="500"
          vfs.zfs.arc_max="20G"
          # low level tuning of the vdev device
          vfs.zfs.vdev.min_pending=4
          vfs.zfs.vdev.max_pending=10
          vfs.zfs.vdev.cache.size=64M
          vfs.zfs.vdev.cache.max="65536" 
          vfs.zfs.vdev.cache.bshift="16"
          # Usefull if you are using Intel-Gigabit NIC
          hw.em.rxd=4096
          hw.em.txd=4096
          hw.em.rx_process_limit="-1"
          kern.maxvnodes=250000
          atapicam_load="YES"
        3. my /etc/sysctl.conf
          #basically energy settings, trying to send most of the cores into c-state 3 but keep one core at C2
          dev.cpu.0.cx_lowest=C2
          dev.cpu.1.cx_lowest=C3
          dev.cpu.2.cx_lowest=C3
          dev.cpu.3.cx_lowest=C3
          dev.cpu.4.cx_lowest=C3
          dev.cpu.5.cx_lowest=C3
          dev.cpu.6.cx_lowest=C3
          dev.cpu.7.cx_lowest=C3
          # XEON supports 200Mhz rate :)
          debug.cpufreq.lowest=200
          # Network Tuning, more buffers etc.
          net.inet.tcp.rfc1323=1
          kern.ipc.maxsockbuf=16777216
          net.inet.tcp.sendspace=1048576
          net.inet.tcp.recvspace=1048576
          net.inet.tcp.sendbuf_max=16777216  
          net.inet.tcp.recvbuf_max=16777216
          net.inet.tcp.sendbuf_auto=1
          net.inet.tcp.recvbuf_auto=1
          net.inet.tcp.sendbuf_inc=16384 
          net.inet.tcp.recvbuf_inc=524288
          net.inet.tcp.inflight.enable=0
          # ZFS tuning
          vfs.zfs.prefetch_disable=0
          vfs.zfs.l2arc_write_max=200000000 # ssd can deliver more than 8MB/sec speed, beware the wearing that comes with it
          vfs.zfs.l2arc_write_boost=380000000 # ssd can deliver more than 8MB/sec speed, beware the wearing that comes with it
          vfs.zfs.l2arc_noprefetch=0
  • if there are any BSD guru’s out there reading this, any hints especially in regards to power consumption are very welcome. The system right now with the specs above, shows ~52W power consumption (with 4x3TB WD red installed) which is more than 100W less as my previous server 🙂

Update 24/Feb/2013: I’ve installed my old 3TB seagate barracuda drives now (4x). Alternate says they have a power consumption of 8W each. I am using the 4 remaining SATA 3GB ports on the Supermicro Board, instead of the 4 remaining 6GB ports on the LSI card. Despite only half the speed (not that they would reach that anyway), they were properly detected in FreeBSD, which means i could send them IDLE/STANDBY commands via camcontrol. So i created a zpool called “FILES ARCHIVE” with those disk and set them to go into standby mode as soon as possible. With this setting, only the 4 WD red drives and SSDs are active, the system consumes ONLY 56watts. Considering my old system used more like 160watts, i am pretty happy with the result 🙂

Leave a Reply

You must be logged in to post a comment.

preload preload preload