Jan 02

Good Morning Admin,

so you updated your proxmox installation and rebooted… waited patiently… and it didn’t come back? uh oh?
Hopefully you have some kind of console to see those nice errors..

Loading initial ramdisk ...
oading, please wait...
[ 2.394422] megaraid_sas 0000:02:00.0: INIT adapter done
modprobe: module unknown not found in modules.dep
mount: mounting ZFS=rpool/ROOT/pve-1 on /root failed: No such file or directory
Target filesystem doesn't have requested /sbin/init.
mount: mounting /dev on /root/dev failed: No such file or directory
No init found. Try passing init= bootarg.
modprobe: module ehci-orion not found in modules.dep


BusyBox v1.22.1 (Debian 1:1.22.0-9+deb8u1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/sh: can't access tty; job control turned off

After lots of fuzzing around, booting one old kernel etc. 
it comes all down to broken -pve5 GRUB packages.

Solution (inside initramfs)

modprobe zfs
zpool import -R /rpool -N rpool
zpool mount -a
mount -t proc proc /rpool/proc
mount -t sysfs sys /rpool/sys/
mount -o bind /dev /rpool/dev/
mount devpts /dev/pts -t devpts
chroot /rpool
/etc/init.d/networking start
/etc/init.d/ssh start
#now you can finally get rid of this console (which seems to timeout every freakin 10seconds on those fujitsu boxes) and login via SSH.

now my tool of choice is aptitude, was searching for the package “grub-pc” using the “/” command. Then chose the previous -pve4 version of it and hit “!” a couple times till it said its was going to downgrade 5 packages. You can also hit “e” to examine which packages exactly. After hitting “g” for GO, it actually downgrades everything. I took the package maintainers version of grub config. After a sync & reboot, it booted the new kernel.

You might wanna block further upgrades of grub using “aptitude hold grub-pc”. Note this works only if aptitude is your tool of choice. Using apt-get will not consider those “holds”.

Hopefully this helps some of you.

Oct 17

OK, so you broke your freebsd /var/db/pkg directory and want to recover it from a backup.

backups look like this:
root@websrv:/var/db/pkg # ls -l /var/backups/
total 19190
-rw-r–r– 1 root wheel 1690 Jul 10 2014 aliases.bak
-rw-r–r– 1 root wheel 475 Jul 1 19:30 group.bak
-rw——- 1 root wheel 1937 Sep 7 10:25 master.passwd.bak
-rw——- 1 root wheel 1954 Aug 23 18:34 master.passwd.bak2
-rw-r–r– 1 root wheel 2429640 Oct 16 03:20 pkg.sql.xz
-rw-r–r– 1 root wheel 2429640 Oct 15 03:01 pkg.sql.xz.1
-rw-r–r– 1 root wheel 2429640 Oct 14 03:09 pkg.sql.xz.2
-rw-r–r– 1 root wheel 2429640 Oct 13 03:01 pkg.sql.xz.3
-rw-r–r– 1 root wheel 2429640 Oct 12 03:01 pkg.sql.xz.4
-rw-r–r– 1 root wheel 2429640 Oct 11 04:00 pkg.sql.xz.5
-rw-r–r– 1 root wheel 2429640 Oct 10 03:01 pkg.sql.xz.6
-rw-r–r– 1 root wheel 2429640 Oct 9 03:01 pkg.sql.xz.7

but it’s not working as advertised…

root@websrv:/var/db/pkg # pkg backup -r /var/backups/pkg.sql.xz
Restoring database:
Restoring: 100%
pkg: sqlite error while executing backup step in file backup.c:99: not an error
pkg: sqlite error — (null)

root@websrv:/tmp # pkg backup -r pkg.sql
Restoring database:
Restoring: 100%
pkg: sqlite error while executing backup step in file backup.c:99: not an error
pkg: sqlite error — (null)

always results in an fresh but empty /var/db/local.sqlite..

root@websrv:/var/backups # pkg info

manual fix

root@websrv:/var/db/pkg # pkg install sqlite3
Updating FreeBSD repository catalogue…
FreeBSD repository is up-to-date.
All repositories are up-to-date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
sqlite3: 3.8.11.1_1

root@websrv:/var/backups # cp pkg.sql.xz /tmp
root@websrv:/var/backups # xz -d /tmp/pkg.sql.xz
root@websrv:/var/backups # cd /var/db/pkg
root@websrv:/var/db/pkg # mv local.sqlite local.sqlite.broken
root@websrv:/var/db/pkg # sqlite3 local.sqlite
SQLite version 3.8.11.1 2015-07-29 20:00:57
Enter “.help” for usage hints.
sqlite> .read /tmp/
Display all 1316 possibilities? (y or n)
sqlite> .read /tmp/pkg.sql

tada

root@websrv:/var/backups # pkg info
ap24-mod_mpm_itk-2.4.7_1 This MPM allows you to run each vhost under a separate uid and gid
apache24-2.4.16_1 Version 2.4.x of Apache web server
apr-1.5.2.1.5.4 Apache Portability Library
autoconf-2.69 Automatically configure source code on many Un*x platforms
autoconf-wrapper-20131203 Wrapper script for GNU autoconf
automake-1.15 GNU Standards-compliant Makefile generator
automake-wrapper-20131203 Wrapper script for GNU automake
binutils-2.25.1 GNU binary tools
bison-2.7.1,1 Parser generator from FSF, (mostly) compatible with Yacc
boost-jam-1.55.0 Build tool from the boost.org
boost-libs-1.55.0_9 Free portable C++ libraries (without Boost.Python)
ca_root_nss-3.20 Root certificate bundle from the Mozilla Project
cmake-3.3.1 Cross-platform Makefile generator
cmake-modules-3.3.1 Modules and Templates for CMake
curl-7.44.0 Non-interactive tool to get files from FTP, GOPHER, HTTP(S) servers
db5-5.3.28_2 The Oracle Berkeley DB, revision 5.3
dialog4ports-0.1.5_2 Console Interface to configure ports

Mar 31

OwnCloudIntroduction

This post is about OwnCloud 6.x which seems finally mature enough to replace most of my current workarounds in a single fine solution.

old Workaround

  • Davical as calendar solution, supports CalDAV and uses PHP (comes with a webfrontend) with postgres as backend database. Been a fine solution so far, but looks like the project got somewhat abandoned.. sometimes the homepage wasn’t even reachable anymore for me. I’ve had quite some issues getting it to work properly on iOS devices after iOS 7.x became available. I found some 3rd party workaround, that fixed some database structure. There was also some issue on OSX iCal, whenever you added an item, it didn’t seem to stick. Only after you manually triggered a fresh sync before adding a new item, it seemed to work. This was one of the hassle’s which made me look for an alternative. As for sharing, my wife simply added my account as well to see my appointments.davical permissions
  • Contacts in OSX. I’ve been using the standard iTunes SYNC with iOS devices (iPhone/iPad) to sync my contacts and groups. If you keep your contacts up2date with pictures and stuff, you will also see the picture of the caller right away. This was a feature i wanted to keep. I haven’t found / or searched long enough, for a replacement to make this available remotely without the need to sync. Sharing more or less didn’t happen. Everyone’s keeping their own contacts up2date.
  • 1Password in OSX. I’ve been using 1password for quite some time but i am not a friend of sync solutions such as dropbox. While AgileBits is telling you all the time, your 1password files are secure and it requires billions of years to crack them, i have my doubts. Only because it is not crackable today, it doesn’t mean some major super duper quantum computer can’t crack it tomorrow. Keeping your files to yourself in the first place, sort of avoids this problem. Even though, they can probably hack everything by then anyways. Anyway, 1password suppors WiFi-Sync, so i used that one to sync my stuff between OSX and iOS devices. It’s a little bothersome, since you have to open the applications everywhere, make sure they are on the same wifi network and even open some special submenu on OSX these days.. not very comfortable .. making the sync a real “sunday morning” task instead of having it just happening in the background. By this time my Windows Gaming Box was not involved in the process, since WiFi sync was not available here.
  • Files. AFP/Samba. For simple file sharing, i’ve been using AFP and/or SAMBA. Remotely connect via VPN.
  • Bookmark Management. I don’t like to store my bookmarks on 3rd party pages. I think bookmarks are a very personal thing. Like what type of books you read. Telling a stranger much about you, your personality.. maybe beliefs, religion, policitical views etc. I’d like to decide on my own, with whom i want to share this with. So my bookmark management was the mozilla weave server. You may use theirs or install your own weave server, which i did. The application depends on a python environment, combined with some database backend (mysql, sqllite). Looking into the database structure, everything seems properly encrypted. (only garbled blobs there). This behind an apache SSL Proxy. Unfortunately this works only with firefox, even though with all OS versions of firefox. I haven’t found a proper way to use this on Safari or Chrome without using some 3rd party website again.firefox sync

new ownCloud era

  • OwnCloud is a solution for all problems above.
  • CalDAV Support. In OwnCloud you get a fine calendar solution with support for multiple calendars and the ability to share them to other OwnCloud Users (with fine grained permissions) or public readonly links.
    • Works perfectly on iOS devices, OSX iCal. Simply add the link, OSX will take care about the rest and show you all calendars automatically your User is supposed to see.
  • CardDAV Support. With OwnCloud you can keep track of all your contacts, with extra fields, pictures etc. You can also have multiple adressbooks (for example corporate, private) and can share them to other OwnCloud users. They will automatically see your adressbooks in their contacts webpage.
    • iOS devices work just fine. They will also show you all adressbooks that have been shared to you. Adding / Updating entries, no problem. You can even update the pictures right from your iPhone.
    • Mac OSX 10.8 / 10.9 ..  here comes the downside, but Apple is to blame. It doesn’t work properly here. Only a single adressbook is shown to you no matter how much adressbooks you have or even if you explicitely asked for a special adressbook via URL (eg. /USERNAME/adressbookname). This includes shared ones. I’ve submitted some feedback on the apple webpage, since obviously the code in the iOS versions was already there and just have to be ported back to OSX.
    • Since the Contacts App in OSX didn’t work properly, i’ve switched to Thunderbird. Here you have to install the “Inverse SoGo Connector” extension in order to use CardDAV adressbooks. After that is pretty much just adding a “remote adressbook”, one by one for each adressbook you want to see. Thunderbird will allow you to choose a default adressbook. One downside here i haven’t quite figured out yet, it seems to ask for your owncloud credentials every time you start it, but it doesn’t seem to ask for it if you just wakeup your MacBook. Here it just fails and you have to restart the app. Maybe there is a workaround i just don’t know yet.
  • Filesharing & Sync. OwnCloud provides a small sync application for almost all Operating Systems (OSX; Linux; Windows, iOS, Android) to access the storage part of their suite. The tool on osx/windows side i’ve tested, simply asks for your credentials and server address, lets you pick a local and a remote directory and keeps them in sync. I picked a single extra directory, so it doesn’t download everything. Everything for example, includes all directories people have shared to you. These directories show up in some special “shared” directory in the root of your “filesystem”.owncloud win
  • 1password. With the sync tool from above, you can simply use 1passwords “synchronization with folder” option and pick your OwnCloud local folder. It will sync all items automatically. My Windows 1password installation could pick that newly created 1password file (its actually a directory) right away and use it. Adding an item on Mac 1password triggered a 1password sync, followed by an owncloud sync UP to the owncloud server, followed by an owncloud sync DOWN to the windows installation, making it kind of instantly available to the windows 1password installation. Finally a working sync between Windows and OSX without a 3rd party.1password win
  • bookmark management. Some form of weave service can be installed as plugin into OwnCloud. Switching from weave to OwnCloud worked right away since switching the old service off in firefox, doesn’t delete any bookmarks. So i disabled the old sync and setup a new one. You need to create a new account and choose your owncloud installation. Everything you need, such as the exact URL you can find in the personal area of your OwnCloud. One more thing tho, once you setup everything, you may want to check on the new “recovery key” in firefox and store it somewhere safe (maybe in 1password?).

Since this is a pretty new installation, i cannot say much about stability but for now it makes a good impression on me so far.

Update:

  • If you run your owncloud installation behind a Pound Proxy, you may need to patch that one so it allows the PATCH http method.
Sep 22
dovecot: master: Warning: service(imap-login): process_limit (256) reached, client connections are being dropped

So you found this error message in your server logs and noticed you cannot login anymore via IMAP because all available ‘slots’ have been consumed. My working theory is, it is related to the IOS7 release and it’s IPv6 support.  With privacy extensions enabled ( See RFC 4941 ) it looks like the IOS device is grabbing a new IPv6 address every time it wakes up. This is a perfect fine behaviour if it weren’t for IMAP IDLE. From my understanding, IMAP idle is sort of a long lasting SSL tunnel with a very long TTL. This way your client doesn’t need to be online all the time and only wakes up, if there anything new on the server ( RFC 2177 ).

Workaround: Break IMAP IDLE. Not really, but at least reduce the TTL for the Tunnel by somewhat. Don’t be to harsh, otherwise your’e mobile device is gonna wake up too often, therefore consuming too much power. I set it to 30minutes.

protocol imap {
 #process_limit = 512
 # process_min_avail = 5
 imap_idle_notify_interval = 30 mins
 mail_max_userip_connections = 10
}
#You can check your configuration by using "doveconf -N" before reloading/restarting.

If it works, dovecot will drop the connections to not responsive ‘expired’ IPv6 IPs, freeing resources.

Sep 22 11:04:36 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=273/1229
Sep 22 11:04:37 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=237/1161
Sep 22 11:04:37 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=189/1073
Sep 22 11:04:37 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=222/1132
Sep 22 11:04:37 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=294/1268
Sep 22 11:04:37 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=273/1229
Sep 22 11:04:37 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=222/1211
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=246/1180
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=288/1258
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=252/1190
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=273/1229
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=267/1219
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=288/1258
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=252/1190
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=165/1104
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=210/1112
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=216/1201
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=237/1161
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=189/1073
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=237/1161
Sep 22 11:04:40 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=273/1229
Sep 22 11:04:41 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=216/1122
Sep 22 11:04:41 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=258/1200
Sep 22 11:04:41 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=201/1172
Sep 22 11:04:41 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=252/1190
Sep 22 11:04:41 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=231/1151
Sep 22 11:04:41 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=231/1151
Sep 22 11:04:41 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=252/1190
Sep 22 11:04:41 mx dovecot: imap(email@server): Disconnected: Disconnected in IDLE bytes=189/1073
Tagged with:
Feb 22

Installation completed successfully.

with 8x 3TB drives, 2 SSD
But lets start from the beginning…

Delivery & Hardware System Setup

Installation

FreeBSD Installation with ZFS Setup

  • since i am pretty much a FreeBSD noob, i have been googl’ing a lot and came up with a few things to consider.
    • Sector Alignment – New hard-drives, including SSDs, show 512k sectorsize to the Operating system but use 4k blocks in reality. This matters because, today, where you have multiple layers in between (harddisk, lvm, filesystem, database tablespaces etc.) .. if you screw this up, every single i/o call will multiply, reducing performance dramatically and also puts more wearing on your SSDs. So make sure you have formatting etc. in place to use 4k sectors.
    • there are a few tools on FreeBSD to simulate 4k blocks. Also using encryption allows you to set fixed block sizes.
    • Another thing to consider when partitioning, leave some space between the start of the disk and your first partition. I started at 2megabytes. This helps because its a multiply of 512k and maybe if you need to replace a disk a few years later, you have some room in case the new drive has different sectors as your previous hard-drive.

     

        1. I used the latest production release, FreeBSD 9.1 as DVD image. Burned it onto the disk and used the following pages to guide me through the installation.
        2. my /boot/loader.conf
          zfs_load="YES"
          aesni_load="YES" #Xeon has hardware support for AES which will get used by geli
          geom_eli_load="YES"
          ahci_load="YES"
          vesa_load="YES" # we want a better console
          if_lagg_load="YES" # required if you want to bond multiple devices
          loader_logo="beastie" 
          vfs.root.mountfrom="zfs:zroot"
          geli_ada0p3_keyfile0_load="YES"
          geli_ada1p3_keyfile0_load="YES"
          geli_ada0p3_keyfile0_type="ada0p3:geli_keyfile0"
          geli_ada1p3_keyfile0_type="ada1p3:geli_keyfile0"
          geli_ada0p3_keyfile0_name="/boot/encryption.key"
          geli_ada1p3_keyfile0_name="/boot/encryption.key"
          hw.snd.latency=7
          vm.kmem_size_max="64G"
          vm.kmem_size="48G"
          # zfs tuning i picked up from all the pages everywhere
          vfs.zfs.prefetch_disable="1"
          vfs.zfs.txg.timeout="120" # i believe this is the timer when to actually write to disk. Beware 120seconds loss of data if you dont have a USV.
          vfs.zfs.txg.synctime_ms="500"
          vfs.zfs.arc_max="20G"
          # low level tuning of the vdev device
          vfs.zfs.vdev.min_pending=4
          vfs.zfs.vdev.max_pending=10
          vfs.zfs.vdev.cache.size=64M
          vfs.zfs.vdev.cache.max="65536" 
          vfs.zfs.vdev.cache.bshift="16"
          # Usefull if you are using Intel-Gigabit NIC
          hw.em.rxd=4096
          hw.em.txd=4096
          hw.em.rx_process_limit="-1"
          kern.maxvnodes=250000
          atapicam_load="YES"
        3. my /etc/sysctl.conf
          #basically energy settings, trying to send most of the cores into c-state 3 but keep one core at C2
          dev.cpu.0.cx_lowest=C2
          dev.cpu.1.cx_lowest=C3
          dev.cpu.2.cx_lowest=C3
          dev.cpu.3.cx_lowest=C3
          dev.cpu.4.cx_lowest=C3
          dev.cpu.5.cx_lowest=C3
          dev.cpu.6.cx_lowest=C3
          dev.cpu.7.cx_lowest=C3
          # XEON supports 200Mhz rate :)
          debug.cpufreq.lowest=200
          # Network Tuning, more buffers etc.
          net.inet.tcp.rfc1323=1
          kern.ipc.maxsockbuf=16777216
          net.inet.tcp.sendspace=1048576
          net.inet.tcp.recvspace=1048576
          net.inet.tcp.sendbuf_max=16777216  
          net.inet.tcp.recvbuf_max=16777216
          net.inet.tcp.sendbuf_auto=1
          net.inet.tcp.recvbuf_auto=1
          net.inet.tcp.sendbuf_inc=16384 
          net.inet.tcp.recvbuf_inc=524288
          net.inet.tcp.inflight.enable=0
          # ZFS tuning
          vfs.zfs.prefetch_disable=0
          vfs.zfs.l2arc_write_max=200000000 # ssd can deliver more than 8MB/sec speed, beware the wearing that comes with it
          vfs.zfs.l2arc_write_boost=380000000 # ssd can deliver more than 8MB/sec speed, beware the wearing that comes with it
          vfs.zfs.l2arc_noprefetch=0
  • if there are any BSD guru’s out there reading this, any hints especially in regards to power consumption are very welcome. The system right now with the specs above, shows ~52W power consumption (with 4x3TB WD red installed) which is more than 100W less as my previous server 🙂

Update 24/Feb/2013: I’ve installed my old 3TB seagate barracuda drives now (4x). Alternate says they have a power consumption of 8W each. I am using the 4 remaining SATA 3GB ports on the Supermicro Board, instead of the 4 remaining 6GB ports on the LSI card. Despite only half the speed (not that they would reach that anyway), they were properly detected in FreeBSD, which means i could send them IDLE/STANDBY commands via camcontrol. So i created a zpool called “FILES ARCHIVE” with those disk and set them to go into standby mode as soon as possible. With this setting, only the 4 WD red drives and SSDs are active, the system consumes ONLY 56watts. Considering my old system used more like 160watts, i am pretty happy with the result 🙂

Feb 10

iTunes Backup greyed out

This is one of the posts that are basically for my increasing digital dementia 🙂 I’ve been wondering what i broke in my iTunes as it was no longer able to create any backups, which i need to use from time to time, jailbreaking the devices. The button in my german iTunes says “Jetzt sichern” but is greyed out. Also the rightclick menu on the device itself does not show any backup function anymore.

I started wondering if it was due to my Libary being on a different server, so the first thing was to create an entirely new library locally, with no effect.
iTunes Backup greyed out

The solution was pretty simple. I google’d around what could have disabled this function instead of continuing to search for a fix to enable it.. and here we go.. Looks like i disabled the function a few month ago. Fortunately the same link that helped me disable backup also shows how to re-enable it 🙂

 

ENABLE backups
defaults delete com.apple.iTunes DeviceBackupsDisabled
DISABLE backups
defaults write com.apple.iTunes DeviceBackupsDisabled -bool YES

Thanks to OSXDaily for the hint 🙂

 

Tagged with:
Dec 30

Mission Goal

So whats this all about… I’ve been running a homeserver for a few years now, using different kinds of linux distro’s. The server usually more or less built from leftovers of previous workstations. The old system also includes 2x 3slot SATA harddisk enclosures, connected to a promise raid controller. The system was a 2.6Ghz core2duo with 4GB memory and 2x 250GB system drives. The system was storing all our data, videos, photos via Samba/Netatalk and took backups (bacula) of my various servers on the net. It was also running a nagios monitoring installation. Everything was encrypted with the usual linux LUKS stuff.

PC V2120

So why a new system?

  • Energy consumption too high. As you may know, energy isn’t as cheap anymore. With the recent changes in Germany and some not so clever decisions made by our fearless leaders, the prices for energy skyrocketed and the pressure to built intelligent less power consuming systems keeps on growing. While i totally agree on shutting down nuclear power plants, i don’t like the way how renewable energy is subsidized and how big energy companies profit by putting the burden on the customer. Actually i think, core infrastructure such as energy, water, public transport etc. should be handled by the government or non-profit organizations, rather than companies.
  • Next generation filesystem. Everyone’s generating lots of data these days. Documents, Scans, Photos and Videos of your family, your music etc. Maybe you already convinced your family to create backups, do this extra step and bought them a time capsule or similar. You have a raid in your homeserver and you are aware of the fact, harddrives wear out over time..  Now considering, you store your digital life for centuries and do everything right, keep replacing disks and so on.. collecting terrabytes of data over the years.. you won’t even notice if any of your data is damaged until it is too late. Here comes ZFS to the rescue, by taking care of the integrity of your files. By generating checksums for all your files and keeping multiple copies of it. It will detect broken files and replace them automatically with the working copy. A nice side effect for ZFS, it includes also compression and deduplication. On Solaris it even includes an encryption layer.

 

Requirements

  1. Data integrity. The ZFS Filesystem is said to require 1GB memory per TB stored, especially if you want to use deduplication, since this feature will hold big hashtables in memory. Afaik, only the Oracle Solaris 11 supports the builtin ZFS encryption layer. But Solaris is a commercial system, requiring you to have some support contract if you use it in production. While this isn’t necessarily a problem for a homeserver, the supported hardware list for solaris is very short. So i will try to stick with FreeBSD and take the performance hit it comes with by using GELI encryption below the ZFS layer. Since i will be using SSD drives for the main OS, i try to get some performance back by adding some SSD caching to the ZFS. I would also love to use ECC memory.
  2. diskspace should be at least 6-8TB usable space, with room for additional drives
  3. As for energy, i am trying to get the most performance at the lowest energy. Performance is required for encryption, compression, occasional realtime transcoding. Depending on how it works out, everything should spin down or step down if it is idle.
  4. Remote Access
  5. Low Noise

Hardware so far…

  • Intel Server Board S1200BTLR – Sockel1155, SandyBridge, 2xGbit, 2x SATA3, 4x SATA2, onboard GFX, up to 4x8GB ECC DDR3 Memory, IPMI 2.0 ~200EUR
  • Intel Xeon E3-1265LV2 – TDP 45W, QuadCore 2.5Ghz, IvyBridge, AES-NI, HT, Turbo Boost, VT flags ~300EUR (wikipedia)
  • Adaptec 7805 – 8x Port SAS/SATA Raid Controller, lowprofile, 1GB cache, support for FreeBSD, support for drives with 4TB size, up to 256 drives using expander. ~500EUR
  • cheaper “dumb” controller LSI SAS 9207-8i for ~200EUR even got Solaris Support.. and FreeBSD.
  • Lian Li PC V2120 – beautiful, low noise chassis with plenty of room for more disks ~400EUR
  • Western Digital Red Harddrive – 3TB SATA3, low 24dB Noise, 24/7 support, Low power: 4,4W read/write, <1W sleep/standby power consumption ~140EUR

 

  • Update: I found a mainboard with USB3 and ECC. Have a look at Supermicro X9SAE.
  • Update: Order is out to 3 different stores. I took the Supermicro X9SAE-V which allows two x8 PCIe3.0 instead of a single x16 PCIe3.0 .. considering i may need more disks some day and a 2nd LSI SAS 9207-8i Controller. Rest as expected, 3x 3TB WD red drives, Xeon E3-1265LV2 but a slight change for the chassis, its a Lian Li PC-P80N now, since its just standing in some dark corner anyway, so no need to look beautiful.. but not getting my hands bloody on the installation and having 10 3,5″ harddrive slots as well as 2 2,5″ slots for 2 ssd drives, is nice. I will be using two Sandisk G25-G3 Extreme SSD 2,5″ 120GB for the system itself and as cache for the ZFS. The reason for the sandisk was it is pretty fast Sata3(6Gbit), 550MB read/510MB write, a good MTBF of 2.500.000 hours and as expected a low 0,6W read/write energy consumption. For the operating system, i will see how a solaris 11.1 installation will work out. If it fails to support anything or does not fully use powersaving features, i may still consider FreeBSD on the 2nd run 🙂
  • Latest Update: No free updates, no solaris.

 

This post will be updated as they come in.

Tagged with:
Feb 23

Another fancy error message that took me some time to figure out. I haven’t even found the entire reason yet, but found a workaround.
The error is thrown by accessing the repository catalog in the Contao Backend.

Ok, since i use a Gentoo Installation, that is always up2date, but sometimes runs into some dependency problems i wasn’t sure if the SOAP part of PHP has some generic error. First try was to search for the WSDL path in the files.

I found it only once, contaoroot/system/config/config.php :
$GLOBALS[‘TL_CONFIG’][‘repository_wsdl’]      = ‘http://www.contao.org/services/repository.wsdl’;
I downloaded the file, placed it locally and changed the value. Worked, so the SOAP part of PHP is working.

After some more fiddling around, checking with tcpdump, it doesn’t even try to connect to the outside world, so i’m assuming i have some other network/naming related issues since this stuff runs on a VM with a RFC1918 private IP but with access to the internet.

Anyway, the better workaround was to use my proxy by adding the following to the config:
$GLOBALS[‘TL_CONFIG’][‘useProxy’] = true;
$GLOBALS[‘TL_CONFIG’][‘proxy_url’] = ‘http://172.20.20.11:3128’;

Original Error Message:
[22-Feb-2012 21:35:50 UTC] PHP Fatal error: Uncaught exception ‘SoapFault’ with message ‘SOAP-ERROR: Parsing WSDL: Couldn’t load from ‘http://www.contao.org/services/repository.wsdl’ : Start tag expected, ‘<‘ not found
‘ thrown in /var/www/xxxxx.de/system/modules/rep_client/RepositoryBackendModule.php on line 139

 

 

Mar 13

Ping Logo
I am using a Squid Proxy in my home network to filter outbound access. The WLAN configuration in my iPhone is using this proxy too, but since there is no ‘exclude from proxy’ option there, everything on any ports gets thrown at this Squid Proxy. The Ping Tool is also using a specific port to receive it’s messages. I also sent a mail to the developer, telling him to consider a more friendly port (80,443) for his app so people could use this app also behind firewalls, proxies which you have … at least in a corporate environments.

No answer so far. So here is the solution, at least to get it to work if you have access to your Squid Proxy.


acl http proto http
acl iphone_push dstdomain push.emailmii.net
acl iphone_push_port port 1919
always_direct allow iphone_push
http_access allow CONNECT iphone_push_port iphone_push
http_access allow http iphone_push_port iphone_push

make sure its BEFORE those entries:

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny all

 

Update: This is obsolete by now. I think they realized to use standard ports.

Tagged with:
Mar 23
  • Update 27 December 09 – minor changes due to new blog system
  • Update 01 May 09 Referring to the “ffmpegthumbnailer” .. it is not a script. It is a real tool which can be found in gentoo portage or here 🙂
  • Update 09 December 08 As there still people coming to this page, i posted my recent WORKING settings to allow for mkv playback as well as the missing bash script.
  • Update 15 Oktober 08 mediatomb.cc got a nice wiki which has alot more details howto setup different kinds of transcodings. Please check them out.
  • Update 28 July 08 Sony Firmware 2.35 finally fixed that problem.

It took a while, but the software (mediatomb) as well as the client (ps3) are finally able to talk to each other. It works almost out of the box with the recent 0.11.0 release for xvid and divx playback on the PS3.

Mediatomb has now the ability to transcode in realtime, allowing you to play other files like matroska .mkv files as well. Mediatomb can forward the file to play to external applications based on mimetype.

A fine howto to get this stuff going under Gentoo can be found That includes a few external scripts for this transcoding stuff. I am using the ffmpeg version (video) for matroska files.

my config.xml looks like this:>

<?xml version=”1.0″ encoding=”UTF-8″?>

<config version=”1″ xmlns=”http://mediatomb.cc/config/1″ xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://mediatomb.cc/config/1 http://mediatomb.cc/config/1.xsd”>
<server>
<ui enabled=”yes”>
<accounts enabled=”no” session-timeout=”30″>
<account user=”mediatomb” password=”mediatomb”/>
</accounts>
</ui>
<name>MediaTomb</name>
<udn>uuid:eef9e7bf-b63f-4248-8add-a33086ac49d8</udn>
<home>/etc/mediatomb</home>
<webroot>/usr/local/share/mediatomb/web</webroot>
<storage>
<sqlite3 enabled=”no”>
<database-file>mediatomb.db</database-file>
</sqlite3>
<mysql enabled=”yes”>
<host>localhost</host>
<username>mediatomb</username>
<database>mediatomb</database>
<password>YOURPASSWORDHERE</password>
</mysql>
</storage>
<protocolInfo extend=”yes”/><!– For PS3 support change to “yes” –>
<!–
Uncomment the lines below to get rid of jerky avi playback on the
DSM320 or to enable subtitles support on the DSM units
–>
<!–
<custom-http-headers>
<add header=”X-User-Agent: redsonic”/>
</custom-http-headers>
<manufacturerURL>redsonic.com</manufacturerURL>
<modelNumber>105</modelNumber>
–>
<!– Uncomment the line below if you have a Telegent TG100 –>
<!–
<upnp-string-limit>101</upnp-string-limit>
–>
</server>
<import hidden-files=”no”>
<scripting script-charset=”UTF-8″>
<virtual-layout type=”builtin”/>
</scripting>
<mappings>
<extension-mimetype ignore-unknown=”no”>
<map from=”mp3″ to=”audio/mpeg”/>
<map from=”ogg” to=”application/ogg”/>
<map from=”asf” to=”video/x-ms-asf”/>
<map from=”asx” to=”video/x-ms-asf”/>
<map from=”wma” to=”audio/x-ms-wma”/>
<map from=”wax” to=”audio/x-ms-wax”/>
<map from=”wmv” to=”video/x-ms-wmv”/>
<map from=”wvx” to=”video/x-ms-wvx”/>
<map from=”wm” to=”video/x-ms-wm”/>
<map from=”wmx” to=”video/x-ms-wmx”/>
<map from=”m3u” to=”audio/x-mpegurl”/>
<map from=”pls” to=”audio/x-scpls”/>
<map from=”flv” to=”video/x-flv”/>
<!– Uncomment the line below for PS3 divx support –>
<map from=”avi” to=”video/x-divx”/>
<map from=”divx” to=”video/x-divx”/>
<map from=”mkv” to=”video/x-matroska”/>
<map from=”mov” to=”video/quicktime”/>
<map from=”qt” to=”video/quicktime”/>
<map from=”mpg” to=”video/mpeg”/>
<map from=”mpeg” to=”video/mpeg”/>
<!– Uncomment the line below for D-Link DSM / ZyXEL DMA-1000 –>
<!– <map from=”avi” to=”video/avi”/> –>
</extension-mimetype>
<mimetype-upnpclass>
<map from=”audio/*” to=”object.item.audioItem.musicTrack”/>
<map from=”video/*” to=”object.item.videoItem”/>
<map from=”image/*” to=”object.item.imageItem”/>
</mimetype-upnpclass>
<mimetype-contenttype>
<treat mimetype=”audio/mpeg” as=”mp3″/>
<treat mimetype=”application/ogg” as=”ogg”/>
<treat mimetype=”audio/x-flac” as=”flac”/>
<treat mimetype=”image/jpeg” as=”jpg”/>
<treat mimetype=”audio/x-mpegurl” as=”playlist”/>
<treat mimetype=”audio/x-scpls” as=”playlist”/>
<treat mimetype=”audio/x-wav” as=”pcm”/>
<treat mimetype=”audio/L16″ as=”pcm”/>
<treat mimetype=”video/x-msvideo” as=”avi”/>
<treat mimetype=”video/mp4″ as=”mp4″/>
<treat mimetype=”audio/mp4″ as=”mp4″/>
<treat mimetype=”video/x-divx” as=”avi”/>
</mimetype-contenttype>
</mappings>
</import>
<transcoding enabled=”yes”>
<mimetype-profile-mappings>
<transcode mimetype=”video/x-flv” using=”vlcmpeg”/>
<transcode mimetype=”application/ogg” using=”vlcmpeg”/>
<transcode mimetype=”application/ogg” using=”oggflac2raw”/>
<transcode mimetype=”audio/x-flac” using=”audio-common”/>
<transcode mimetype=”video/x-divx” using=”video-common”/>
<transcode mimetype=”video/x-matroska” using=”video-common”/>
<transcode mimetype=”video/quicktime” using=”video-common”/>
</mimetype-profile-mappings>
<profiles>
<profile name=”oggflac2raw” enabled=”yes” type=”external”>
<mimetype>audio/L16</mimetype>
<accept-url>no</accept-url>
<first-resource>yes</first-resource>
<accept-ogg-theora>no</accept-ogg-theora>
<agent command=”ogg123″ arguments=”-d raw -f %out %in”/>
<buffer size=”1048576″ chunk-size=”131072″ fill-size=”262144″/>
</profile>
<profile name=”vlcmpeg” enabled=”yes” type=”external”>
<mimetype>video/mpeg</mimetype>
<accept-url>yes</accept-url>
<first-resource>yes</first-resource>
<accept-ogg-theora>yes</accept-ogg-theora>
<agent command=”vlc” arguments=”-I dummy %in –sout #transcode{venc=ffmpeg,vcodec=mp2v,vb=4096,fps=25,aenc=ffmpeg,acodec=mpga,ab=192,samplerate=44100,channels=2}:standard{access=file,mux=ps,dst=%out} vlc:quit”/>
<buffer size=”14400000″ chunk-size=”512000″ fill-size=”120000″/>
</profile>
<profile name=”audio-common” enabled=”yes” type=”external”>
<mimetype>audio/x-wav</mimetype>
<accept-url>yes</accept-url>
<first-resource>yes</first-resource>
<accept-ogg-theora>no</accept-ogg-theora>
<agent command=”mediatomb-transcode-audio” arguments=”%in %out”/>
<buffer size=”1048576″ chunk-size=”131072″ fill-size=”262144″/>
</profile>
<profile name=”video-common” enabled=”yes” type=”external”>
<mimetype>video/mpeg</mimetype>
<accept-url>yes</accept-url>
<first-resource>yes</first-resource>
<accept-ogg-theora>yes</accept-ogg-theora>
<agent command=”mediatomb-transcode-video-ffmpeg” arguments=”%in %out”/>
<buffer size=”14400000″ chunk-size=”512000″ fill-size=”120000″/>
</profile>
</profiles>
</transcoding>
</config>


# cat /usr/bin/mediatomb-transcode-video-ffmpeg
INPUT="$1"
OUTPUT="$2"
VIDEO_CODEC="mpeg2video"
VIDEO_BITRATE="4096k"
AUDIO_CODEC="mp2"
AUDIO_BITRATE="192k"
AUDIO_SAMPLERATE="48000"
AUDIO_CHANNELS="2"
FORMAT="dvd"
exec /usr/bin/ffmpeg -threads 2 -i "${INPUT}" -vcodec ${VIDEO_CODEC} -b ${VIDEO_BITRATE} \
-acodec ${AUDIO_CODEC} -ab ${AUDIO_BITRATE} -ar ${AUDIO_SAMPLERATE} -ac ${AUDIO_CHANNELS} \
-f ${FORMAT} - > "${OUTPUT}" #2>/dev/null

Hope that helps you guys out there 🙂

Tagged with:
preload preload preload