ZFS root with gmirror

One of the people who influenced me to with BSD and specifically with FreeBSD is Jimmy B. Lim. I also met him online way way back in the 90's with Team FDN's core group Error 404. His words of challenge always echoes in my mind whenever I am faced with despair in front of my b0x... "Mag windows ka na lang!" (Just use Windows!) haha!

This approach with ZFS and mirroring has raised a question in my head with my first glance on his post from the mailing list. Why use gmirror when you can already use ZFS to mirror hard disk drives? I asked him this during one of our YM sessions and his answer is quite simple. This approach is much cleaner and with the least administrative effort. You don't need to manually edit your fstab and do other uncessary actions with your box :)

Let's get our hands dirty and configure our gmirror root[0]!



1. Install 2 identical hard drive on your box
2. Install FreeBSD-7.1
3. Partition your hard drive:
ad0s1 6GB (for 2GB RAM and this will be change if you want more
swap if your RAM is more than 4GB)
ad0s2
4. Make ad0s1a 2GB (just enough for bootdir, I know this is kinda big,
but it is much better to have spare than to have few space)
5. Make ad0s1b 4GB (twice the size of 2GB RAM)
6. Make ad0s2d as your ZFS partition
7. Partition your second drive (don't for get to make it bootable of
course) according the first drive. e.g. ad1)
8. Install FreeBSD-7.1
9. Configure gmirror root[0]
a. Setup gmirror on the second drive
gmirror label -v -n -b round-robin gm0 /dev/ad1s1
gmirror load
b. Format the gmirror partition and mount to /mnt
newfs -U /dev/mirror/gm0a
mount /dev/mirror/gm0a /mnt
c. Copy the / to newly created gmirror
dump -L -0 -f- / | (cd /mnt && restore -r -v -f-)
d. Setup the fstab for gmirror
vi /mnt/etc/fstab
# Device Mountpoint FStype Options
Dump Pass#
/dev/mirror/gm0b none swap sw
0 0
/dev/mirror/gm0a / ufs rw
1 1
e. Force to boot to the second drive and load gmirror
echo 'geom_mirror_load="YES"' >>/mnt/boot/loader.conf
echo ad1 | sed -e
's;^\([^0-9]*\)\([0-9][0-9]*\)$;1:\1(\2,a)/boot/loader;' \
>/boot.config
f. Reboot and cross your fingers
g. Once everything goes as we expected, insert the first drive on
the gmirror
gmirror configure -a gm0
gmirror insert -p 1 gm0 /dev/ad0s1
h. From this point the gmirror is now rebuilding the first drive
gmirror status
i. Once the rebuild is finished, reboot the server
10. Set up ZFS root[1]
mount -w /
zpool create tank /dev/ad0s2d
zfs set mountpoint=none tank
zfs create tank/root
zfs create tank/usr
zfs create tank/var
zfs create tank/tmp
zfs create tank/home
zfs set mountpoint=/tank tank/root
zfs set mountpoint=/tank/usr tank/usr
zfs set mountpoint=/tank/var tank/var
zfs set mountpoint=/tank/tmp tank/tmp
zfs set mountpoint=/tank/home tank/home

11. Check zfs settings
df -h
zfs list

12. Enable ZFS on boot
echo 'zfs_enable="YES"' >> /etc/rc.conf

13. Copy / to zfs root

13. Copy / to ZFS root
find -x / | cpio -pmd /tank

14. Remove /tank/boot to load gmirror boot
rm -rf /tank/boot
mkdir /tank/bootdir
cd /tank
ln -s bootdir/boot boot

15. Load ZFS on boot time
echo 'zfs_load="YES"' >> /boot/loader.conf
echo 'vfs.root.mountfrom="zfs:tank/root"' >> /boot/loader.conf

16. Edit fstab
/dev/mirror/gm0a /bootdir ufs
rw 1 1

17. Setup the mounting points of ZFS
zfs set mountpoint=/tmp tank/tmp
zfs set mountpoint=/usr tank/usr
zfs set mountpoint=/var tank/var
zfs set mountpoint=/home tank/home

18. Set the mountpoint legacy
cd /
zfs set mountpoint=legacy tank/root

19. Reboot

20. If everything goes as we expected, chech the ZFS root
df -h

21. Setup the secondary drive to mirror ZFS
zpool attach tank ad0s2d ad1s2d
zpool status

Thanks Jim!

"Now What?"

Mirroring a ZFS Root on FreeBSD

You guys have to check on FreeBSD Installation and ZFS first if you haven't configured your FreeBSD b0x's root on ZFS as this article is just a continuation from that post ;)

Now let's get our hands on having a mirror or RAID-1 with ZFS and FreeBSD.

• Well first off, we have to install the hard disk to your system and boot up
• Login as root
• Type in sysinstall
• Select Custom Installation
• Partition the Hard Disk (Use A = Use Entire Disk and Press Q = Quit)
• Label the Hard Disk (Use A = Auto Defaults)
• Select the /usr, /var, /tmp Mount Points one at a time and press D (Delete)
• Leave only the / and the swap partitions
• Press C to create a new partition and then type /anything as the Mount Point
• Select the newly created partition and press M (Mount pt) to clear the field by deleting /anything and then press on W to write on the disk and Q to quit
• Press on
  
• Exit from sysinstall

And now we could check out initially what we have:

#zpool status

Then we add the 2nd drive to our mirror (our second drive comes up as ad6)

# zpool attach tank ad4s1d ad6s1d
# zpool status

It will now show the two hard disks on a mirror

pool: tank
state: ONLINE
scrub: none requested


config:

     NAME        STATE     READ WRITE CKSUM
     tank        ONLINE       0     0     0
       mirror    ONLINE       0     0     0
         ad4s1d  ONLINE       0     0     0
         ad6s1d  ONLINE       0     0     0

errors: No known data errors

Seeing that we have a new mirror, we now copy the contents from our UFS slice to the second drive.

# newfs /dev/ad6s1a
# mkdir /mnt/bootdir
# mount /dev/ad6s1a /mnt/bootdir
# find -x /bootdir | cpio -pmd /mnt

Now supposed that we have a failure on the 1st HDD, we have configured the 2nd HDD to be ready.

We also have to stop fsck during the bootup process should the first HDD fail (in this case ad4). Edit /etc/fstab so our UFS slice is mounted in the right location for when ZFS boots.

We also NEED to make sure that the /bootdir/zfs/zpool.cache file is the same on both HDDs and edit /etc/fstab as:

/dev/ad4s1a  /bootdir       ufs      rw      1       0
/dev/ad6s1a  /mnt/bootdir   ufs      rw      1       0

And that's it!


"Now What?"

FreeBSD Installation and ZFS

Its very important that you learn something regardless whether its everyday, every week, every month or at anytime. In my case I learned something new from the alphabet. Having to use the letters C, D, and M apart from A makes me feel good that I am moving forward. Although I'm still using Sysinstall for my FreeBSD Installation. The important thing is I learned to use these letters on my FreeBSD 7.1 Installation sitting on a ZFS root.

I am using an AMD Phenom Quad-Core Processor with 2GB of memory b0x under FreeBSD 7.1 AMD64 over this installation process. And then I followed the instructions that was given here http://www.ish.com.au/solutions/articles/freebsdzfs. I modified some intructions from here because of the problems that I encountered and because of the fact that I'm an idiot.

ZFS on FreeBSD Installation

The installer does not support ZFS natively so we'd have to make our hands dirty.

• Boot up on the FreeBSD 7.1 CD Installer
• Choose the Country and Locale
• Choose Custom Installation
• Partition the Hard Disk (Use A = Use Entire Disk and Press Q = Quit)
• Select Standard (Install a standard MBR)
• Label the Hard Disk (Use A = Auto Defaults)
• Select the /usr, /var, /tmp Mount Points one at a time and press D (Delete)
• Leave only the / and the swap partitions
  
• Press C to create a new partition and then type /anything as the Mount Point
• Select the newly created partition and press M (Mount pt) to clear the field by deleting /anything and then press Q to quit
• Select on Distribution and choose Minimal Install
• Select Media and choose CD/DVD
• Select on Commit to install the Operating System

Create the ZFS Pool

• After the Installation has been completed, remove the CD installer and reboot
• Boot on the FreeBSD partition and press 4 to boot in Single User Mode
• Accept the /bin/sh shell by hitting the ENTER key

I am currently using a SATA device at ad4. Follow the instructions below:

# mount -w /
# zpool create tank /dev/ad4s1d
# zfs set mountpoint=none tank
# zfs create tank/root
# zfs create tank/usr
# zfs create tank/var
# zfs set mountpoint=/tank tank/root
# zfs set mountpoint=/tank/usr tank/usr
# zfs set mountpoint=/tank/var tank/var
# zfs set mountpoint=/tank/tmp tank/tmp
# zfs create tank/tmp

And to see what we've done so far

#df -h

And...

#zfs list

Now to enable ZFS on boot...

# echo 'zfs_enable="YES"' >> /etc/rc.conf

And to give us a workable FreeBSD installation under ZFS...

# find -x / | cpio -pmd /tank 

Solving the FreeBSD Boot Problem

# rm -rf /tank/boot
# mkdir /tank/bootdir
# cd /tank
# ln -s bootdir/boot boot
# echo 'zfs_load="YES"' >>  /boot/loader.conf
# echo 'vfs.root.mountfrom="zfs:tank/root"'  >>  /boot/loader.conf

Next is editing our /etc/fstab to:

/dev/ad4s1a  /bootdir        ufs     rw      1       1

Setting the true mountpoints:

# zfs set mountpoint=/tmp tank/tmp
# zfs set mountpoint=/usr tank/usr
# zfs set mountpoint=/var tank/var

And for the legacy:

# cd /
# zfs set mountpoint=legacy tank/root

Reboot and we're ready to go!

# df -h

"Now What?"

FreeBSD Tuning and ZFS

INTRODUCTION

Norbert Copones (http://www.feu-nrmf.ph/norbert/weblog/) have been a very BAD influence for me since puberty. One of the many things he influenced me with is the Z File System or ZFS. I tried Open Solaris 2008.11 first but I was bored to its performance most specially with the gnome desktop. How I wish I could take it off and just have a very minimal installation -- just like Norbert did. Now, since my first love have always been FreeBSD, I plan to walk further on this path with ZFS.

The ZFS was developed by Sun™ that uses a pooled storage method. There's really a lot of features that makes this file system the most advanced file system. But unfortunately, I don't really understand them that much. So as of this current writing, below are the things I do understand, and how I understand it (corrections are VERY welcome).

• Redundancy - still is available when something bad happens
• Mirroring - much like using RAID1 on ZFS or having a hard disk/s with an exact copy to another disk and keeping the process running eventhough 1 or more (but not all) disk fails.
• Snapshots - Keeping a state of the hard disk at a given time.
• Rollback - A recovery process from a Snapshot.

FreeBSD 7.1 CONFIGURATION AND TUNING FOR ZFS

I used an AMD Phenom Quad-Core Processor with 2GB of memory b0x under FreeBSD 7.1 AMD64. I started on reading from the FBSD Handbook (http://www.freebsd.org/doc/en/books/handbook/filesystems-zfs.html). I also used this reference to modify my b0x http://wiki.freebsd.org/ZFSTuningGuide.

I started having this configuration added to my FBSD b0x by editing the /boot/loader.conf

vm.kmem_size_max="1024"
vm.kmem_size="1024"
vfs.zfs.arc_max="100M"

I also added a line to my rc.conf file by typing:

Then, I restarted my b0x and unfortunately, the result was a kernel panic :(

Afterwhich, I chose Escape to loader prompt from the FBSD boot menu and used unset to remove the above variables (ex. unset vm.kmem_size_max) and executed boot.

I started reviewing my configuration and realized that the reason for the panic was simply me being stupid. I forgot to put "M" to specify the file size. And so I edited the /boot/loader.conf file and put the values below:

vm.kmem_size_max="1024M"
vm.kmem_size="1024M"
vfs.zfs.arc_max="100M"

I rebooted my b0x and typed in

# zpool list
no pools available

Yeah! It seems to be working just fine now.

Edit: I ended up with the config below for my server

zfs_load="YES"
vfs.root.mountfrom="zfs:zfbsd/root"

vm.kmem_size_max="1024M"
vm.kmem_size="1024M"
vfs.zfs.zil_disable=1
vfs.zfs.arc_max="100M"
vfs.zfs.vdev.cache.size="5M"

"Now what?"