1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
|
GNU/Linux Debian Wheezy Server Bringup w/ ZFS
2013 May 30th
Sven Gothel
JogAmp Community
Creative Commons Attribution 3.0 License. http://creativecommons.org/licenses/by/3.0/
+++
Small Pool:
1x ZPool
1x VDEV 2x n TB raidz1 ( 1 + 1 )
or better
1x Mirror
1x VDEV 2x n TB mirror ( 1 + 1 )
Big Pool:
1x ZPool
2x VDEV 3x n TB raidz1 ( 2 + 1 )
+++
Tuning:
- no dedup (huge memory consumption / performance)
- no compression (cheap .. but why ?)
- max ARC cache .. 50/50 ?
- http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Limiting_the_ARC_Cache
arc_c_min= (16MB?)
arc_c_max= 17179869184 (16GB) (50% of 32GB)
arc.c = arc.c_max
arc.p = arc.c / 2
or
# ARC: 0 - 12 GB
echo options zfs zfs_arc_max=12884901888 >> /etc/modprobe.d/zfs.conf
echo zfs zfs_arc_max=12884901888 >> /etc/initramfs-tools/modules
17179869184 # 0x400000000 (16 GB)
12884901888 # 0x300000000 (12 GB)
8589934592 # 0x200000000 ( 8 GB)
4294967296 # 0x100000000 ( 4 GB)
1073741824 # 0x040000000 ( 1 GB)
536870912 # 0x020000000 (.5 GB)
Note: 'cat /sys/module/zfs/parameters/zfs_arc_max' shows you the current value on a running system!
- Dedup Ram: 5GB of RAM per Terrabyte of storage ?
- Advanced Format 4096 byte blocks -> ashift=12 ?!
+++
Install:
1 Boot in Debian Rescue mode
1.0 Install on USB drive
- No DESKTOP, ..
- Leave 1-2GB available for extra ext4 boot partition
to mitigate grub's lack of ZFS support!!
/dev/sdc USB drive
/dev/sdc1 root
/dev/sdc2 boot2
1.1 Network Setup ..
1.1.1 Manual Setup /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address a.b.c.d
netmask 255.255.255.255
dns-nameservers a.b.c.d 1.2.3.4 5.6.7.8
gateway 1.2.3.4
iface eth0 inet6 static
address a:b:c:d:::1:1
netmask 64
up ip -6 route add fe80::1 dev eth0
down ip -6 route del fe80::1 dev eth0
up ip -6 route add default via fe80::1 dev eth0
down ip -6 route del default via fe80::1 dev eth0
- echo jogamp.org > /etc/hostname
/etc/init.d/hostname.sh start
2 Boot USB drive
2.0 Old md superblock ?
Example: /dev/sdb is autodetected as md0:
- mdadm --manage --stop /dev/md0
- mdadm --zero-superblock /dev/sdb
2.1 Fix /etc/apt/sources.list
- remove dvd driver
- add 'wheezy main contrib non-free' net source
- cp -a apt-sources.list /etc/apt/sources.list
'apt-sources.list':
deb http://ftp.debian.org/debian/ wheezy main contrib non-free
deb-src http://ftp.debian.org/debian/ wheezy main contrib non-free
deb http://security.debian.org/ wheezy/updates main contrib
deb-src http://security.debian.org/ wheezy/updates main contrib
deb http://ftp.de.debian.org/debian/ wheezy-updates main contrib non-free
deb-src http://ftp.de.debian.org/debian/ wheezy-updates main contrib non-free
2.2 Install ZFS on USB drive (Script './apt-install-zfs_kernel_etc.sh'):
- wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_1%7Ewheezy_all.deb
- dpkg -i zfsonlinux_1~wheezy_all.deb
- apt-get update
- apt-get upgrade
- apt-get install linux-base linux-image-amd64 linux-headers-amd64 util-linux
- apt-get install debian-zfs libzfs-dev
- apt-get install zfs-initramfs
- apt-get install vim
- Edit /etc/default/zfs
- ZFS_MOUNT='no'
+ ZFS_MOUNT='yes'
- Fix /usr/share/initramfs-tools/scripts/zfs issues:
--- /usr/share/initramfs-tools/scripts/zfs 2013-03-27 04:14:41.000000000 +0100
+++ initramfs-tools.scripts.zfs 2013-06-04 04:20:23.278131866 +0200
@@ -104,7 +104,7 @@
echo ""
echo "Manually import the root pool at the command prompt and then exit."
echo "Hint: Try: zpool import -f -R / -N $ZFS_RPOOL"
- /bin/sh
+ # /bin/sh
fi
if [ -z "$ZFS_BOOTFS" ]
- ARC Mem config (assume 32GB ram)
# ARC: 0 - 12 GB
echo options zfs zfs_arc_max=12884901888 >> /etc/modprobe.d/zfs.conf
echo zfs zfs_arc_max=12884901888 >> /etc/initramfs-tools/modules
depmod -a
Note: 'cat /sys/module/zfs/parameters/zfs_arc_max' shows you the current value on a running system!
2.3 Gather some specs ..
- Gather hdd UUIDs
- $ ls -l /dev/disk/by-uuid
lrwxrwxrwx 1 root root 10 11. Okt 18:02 53cdad3b-4b01-4a6c-a099-be1cdf1acf6d -> ../../sda2
- Check whether hdd uses Advanced Formating, i.e. blk size 4096 instead of 512.
- apt-get install smartmontools
- smartctl -a /dev/sda, .. etc .. see whether (Adv. Format) is listed
- if any disk uses Adv. Format -> "zfs create -o ashift=12 jogamp07 .."
However, to be future-proof, it is recommended to always use 4096 block size for ZFS.
You won't be able to change your pool later.
2.4 Config zfs on sda sdb
- Below we use 'jogamp07' as the zfs pool name,
it's best to replace it w/ a meaningful name, i.e. your hostname.
Below is scripted in './zfs01-create_config-pool.sh' (using jogamp07 as pool name)
- Create zpool
- mirror: if cheap redundancy is more important than size and disks == 2
- zpool create -f -o ashift=12 jogamp07 mirror /dev/disk/by-id/uuid1 /dev/disk/by-id/uuid2
- .. else zraid:
- zpool create -f -o ashift=12 jogamp07 raidz1 /dev/disk/by-id/ata...
- -f -> overwrite existing partition table / data
- ashift=12 -> blocksize 2096
- Config zpool
- zpool autoexpand=on jogamp07
- zpool autoreplace=on jogamp07
- zpool listsnapshots=off jogamp07
- Config root dataset (for all datasets)
- zfs set dedup=off jogamp07
- zfs set compression=off jogamp07
- zfs set atime=off jogamp07
- zfs set mountpoint=none jogamp07
- Swap (Ram 32G, CPU x86_64 == 4k page size)
- zfs create jogamp07/swap -V 33G -b 4K
- zfs set checksum=off jogamp07/swap
ZVOL should be either:
/dev/zvol/jogamp07/swap -> ../../zd0
/dev/zvol/swap
/dev/jogamp07/swap
- mkswap -f /dev/zvol/jogamp07/swap
Below is scripted in './zfs02-create-datasets.sh' (using jogamp07 as pool name)
- Create Datasets
Separation of system/* and the other (users, ..)
allows moving onto another OS/distri while maintaining the data.
- zfs create -o mountpoint=none jogamp07/system
- zfs create -o mountpoint=/ jogamp07/system/debian7_01
# Skip: Will fail w/ zfs-initramfs, mount of /var is too late
# Skip: zfs create jogamp07/system/debian7_01/var
- zfs create -o mountpoint=/home jogamp07/users
- zfs create -o mountpoint=/root jogamp07/users/root
- zfs create -o mountpoint=/data jogamp07/data
- zfs create -o compression=gzip jogamp07/data/backup
- zfs create -o mountpoint=/srv jogamp07/services
- zfs create -o mountpoint=/data jogamp07/backup
- zfs set readonly=on jogamp07/backup
- zfs list -o name,readonly,compression
Below is scripted in './zfs03-export_import.sh' (using jogamp07 as pool name)
- Set zfs pool bootfs
- zpool set bootfs=jogamp07/system/debian7_01 jogamp07
- Export zfs pool
- zpool export jogamp07
- Import zfs pool
- zpool import -f -R /mnt/new jogamp07
- Ensure zpool.cache exists:
- zpool set cachefile=/etc/zfs/zpool.cache jogamp07
- Copy zpool.cache:
- mkdir -p /mnt/new/etc/zfs
- cp -a /etc/zfs/zpool.cache /mnt/new/etc/zfs
2.5 Grub Config
- cp grub_custom.cfg /boot/grub/custom.cfg
- edit /boot/grub/custom.cfg to match your partitions / UUID, e.g.:
menuentry "zfs_01" {
load_video
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd2,msdos1)'
search --no-floppy --fs-uuid --set=root 187bf74d-d4c3-4138-a61a-d4bfb4bc5052
linux /boot/vmlinuz-3.2.0-4-amd64 boot=zfs rpool=jogamp07 bootfs=jogamp07/system/debian7_01 ro
initrd /boot/initrd.img-3.2.0-4-amd64
}
- If not yet installed:
- grub-install --no-floppy /dev/sdc (the USB device!)
- Edit /etc/default/grub:
#GRUB_DEFAULT=0
GRUB_DEFAULT="zfs_01"
- update-grub
2.6 Populate ZFS rootfs
Below is scripted in './zfs04-system-populate.sh'
- See [C1]
- apt-get install debootstrap
- debootstrap --arch amd64 stable /mnt/new ftp://ftp.de.debian.org/debian/
- Setup /mnt/new/etc/network/interfaces
- In case you use same /etc/network/interfaces (scripted):
mv /mnt/new/etc/network/interfaces /mnt/new/etc/network/interfaces.orig
cp -a /etc/network/interfaces /mnt/new/etc/network/interfaces
cp -a /etc/udev/rules.d/70-persistent-net.rules /mnt/new/etc/udev/rules.d/
- Or redo 1.1.1 (Mind that you do it for rootfs on /mnt/new)
- Redo: 2.1 Fix /etc/apt/sources.list (probably just the servers)
-In case you use same apt sources (scripted):
mv /mnt/new/etc/apt/source.list /mnt/new/etc/apt/sources.list.orig
cp -a /etc/apt/sources.list /mnt/new/etc/apt/source.list
- Or redo 2.1
2.6 Chroot ZFS rootfs
- Mount essentials # './chroot-mount.sh':
mount --bind /dev /mnt/new/dev
mount --bind /proc /mnt/new/proc
mount --bind /sys /mnt/new/sys
- chroot /mnt/new /bin/bash # './chroot-chroot.sh'
2.7 Refine / Config System
- Root passord: passwd
- apt-get update
- apt-get upgrade
- apt-get install vim locales
- dpkg-reconfigure locales tzdata
LANG: en_US.UTF-8, LANGUAGE=en_US:en
TZ: your choice
- apt-get install ssh ntp htop iotop rsync xz-utils p7zip-full
2.8 ZFS on new System
- Redo: 2.2 Install ZFS on USB drive (Script './apt-install-zfs_kernel_etc.sh')
2.9 Leave New System / Final Touches
- Mount USB rootfs in new system
- mkdir -p /mnt/usbroot
- Use root partition on USB stick (-> 1.0)
- Edit /etc/fstab, i.e.:
UUID=187bf74d-d4c3-4138-a61a-d4bfb4bc5052 /mnt/usbroot ext4 errors=remount-ro 0 2
- mount /mnt/usbroot
- rm -rf /boot
- ln -s /mnt/usbroot/boot /boot
- umount /mnt/usbroot
- Exit from chroot
- umount proc/dev/sys './chroot-umount.sh'
You may need to:
- umount -l /mnt/new/dev
Reset zfs pool cache ..
- zpool export jogamp07
- zpool import -N jogamp07
- zpool set cachefile=/etc/zfs/zpool.cache jogamp07
- update-initramfs -u -k all
-
# zpool export jogamp07
: now export all other pools too
# zpool import -f -N jogamp07
: now import all other pools too
# zpool set cachefile=/etc/zfs/zpool.cache jogamp07
# mount -t zfs -o zfsutil jogamp07/system/debian7_01 /mnt/new
: do not mount any other filesystem
# cp /etc/zfs/zpool.cache /mnt/new/etc/zfs/zpool.cache
# umount /mnt/new
# update-initramfs -u -k all
# zpool export rpool
: now export all other pools too
# zpool import -d /dev/disk/by-id -f -N rpool
: now import all other pools too
# mount -t zfs -o zfsutil rpool/ROOT/debian-1 /root
: do not mount any other filesystem
# cp /etc/zfs/zpool.cache /root/etc/zfs/zpool.cache
# exit
2.10 Grub on target
cp -a /mnt/usbroot/etc/default/grub /etc/default/grub
apt-get install grub-pc grub-common grub-pc-bin grub2-common
X.1 Skip: Grub 2.00 Derivates / Boot
Note: All failed detecting ZFS rootfs properly!
- Manually build grub 2.00 (ZFS support) [C2]
- dpkg -P grub2 grub-pc grub-common grub-pc-bin grub2-common
- Build Grub
- apt-get install bzr
- apt-get install gettext bison flex libdevmapper-dev
- apt-get install python autoconf automake autogen
Do the following from a user account:
- Skip: Grub 2.00 (not working .. zfs 'unknown filesystem')
wget ftp://ftp.gnu.org/gnu/grub/grub-2.00.tar.gz
tar xzf grub-2.00.tar.gz
cd grub-2.00
- Skip Using bazar repo revno: 5022
bzr branch http://bzr.savannah.gnu.org/r/grub/trunk/grub
git clone --branch snapshot/ubuntu/raring https://github.com/zfsonlinux/grub.git zfs-grub
cd grub
./autogen.sh
./configure --disable-werror --enable-libzfs --enable-device-mapper --enable-efiemu
make
sudo make install
- Skip Using zfs-grub (or ubuntu grub)
git clone --branch snapshot/ubuntu/raring https://github.com/zfsonlinux/grub.git zfs-grub
cd zfs-grub
for i in `cat debian/patches/series`; do patch -p1 < debian/patches/$i 2>&1 | tee -a patch.log ; done
./autogen.sh
./configure --disable-werror --enable-libzfs --enable-device-mapper --enable-efiemu
make
sudo make install
- Install ..
update-initramfs -u -k all
grub-install --no-floppy /dev/sdc (the USB device!)
+++
set timeout=3
set default=0
# Funtoo
menuentry "JogAmp 01" {
insmod zfs
linux /vmlinuz-3.2.0-4-rt-amd64 root=jogamp07/system/debian7_01
initrd /initrd.img-3.2.0-4-rt-amd64
}
+++
References:
[A1] https://en.wikipedia.org/wiki/ZFS
[A2] http://wiki.illumos.org/display/illumos/ZFS
[A3] http://docs.huihoo.com/opensolaris/solaris-zfs-administration-guide/html/index.html
[A4] http://cuddletech.com/ZFSNinja-Slides.pdf
[A5] http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
[A6] http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide
[A7] http://www.cuddletech.com/blog/pivot/entry.php?id=983 (ZFS Compression)
[A8] https://blogs.oracle.com/dap/entry/zfs_compression
[B1] http://zfsonlinux.org/
[B2] http://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/
[B3] https://wiki.freebsd.org/ZFSTuningGuide
[B4] http://stoneyforest.net/~chris/blog/freebsd/zfs/maint.html
[B5] http://www.funtoo.org/wiki/ZFS_Install_Guide
[C1] http://www.howtoforge.com/installing-debian-wheezy-testing-with-debootstrap-from-a-grml-live-linux-p2
[C2] http://www.gnu.org/software/grub/manual/grub.html
[_1] http://wiki.debian.org/AptPreferences#Pinning
|
