Changes between Version 17 and Version 18 of Linux/Ubuntu/HardyRAID5EncryptedLVM
- Timestamp:
- 09/02/09 03:27:04 (19 months ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
Linux/Ubuntu/HardyRAID5EncryptedLVM
v17 v18 1 1 [[PageOutline]] 2 2 3 = RAID-5 Encrypted with Logical Volume Management = 3 4 These instructions in this tutorial should apply to Intrepid (and possibly Jaunty) as well as Hardy. Some slight adjustments might be necessary due to changes in various system libraries and tools. The key-script has been updated to work with Intrepid. 4 These instructions in this tutorial should apply to Intrepid (and possibly Jaunty) as well as Hardy. Some slight adjustments might be necessary due to changes in various system libraries and tools. The key-script has been updated to work with Intrepid. 5 5 6 6 These instructions show how to use the Ubuntu Hardy Desktop Live CD to build and install to a secure encrypted multi-disk RAID system that requires a key-file on an external USB memory stick in order to start. It works around several ''bugs'' in the Ubiquity installer in order to work with ''md'' software RAID arrays (thus avoiding the need to use the alternate text-based installer CD image). In [wiki:Linux/Ubuntu/HardyEncryptedLVM another article there are similar instructions for encrypted non-RAID systems] such as notebooks and laptops that have a single disk drive. … … 15 15 16 16 Access to the Internet is required to install packages not on the LiveCD. Alternatively, download the .deb packages for [http://packages.ubuntu.com/hardy/mdadm mdadm], [http://packages.ubuntu.com/hardy/cryptsetup cryptsetup] and [http://packages.ubuntu.com/hardy/lvm2 lvm2] and make them available locally on a USB memory stick, CD, or floppy disk (they require about 1MB) and install them using: 17 17 18 {{{ 18 19 dpkg -i <package>.deb 19 20 }}} 20 21 21 == Organisation == 22 23 The physical system looks like this:[[BR]] 24 [[Image(RAIDencryptedLVM-physical.png)]] 25 26 The boot files and swap are in RAID-1 (mirrored) arrays that are accessible to GRUB and BIOS:[[BR]] 27 [[Image(RAIDencryptedLVM-logical-RAID1.png)]] 28 29 The operating system and user data are in an encrypted RAID-5 (striped plus parity) array:[[BR]] 30 [[Image(RAIDencryptedLVM-logical-RAID5.png)]] 31 22 The physical system looks like this:[[BR]] [[Image(RAIDencryptedLVM-physical.png)]] 23 24 The boot files and swap are in RAID-1 (mirrored) arrays that are accessible to GRUB and BIOS:[[BR]] [[Image(RAIDencryptedLVM-logical-RAID1.png)]] 25 26 The operating system and user data are in an encrypted RAID-5 (striped plus parity) array:[[BR]] [[Image(RAIDencryptedLVM-logical-RAID5.png)]] 32 27 33 28 == Boot from the Desktop Live CD == 34 35 29 There is no operating system or other data on the disks so the Live CD environment is used to prepare the disks prior to installing Hardy. 36 30 … … 40 34 sudo su 41 35 }}} 42 43 36 == Randomise the disk surface == 44 45 37 By ensuring every sector of the disks is written with random data, a potential attacker will have great difficulty locating encrypted data that they might want to try to decrypt. If the surface of the disk was written with zeros (using if=/dev/zero) or some other predictable values encrypted data would be easy to identify if the disk were to fall into hostile hands. 46 38 47 39 This is likely to take a long time - '''possibly 24 hours or more''' - even with running one background process for each disk: 40 48 41 {{{ 49 42 for dr in a b c d; do DEV="/dev/sd${dr}"; sh -c "dd if=/dev/urandom of=$DEV bs=512"& done 50 43 }}} 51 52 44 Check the progress by sending the '''USR1''' signal to the `dd` processes: 45 53 46 {{{ 54 47 ps -ef | sed -n 's/[a-z ]*\([0-9]*\).* dd.*/\1/p' | while read PID; do kill -USR1 $PID; done 55 48 }}} 56 57 49 == Partition the disks == 58 59 50 Each disk is identically partitioned. The RAID-5 array will use the majority of the space. GRUB and BIOS require a ''regular'' disk layout in order to boot the system, so a small partition for /boot is first on the disks. 60 51 … … 62 53 63 54 We repeat the same partitioning procedure for each disk, creating a 3GB boot/swap partition and put the remainder of the disk in the second partition: 55 64 56 {{{ 65 57 for dr in a b c d; do DEV="/dev/sd${dr}"; echo -e "o\nn\np\n1\n\n+3G\nn\np\n2\n\n\nt\n1\nfd\nt\n2\nfd\np\nw\n" | fdisk $DEV; done 66 58 }}} 67 59 '''Note:''' At this point, if fdisk reports the kernel can't re-read the partition tables: 60 68 61 {{{ 69 62 WARNING: Re-reading the partition table failed with error 16: Device or resource busy. … … 72 65 }}} 73 66 you will need to restart the system: 67 74 68 {{{ 75 69 shutdown -r now … … 78 72 79 73 == Create the RAID arrays == 80 81 74 The RAID arrays will be made up of groups of partitions. 82 75 83 76 Install the Linux RAID package: 77 84 78 {{{ 85 79 apt-get install mdadm 86 80 }}} 87 88 81 First the mirror array for /boot, which will use the first partition on two disks on different IDE channels (to avoid master/slave bootlenecks): 82 89 83 {{{ 90 84 mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdc1 91 85 mdadm: array /dev/md0 started. 92 86 }}} 93 94 87 Now create a mirror array for swap & hibernate: 88 95 89 {{{ 96 90 mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdd1 97 91 mdadm: array /dev/md1 started. 98 92 }}} 99 100 93 All remaining space (in the second partition) is allocated to the RAID-5 array. Its capacity will be the number of disks in the array minus one, multiplied by the size of the partition: (N - 1) x C : (4 - 1) x 58GB = 174GB. 94 101 95 {{{ 102 96 mdadm --create /dev/md2 --level=5 --raid-devices=4 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2 103 97 mdadm: array /dev/md2 started. 104 98 }}} 105 106 99 Check the array build status: 100 107 101 {{{ 108 102 cat /proc/mdstat … … 122 116 unused devices: <none> 123 117 }}} 124 125 118 Wait until the arrays are finished building. Press Ctrl+C to interrupt `watch` when all the arrays have finished building: 119 126 120 {{{ 127 121 watch -n 30 cat /proc/mdstat … … 139 133 unused devices: <none> 140 134 }}} 141 142 135 == Encryption == 143 144 136 First, install the cryptography package: 137 145 138 {{{ 146 139 apt-get install cryptsetup 147 140 }}} 148 149 141 === Choosing a key-file === 150 151 142 The key-file is kept on an external USB memory stick. There are various ways to create and store the key-file. Many guides recommend generating a random key from `/dev/random` but in my opinion anyone that managed to get access to the memory stick could easily locate the key-file because of its totally random contents, unless many 'fake' keys were also on the memory stick. 152 143 … … 158 149 159 150 Load the kernel module: 151 160 152 {{{ 161 153 modprobe dm-crypt 162 154 }}} 163 164 155 === Encrypt the arrays === 165 166 156 Now encrypt the RAID-5 array: 157 167 158 {{{ 168 159 cryptsetup --hash sha512 --key-size 256 --cipher aes-cbc-essiv:sha256 \ … … 176 167 Command successful 177 168 }}} 178 179 169 Open the encrypted device, giving it the name ''md2encrypted'': 170 180 171 {{{ 181 172 cryptsetup --key-file /media/casper-rw/home/tj/Media/theme-song.mp3 luksOpen /dev/md2 md2encrypted … … 184 175 }}} 185 176 There will now be a new device: 177 186 178 {{{ 187 179 ls -1 /dev/mapper … … 189 181 md2encrypted 190 182 }}} 191 192 183 == Configure Logical Volume Management (LVM) == 193 194 184 First install the LVM package: 185 195 186 {{{ 196 187 apt-get install lvm2 197 188 }}} 198 199 189 ==== LVM Tools Confuse Megabytes with Mebibytes ==== 200 201 190 The LVM tool reports can be confusing because they use the wrong size suffixes (such as MB and GB). The problem is, a gigabyte (GB) is 1,000MB, a megabyte (MB) is 1,000 kilobytes, and a kilobyte (KB) is 1,000 bytes. However, LVM uses binary-based calculations, not the decimal, with KB = 1,024, MB = 1,024KB, and GB = 1,024MB. LVM ''should'' use [http://en.wikipedia.org/wiki/Mebibytes MiB] and [http://en.wikipedia.org/wiki/Gibibyte GiB] suffixes to indicate binary-based measurements. 202 191 … … 206 195 207 196 ==== Determine the Number and Size of Logical Extents in Logical Volumes ==== 208 209 197 Logical volume size is defined as the number of ''logical extents'' (LE), the size of which is the same for all logical volumes in a volume group and is the same as the ''physical extent'' (PE) size. Use `vgdisplay` to find out: 198 210 199 {{{ 211 200 vgdisplay VGraid5 … … 232 221 VG UUID V0kRhd-oFLa-hq21-9eCs-kAnm-e1BW-xK7GPT 233 222 }}} 234 The '''PE Size''' (physical extent size) for this volume group is 4.00''MB''. The '''Total PE''' (number of extents) is 43,794. 223 The '''PE Size''' (physical extent size) for this volume group is 4.00''MB''. The '''Total PE''' (number of extents) is 43,794. 235 224 236 225 So, in fact, the PE Size is 4.00''MiB'' (4.00 x 1024 x 1024 = 4,194,304 bytes), or 4.194304''MB'' (4.194304 x 1000 x 1000 = 4,194,304 bytes). 237 226 238 227 To make later calculations easier set up some definitions: 228 239 229 {{{ 240 230 export GB=1000000000 … … 243 233 export MiB=1048576 244 234 }}} 245 246 235 === Encrypted Volume === 247 248 236 Create the physical volume and volume group: 237 249 238 {{{ 250 239 pvcreate /dev/mapper/md2encrypted … … 259 248 Volume group "VGraid5" successfully created 260 249 }}} 261 262 250 Create the logical volume 'root' using 18GB in the volume group 'VGraid5': 251 263 252 {{{ 264 253 export PES_BYTES=$(echo "$(vgdisplay VGraid5 | sed -n 's/.*PE Size *\([0-9\.]*\) .*/\1/p') * $MiB" | bc) … … 268 257 Logical volume "root" created 269 258 }}} 270 271 259 Repeat for `/var/` (10GB) and `/home/` (75% of remaining free space): 260 272 261 {{{ 273 262 EXTENTS=$(echo "10 * $GB / $PES_BYTES" | bc); echo $EXTENTS … … 280 269 Logical volume "home" created 281 270 }}} 282 283 271 Using `vgdisplay` you can see this will leave ~34GB of free space. If or when one of the existing logical volumes reaches capacity simply use `lvextend` to allocate more extents from the volume group - no need to shuffle data or partitions around. 272 284 273 {{{ 285 274 vgdisplay VGraid5 | grep 'Free PE' 286 275 Free PE / Size 8789 / 34.33 GB 287 276 }}} 288 289 277 Check the devices are available: 278 290 279 {{{ 291 280 ls -1 /dev/mapper … … 296 285 VGraid5-var 297 286 }}} 298 299 287 Now the system has all the devices ready for formatting with file-systems, so installation of the operating system can begin. 300 288 301 289 == Install Ubuntu == 302 303 290 === Fix a bug in the partition manager scripts === 304 305 291 Ubiquity (the Ubunutu Live CD installer) depends on a set of scripts to partition the system disks. These are extremely complicated due to the vast number of permutations of storage device types the user might want to install to. As a result of ancient bugs it doesn't support installing to ''plain'' multiple-disk (md) arrays (/dev/md*), although it ''does'' support LVM devices (/dev/mapper/*) on md arrays. 306 292 … … 308 294 309 295 There is a quick-and-dirty workaround to allow Ubiquity to see the md arrays. It is a patch that simply comments out a line that ''ignores'' md devices. It is possible it could cause other issues so beware, but from my experience in the scenario described in this article, it was successful. Run this command before starting the installer: 296 310 297 {{{ 311 298 sed -i 's,^\([\t ]*grep -v .^/dev/md. |\),#\1,' /lib/partman/init.d/30parted 312 299 }}} 313 300 Check the line has had a # prefixed to comment it out: 301 314 302 {{{ 315 303 grep '/dev/md' /lib/partman/init.d/30parted 316 304 # grep -v '^/dev/md' | 317 305 }}} 318 319 306 The md devices also need to be manually formatted (''not'' partitioned though) because Ubiquity can't handle that: 307 320 308 {{{ 321 309 mkfs.ext3 -L boot /dev/md0 … … 324 312 325 313 === Configuring === 326 327 314 Run the Ubuntu installer by double-clicking the '''Install''' icon on the Live CD desktop. Select the language, time-zone and keyboard layout. 328 315 … … 334 321 335 322 The ''device'' list will show amongst others: 323 336 324 {{{ 337 325 /dev/md0 … … 350 338 '''Note:''' Because of a [https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/245855 bug in the Desktop Live CD installer] it is not possible to create the swap partition during installation. Therefore the installer will show a dialog with the message: 351 339 352 You have not selected any partitions for use as swap space. Enabling swap space is recommended so that the system can make better use of available physical memory, and so that it behaves better when physical memory is scarce. You may experience installation problems if you do not have enough physical memory. 353 If you do not go back to the partitioning menu and assign a swap partition, the installation will continue without swap space. 340 You have not selected any partitions for use as swap space. Enabling swap space is recommended so that the system can make better use of available physical memory, and so that it behaves better when physical memory is scarce. You may experience installation problems if you do not have enough physical memory. If you do not go back to the partitioning menu and assign a swap partition, the installation will continue without swap space. 354 341 355 342 Press the '''Continue''' button. … … 361 348 Press the '''Advanced...''' button and '''disable''' (un-tick) "Install boot loader". This must be done manually after the installer has finished. 362 349 363 Finally, when you're happy with all the choices, press the '''Install''' button and wait whilst the installer runs. 350 Finally, when you're happy with all the choices, press the '''Install''' button and wait whilst the installer runs. 364 351 365 352 When it finishes '''DO NOT''' restart the system - some additional configuration is required before the system is restarted so it will boot successfully. 366 353 367 354 === Post-Installation configuration === 368 369 355 Now a series of additional steps is required to ensure the new system can successfully boot. 370 356 371 357 ==== Mount the Target System ==== 372 358 Prepare the installation for access using chroot: 359 373 360 {{{ 374 361 mkdir /mnt/target … … 380 367 mount -o bind /dev /mnt/target/dev 381 368 }}} 382 383 369 ==== Swap ==== 384 370 Format the swap partition and add it to the file-system table so it is started automatically: 371 385 372 {{{ 386 373 mkswap /dev/md1 387 374 echo "UUID=$(vol_id --uuid /dev/md1) none swap sw 0 0" >> /mnt/target/etc/fstab 388 375 }}} 389 390 376 ==== GRUB (boot loader) ==== 391 377 GRUB needs to be installed to both physical disks that make up the /dev/md0 mirror. 392 378 393 379 Install GRUB: 380 394 381 {{{ 395 382 chroot /mnt/target /bin/bash -c "apt-get install grub" … … 397 384 chroot /mnt/target /bin/bash -c "cp /usr/lib/grub/*/{,e2fs_}stage* /boot/grub" 398 385 }}} 399 400 386 Write the master boot record (MBR) to sector #0 of both disks. Start GRUB then issue a series of commands to locate and then install the boot loader files: 387 401 388 {{{ 402 389 grub … … 432 419 grub> quit 433 420 }}} 434 435 421 Write GRUB configuration: 422 436 423 {{{ 437 424 chroot /mnt/target /bin/bash -c "update-grub" … … 449 436 Updating /boot/grub/menu.lst ... done 450 437 }}} 451 452 438 ==== Install mdadm, cryptsetup, and lvm2 ==== 453 439 The installer didn't install the packages needed for the system to understand the disk configuration, so install them manually: 440 454 441 {{{ 455 442 chroot /mnt/target /bin/bash -c "apt-get install mdadm cryptsetup lvm2" 456 443 }}} 457 458 444 ==== Multiple Disk Configuration ==== 459 445 The `mdadm.conf` should have been written automatically when the mdadm package was installed into the chroot environment. Check if the existing file has entries for the three RAID devices: 446 460 447 {{{ 461 448 grep md /mnt/target/etc/mdadm/mdadm.conf … … 469 456 470 457 Otherwise the file can be copied from the Live CD environment: 458 471 459 {{{ 472 460 cp /etc/mdadm/mdadm.conf /mnt/target/etc/mdadm/mdadm.conf 473 461 }}} 474 475 462 Alternatively, it can be written by `mdadm` itself. 463 476 464 {{{ 477 465 mdadm --detail --brief /dev/md* >> /mnt/target/etc/mdadm/mdadm.conf 478 466 }}} 479 480 467 ==== Encrypted Disk Configuration ==== 481 468 Write the configuration of /dev/mapper/md2encrypted so the system knows how to open it. It will use a shell script that is executed early in the boot process from the initrd image: 469 482 470 {{{ 483 471 echo "md2encrypted /dev/disk/by-uuid/$(vol_id --uuid /dev/md2) /home/tj/Media/theme-song.mp3 luks,keyscript=/usr/local/sbin/crypto-usb-key.sh" >> /mnt/target/etc/crypttab 484 472 }}} 485 486 473 There are several suitable shell scripts already existing. I chose to modify one written by Wejn and Rodolfo Garcia published at [http://wejn.org/how-to-make-passwordless-cryptsetup.html How to setup passwordless disk encryption in Debian Etch]. It is [http://tjworld.net/raw-attachment/wiki/Linux/Ubuntu/HardyRAID5EncryptedLVM/crypto-usb-key.sh attached to this article] ready for downloading. 487 474 … … 499 486 * Added comments 500 487 501 '''Note:''' Thanks to feedback from Dave at my-iop I've since fixed a password-reading bug and added more functionality to the script. It should now deal with opening key-files from mounted disks (great if you store key-files for some volumes inside the initial encrypted volume). It simply checks for the existence of the key-file and if it finds it bypasses all the USB functionality and passes the contents of the key-file back to the crypto manager.488 '''Note:''' Thanks to feedback from Dave at my-iop I've since fixed a password-reading bug and added more functionality to the script. It should now deal with opening key-files from mounted disks (great if you store key-files for some volumes inside the initial encrypted volume). It simply checks for the existence of the key-file and if it finds it bypasses all the USB functionality and passes the contents of the key-file back to the crypto manager. 502 489 503 490 Copy it into the new system (ensure the path and name match that specified in /mnt/target/etc/crypttab): 491 504 492 {{{ 505 493 wget http://tjworld.net/raw-attachment/wiki/Linux/Ubuntu/HardyRAID5EncryptedLVM/crypto-usb-key.sh \ 506 494 -O /mnt/target/usr/local/sbin/crypto-usb-key.sh 507 495 chroot /mnt/target /bin/bash -c "chmod a+x /usr/local/sbin/crypto-usb-key.sh" 508 }}} 509 496 }}} 510 497 ==== Update Initial RAM Disk (initrd) ==== 511 512 498 {{{ 513 499 chroot /mnt/target /bin/bash -c "update-initramfs -u all" 514 500 update-initramfs: Generating /boot/initrd.img-2.6.24-19-generic 515 501 }}} 516 517 502 == Restart System and Test == 518 519 503 With everything written to the disks it is time to restart: 504 520 505 {{{ 521 506 shutdown -r now … … 530 515 531 516 === Debugging crypto-usb-key.sh === 532 533 517 If that fails you'll need to enable debugging within the script by changing: 518 534 519 {{{ 535 520 DEBUG=$FALSE 536 521 }}} 537 522 to 523 538 524 {{{ 539 525 DEBUG=$TRUE 540 526 }}} 541 542 527 To do this the initial RAM disk image needs updating. That means the encrypted volume must be opened and mounted and the chroot environment recreated from the Live CD. 543 528 … … 545 530 546 531 Restart the system using the Live CD and open a terminal, then: 532 547 533 {{{ 548 534 sudo su … … 551 537 }}} 552 538 The script requires the key-file name be assigned to the environmental variable KEYFILE: 539 553 540 {{{ 554 541 KEYFILE="/home/tj/Media/theme-song.mp3" … … 556 543 }}} 557 544 Now make the changes and update the initial RAM disk image: 545 558 546 {{{ 559 547 sed -i 's/^\(DEBUG=\)$FALSE/\1$TRUE/' /mnt/target/usr/local/sbin/update-usb-key.sh 560 548 update-initramfs -u all 561 549 }}} 562 563 550 You might also want to remove the "splash" option from the kernel command-line in the GRUB configuration (either in the file /boot/grub/menu.lst or by pressing Escape when GRUB is starting after a reboot to edit the menu directly). This will ensure that the large number of debug messages from the script are easily readable on the console, rather than scrolling up too fast in usplash. 564 551 … … 568 555 569 556 == References == 570 571 [https://wiki.ubuntu.com/LiveUsbPendrivePersistent Installing Ubuntu on USB pendrive using Linux][[BR]] 572 [http://mazeoflies.com/articles/2008/06/09/ubuntu-hard-drive-encryption-with-external-key Ubuntu hard drive encryption with external key][[BR]] 573 [https://help.ubuntu.com/community/EncryptedFilesystemLVMHowto Installing Ubuntu 7.04 on an Encrypted LVM Partition For Root, Swap, and Home][[BR]] 574 [http://wejn.org/how-to-make-passwordless-cryptsetup.html How to setup passwordless disk encryption in Debian Etch][[BR]] 575 [http://linuxgazette.net/140/pfeiffer.html Encrypted Storage with LUKS, RAID and LVM2][[BR]] 576 [http://www.gagme.com/greg/linux/raid-lvm.php Managing RAID and LVM with Linux (v0.5)][[BR]] 577 [http://www.howtoforge.com/linux_lvm_p6 A Beginner's Guide To LVM - LVM On RAID1][[BR]] 578 [http://www.mythtv.org/wiki/index.php/RAID#RAID_0.2B1_.28or_1.2B0.2C_01.2C_10.29 RAID][[BR]] 557 [https://wiki.ubuntu.com/LiveUsbPendrivePersistent Installing Ubuntu on USB pendrive using Linux][[BR]] [http://mazeoflies.com/articles/2008/06/09/ubuntu-hard-drive-encryption-with-external-key Ubuntu hard drive encryption with external key][[BR]] [https://help.ubuntu.com/community/EncryptedFilesystemLVMHowto Installing Ubuntu 7.04 on an Encrypted LVM Partition For Root, Swap, and Home][[BR]] [http://wejn.org/how-to-make-passwordless-cryptsetup.html How to setup passwordless disk encryption in Debian Etch][[BR]] [http://linuxgazette.net/140/pfeiffer.html Encrypted Storage with LUKS, RAID and LVM2][[BR]] [http://www.gagme.com/greg/linux/raid-lvm.php Managing RAID and LVM with Linux (v0.5)][[BR]] [http://www.howtoforge.com/linux_lvm_p6 A Beginner's Guide To LVM - LVM On RAID1][[BR]] [http://www.mythtv.org/wiki/index.php/RAID#RAID_0.2B1_.28or_1.2B0.2C_01.2C_10.29 RAID] 579 558 580 559 == Updates == 581 560 4 December 2008 '''crypto-usb-key.sh''': Dropped detection of specific USB devices because Intrepid and kernel 2.6.27 no longer include "usb" in the /sys/block/.../device path. Now the script relies purely on the 'removable' flag in determining which devices to look on for the keyfile. Fixed msg() not printing to console when using Intrepid. Simplified detection of running usplash. 582 561 583 584 9 February 2009 '''crypto-usb-key.sh''': Added support for Jaunty, made script work in initrd and after root is mounted, use vol_id for FSTYPE and LABEL if possible, remove reliance on basename. 562 9 February 2009 '''crypto-usb-key.sh''': Added support for Jaunty, made script work in initrd and after root is mounted, use vol_id for FSTYPE and LABEL if possible, remove reliance on basename, minimise wait for USB device to settle by monitoring dmesg for attache event.
