Changes between Initial Version and Version 1 of Howto/ResizeQemuDiskImages


Ignore:
Timestamp:
04/05/08 20:26:58 (10 years ago)
Author:
tj
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Howto/ResizeQemuDiskImages

    v1 v1  
     1[[PageOutline]] 
     2= How To Resize (expand) QEMU qcow/qcow2/raw Disk Images = 
     3 
     4First published June 2007 [http://qemu-forum.ipi.fi/viewtopic.php?f=22&t=3756 in the QEMU forums] 
     5 
     6This article explains how to reliably enlarge a QEMU qcow or raw image that contains an NTFS or FAT32 bootable partition. 
     7 
     8Like a lot of other people I needed to increase the size of a QCOW disk image to 10GB from the original maximum size of 5GB, I searched all over and could only find three web pages that described methods of doing it, each of which was imperfect and not totally reliable. There were tens of pages discussing how it is impossible. 
     9 
     10My scenario is QEMU 0.9 running on Ubuntu 7.04 Feisty (kernel 2.6.20-16) with Windows XP in a 5GB qcow disk image. 
     11 
     12After trying the various suggestions for how to resize the image I always got the dreaded: 
     13{{{ 
     14A disk read error occurred 
     15Press Ctrl+Alt+Del to restart 
     16}}} 
     17I read of a suggestion by Fabrice that it was to do with "CHS translation" but no one seemed to have taken it further. 
     18 
     19'''If you just want the solution''' rather than reading the technical details skip to "'''How to enlarge a QEMU qcow or raw image that contains an NTFS bootable partition'''" further down. 
     20 
     21== Discovering why it fails == 
     22 
     23First I converted the qcow image to raw: 
     24{{{ 
     25$ qemu-img convert -f qcow hda.qcow -O raw hda.raw 
     26}}} 
     27We are going to require super-user (root) privileges for most of the following operations so lets switch to super-user: 
     28{{{ 
     29$ sudo su 
     30Password: 
     31root $ 
     32}}} 
     33Using the loopback device in Linux I attached the raw image: 
     34{{{ 
     35$ losetup /dev/loop1 hda.raw 
     36}}} 
     37Then I inspected the Master Boot Record (MBR) using fdisk: 
     38{{{ 
     39$ fdisk -u /dev/loop1 
     40 
     41Command (m for help): p 
     42 
     43Disk /dev/loop1: 4194 MB, 4194816000 bytes 
     44128 heads, 63 sectors/track, 1015 cylinders, total 8193000 sectors 
     45Units = sectors of 1 * 512 = 512 bytes 
     46 
     47      Device Boot      Start         End      Blocks   Id  System 
     48/dev/loop1p1   *          63     8176895     4088416+   7  HPFS/NTFS 
     49 
     50Command (m for help): q    
     51 
     52}}} 
     53Notice that fdisk reports the disk has '''128 (0x80) heads'''. 
     54 
     55Now lets examine the NTFS partition's boot sector. To do this we need to detach the 'physical' disk: 
     56{{{ 
     57$ losetup -d /dev/loop1 
     58}}} 
     59and attach the 'logical' disk contained in partition  1. This starts at sector offset 63 (0x3F) according to fdisk, which is 32256 bytes (63 x 512) into the image file:  
     60{{{ 
     61$ losetup -o32256 /dev/loop1 hda.raw 
     62}}} 
     63Just to prove this is an NTFS partition we have attached lets check it: 
     64{{{ 
     65$ ntfsresize -i /dev/loop1 
     66ntfsresize v1.13.1 (libntfs 9:0:0) 
     67Device name        : /dev/loop1 
     68NTFS volume version: 3.1 
     69Cluster size       : 4096 bytes 
     70Current volume size: 4186538496 bytes (4187 MB) 
     71Current device size: 4194783744 bytes (4195 MB) 
     72Checking filesystem consistency ... 
     73100.00 percent completed 
     74Accounting clusters ... 
     75Space in use       : 3265 MB (78.0%) 
     76Collecting resizing constraints ... 
     77You might resize at 3264757760 bytes or 3265 MB (freeing 922 MB). 
     78Please make a test run using both the -n and -s options before real resizing! 
     79 
     80}}} 
     81Now let's examine the BIOS Parameter Block (BPB) of the partition's boot sector: 
     82{{{ 
     83$ dd if=/dev/loop1 of=bootsector.bin bs=512 count=1 
     84$ losetup -d /dev/loop1 
     85}}} 
     86I use '''hexdump''' to examine the data statically: 
     87{{{ 
     88$ hexdump -C bootsector.bin 
     89}}} 
     90The BPB bytes we are interested in are the ones describing the geometry of the drive: 
     91{{{ 
     92 
     93Offset        Purpose 
     940x18           Sectors per track 
     950x1A           Heads (tracks per cylinder) 
     96 
     97}}} 
     98So here is the fragment that shows these: 
     99{{{ 
     10000000000  eb 52 90 4e 54 46 53 20  20 20 20 00 02 08 00 00  |.R.NTFS    .....| 
     10100000010  00 00 00 00 00 f8 00 00  3f 00 80 00 3f 00 00 00  |........?...?...| 
     102 
     103}}} 
     104You can see at offset '''0x18''' "3f 00" (63) and at '''0x1A''' "80 00" (128). 
     105 
     106When a disk image is resized by a considerable margin the number of heads    reported for the entire disk image is likely to increase, but QEMU doesn't know the embedded Operating System (in this case Windows XP) has stored the disk geometry in its BPB. 
     107 
     108As a result when the QEMU BIOS tries to load the OS it gets the correct disk geometry from the MBR but when the MBR passed execution to the boot sector code in the partition, the BPB causes the boot-code to get confused and the system fails to boot.  
     109 
     110'''So the solution is''' to edit the single byte in the BPB that describes the number of heads (offset 0x1A) so it matches whatever fdisk reports. 
     111 
     112== How to enlarge a QEMU qcow or raw image that contains an NTFS bootable partition == 
     113 
     114''Make sure you have plenty of disk space'' for the following operations. Adjust paths to point to alternative disks that have space if necessary. 
     115 
     116'''Note:''' If you are pressed for disk space or don't want to wait a long time when the file-system is copied from the original image to the new image, read the next post in this thread for how you can enlarge the original file directly. It is more dangerous since it is changing your only copy of the original. 
     117 
     118Convert the compressed qcow image to raw: 
     119{{{ 
     120$ qemu-img convert -f qcow hda.qcow -O raw hda.raw 
     121}}} 
     122Most of the commands will require super-user privileges so switch now to save using sudo for every command: 
     123{{{ 
     124$ sudo su 
     125}}} 
     126Create the new raw disk image. If you are working on ext2 or ext3 this will create a sparse image (disk space won't actually be used until non-zero data is written to the file). The value after '''seek=''' is the size of the new disk in sectors. In this case 20971520 x 512 = 10GB: 
     127{{{ 
     128$ dd if=/dev/zero of=hdb.raw bs=512 count=0 seek=20971520 
     129$ ls -l hdb.raw 
     130 -rw-r--r-- 1 root root 10737418240 2007-06-14 15:14 hdb.raw 
     131}}} 
     132Attach the new drive: 
     133{{{ 
     134$ losetup /dev/loop0 hdb.raw 
     135}}} 
     136Use fdisk to create the new, larger, NTFS partition table entry: 
     137{{{ 
     138$ fdisk -u /dev/loop0 
     139Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel 
     140Building a new DOS disklabel. Changes will remain in memory only, 
     141until you decide to write them. After that, of course, the previous 
     142content won't be recoverable. 
     143 
     144The number of cylinders for this disk is set to 1305. 
     145There is nothing wrong with that, but this is larger than 1024, 
     146and could in certain setups cause problems with: 
     1471) software that runs at boot time (e.g., old versions of LILO) 
     1482) booting and partitioning software from other OSs 
     149   (e.g., DOS FDISK, OS/2 FDISK) 
     150Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) 
     151 
     152Command (m for help): 
     153}}} 
     154Create a '''n'''ew partition (press n p 1 <enter> <enter> ): 
     155{{{ 
     156Command (m for help): n 
     157Command action 
     158   e   extended 
     159   p   primary partition (1-4) 
     160p 
     161Partition number (1-4): 1 
     162First sector (63-20971519, default 63):  
     163Using default value 63 
     164Last sector or +size or +sizeM or +sizeK (63-20971519, default 20971519):  
     165Using default value 20971519 
     166}}} 
     167Set the partition '''t'''ype to NTFS (press t 7): 
     168{{{ 
     169Command (m for help): t 
     170Selected partition 1 
     171Hex code (type L to list codes): 7 
     172Changed system type of partition 1 to 7 (HPFS/NTFS) 
     173}}} 
     174Set the partition to be bootable aka '''a'''ctive (press a 1): 
     175{{{ 
     176Command (m for help): a 
     177Partition number (1-4): 1 
     178}}} 
     179Display aka '''p'''rint the new configuration (press p): 
     180{{{ 
     181Command (m for help): p 
     182 
     183Disk /dev/loop0: 10.7 GB, 10737418240 bytes 
     184255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors 
     185Units = sectors of 1 * 512 = 512 bytes 
     186 
     187      Device Boot      Start         End      Blocks   Id  System 
     188/dev/loop0p1   *          63    20971519    10485728+   7  HPFS/NTFS 
     189}}} 
     190'''Notice the number of heads is 255?''' This is the value we will have to patch into the NTFS BPB in the partition boot sector later. 
     191 
     192Now '''w'''rite the changes to the disk (press w): 
     193{{{ 
     194Command (m for help): w 
     195The partition table has been altered! 
     196 
     197Calling ioctl() to re-read partition table. 
     198 
     199WARNING: Re-reading the partition table failed with error 22: Invalid argument. 
     200The kernel still uses the old table. 
     201The new table will be used at the next reboot. 
     202Syncing disks. 
     203}}} 
     204We now have a bootable disk image with a partition table, but no Operating System to boot. 
     205 
     206Detach the disk image: 
     207{{{ 
     208$ losetup -d /dev/loop0 
     209}}} 
     210Re-attach the image so the Linux kernel will read the new partition table, and then check it with fdisk: 
     211{{{ 
     212$ losetup /dev/loop0 hdb.raw 
     213$  fdisk -ul /dev/loop0 
     214 
     215Disk /dev/loop0: 10.7 GB, 10737418240 bytes 
     216255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors 
     217Units = sectors of 1 * 512 = 512 bytes 
     218 
     219      Device Boot      Start         End      Blocks   Id  System 
     220/dev/loop0p1   *          63    20971519    10485728+   7  HPFS/NTFS 
     221}}} 
     222 
     223Copy '''''just''''' the MBR boot-code into the new disk's MBR: 
     224{{{ 
     225$ dd if=hda.raw of=/dev/loop0 bs=1 count=446 
     226}}} 
     227This is the code the BIOS boot-loader executes when the system boots from this disk device. It is also what a Windows/DOS '''fdisk /mbr''' installs. 
     228 
     229Now copy the NTFS partition from the original image into the new image. Remember, the file-system partitions start at sector offset 63 according to fdisk for both images: 
     230{{{ 
     231$ dd if=hda.raw of=/dev/loop0 bs=512 skip=63 seek=63 
     232}}} 
     233Detach the disk image: 
     234{{{ 
     235$ losetup -d /dev/loop0 
     236}}} 
     237Attach just the file-system partition. It starts at sector 63, which is offset 63 x 512 = 32256: 
     238{{{ 
     239$ losetup -o32256 /dev/loop0 hdb.raw 
     240}}} 
     241Confirm it is a valid NTFS partition: 
     242{{{ 
     243$ ntfsresize -i /dev/loop0 
     244ntfsresize v1.13.1 (libntfs 9:0:0) 
     245Device name        : /dev/loop0 
     246NTFS volume version: 3.1 
     247Cluster size       : 4096 bytes 
     248Current volume size: 4186538496 bytes (4187 MB) 
     249Current device size: 10737385984 bytes (10738 MB) 
     250Checking filesystem consistency ... 
     251100.00 percent completed 
     252Accounting clusters ... 
     253Space in use       : 3265 MB (78.0%) 
     254Collecting resizing constraints ... 
     255You might resize at 3264757760 bytes or 3265 MB (freeing 922 MB). 
     256Please make a test run using both the -n and -s options before real resizing! 
     257}}} 
     258Now it is time to edit the NTFS BPB. I use the tool '''hexedit''' which you may need to install: 
     259{{{ 
     260$ apt-get install hexedit 
     261}}} 
     262Load the new disk image into hexedit: 
     263{{{ 
     264$ hexedit hdb.raw 
     265}}} 
     266Skip to sector 63, which is offset 0x7E00 (press <Enter> 7E00 <Enter>): 
     267{{{ 
     26800007E00   EB 52 90 4E  54 46 53 20  20 20 20 00  02 08 00 00  .R.NTFS    ..... 
     26900007E10   00 00 00 00  00 F8 00 00  3F 00 80 00  3F 00 00 00  ........?...?... 
     270 
     271}}} 
     272Move the cursor to the BPB's offset 0x1A, which is 0x7E00 + 0x1A = 0x7E1A here. 
     273 
     274You can now over-type the hex-value '''80 (128)''' with the correct number of heads, which in this case is '''FF (255)''': 
     275{{{ 
     27600007E10   00 00 00 00  00 F8 00 00  3F 00 FF 00  3F 00 00 00  ........?...?... 
     277}}} 
     278Save the change by pressing Ctrl-X and then 'y' to confirm the write. 
     279 
     280The image is now ready for QEMU. 
     281{{{ 
     282$ qemu -boot c -snapshot -m 512 -hda 'hdb.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu  
     283}}} 
     284If that boots successfully we can now resize the NTFS file-system: 
     285{{{ 
     286$ ntfsresize /dev/loop0 
     287ntfsresize v1.13.1 (libntfs 9:0:0) 
     288Device name        : /dev/loop0 
     289NTFS volume version: 3.1 
     290Cluster size       : 4096 bytes 
     291Current volume size: 4186538496 bytes (4187 MB) 
     292Current device size: 10737385984 bytes (10738 MB) 
     293New volume size    : 10737381888 bytes (10738 MB) 
     294Checking filesystem consistency ... 
     295100.00 percent completed 
     296Accounting clusters ... 
     297Space in use       : 3263 MB (77.9%) 
     298Collecting resizing constraints ... 
     299WARNING: Every sanity check passed and only the dangerous operations left. 
     300Make sure that important data has been backed up! Power outage or computer 
     301crash may result major data loss! 
     302Are you sure you want to proceed (y/[n])? y 
     303Schedule chkdsk for NTFS consistency check at Windows boot time ... 
     304Resetting $LogFile ... (this might take a while) 
     305Updating $BadClust file ... 
     306Updating $Bitmap file ... 
     307Updating Boot record ... 
     308Syncing device ... 
     309Successfully resized NTFS on device '/dev/loop0'. 
     310}}} 
     311If you get the error: 
     312{{{ 
     313$ ntfsresize /dev/loop0 
     314ntfsresize v1.13.1 (libntfs 9:0:0) 
     315ERROR(95): Opening '/dev/loop0' as NTFS failed: Operation not supported 
     316The NTFS journal file is unclean. Please shutdown Windows properly before 
     317using this software! Note, if you have run chkdsk previously then boot 
     318Windows again which will automatically initialize the journal correctly. 
     319}}} 
     320you should do a complete startup/shutdown sequence of Windows so the disk is marked clean: 
     321{{{ 
     322$ losetup -d /dev/loop0 
     323$ qemu -boot c -m 512 -hda 'hdb.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu  
     324}}} 
     325All that is left to do is detach it: 
     326{{{ 
     327$ losetup -d /dev/loop0 
     328}}} 
     329and convert the raw image back to a compressed qcow image.  I usually explicitly make it qcow2, the later version: 
     330{{{ 
     331$ qemu-img convert -f raw hdb.raw -O qcow2 hdb.qcow2 
     332}}} 
     333Make sure the qcow image boots: 
     334{{{ 
     335$ qemu -boot c -snapshot -m 512 -hda 'hdb.qcow2' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu  
     336}}} 
     337Delete the large raw images and the original qcow image: 
     338{{{ 
     339$ rm hdb.raw hda.raw hda.qcow 
     340}}} 
     341The final step is to set the owner to your regular user, because the file was created as root, and exit super-user: 
     342{{{ 
     343$ chown <username>:<username> hdb.qcow2 
     344$ exit 
     345}}} 
     346 
     347I hope this helps - it took a while to perfect the process  :D 
     348 
     349== How to enlarge without using twice the disk space == 
     350 
     351This method is an alternative to the one described in the previous post. It is intended for more confident users, where you are pressed for disk space or don't want to wait a long time when the file-system is copied from the original image to the new image. It is more dangerous since it is changing your only copy of the original. 
     352 
     353Convert the compressed qcow image to raw: 
     354{{{ 
     355$ qemu-img convert -f qcow hda.qcow -O raw hda.raw 
     356}}} 
     357Most of the commands will require super-user privileges so switch now to save using sudo for every command: 
     358{{{ 
     359$ sudo su 
     360}}} 
     361'''Enlarge''' the '''''existing''''' raw disk image. If you are working on ext2 or ext3 this will create a sparse image (disk space won't actually be used until non-zero data is written to the file). The value after '''seek=''' is the size of the new disk in sectors. In this case 20971520 x 512 = 10GB: 
     362{{{ 
     363$ dd if=/dev/zero of=hda.raw bs=512 count=0 seek=20971520 
     364$ ls -l hda.raw 
     365 -rw-r--r-- 1 root root 10737418240 2007-06-18 12:10 hda.raw 
     366}}} 
     367Attach the enlarged drive image: 
     368{{{ 
     369$ losetup /dev/loop0 hda.raw 
     370}}} 
     371Use fdisk to check how many heads the disk is using: 
     372{{{ 
     373$ fdisk -ul /dev/loop0 
     374 
     375Disk /dev/loop0: 10.7 GB, 10737418240 bytes 
     376255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors 
     377Units = sectors of 1 * 512 = 512 bytes 
     378 
     379      Device Boot      Start         End      Blocks   Id  System 
     380/dev/loop1p1   *          63     8176895     4088416+   7  HPFS/NTFS 
     381}}}  
     382'''Notice the number of heads is 255?''' This is the value we will have to patch into the NTFS BPB in the partition boot sector later. 
     383 
     384Detach the drive image: 
     385{{{ 
     386$ losetup -d /dev/loop0 
     387}}} 
     388Attach just the file-system partition. It starts at sector 63, which is offset 63 x 512 = 32256: 
     389{{{ 
     390$ losetup -o32256 /dev/loop0 hda.raw 
     391}}} 
     392Confirm it is a valid NTFS partition: 
     393{{{ 
     394$ ntfsresize -i /dev/loop0 
     395ntfsresize v1.13.1 (libntfs 9:0:0) 
     396Device name        : /dev/loop0 
     397NTFS volume version: 3.1 
     398Cluster size       : 4096 bytes 
     399Current volume size: 4186538496 bytes (4187 MB) 
     400Current device size: 10737385984 bytes (10738 MB) 
     401Checking filesystem consistency ... 
     402100.00 percent completed 
     403Accounting clusters ... 
     404Space in use       : 3265 MB (78.0%) 
     405Collecting resizing constraints ... 
     406You might resize at 3264757760 bytes or 3265 MB (freeing 922 MB). 
     407Please make a test run using both the -n and -s options before real resizing! 
     408}}} 
     409Now it is time to edit the NTFS BPB. I use the tool '''hexedit''' which you may need to install: 
     410{{{ 
     411$ apt-get install hexedit 
     412}}} 
     413Load the new disk image into hexedit: 
     414{{{ 
     415$ hexedit hda.raw 
     416}}} 
     417Skip to sector 63, which is offset 0x7E00 (press <Enter> 7E00 <Enter>): 
     418{{{ 
     41900007E00   EB 52 90 4E  54 46 53 20  20 20 20 00  02 08 00 00  .R.NTFS    ..... 
     42000007E10   00 00 00 00  00 F8 00 00  3F 00 80 00  3F 00 00 00  ........?...?... 
     421 
     422}}} 
     423Move the cursor to the BPB's offset 0x1A, which is 0x7E00 + 0x1A = 0x7E1A here. 
     424 
     425You can now over-type the hex-value '''80 (128)''' with the correct number of heads, which in this case is '''FF (255)''': 
     426{{{ 
     42700007E10   00 00 00 00  00 F8 00 00  3F 00 FF 00  3F 00 00 00  ........?...?... 
     428}}} 
     429Save the change by pressing Ctrl-X and then 'y' to confirm the write. 
     430 
     431Detach the drive image: 
     432{{{ 
     433$ losetup -d /dev/loop0 
     434}}} 
     435The image is now ready for QEMU. 
     436{{{ 
     437$ qemu -boot c -snapshot -m 512 -hda 'hda.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu  
     438}}} 
     439If that boots successfully we can now resize the NTFS file-system: 
     440 
     441Attach the file-system image: 
     442{{{ 
     443$ losetup -o32256 /dev/loop0 hda.raw 
     444}}} 
     445Resize to fill the drive-image: 
     446{{{ 
     447$ ntfsresize /dev/loop0 
     448ntfsresize v1.13.1 (libntfs 9:0:0) 
     449Device name        : /dev/loop0 
     450NTFS volume version: 3.1 
     451Cluster size       : 4096 bytes 
     452Current volume size: 4186538496 bytes (4187 MB) 
     453Current device size: 10737385984 bytes (10738 MB) 
     454New volume size    : 10737381888 bytes (10738 MB) 
     455Checking filesystem consistency ... 
     456100.00 percent completed 
     457Accounting clusters ... 
     458Space in use       : 3263 MB (77.9%) 
     459Collecting resizing constraints ... 
     460WARNING: Every sanity check passed and only the dangerous operations left. 
     461Make sure that important data has been backed up! Power outage or computer 
     462crash may result major data loss! 
     463Are you sure you want to proceed (y/[n])? y 
     464Schedule chkdsk for NTFS consistency check at Windows boot time ... 
     465Resetting $LogFile ... (this might take a while) 
     466Updating $BadClust file ... 
     467Updating $Bitmap file ... 
     468Updating Boot record ... 
     469Syncing device ... 
     470Successfully resized NTFS on device '/dev/loop0'. 
     471}}} 
     472If you get the error: 
     473{{{ 
     474$ ntfsresize /dev/loop0 
     475ntfsresize v1.13.1 (libntfs 9:0:0) 
     476ERROR(95): Opening '/dev/loop0' as NTFS failed: Operation not supported 
     477The NTFS journal file is unclean. Please shutdown Windows properly before 
     478using this software! Note, if you have run chkdsk previously then boot 
     479Windows again which will automatically initialize the journal correctly. 
     480}}} 
     481you should do a complete startup/shutdown sequence of Windows so the disk is marked clean: 
     482{{{ 
     483$ losetup -d /dev/loop0 
     484$ qemu -boot c -m 512 -hda 'hda.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu  
     485}}} 
     486All that is left to do is detach it: 
     487{{{ 
     488$ losetup -d /dev/loop0 
     489}}} 
     490and convert the raw image back to a compressed qcow image.  I usually explicitly make it qcow2, the later version: 
     491{{{ 
     492$ qemu-img convert -f raw hda.raw -O qcow2 hda.qcow2 
     493}}} 
     494Make sure the qcow image boots: 
     495{{{ 
     496$ qemu -boot c -snapshot -m 512 -hda 'hda.qcow2' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu  
     497}}} 
     498Delete the large raw image and the original qcow image: 
     499{{{ 
     500$ rm hda.raw hda.qcow 
     501}}} 
     502The final step is to set the owner to your regular user, because the file was created as root, and exit super-user: 
     503{{{ 
     504$ chown <username>:<username> hda.qcow2 
     505$ exit 
     506}}}