wiki:Howto/ResizeQemuDiskImages

How To Resize (expand) QEMU qcow/qcow2/raw Disk Images

First published June 2007  in the QEMU forums

This article explains how to reliably enlarge a QEMU qcow or raw image that contains an NTFS or FAT32 bootable partition.

Like a lot of other people I needed to increase the size of a QCOW disk image to 10GB from the original maximum size of 5GB, I searched all over and could only find three web pages that described methods of doing it, each of which was imperfect and not totally reliable. There were tens of pages discussing how it is impossible.

My scenario is QEMU 0.9 running on Ubuntu 7.04 Feisty (kernel 2.6.20-16) with Windows XP in a 5GB qcow disk image.

After trying the various suggestions for how to resize the image I always got the dreaded:

A disk read error occurred
Press Ctrl+Alt+Del to restart

I read of a suggestion by Fabrice that it was to do with "CHS translation" but no one seemed to have taken it further.

If you just want the solution rather than reading the technical details skip to "How to enlarge a QEMU qcow or raw image that contains an NTFS bootable partition" further down.

Discovering why it fails

First I converted the qcow image to raw:

$ qemu-img convert -f qcow hda.qcow -O raw hda.raw

We are going to require super-user (root) privileges for most of the following operations so lets switch to super-user:

$ sudo su
Password:
root $

Using the loopback device in Linux I attached the raw image:

$ losetup /dev/loop1 hda.raw

Then I inspected the Master Boot Record (MBR) using fdisk:

$ fdisk -u /dev/loop1

Command (m for help): p

Disk /dev/loop1: 4194 MB, 4194816000 bytes
128 heads, 63 sectors/track, 1015 cylinders, total 8193000 sectors
Units = sectors of 1 * 512 = 512 bytes

      Device Boot      Start         End      Blocks   Id  System
/dev/loop1p1   *          63     8176895     4088416+   7  HPFS/NTFS

Command (m for help): q   

Notice that fdisk reports the disk has 128 (0x80) heads.

Now lets examine the NTFS partition's boot sector. To do this we need to detach the 'physical' disk:

$ losetup -d /dev/loop1

and attach the 'logical' disk contained in partition 1. This starts at sector offset 63 (0x3F) according to fdisk, which is 32256 bytes (63 x 512) into the image file:

$ losetup -o32256 /dev/loop1 hda.raw

Just to prove this is an NTFS partition we have attached lets check it:

$ ntfsresize -i /dev/loop1
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name        : /dev/loop1
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 4194783744 bytes (4195 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use       : 3265 MB (78.0%)
Collecting resizing constraints ...
You might resize at 3264757760 bytes or 3265 MB (freeing 922 MB).
Please make a test run using both the -n and -s options before real resizing!

Now let's examine the BIOS Parameter Block (BPB) of the partition's boot sector:

$ dd if=/dev/loop1 of=bootsector.bin bs=512 count=1
$ losetup -d /dev/loop1

I use hexdump to examine the data statically:

$ hexdump -C bootsector.bin

The BPB bytes we are interested in are the ones describing the geometry of the drive:

Offset        Purpose
0x18           Sectors per track
0x1A           Heads (tracks per cylinder)

So here is the fragment that shows these:

00000000  eb 52 90 4e 54 46 53 20  20 20 20 00 02 08 00 00  |.R.NTFS    .....|
00000010  00 00 00 00 00 f8 00 00  3f 00 80 00 3f 00 00 00  |........?...?...|

You can see at offset 0x18 "3f 00" (63) and at 0x1A "80 00" (128).

When a disk image is resized by a considerable margin the number of heads reported for the entire disk image is likely to increase, but QEMU doesn't know the embedded Operating System (in this case Windows XP) has stored the disk geometry in its BPB.

As a result when the QEMU BIOS tries to load the OS it gets the correct disk geometry from the MBR but when the MBR passed execution to the boot sector code in the partition, the BPB causes the boot-code to get confused and the system fails to boot.

So the solution is to edit the single byte in the BPB that describes the number of heads (offset 0x1A) so it matches whatever fdisk reports.

How to enlarge a QEMU qcow or raw image that contains an NTFS bootable partition

Make sure you have plenty of disk space for the following operations. Adjust paths to point to alternative disks that have space if necessary.

Note: If you are pressed for disk space or don't want to wait a long time when the file-system is copied from the original image to the new image, read the next post in this thread for how you can enlarge the original file directly. It is more dangerous since it is changing your only copy of the original.

Convert the compressed qcow image to raw:

$ qemu-img convert -f qcow hda.qcow -O raw hda.raw

Most of the commands will require super-user privileges so switch now to save using sudo for every command:

$ sudo su

Create the new raw disk image. If you are working on ext2 or ext3 this will create a sparse image (disk space won't actually be used until non-zero data is written to the file). The value after seek= is the size of the new disk in sectors. In this case 20971520 x 512 = 10GB:

$ dd if=/dev/zero of=hdb.raw bs=512 count=0 seek=20971520
$ ls -l hdb.raw
 -rw-r--r-- 1 root root 10737418240 2007-06-14 15:14 hdb.raw

Attach the new drive:

$ losetup /dev/loop0 hdb.raw

Use fdisk to create the new, larger, NTFS partition table entry:

$ fdisk -u /dev/loop0
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help):

Create a new partition (press n p 1 <enter> <enter> ):

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (63-20971519, default 63): 
Using default value 63
Last sector or +size or +sizeM or +sizeK (63-20971519, default 20971519): 
Using default value 20971519

Set the partition type to NTFS (press t 7):

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 7
Changed system type of partition 1 to 7 (HPFS/NTFS)

Set the partition to be bootable aka active (press a 1):

Command (m for help): a
Partition number (1-4): 1

Display aka print the new configuration (press p):

Command (m for help): p

Disk /dev/loop0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1   *          63    20971519    10485728+   7  HPFS/NTFS

Notice the number of heads is 255? This is the value we will have to patch into the NTFS BPB in the partition boot sector later.

Now write the changes to the disk (press w):

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

We now have a bootable disk image with a partition table, but no Operating System to boot.

Detach the disk image:

$ losetup -d /dev/loop0

Re-attach the image so the Linux kernel will read the new partition table, and then check it with fdisk:

$ losetup /dev/loop0 hdb.raw
$  fdisk -ul /dev/loop0

Disk /dev/loop0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1   *          63    20971519    10485728+   7  HPFS/NTFS

Copy just the MBR boot-code into the new disk's MBR:

$ dd if=hda.raw of=/dev/loop0 bs=1 count=446

This is the code the BIOS boot-loader executes when the system boots from this disk device. It is also what a Windows/DOS fdisk /mbr installs.

Now copy the NTFS partition from the original image into the new image. Remember, the file-system partitions start at sector offset 63 according to fdisk for both images:

$ dd if=hda.raw of=/dev/loop0 bs=512 skip=63 seek=63

Detach the disk image:

$ losetup -d /dev/loop0

Attach just the file-system partition. It starts at sector 63, which is offset 63 x 512 = 32256:

$ losetup -o32256 /dev/loop0 hdb.raw

Confirm it is a valid NTFS partition:

$ ntfsresize -i /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name        : /dev/loop0
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 10737385984 bytes (10738 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use       : 3265 MB (78.0%)
Collecting resizing constraints ...
You might resize at 3264757760 bytes or 3265 MB (freeing 922 MB).
Please make a test run using both the -n and -s options before real resizing!

Now it is time to edit the NTFS BPB. I use the tool hexedit which you may need to install:

$ apt-get install hexedit

Load the new disk image into hexedit:

$ hexedit hdb.raw

Skip to sector 63, which is offset 0x7E00 (press <Enter> 7E00 <Enter>):

00007E00   EB 52 90 4E  54 46 53 20  20 20 20 00  02 08 00 00  .R.NTFS    .....
00007E10   00 00 00 00  00 F8 00 00  3F 00 80 00  3F 00 00 00  ........?...?...

Move the cursor to the BPB's offset 0x1A, which is 0x7E00 + 0x1A = 0x7E1A here.

You can now over-type the hex-value 80 (128) with the correct number of heads, which in this case is FF (255):

00007E10   00 00 00 00  00 F8 00 00  3F 00 FF 00  3F 00 00 00  ........?...?...

Save the change by pressing Ctrl-X and then 'y' to confirm the write.

The image is now ready for QEMU.

$ qemu -boot c -snapshot -m 512 -hda 'hdb.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu 

If that boots successfully we can now resize the NTFS file-system:

$ ntfsresize /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name        : /dev/loop0
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 10737385984 bytes (10738 MB)
New volume size    : 10737381888 bytes (10738 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use       : 3263 MB (77.9%)
Collecting resizing constraints ...
WARNING: Every sanity check passed and only the dangerous operations left.
Make sure that important data has been backed up! Power outage or computer
crash may result major data loss!
Are you sure you want to proceed (y/[n])? y
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
Syncing device ...
Successfully resized NTFS on device '/dev/loop0'.

If you get the error:

$ ntfsresize /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
ERROR(95): Opening '/dev/loop0' as NTFS failed: Operation not supported
The NTFS journal file is unclean. Please shutdown Windows properly before
using this software! Note, if you have run chkdsk previously then boot
Windows again which will automatically initialize the journal correctly.

you should do a complete startup/shutdown sequence of Windows so the disk is marked clean:

$ losetup -d /dev/loop0
$ qemu -boot c -m 512 -hda 'hdb.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu 

All that is left to do is detach it:

$ losetup -d /dev/loop0

and convert the raw image back to a compressed qcow image. I usually explicitly make it qcow2, the later version:

$ qemu-img convert -f raw hdb.raw -O qcow2 hdb.qcow2

Make sure the qcow image boots:

$ qemu -boot c -snapshot -m 512 -hda 'hdb.qcow2' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu 

Delete the large raw images and the original qcow image:

$ rm hdb.raw hda.raw hda.qcow

The final step is to set the owner to your regular user, because the file was created as root, and exit super-user:

$ chown <username>:<username> hdb.qcow2
$ exit

I hope this helps - it took a while to perfect the process :D

How to enlarge without using twice the disk space

This method is an alternative to the one described in the previous post. It is intended for more confident users, where you are pressed for disk space or don't want to wait a long time when the file-system is copied from the original image to the new image. It is more dangerous since it is changing your only copy of the original.

Convert the compressed qcow image to raw:

$ qemu-img convert -f qcow hda.qcow -O raw hda.raw

Most of the commands will require super-user privileges so switch now to save using sudo for every command:

$ sudo su

Enlarge the existing raw disk image. If you are working on ext2 or ext3 this will create a sparse image (disk space won't actually be used until non-zero data is written to the file). The value after seek= is the size of the new disk in sectors. In this case 20971520 x 512 = 10GB:

$ dd if=/dev/zero of=hda.raw bs=512 count=0 seek=20971520
$ ls -l hda.raw
 -rw-r--r-- 1 root root 10737418240 2007-06-18 12:10 hda.raw

Attach the enlarged drive image:

$ losetup /dev/loop0 hda.raw

Use fdisk to check how many heads the disk is using:

$ fdisk -ul /dev/loop0

Disk /dev/loop0: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes

      Device Boot      Start         End      Blocks   Id  System
/dev/loop1p1   *          63     8176895     4088416+   7  HPFS/NTFS

Notice the number of heads is 255? This is the value we will have to patch into the NTFS BPB in the partition boot sector later.

Detach the drive image:

$ losetup -d /dev/loop0

Attach just the file-system partition. It starts at sector 63, which is offset 63 x 512 = 32256:

$ losetup -o32256 /dev/loop0 hda.raw

Confirm it is a valid NTFS partition:

$ ntfsresize -i /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name        : /dev/loop0
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 10737385984 bytes (10738 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use       : 3265 MB (78.0%)
Collecting resizing constraints ...
You might resize at 3264757760 bytes or 3265 MB (freeing 922 MB).
Please make a test run using both the -n and -s options before real resizing!

Now it is time to edit the NTFS BPB. I use the tool hexedit which you may need to install:

$ apt-get install hexedit

Load the new disk image into hexedit:

$ hexedit hda.raw

Skip to sector 63, which is offset 0x7E00 (press <Enter> 7E00 <Enter>):

00007E00   EB 52 90 4E  54 46 53 20  20 20 20 00  02 08 00 00  .R.NTFS    .....
00007E10   00 00 00 00  00 F8 00 00  3F 00 80 00  3F 00 00 00  ........?...?...

Move the cursor to the BPB's offset 0x1A, which is 0x7E00 + 0x1A = 0x7E1A here.

You can now over-type the hex-value 80 (128) with the correct number of heads, which in this case is FF (255):

00007E10   00 00 00 00  00 F8 00 00  3F 00 FF 00  3F 00 00 00  ........?...?...

Save the change by pressing Ctrl-X and then 'y' to confirm the write.

Detach the drive image:

$ losetup -d /dev/loop0

The image is now ready for QEMU.

$ qemu -boot c -snapshot -m 512 -hda 'hda.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu 

If that boots successfully we can now resize the NTFS file-system:

Attach the file-system image:

$ losetup -o32256 /dev/loop0 hda.raw

Resize to fill the drive-image:

$ ntfsresize /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
Device name        : /dev/loop0
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 4186538496 bytes (4187 MB)
Current device size: 10737385984 bytes (10738 MB)
New volume size    : 10737381888 bytes (10738 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use       : 3263 MB (77.9%)
Collecting resizing constraints ...
WARNING: Every sanity check passed and only the dangerous operations left.
Make sure that important data has been backed up! Power outage or computer
crash may result major data loss!
Are you sure you want to proceed (y/[n])? y
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
Syncing device ...
Successfully resized NTFS on device '/dev/loop0'.

If you get the error:

$ ntfsresize /dev/loop0
ntfsresize v1.13.1 (libntfs 9:0:0)
ERROR(95): Opening '/dev/loop0' as NTFS failed: Operation not supported
The NTFS journal file is unclean. Please shutdown Windows properly before
using this software! Note, if you have run chkdsk previously then boot
Windows again which will automatically initialize the journal correctly.

you should do a complete startup/shutdown sequence of Windows so the disk is marked clean:

$ losetup -d /dev/loop0
$ qemu -boot c -m 512 -hda 'hda.raw' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu 

All that is left to do is detach it:

$ losetup -d /dev/loop0

and convert the raw image back to a compressed qcow image. I usually explicitly make it qcow2, the later version:

$ qemu-img convert -f raw hda.raw -O qcow2 hda.qcow2

Make sure the qcow image boots:

$ qemu -boot c -snapshot -m 512 -hda 'hda.qcow2' -net nic,vlan=0 -net user,vlan=0 -localtime -soundhw es1370 -kernel-kqemu 

Delete the large raw image and the original qcow image:

$ rm hda.raw hda.qcow

The final step is to set the owner to your regular user, because the file was created as root, and exit super-user:

$ chown <username>:<username> hda.qcow2
$ exit