CSE 522S: Studio 14

Virtual Machine Performance with QEMU and KVM

A container is not a virtual machine in the traditional sense. Virtual machines contain a complete operating system, running on top of a hypervisor that is managed by the underlying host operating system. The biggest advantage to hardware virtualization is that it is easy to run many virtual machines with radically different operating systems on a single host. With containers, both the host and the containers share the same kernel. This means that containers utilize fewer system resources but must be based on the same underlying operating system (i.e., Linux).

—Sean P. Kane & Karl Matthias, Docker Up & Running, 2nd Edition

Virtual machines provide machine abstractions that make workload consolidation, isolation, and even full machine emulation possible. Today's studio will focus on the QEMU (pronunciation varies — I like "Q-em-you") machine emulator and the KVM hypervisor, showing you how to boot virtual machines and explore the differences between software emulation and hardware virtualization in Linux. Most modern day CPUs, including the Raspberry Pi's Cortex A53 CPU, implement virtualization extensions that can drastically improve the performance of virtualized applications. We will analyze the performance of the ARM core's second level address translation (SLAT) mechanism that allows the hardware to not only translate guest virtual to guest physical addresses, but also to translate guest physical to host physical addresses, all without requiring hypervisor intervention.

In this studio, you will:

  1. Use QEMU to boot virtual machines on the Raspberry Pi
  2. Measure the time it takes to boot a hardware virtualized VM
  3. Boot a SLAT-enabled virtual machine on the Raspberry Pi
  4. Measure the performance of two different virtualized workloads
  5. Compare these measurements with the same workloads running natively on the Raspberry Pi

Please complete the required exercises below. We encourage you to please work in groups of 2 or 3 people on each studio (and the groups are allowed to change from studio to studio) though if you would prefer to complete any studio by yourself that is allowed.

As you work through these exercises, please record your answers, and when finished upload them along with the relevant source code to the appropriate spot on Canvas.

Make sure that the name of each person who worked on these exercises is listed in the first answer, and make sure you number each of your responses so it is easy to match your responses with each exercise.

Required Exercises

  1. As the answer to the first exercise, please list the names of the people who worked together on this studio.

  2. The KVM hypervisor is a features of the Linux kernel that provides hardware virtualization, and may be leveraged by QEMU to improve virtual machine performance. The Raspberry Pi OS distribution that you have been using does not have KVM compiled into the Linux kernel. As such, for today's studio (and the remaining studios for this semester), you will be running a different Linux distribution directly on your Raspberry Pi.

    This will involve writing a new image onto a MicroSD card. Because you are likely using your Raspberry Pi for your course project or other studio work, you may need another MicroSD card for this studio. Alternatively, you can use your laptop or desktop to create a full image backup of your current MicroSD card. Even if you have a second card available, it's a good idea to create a backup image in case your card gets lost or stolen.

    The procedure varies by the operating system you're using on your laptop or desktop. The instructions here are based on this page, which provides instructions for installing an image onto your Raspberry Pi.


    1. Run lsblk -p to see the block devices connected to your machine.
    2. Connect the MicroSD card to your laptop/desktop.
    3. Run lsblk -p again, noting the new entry corresponding to the MicroSD card.
    4. Make sure the card was not mounted (the MOUNTPOINT column should be empty). If it was mounted, unmount it.
    5. Run the command:
      sudo dd if=/dev/sdX of=backup.img bs=4M conv=fsync
      (replace sdX with the device entry for your SD card).

    Mac OS

    1. Run diskutil list to see the disks connected to your machine.
    2. Connect the MicroSD card to your laptop/desktop.
    3. Run diskutil list again, noting the new entry corresponding to the MicroSD card.
    4. Unmount the SD card with the command:
      diskutil unmountDisk /dev/diskX
      (replace diskX with the device entry for your SD card).
    5. Run the command:
      sudo dd bs=1m if=/dev/rdiskX of=backup.img; sync
      (note that here, we use rdiskX instead of diskX, indicating raw disk access, which speeds up the copy process)


    1. Connect the MicroSD card to your laptop/desktop.
    2. Download, install, and open the Win32 Disk Imager.
    3. Select the device letter corresponding to your MicroSD card. The card should have two partitions, and therefore two drive letters. Back up the drive that shows as unformatted (not the boot disk). This is formatted, but using a filesystem that Windows does not support.
    4. In the "Image File" area, enter the location where you would like to save your backup file. Name it with the extension ".img".
    5. Click the Read button.

    Please go ahead and make a backup of your SD card for this exercise. As the answer to this exercise, please (1) indicate how long the backup took to complete, and (2) write the size of the backup ".img" file you created.

  3. Now, you are ready to install a new Linux distribution onto your Raspberry Pi. Download and install the Raspberry Pi Imager. Connect a MicroSD card to your laptop/desktop, then run the software.

    Click "CHOOSE OS" > "Other general-purpose OS" > "Ubuntu," then select the 64-bit version of the Ubuntu Server 20.xx.x LTS. Then click "CHOOSE STORAGE" and select your MicroSD card from the list.

    Once the card has been written, eject it from your computer, put it into your Raspberry Pi, and boot it. The Ubuntu Server distribution we're using has SSH enabled by default, so you can log into it remotely if you do not have a keyboard, monitor, and mouse.

    The username and password are both ubuntu; note that you will be prompted to change your password when you first log in. If you are connected over SSH, your session will disconnect when you change the password. You can connect again and authenticate with your new password.

    Once you have set the password, you will want to change the hostname to something unique (similarly to when you originally set up your Raspberry Pi). Open a terminal window as root (e.g., sudo su), then (1) set the hostname with the hostname command, then (2) have the new hostname persist across reboots:

    hostname > /etc/hostname

    Next, you'll need to disable the automatic update feature in Ubuntu Server, to keep it from preventing you from installing new software packages. This can be done easily with the following commands:

    sudo apt-get update
    sudo systemctl mask apt-daily.service apt-daily-upgrade.service
    sudo systemctl disable apt-daily.service apt-daily.timer apt-daily-upgrade apt-daily-upgrade.timer

    Once you've done this, reboot your Raspberry Pi (sudo reboot now).

    Once the Raspberry Pi is back on, issue the uname -a command. As the answer to this exercise, please show its output.

  4. For this exercise, you will run an Ubuntu virtual machine on top of your Ubuntu OS using QEMU! First, install QEMU and a related utility:

    sudo apt-get install qemu qemu-utils qemu-system qemu-kvm mkisofs

    The remainder of this exercise is based on, though does not exactly follow, the first post on this page: https://www.raspberrypi.org/forums/viewtopic.php?t=224057. If you are interested in learning more about what each step is doing, feel free to read the post. All steps should be performed on your Raspberry Pi running Ubuntu Server.

    1. Download an EFI BIOS for QEMU:
      wget http://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/latest/QEMU-AARCH64/RELEASE_CLANG35/QEMU_EFI.fd

    2. Download an image for Ubuntu Bionic server's ARM 64-bit release:
      wget https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-arm64.img

    3. Make a copy of the image in QEMU's copy on write format:
      qemu-img create -f qcow2 -b bionic-server-cloudimg-arm64.img bionic-image-01.img

    4. You can configure settings for the Ubuntu OS in the virtual machine before it's even launched. This will simplify the login process, and also keep it from requiring a network connection on first boot. Create the following two files:

    5. meta-data


      instance-id: kvm-bionic-01
      local-hostname: kvm-bionic

    6. user-data


      password: cse522studio
      chpasswd: { expire: False }
      ssh_pwauth: True

      (if you like, you can change the password to something else of your choosing)

    7. Create an Ubuntu installation disk image that packages the configuration data:
      mkisofs -o seed-kvm-bionic-01.iso -V cidata -J -rock user-data meta-data

    Now, you are ready to launch your virtual machine!

    Run the following command:

    sudo qemu-system-aarch64 -M virt -cpu host -m 256M -smp 2 -nographic -bios QEMU_EFI.fd -cdrom seed-kvm-bionic-01.iso -drive if=none,file=bionic-image-01.img,id=hd0 -device virtio-blk-device,drive=hd0 -device virtio-net-device,netdev=vmnic -netdev user,id=vmnic -accel kvm

    (If you see an error displayed on the boot screen, you can press ENTER to continue.)

    Once the virtual machine boots (which may take a minute or two), log in using the username ubuntu and the password you set in the user-data configuration file. Issue the uname -a command. As the answer to this exercise, please (1) show the output of that command, and (2) explain how it is similar or different from the output in the previous exercise.

    Note: if you need to exit your virtual machine without waiting for it to boot, then gracefully shutting it down, you can do so by issuing a special keyboard command to QEMU. Press "CTRL+a", then press the "x" key (no longer holding CTRL or "a").

  5. Teardown the virtual machine by typing shutdown -h now in the guest console. Open up a separate terminal window, issue the following command:

    time while [ 1 -eq 1 ]; do sleep 10; done

    Now, restart the VM by running the same QEMU command line to boot the VM, and, once you have logged in to the VM, type "Ctrl+C" in the window in which you ran the time command. As the answer to this exercise, list the amount of time it took to boot the VM.

    While this may take a minute or two, this is likely much faster than if you ran the VM using emulation, instead of hardware virtualization! Notice the "-accel kvm" and "-cpu host" options from the QEMU command line. On my Raspberry Pi 3 Model B+, I waited 20 minutes for VM to boot before getting impatient and forcing QEMU to quit!

    When you are finished, shut down the virtual machine again.

  6. Download two C applications onto your Raspberry Pi:



    (hint: use wget to download them directly)

    Read over these two programs. As the answer to this exercise, explain in just a couple of sentences what each program is doing and whether their memory access patterns are likely to have good locality or not.

  7. Compile these programs:

    gcc random_access.c -o random_access
    gcc dense_mm.c -o dense_mm

    and then, using the time command, measure the time it takes to run each of these programs. Run dense_mm with an input of 512, and run random_access with an input of 64. As the answer to this exercise, show the output from each program and the amount of time each took to run.

  8. Finally, you will run and time these programs in your virtual machine. Run the virtual machine as you did in the earlier exercise. Once it has booted, and you have logged in, (1) download the same two C programs in the VM, and (2) install the gcc compiler with:

    sudo apt-get install gcc

    Then, compile these programs in the guest and, again using the time command, measure the time it takes to run each of these programs. Run dense_mm with an input of 512, and run random_access with an input of 64.

    As the answer to this exercise, show the amount of time each program took to run, and compare this with the amount of time each took to run natively on the host OS. If one program experiences more or less overhead than the other when running virtualized, explain what it is about the program's workload that likely explains the performance difference.

  9. Things to Turn In:

Page updated Thursday, March 24, 2022, by Marion Sudvarg and Chris Gill.