Mac os hypervisor framework

Mac os hypervisor framework

The xhyve hypervisor is a port of bhyve to macOS. It is built on top of Hypervisor.framework in OS X 10.10 Yosemite and higher, runs entirely in userspace, and has no other dependencies. It can run FreeBSD, some Linux distributions, and Windows 10 and may gain support for other guest operating systems in the future.

• OS X 10.10.3 Yosemite or later
• a 2010 or later Mac (i.e. a CPU that supports EPT: sysctl kern.hv_support = 1)

If you have homebrew, then simply:

The —HEAD in the brew command ensures that you always get the latest changes, even if the homebrew database is not yet updated. If for any reason you don’t want that simply do brew install xhyve .

If you have MacPorts, then simply:

MacPorts is up to date with the GitHub ref listed in the port info

The resulting binary will be in build/Release/xhyve

See below for steps to boot various OSs

bhyve is the FreeBSD hypervisor, roughly analogous to KVM + QEMU on Linux. It has a focus on simplicity.

It exposes the following peripherals to virtual machines:

• Local x(2)APIC
• IO-APIC
• 8259A PIC
• 8253/8254 PIT
• HPET
• PM Timer
• RTC
• PS/2 Keyboard and Mouse (via VNC)
• PCI
  • host bridge
  • passthrough
  • UART
  • AHCI (i.e. HDD and CD)
  • VirtIO block device
  • VirtIO networking
  • VirtIO RNG
  • Intel e1000 (aka e82545)
  • VGA/Framebuffer (exposed with a minimal VNC server)
  • XHCI USB support with one device defined — a tablet for Windows guest support

vmm.ko

The bhyve FreeBSD kernel module. Manages VM and vCPU objects, the guest physical address space and handles guest interaction with PIC, PIT, HPET, PM Timer, x(2)APIC and I/O-APIC. Contains a minimal x86 emulator to decode guest MMIO. Executes the two innermost vCPU runloops (VMX/SVM and interrupts/timers/paging). Has backends for Intel VMX and AMD SVM. Provides an ioctl and mmap API to userspace.

libvmmapi

Thin abstraction layer between the vmm.ko ioctl interface and the userspace C API.

bhyve

The userspace bhyve component (kind of a very light-weight QEMU) that executes virtual machines. Runs the guest I/O vCPU runloops. Manages ACPI, PCI and all non in-kernel devices. Interacts with vmm.ko through libvmmapi.

bhyvectl

Somewhat superfluous utility to introspect and manage the life cycle of virtual machines. Virtual machines and vCPUs can exist as kernel objects independently of a bhyve host process. Typically used to delete VM objects after use. Odd architectural choice.

bhyveload

Userspace port of the FreeBSD bootloader. This is a cumbersome workaround to bootstrap a FreeBSD guest operating system without using firmware. It creates a VM object, loads the FreeBSD kernel into guest memory, sets up the initial vCPU state and then exits. Only then a VM can be executed by bhyve.

grub2-bhyve

Performs the same function as bhyveload but is a userspace port of GRUB2. It is used to bootstrap guest operating systems other than FreeBSD, i.e. Linux, OpenBSD and NetBSD.

xhyve shares most of the code with bhyve but is architecturally very different. Hypervisor.framework provides an interface to the VMX VMCS guest state and a safe subset of the VMCS control fields, thus making userspace hypervisors without any additional kernel extensions possible. The VMX host state and all aspects of nested paging are handled by the macOS kernel, you can manage the guest physical address space simply through mapping of regions of your own address space.

xhyve is equivalent to the bhyve process but gains a subset of a userspace port of the vmm kernel module. SVM, PCI passthrough and the VMX host and EPT aspects are dropped. The vmm component provides a libvmmapi compatible interface to xhyve. Hypervisor.framework seems to enforce a strict 1:1 relationship between a host process/VM and host thread/vCPU, that means VMs and vCPUs can only be interacted with by the processes and threads that created them. Therefore, unlike bhyve, xhyve needs to adhere to a single process model. Multiple virtual machines can be created by launching multiple instances of xhyve. xhyve retains most of the bhyve command line interface.

bhyvectl, bhyveload and grub2-bhyve are incompatible with a single process model and are dropped. xhyve supports the Linux kexec protocol, a very simple and straightforward way to bootstrap a Linux kernel. It takes a bzImage and optionally initrd image and kernel parameter string as input.

xhyve can now boot an OS via EFI. The BSD-licensed TianoCore EFI built for bhyve can be used to boot Windows and other OSs.

If you want the same IP address across VM reboots, assign a UUID to a particular VM:

Optional:

If you need more advanced networking and already have a configured TAP device you can use it with:

Источник

Using the OS X 10.10 Hypervisor Framework: A Simple DOS Emulator

Since Version 10.10 (Yosemite), OS X contains Hypervisor.framework, which provides a thin user mode abstraction of the Intel VT features. It enables apps to use virtualization without the need of a kernel extension (KEXT) – which makes them compatible with the OS X App Store guidelines.

The idea is that the OS takes care of memory management (including nested paging) as well as scheduling virtual CPUs like normal threads. All we have to do is create a virtual CPU (or more!), set up all its state, assign it some memory, and run it… and then handle all “VM exits” – Intel lingo for hypervisor traps.

There is no real documentation, but the headers contain a decent amount of information. Here are some declarations from Hypervisor/hv.h:

* @abstract Creates a VM instance for the current task

* @param flags RESERVED

* @result 0 on success or error code

extern hv_return_t hv_vm_create(hv_vm_options_t flags) __HV_10_10;

* @abstract Maps a region in the virtual address space of the current task

* into the guest physical address space of the VM

* @param uva Page aligned virtual address in the current task

* @param gpa Page aligned address in the guest physical address space

* @param size Size in bytes of the region to be mapped

* @param flags READ, WRITE and EXECUTE permissions of the region

* @result 0 on success or error code

extern hv_return_t hv_vm_map(hv_uvaddr_t uva, hv_gpaddr_t gpa, size_t size,

hv_memory_flags_t flags) __HV_10_10;

* @abstract Creates a vCPU instance for the current thread

* @param vcpu Pointer to the vCPU ID (written on success)

* @param flags RESERVED

* @result 0 on success or error code

extern hv_return_t hv_vcpu_create(hv_vcpuid_t *vcpu,

hv_vcpu_options_t flags) __HV_10_10;

* @abstract Executes a vCPU

* @param vcpu vCPU ID

* @result 0 on success or error code

* Call blocks until the next VMEXIT of the vCPU

* Must be called by the owning thread

extern hv_return_t hv_vcpu_run(hv_vcpuid_t vcpu) __HV_10_10;

So let’s create a virtual machine that runs simple DOS applications in 16 bit real mode, and trap all “int” DOS system calls – similar to DOSBox.

First, we need to create a VM:

This creates a VM for the current Mach task (i.e. UNIX process). It’s implicit, so it doesn’t return anything. Then we allocate some memory and assign it to the VM:

And we need to create a virtual CPU:

Now comes the annoying part: Set up the CPU state. If the state is illegal or inconsistent, the CPU will refuse to run. You will need to refer to the Intel Manual 3C for all the context. Luckily, most virtual machines start from 16 bit real mode, and mode changes will be done by the boot loader or operating system inside the VM, so you won’t have to worry about setting up any other state than real mode state. Real mode state setup looks something like this:

After that, we should populate RAM with the code we want to execute:

…and assign the GPRs the proper initial state – including the instruction pointer, which will point to the code:

The virtual CPU is fully set up, we can now run it!

This call runs the virtual CPU (while blocking the calling thread) until its time slice expires or a “VM exit” happens. A VM exit is a hypervisor-class exception, i.e. an event in the VM that the hypervisor wants to trap. We can trap events like exceptions, certain privileged instructions (CPUID, HLT, RDTSC, RDMSR, …) and control register (CR0, CR2, CR3, CR4, …) accesses.

After hv_vcpu_run() returns, we need to read the exit reason and act upon it, and run the virtual CPU again. Here is a minimal loop to handle VM exits:

EXIT_REASON_EXT_INTR is caused by host interrupts (usually it means that the time slice is up), so we will just ignore it. EXIT_REASON_EPT_FAULT happens every time the guest accesses a page for the first time, or when the guest accesses an unmapped page – this way we can emulate MMIO. In our case, we can also ignore those.

For emulating DOS, we are catching EXIT_REASON_EXCEPTION, which is caused by the int instruction (if caught). We can get the number of the interrupt from the virtual CPU state without decoding instructions:

…and emulate the system call. We can read and write GPRs using the hv_vcpu_read_register() and hv_vcpu_write_register() calls.

hvdos – a simple DOS Emulator for OS X

The full source of hvdos, a simple DOS emulator using the OS X Hypervisor framework, is available at github.com/mist64/hvdos.

It contains an adapted version of the libcpu DOS system call library and manages to run (parts of) some .COM files. A good demo is the pkunzjr.com ZIP decompression tool.

Creating your own Hypervisor

hvdos can serve as a template for your own Hypervisor.framework experiments. It contains wrapper functions for error handling, a header that defines all Intel VT constants (taken from FreeBSD), complete 16 bit real mode initialization, as well as a few helper functions to set up the fields VMCS_PIN_BASED_CTLS, VMCS_PRI_PROC_BASED_CTLS, VMCS_SEC_PROC_BASED_CTLS and VMCS_ENTRY_CTLS properly. These are needed to define, among other things, which events cause VM exits.

You can easily add more CPUs by creating one POSIX thread per virtual CPU. For every thread, you create a virtual CPU and run a VM exit main loop.

You can for example start writing an IBM PC emulator by running Bochs BIOS and trapping I/O accesses, or running MS-DOS without BIOS by trapping BIOS int calls.

Or you could bridge an existing open source solution (QEMU, QEMU+KVM, VirtualBox, DOSBox, …) to use Hypervisor.framework…

6 thoughts on “Using the OS X 10.10 Hypervisor Framework: A Simple DOS Emulator”

Thank you very much for sharing the code! I found it especially interesting that int 0x21 would normally generate #GP (due to IDTR.limit being 0, which is below 4 * 0x22), but since #GP is masked it generates EXIT_REASON_EXCEPTION. But quite unexpectedly vectoring info doesn’t point to #GP, but to the actual interrupt. It even has type 4 (software interrupt) in bits 10:8. This is very cool, but I can’t seem to find where this particular behaviour is documented. Is this because the code is running in real mode?

VMCS_EXIT_INTR_INFO records the actual VM exit reason (#GP), a valid VMCS_IDT_VECTORING_INFO (0x21) tells you that it was indirectly caused by attempting to deliver an event through the IDT (which doesn’t exist).

A roundabout way of trapping software interrupts directly to the hypervisor 🙂

Ouch, thanks! Turns out I conflated the two (they have a similar enough format and I completely missed that Tables 24-15 and 24-16 have different headers, and that those are different fields, not merely different formats for the same field). Now it’s finally clear what’s going on. 🙂

When will this be available in the app store? Can I load a real version of Microsoft DOS?

Hello. Thank you for such a great post.
I was wondering how is it that hvdos handles real-mode execution: reading Intel Manuals, the “unrestricted guest” bit in the secondary processor-based control bitmap must be set to allow the bits in CR0 that correspond to protected mode and paging to be 0. Is this done, and if so, how? (I don’t see this “unrestricted guest” bit being set in hvdos’s code). If not, how is the VMCS actually setup so that real-mode code can be executed in the VM?
Thank you in advance.

Leave a Comment Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Источник

Mac os hypervisor framework

Copy raw contents

App Store problems

Do you see the «Your device or computer could not be verified» message when you try to login to the App Store? If yes, here are the steps to fix it.

Make sure that your wired ethernet connection is called «en0» (and not «en1» or something else). Use «ifconfig» command to verify this.

If the wired ethernet connection is not called «en0», then then go to Network in System Preferences and delete all the devices, and apply the changes. Next, go to the console and type in sudo rm /Library/Preferences/SystemConfiguration/NetworkInterfaces.plist . Finally reboot, and then use the App Store without problems.

This fix was found by Glnk2012 of https://www.tonymacx86.com/ site.

Also tweaking the smbios.plist file can help (?).

Change resolution in OpenCore

Ensure that the OVMF resolution is set equal to resolution set in your OpenCore qcow2 file (default is 1024×768). This can be done via the OVMF menu, which you can reach with a press of the ESC button during the OVMF boot logo (before OpenCore boot screen appears). In the OVMF menu settings, set Device Manager -> OVMF Platform Configuration -> Change Preferred Resolution for Next Boot to the desired value (default is 1024×768). Commit changes and exit the OVMF menu.

Note: The macOS VM’s resolution can be changed via Settings -> Displays option easily.

GPU passthrough notes

These steps will need to be adapted for your particular setup. A host machine with IOMMU support is required. Consult this Arch Wiki article for general-purpose guidance and details.

I am running Ubuntu 20.04.2 LTS on Intel i5-6500 + ASUS Z170-AR motherboard + AMD RX 570 GPU (May 2021).

Blacklist the required kernel modules.

Find details of the PCIe devices to passthrough.

Enable IOMMU support and configure VFIO.

Append the given line to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub .

Intel CPU Systems

iommu=pt intel_iommu=on vfio-pci.ids=1002:67df,1002:aaf0 kvm.ignore_msrs=1 video=vesafb:off,efifb:off

AMD CPU Systems

Tweak module configuration a bit according to the following output (thanks to Mathias Hueber).

Update GRUB, initramfs, and then reboot.

In the BIOS setup, set the Primary Display to IGFX (onboard graphics).

Verify that the IOMMU is enabled, and vfio-pci is working as expected. Verify that the expected devices are using vfio-pci as their kernel driver

Fix permisions for the /dev/vfio/1 device (modify as needed):

Open /etc/security/limits.conf file and add the following lines:

Thanks to Heiko Sieger for this solution.

Confirm the contents of boot-passthrough.sh and run it to boot macOS with GPU passthrough.

To reuse the keyboard and mouse devices from the host, setup «Automatic login» in System Preferences in macOS and configure Synergy software.

USB passthrough notes

These steps will need to be adapted for your particular setup.

Isolate the passthrough PCIe devices with vfio-pci, with the help of lspci -nnk command.

Add 1b21:1242 to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub file in the required format. See GPU passthrough notes (above) for details.

Update initramfs, and then reboot.

Use the helper scripts to isolate the USB controller.

Add -device vfio-pci,host=03:00.0,bus=pcie.0 \ line to boot-passthrough.sh .

Boot the VM, and devices attached to the ASMedia USB controller should just work under macOS.

I installed «synergy-v1.8.8-stable-MacOSX-x86_64.dmg» on the macOS guest and configured it as a client.

For automatically starting Synergy on macOS, add Synergy to «Login Items», System Preferences -> Users & Groups -> Select your user account -> Login Items -> Add a login item

On the Linux host machine, install «synergy-v1.8.8-stable-Linux-x86_64.deb» or newer, configure

/.synergy.conf and run synergys command.

The included .synergy.conf will need to be adapted according to your setup.

Virtual Sound Device

Warning: The OpenCore distribution that comes with OSX-KVM already has VoodooHDA OC . Do NOT mix VoodooHDA with AppleALC. You may want to consider HDA passthrough if it is practical or use HDMI audio instead

Note: The emulated sound output can be choppy, and distorted. Use Sound Card / USB Sound Card passthrough instead.

Note: It seems that playback of Flash videos requires an audio device to be present.

Building QEMU from source

Connect iPhone / iPad to macOS guest

iDevices can be passed through in two ways: USB or USB OTA.

VFIO USB Passthrough:

Exposing AES-NI instructions to macOS

Add +aes argument to the -cpu option in boot-macOS.sh file.

Other host CPU features can be similarly exposed to the macOS guest.

The following command can be used on macOS to verify that AES-NI instructions are exposed,

On machines with OpenSSL installed, the following two commands can be used to check AES-NI performance,

Exposing AVX and AVX2 instructions to macOS

Exposing AVX and AVX2 instructions to macOS requires support for these instructions on the host CPU.

The boot-clover.sh script already exposes AVX and AVX2 instructions to the macOS guest by default. Modify or comment out the MY_OPTIONS line in boot-clover.sh file in case you are having problems.

To enable AVX2, do the following change,

Clover boot menu -> Options -> Binaries patching -> Fake CPUID -> 0x0306C0 # for Haswell

For details, see this wiki page.

Once enabled, the following commands can be used to confirm the presence of AVX and AVX2 instructions on the macOS guest.

Enabling Hypervisor.Framework (Nested Virtualization / Docker for Mac / Android Emulator / etc)

Docker for Mac, the Android Emulator and other virtualization products require nested virtualization in the form of the Hypervisor Framework to work on macOS.

Use the sysctl kern.hv_support (output 1 is good) command to check if Hypervisor is enabled within the macOS VM.

If Hypervisor is not enabled, check that you have the required CPU features. Run the sysctl -a | grep machdep.cpu.features command and look for the presence of VMX flag.

If the VMX flag is missing, use the following steps to enable it:

Make sure that kvm_intel module is loaded properly. This is documented in our main documentation.

Make sure the VM is booted with VMX support passed through using one of the two below strategies:

You may add vmx,rdtscp arguments to the -cpu option in boot-macOS.sh file (easier option).

You may add +vmx, to the front of MY_OPTIONS in the boot script while changing -cpu Penryn to -cpu Skylake-Client or any other suitable supported CPU.

Note: Host CPU passthrough is troublesome and not generally recommended.

Note: You may need to Reset NVRAM on the next reboot, but after that you should see a 1 when you re-check sysctl kern.hv_support .

Using virtio-blk-pci with macOS

Newer macOS (namely Mojave+) have support for some virtio drivers.

This can be enabled by applying the following change to boot-macOS-NG.sh to get some performance gain.

Permission problems with libvirt / qemu?

Extract .pkg files

http://mackyle.github.io/xar/ is unmaintained and may fail for many .pkg files.

Use a modern version of 7-Zip instead.

QEMU quits with gtk initialization failed

Append the display=none argument to your QEMU execution script (this has already been done for boot-passthrough.sh )

ISO/DMG ( createinstallmedia generated) install medium not detected

Attach physical drive to QEMU VM

Note: If using NVMe, passing the controller may be a better option then passing it as a block device

Run ls -la /dev/disk/by-id/ to get the unique mapping for the device you want to attach to the VM (like sda , sdb , nvme0n1 , while you can attach only a partition like sda1 , this is not recommended)

Then edit your QEMU launch script and add these lines (adapt to it your hardware), then launch the script using sudo (because you cannot write to a block device without root permissions)

Run the Virtual Machine on Boot

Edit your QEMU launch script and set the absolute path of OSX-KVM as the value of REPO_PATH

Edit /etc/rc.local and add the absolute path of the script (with or without sudo depending on your needs) to the bottom of the script.

Setup SSH for internal remote access

Presuming your network interface has a statically defined internal IP (on Ubuntu).

Consider using CMMChris’s RadeonBoost.kext for the RX480, RX580, RX590 and Radeon VII GPUs.

USB passthrough notes

USB 3.0 flash drive

The following USB configuration works for usb passthrough of a USB 3.0 flash drive to Fedora 25 guest.

The following USB configuration works for usb passthrough of a Moto G3 phone to Fedora 25 guest.

The following USB configuration works for usb passthrough of a CoolerMaster keyboard to macOS Sierra guest!

Virtual USB disk

The following USB configuration works for attaching a virtual USB disk to macOS Sierra guest. Use «qemu-img» to create «disk.raw» virtual disk.

However USB passthrough of EHCI, and XHCI (USB 3.0) devices does not work with macOS Sierra. See https://bugs.launchpad.net/qemu/+bug/1509336 for confirmation. According to this bug report, USB passthrough does not work with versions >= Mac OS X El Capitan guests.

It seems that this problem can be fixed by using OVMF + Clover.

Update: OVMF + Clover doesn’t help. It seems that macOS is missing the required drivers for the EHCI, and XHCI controllers that are exposed by QEMU.

Generate macOS Mojave / Catalina installation ISO.

This step currently needs to be run on an existing macOS system.

Tweaks for macOS

Disable Energy Saver in System Preferences .

Disable Screen Saver in System Preferences -> Desktop & Screen Saver .

Turn off indexing using the following command.

Enable Remote Login (aka SSH) via System Preferences -> Sharing .

Snapshot Debugging Tips

Get savevm to work:

Ensure that you have plenty of free space in /var/tmp and / .

To use a separate storage location for storing snapshots, use the following trick (from zimbatm ):

‘Fix’ weird boot problems

‘Fix’ time drift problems

Run the following command periodically from root’s crontab:

Pass through all CPU cores / threads

macOS requires a core count that is a power of 2, but some modern CPUs have odd counts — like 6 cores and 12 threads.

So how can we harness the entire CPU in the VM?

There are strategies that mix smp/sockets/cores/threads/maxcpu arguments and use odd socket counts to arrive at even core counts, and even let you specify that some of the cores are hyperthreaded.

Specifically for the case of an Intel i7 processor with 6 cores and 12 total threads, altering the boot script to contain these variables and the following modified SMP line results in a full CPU core/thread pass through:

The -smp line should read something like the following:

Trouble with iMessage?

Enable rc.local functionality on moden Ubuntu versions

Create /etc/rc.local file with the following content, if it doesn’t exist:

Make this file executable, if required:

Create /etc/systemd/system/rc-local.service with the following content, if it doesn’t exist:

Enable rc.local systemd service:

These notes are borrowed from various multiple internet resources.

Источник