Getting started guide for Intel NUC

The Intel® NUC (NUC6CAYH) is the primary tested platform for ACRN development, and its setup is described below.

Hardware setup

Two Apollo Lake Intel platforms, described in Supported Hardware, are currently supported for ACRN development:

Firmware update on the NUC

You may need to update to the latest UEFI firmware for the NUC hardware. Follow these BIOS Update Instructions for downloading and flashing an updated BIOS for the NUC.

Software setup

Set up a Clear Linux Operating System

Currently, an installable version of ARCN does not exist. Therefore, you need to setup a base Clear Linux OS and you’ll build and bootstrap ACRN on your platform. You’ll need a network connection for your platform to complete this setup.

Note

ACRN requires Clear Linux version 22140 or newer. The instructions below have been validated with version 22140 and need some adjustment to work with newer versions. You will see a note when the instruction needs to be adjusted.

  1. Download the compressed Clear installer image from https://download.clearlinux.org/releases/22140/clear/clear-22140-installer.img.xz and follow the Clear Linux installation guide as a starting point for installing Clear Linux onto your platform. Follow the recommended options for choosing an Automatic installation type, and using the platform’s storage as the target device for installation (overwriting the existing data and creating three partitions on the platform’s storage drive).

  2. After installation is complete, boot into Clear Linux, login as root, and set a password.

  3. Clear Linux is set to automatically update itself. We recommend that you disable this feature to have more control over when the updates happen. Use this command to disable the autoupdate feature:

    # swupd autoupdate --disable
    
  4. Use the swupd bundle-add command and add these Clear Linux bundles:

    # swupd bundle-add vim network-basic service-os kernel-pk
    
    Table 1 Clear Linux bundles
    Bundle Description
    vim vim text editor
    network-basic Run network utilities and modify network settings
    service-os Add the acrn hypervisor, the acrn devicemodel and Service OS kernel
    kernel-pk Run the Intel “PK” kernel(product kernel source) and enterprise-style kernel with backports

Add the ACRN hypervisor to the EFI Partition

In order to boot the ACRN SOS on the platform, you’ll need to add it to the EFI partition. Follow these steps:

  1. Mount the EFI partition and verify you have the following files:

    # mount /dev/sda1 /mnt
    
    # ls -1 /mnt/EFI/org.clearlinux
    bootloaderx64.efi
    kernel-org.clearlinux.native.4.16.6-563
    kernel-org.clearlinux.pk414-sos.4.14.34-28
    kernel-org.clearlinux.pk414-standard.4.14.34-28
    loaderx64.efi
    

    Note

    The Clear Linux project releases updates often, sometimes twice a day, so make note of the specific kernel versions (*-sos and *-standard) listed on your system, as you will need them later.

    Note

    The EFI System Partition (ESP) may be different based on your hardware. It will typically be something like /dev/mmcblk0p1 on platforms that have an on-board eMMC or /dev/nvme0n1p1 if your system has a non-volatile storage media attached via a PCI Express (PCIe) bus (NVMe).

  2. Put the acrn.efi hypervisor application (included in the Clear Linux release) on the EFI partition with:

    # mkdir /mnt/EFI/acrn
    # cp /usr/lib/acrn/acrn.efi /mnt/EFI/acrn/
    
  3. Configure the EFI firmware to boot the ACRN hypervisor by default

    The ACRN hypervisor (acrn.efi) is an EFI executable loaded directly by the platform EFI firmware. It then in turns loads the Service OS bootloader. Use the efibootmgr utility to configure the EFI firmware and add a new entry that loads the ACRN hypervisor.

    # efibootmgr -c -l "\EFI\acrn\acrn.efi" -d /dev/sda -p 1 -L ACRN
    

    Note

    Be aware that a Clearlinux update that includes a kernel upgrade will reset the boot option changes you just made. A Clearlinux update could happen automatically (if you have not disabled it as described above), if you later install a new bundle to your system, or simply if you decide to trigger an update manually. Whenever that happens, double-check the platform boot order using efibootmgr -v and modify it if needed.

    The ACRN hypervisor (acrn.efi) accepts two command-line parameters that tweak its behaviour:

    1. bootloader=: this sets the EFI executable to be loaded once the hypervisor is up and running. This is typically the bootloader of the Service OS and the default value is to use the Clearlinux bootloader, i.e.: \EFI\org.clearlinux\bootloaderx64.efi.
    2. uart=: this tells the hypervisor where the serial port (UART) is found or whether it should be disabled. There are three forms for this parameter:
      1. uart=disabled: this disables the serial port completely
      2. uart=mmio@<MMIO address>: this sets the serial port MMIO address
      3. uart=port@<port address>: this sets the serial port address

    Here is a more complete example of how to configure the EFI firmware to load the ACRN hypervisor and set these parameters.

    # efibootmgr -c -l "\EFI\acrn\acrn.efi" -d /dev/sda -p 1 -L "ACRN Hypervisor" \
          -u "bootloader=\EFI\org.clearlinux\bootloaderx64.efi uart=mmio@0x9141e000"
    
  4. Create a boot entry for the ACRN Service OS by copying a provided acrn.conf and editing it to account for the kernel versions noted in a previous step.

    It must contain these settings:

    Setting Description
    title Text to show in the boot menu
    linux Linux kernel for the Service OS (*-sos)
    options Options to pass to the Service OS kernel (kernel parameters)

    A starter acrn.conf configuration file is included in the Clear Linux release and is also available in the acrn-hypervisor/hypervisor GitHub repo as acrn.conf as shown here:

    Code Block 1 hypervisor/bsp/uefi/clearlinux/acrn.conf
    title The ACRN Service OS
    linux   /EFI/org.clearlinux/kernel-org.clearlinux.pk414-sos.4.14.23-19
    options pci_devices_ignore=(0:18:2) maxcpus=1 console=tty0 console=ttyS0 i915.nuclear_pageflip=1 root=PARTUUID=<UUID of rootfs partition> rw rootwait ignore_loglevel no_timer_check consoleblank=0 i915.tsd_init=7 i915.tsd_delay=2000 i915.avail_planes_per_pipe=0x01010F i915.domain_plane_owners=0x011111110000 i915.enable_guc_loading=0 i915.enable_guc_submission=0 i915.enable_preemption=1 i915.context_priority_mode=2 i915.enable_gvt=1 i915.enable_initial_modeset=1 hvlog=2M@0x1FE00000
    

    On the platform, copy the acrn.conf file to the EFI partition we mounted earlier:

    # cp /usr/share/acrn/samples/nuc/acrn.conf /mnt/loader/entries/
    

    You will need to edit this file to adjust the kernel version (linux section) and also insert the PARTUUID of your /dev/sda3 partition (root=PARTUUID=<><UUID of rootfs partition>) in the options section.

    Use blkid to find out what your /dev/sda3 PARTUUID value is.

    Note

    It is also possible to use the device name directly, e.g. root=/dev/sda3

  5. Add a timeout period for Systemd-Boot to wait, otherwise it will not present the boot menu and will always boot the base Clear Linux

    # clr-boot-manager set-timeout 20
    # clr-boot-manager update
    
  6. Reboot and select “The ACRN Service OS” to boot, as shown in Figure 11:

    ../_images/gsg-bootmenu.png

    Figure 11 ACRN Service OS Boot menu

  7. After booting up the ACRN hypervisor, the Service OS will be launched automatically by default, as shown in Figure 12:

    ../_images/gsg-sos-console.png

    Figure 12 Service OS Console

    Note

    You may need to hit Enter to get a clean login prompt

  8. From here you can login as root using the password you set previously when you installed Clear Linux.

ACRN Network Bridge

ACRN bridge has been setup as a part of systemd services for device communication. The default bridge creates acrn_br0 which is the bridge and acrn_tap0 as an initial setup. The files can be found in /usr/lib/systemd/network. No additional setup is needed since systemd-networkd is automatically enabled after a system restart.

Set up Reference UOS

  1. On your platform, download the pre-built reference Clear Linux UOS image into your (root) home directory:

    # cd ~
    # curl -O https://download.clearlinux.org/releases/22140/clear/clear-22140-kvm.img.xz
    

    Note

    In case you want to use or try out a newer version of Clear Linux as the UOS, you can download the latest from http://download.clearlinux.org/image. Make sure to adjust the steps described below accordingly (image file name and kernel modules version).

  2. Uncompress it:

    # unxz clear-22140-kvm.img.xz
    
  3. Deploy the UOS kernel modules to UOS virtual disk image (note: you’ll need to use the same standard image version number noted in step 1 above):

    # losetup -f -P --show /root/clear-22140-kvm.img
    # mount /dev/loop0p3 /mnt
    # cp -r /usr/lib/modules/4.14.34-28.pk414-standard /mnt/lib/modules/
    # umount /mnt
    # sync
    
  4. Edit and Run the launch_uos.sh script to launch the UOS.

    A sample launch_uos.sh is included in the Clear Linux release, and is also available in the acrn-hypervisor/devicemodel GitHub repo (in the samples folder) as shown here:

    Code Block 2 devicemodel/samples/nuc/launch_uos.sh
    #!/bin/bash
    
    function launch_clear()
    {
    vm_name=vm$1
    
    
    #check if the vm is running or not
    vm_ps=$(pgrep -a -f acrn-dm)
    result=$(echo $vm_ps | grep "${vm_name}")
    if [[ "$result" != "" ]]; then
      echo "$vm_name is running, can't create twice!"
      exit
    fi
    
    #for memsize setting
    mem_size=1000M
    
    # make sure there is enough 2M hugepages in the pool
    echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    
    acrn-dm -A -m $mem_size -c $2 -s 0:0,hostbridge -s 1:0,lpc -l com1,stdio \
      -s 2,pci-gvt -G "$3" \
      -s 5,virtio-console,@pty:pty_port \
      -s 6,virtio-hyper_dmabuf \
      -s 3,virtio-blk,/root/clear-21260-kvm.img \
      -s 4,virtio-net,tap0 \
      -k /usr/lib/kernel/org.clearlinux.pk414-standard.4.14.23-19 \
      -B "root=/dev/vda3 rw rootwait maxcpus=$2 nohpet console=tty0 console=hvc0 \
      console=ttyS0 no_timer_check ignore_loglevel log_buf_len=16M \
      consoleblank=0 tsc=reliable i915.avail_planes_per_pipe=$4 \
      i915.enable_hangcheck=0 i915.nuclear_pageflip=1 i915.enable_guc_loading=0 \
      i915.enable_guc_submission=0" $vm_name
    }
    
    launch_clear 2 1 "64 448 8" 0x070F00 clear
    

    Note

    In case you have downloaded a different Clear Linux image than the one above (clear-22140-kvm.img.xz), you will need to modify the Clear Linux file name and version number highlighted above (the -s 3,virtio-blk argument) to match what you have downloaded above. Likewise, you may need to adjust the kernel file name on the second line highlighted (check the exact name to be used using: ls /usr/lib/kernel/org.clearlinux*-standard*).

    By default, the script is located in the /usr/share/acrn/samples/nuc/ directory. You can edit it there, and then run it to launch the User OS:

    # cd /usr/share/acrn/samples/nuc/
    # ./launch_uos.sh
    
  5. At this point, you’ve successfully booted the ACRN hypervisor, SOS, and UOS:

    ../_images/gsg-successful-boot.png

Device Manager memory allocation mechanism

The ACRN Device Manager (DM) virtual memory allocation uses the HugeTLB mechanism. (You can read more about HugeTLB in the linux kernel for more information about how this mechanism works.)

For hugeTLB to work, you’ll need to reserve huge pages:

  • For a (large) 1GB huge page reservation, add hugepagesz=1G hugepages=reserved_pg_num (for example, hugepagesz=1G hugepages=4) to the SOS cmdline in acrn.conf (for EFI)

  • For a (smaller) 2MB huge page reservation, after the SOS starts up, run the command:

    echo reserved_pg_num > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
    

Note

You can use 2M reserving method to do reservation for 1G page size, but it may fail. For an EFI platform, you may skip 1G page reservation by using a 2M page, but make sure your huge page reservation size is large enough for your usage.

Build ACRN from Source

If you would like to build ACRN hypervisor and device model from source, follow these steps.

Install build tools and dependencies

ARCN development is supported on popular Linux distributions, each with their own way to install development tools:

Note

ACRN uses menuconfig, a python3 text-based user interface (TUI) for configuring hypervisor options and using python’s kconfiglib library.

  • On a Clear Linux development system, install the os-clr-on-clr bundle to get the necessary tools:

    $ sudo swupd bundle-add os-clr-on-clr
    $ sudo swupd bundle-add python3-basic
    $ sudo pip3 install kconfiglib
    
  • On a Ubuntu/Debian development system:

    $ sudo apt install gcc \
         git \
         make \
         gnu-efi \
         libssl-dev \
         libpciaccess-dev \
         uuid-dev \
         libsystemd-dev \
         libevent-dev \
         libxml2-dev \
         libusb-1.0-0-dev \
         python3 \
         python3-pip
    $ sudo pip3 install kconfiglib
    

    Note

    Ubuntu 14.04 requires libsystemd-journal-dev instead of libsystemd-dev as indicated above.

  • On a Fedora/Redhat development system:

    $ sudo dnf install gcc \
         git \
         make \
         findutils \
         gnu-efi-devel \
         libuuid-devel \
         openssl-devel \
         libpciaccess-devel \
         systemd-devel \
         libxml2-devel \
         libevent-devel \
         libusbx-devel \
         python3 \
         python3-pip
    $ sudo pip3 install kconfiglib
    
  • On a CentOS development system:

    $ sudo yum install gcc \
            git \
            make \
            gnu-efi-devel \
            libuuid-devel \
            openssl-devel \
            libpciaccess-devel \
            systemd-devel \
            libxml2-devel \
            libevent-devel \
            libusbx-devel \
            python34 \
            python34-pip
    $ sudo pip3 install kconfiglib
    

    Note

    You may need to install EPEL for installing python3 via yum for CentOS 7. For CentOS 6 you need to install pip manually. Please refer to https://pip.pypa.io/en/stable/installing for details.

Build the hypervisor, device model and tools

The acrn-hypervisor repository has three main components in it:

  1. The ACRN hypervisor code located in the hypervisor directory
  2. The ACRN devicemodel code located in the devicemodel directory
  3. The ACRN tools source code located in the tools directory

You can build all these components in one go as follows:

$ git clone https://github.com/projectacrn/acrn-hypervisor
$ cd acrn-hypervisor
$ make

The build results are found in the build directory.

Note

if you wish to use a different target folder for the build artefacts, set the O (that is capital letter ‘O’) to the desired value. Example: make O=build-uefi PLATFORM=uefi.

You can also build these components individually. The following steps assume that you have already cloned the acrn-hypervisor repository and are using it as the current working directory.

  1. Build the ACRN hypervisor.

    $ cd hypervisor
    $ make PLATFORM=uefi
    

    The build results are found in the build directory.

  2. Build the ACRN device model (included in the acrn-hypervisor repo):

    $ cd ../devicemodel
    $ make
    

    The build results are found in the build directory.

  3. Build the ACRN tools (included in the acrn-hypervisor repo):

    $ cd ../tools
    $ for d in */; do make -C "$d"; done
    

Follow the same instructions to boot and test the images you created from your build.

Generate the hypervisor configurations

The ACRN hypervisor leverages Kconfig to manage configurations, powered by Kconfiglib. A default configuration is generated based on the platform you have selected via the PLATFORM= command line parameter. You can make further changes to that default configuration to adjust to your specific requirements.

To generate hypervisor configurations, you need to build the hypervisor individually. The following steps generate a default but complete configuration, based on the platform selected, assuming that you are under the top-level directory of acrn-hypervisor. The configuration file, named .config, can be found under the target folder of your build.

$ cd hypervisor
$ make defconfig PLATFORM=uefi

The PLATFORM specified is used to select a defconfig under arch/x86/configs/. The other command-line based options (e.g. RELEASE) take no effects when generating a defconfig.

Modify the hypervisor configurations

To modify the hypervisor configurations, you can either edit .config manually, or invoke a TUI-based menuconfig, powered by kconfiglib, by executing make menuconfig. As an example, the following commands, assuming that you are under the top-level directory of acrn-hypervisor, generate a default configuration file for UEFI, allow you to modify some configurations and build the hypervisor using the updated .config.

$ cd hypervisor
$ make defconfig PLATFORM=uefi
$ make menuconfig              # Modify the configurations per your needs
$ make                         # Build the hypervisor with the new .config

Note

Menuconfig is python3 only.

Refer to the help on menuconfig for a detailed guide on the interface.

$ pydoc3 menuconfig

Create a new default configuration

Currently the ACRN hypervisor looks for default configurations under hypervisor/arch/x86/configs/<PLATFORM>.config, where <PLATFORM> is the specified platform. The following steps allow you to create a defconfig for another platform based on a current one.

$ cd hypervisor
$ make defconfig PLATFORM=uefi
$ make menuconfig         # Modify the configurations
$ make savedefconfig      # The minimized config reside at build/defconfig
$ cp build/defconfig arch/x86/configs/xxx.config

Then you can re-use that configuration by passing the name (xxx in the example above) to ‘PLATFORM=’:

$ make defconfig PLATFORM=xxx