Getting Started Guide for ACRN Industry Scenario with ROScube-I

Verified version

  • Ubuntu version: 18.04

  • GCC version: 7.5.0

  • ACRN-hypervisor branch: v2.1

  • ACRN-Kernel (Service VM kernel): v2.1

  • RT kernel for Ubuntu User VM OS: Linux kernel 4.19.59 with Xenomai 3.1

  • HW: ROScube-I

    ADLINK ROScube-I is a real-time ROS 2-enabled robotic controller based on Intel® Xeon® 9th Gen Intel® Core™ i7/i3 and 8th Gen Intel® Core™ i5 processors. It features comprehensive I/O connectivity supporting a wide variety of sensors and actuators for unlimited robotic applications.

Architecture

This tutorial will show you how to install the ACRN Industry Scenario on ROScube-I. The scenario is shown here:

../../_images/rqi-acrn-architecture.png
  • Service VM: Used to launch the User VM and real-time VM.

  • User VM: Run ROS 2 application in this VM, such as for SLAM or navigation.

  • Real-time VM: Run critical tasks in this VM, such as the base driver.

Prerequisites

  • Connect the ROScube-I as shown here:

    • HDMI for monitor.

    • Network on Ethernet port 1.

    • Keyboard and mouse on USB.

    ../../_images/rqi-acrn-hw-connection.jpg
  • Install Ubuntu 18.04 on ROScube-I.

  • Modify the following BIOS settings.

    Hyper-threading

    Advanced -> CPU Configuration

    Disabled

    Intel (VMX) Virtualization

    Advanced -> CPU Configuration

    Enabled

    Intel(R) SpeedStep(tm)

    Advanced -> CPU Configuration

    Disabled

    Intel(R) Speed Shift Technology

    Advanced -> CPU configuration

    Disabled

    Turbo Mode

    Advanced -> CPU configuration

    Disabled

    C States

    Advanced -> CPU configuration

    Disabled

    VT-d

    Chipset -> System Agent (SA) Configuration

    Enabled

    DVMT-Pre Allocated

    Chipset -> System Agent (SA) Configuration -> Graphics Configuration

    64M

Install ACRN hypervisor

Set up Environment

  1. Open /etc/default/grub/ and add idle=nomwait intel_pstate=disable to the end of GRUB_CMDLINE_LINUX_DEFAULT.

    ../../_images/rqi-acrn-grub.png
  2. Update GRUB and then reboot.

    sudo update-grub
    sudo reboot
    
  3. Install the necessary libraries:

    sudo apt update
    sudo apt install -y gcc git make gnu-efi libssl-dev libpciaccess-dev \
      uuid-dev libsystemd-dev libevent-dev libxml2-dev \
      libusb-1.0-0-dev python3 python3-pip libblkid-dev \
      e2fslibs-dev pkg-config libnuma-dev liblz4-tool flex bison
    sudo pip3 install kconfiglib
    
  4. Get code from GitHub.

    mkdir ~/acrn && cd ~/acrn
    git clone https://github.com/projectacrn/acrn-hypervisor -b release_2.1
    cd acrn-hypervisor
    

Configure Hypervisor

  1. Parse system information.

    sudo apt install -y cpuid msr-tools
    cd ~/acrn/acrn-hypervisor/misc/acrn-config/target/
    sudo python3 board_parser.py ros-cube-cfl
    cp ~/acrn/acrn-hypervisor/misc/acrn-config/target/out/ros-cube-cfl.xml \
      ~/acrn/acrn-hypervisor/misc/acrn-config/xmls/board-xmls/
    
  2. Run ACRN configuration app and it will open a browser page.

    cd ~/acrn/acrn-hypervisor/misc/acrn-config/config_app
    sudo pip3 install -r requirements
    python3 app.py
    
    ../../_images/rqi-acrn-config-web.png
  3. Select “Import Board info”.

    ../../_images/rqi-acrn-config-import-board.png
  4. Select target board name.

    ../../_images/rqi-acrn-config-select-board.png
  5. Select “Scenario Setting” and choose “Load a default scenario”.

    ../../_images/rqi-acrn-config-scenario-settings.png
  6. Settings “HV”: You can ignore this if your RAM is <= 16GB.

    ../../_images/rqi-acrn-config-hv-settings.png
  7. Settings “VM0”: Select the hard disk currently used.

    ../../_images/rqi-acrn-config-vm0-settings.png
  8. Settings “VM1”: Enable all the cpu_affinity. You can press + to increase CPU ID. This doesn’t mean to attach all CPUs to the VM. The CPU number can be adjusted later.

    ../../_images/rqi-acrn-config-vm1-settings.png
  9. Settings “VM2”: Set up RT flags and enable all the cpu_affinity.

    ../../_images/rqi-acrn-config-vm2-settings1.png
    ../../_images/rqi-acrn-config-vm2-settings2.png
  10. Export XML.

    ../../_images/rqi-acrn-config-export-xml.png
    ../../_images/rqi-acrn-config-export-xml-submit.png
  11. Generate configuration files.

    ../../_images/rqi-acrn-config-generate-config.png
    ../../_images/rqi-acrn-config-generate-config-submit.png
  12. Close the browser and stop the process (Ctrl+C).

  13. Optional: Patch the hypervisor if you want to passthrough GPIO to VM.

    cd ~/acrn/acrn-hypervisor
    wget https://raw.githubusercontent.com/Adlink-ROS/ROScube_ACRN_guide/v2.1/patch/0001-Fix-ROScube-I-gpio-pin-assignment-table.patch
    git apply 0001-Fix-ROScube-I-gpio-pin-assignment-table.patch
    
  14. Build hypervisor

    cd ~/acrn/acrn-hypervisor
    make all \
       BOARD_FILE=misc/acrn-config/xmls/board-xmls/ros-cube-cfl.xml \
       SCENARIO_FILE=misc/acrn-config/xmls/config-xmls/ros-cube-cfl/user_defined/industry_ROS2SystemOS.xml \
       RELEASE=0
    
  15. Install hypervisor

    sudo make install
    sudo mkdir /boot/acrn
    sudo cp ~/acrn/acrn-hypervisor/build/hypervisor/acrn.bin /boot/acrn/
    

Install Service VM kernel

Build Service VM kernel

  1. Get code from GitHub

    cd ~/acrn
    git clone https://github.com/projectacrn/acrn-kernel -b release_2.1
    cd acrn-kernel
    
  2. Restore default ACRN configuration.

    cp kernel_config_uefi_sos .config
    make olddefconfig
    sed -ri '/CONFIG_LOCALVERSION=/s/=.+/="-ROS2SystemSOS"/g' .config
    sed -i '/CONFIG_PINCTRL_CANNONLAKE/c\CONFIG_PINCTRL_CANNONLAKE=m' .config
    
  3. Build Service VM kernel. It will take some time.

    make all
    
  4. Install kernel and module.

    sudo make modules_install
    sudo cp arch/x86/boot/bzImage /boot/acrn-ROS2SystemSOS
    

Update Grub

  1. Get the UUID and PARTUUID.

    sudo blkid /dev/sda*
    

    Note

    The UUID and PARTUUID we need should be /dev/sda2, which is TYPE="ext4", as shown in the following graph:

    ../../_images/rqi-acrn-blkid.png
  2. Update /etc/grub.d/40_custom as below. Remember to edit <UUID> and <PARTUUID> to your system’s values.

    menuentry "ACRN Multiboot Ubuntu Service VM" --id ubuntu-service-vm {
      load_video
      insmod gzio
      insmod part_gpt
      insmod ext2
    
      search --no-floppy --fs-uuid --set <UUID>
      echo 'loading ACRN Service VM...'
      multiboot2 /boot/acrn/acrn.bin  root=PARTUUID="<PARTUUID>"
      module2 /boot/acrn-ROS2SystemSOS Linux_bzImage
    }
    
    ../../_images/rqi-acrn-grun-40_custom.png
  3. Update /etc/default/grub to make GRUB menu visible and load Service VM as default.

    GRUB_DEFAULT=ubuntu-service-vm
    #GRUB_TIMEOUT_STYLE=hidden
    GRUB_TIMEOUT=5
    
  4. Then update GRUB and reboot.

    sudo update-grub
    sudo reboot
    
  5. ACRN Multiboot Ubuntu Service VM entry will be shown in the GRUB menu. Choose it to load ACRN. You can check that the installation is successful by using dmesg.

    sudo dmesg | grep ACRN
    
    ../../_images/rqi-acrn-dmesg.png

Install User VM

Before create User VM

  1. Download Ubuntu image (Here we use Ubuntu 18.04 LTS for example):

  2. Install necessary packages.

    sudo apt install qemu-kvm libvirt-clients libvirt-daemon-system \
      bridge-utils virt-manager ovmf
    sudo reboot
    

Create User VM image

Note

Reboot into the native Linux kernel (not the ACRN kernel) and create User VM image.

  1. Start virtual machine manager application.

    sudo virt-manager
    
  2. Create a new virtual machine.

    ../../_images/rqi-acrn-kvm-new-vm.png
  3. Select your ISO image path.

    ../../_images/rqi-acrn-kvm-choose-iso.png
  4. Select CPU and RAM for the VM. You can modify as high as you can to accelerate the installation time. The settings here are not related to the resource of the User VM on ACRN, which can be decided later.

    ../../_images/rqi-acrn-kvm-cpu-ram.png
  5. Select disk size you want. Note that this can’t be modified after creating image!

    ../../_images/rqi-acrn-kvm-storage.png
  6. Edit image name and select “Customize configuration before install”.

    ../../_images/rqi-acrn-kvm-name.png
  7. Select correct Firmware, apply it, and Begin Installation.

    ../../_images/rqi-acrn-kvm-firmware.png
  8. Now you’ll see the installation page of Ubuntu. After installing Ubuntu, you can also install some necessary packages, such as ssh, vim, and ROS 2. We’ll clone the image for real-time VM to save time.

  9. To install ROS 2, refer to Installing ROS 2 via Debian Packages

  10. Optional: Use ACRN kernel if you want to passthrough GPIO to User VM.

    sudo apt install git build-essential bison flex libelf-dev libssl-dev liblz4-tool
    
    # Clone code
    git clone -b release_2.1 https://github.com/projectacrn/acrn-kernel
    cd acrn-kernel
    
    # Set up kernel config
    cp kernel_config_uos .config
    make olddefconfig
    export ACRN_KERNEL_UOS=`make kernelversion`
    export UOS="ROS2SystemUOS"
    export BOOT_DEFAULT="${ACRN_KERNEL_UOS}-${UOS}"
    sed -ri "/CONFIG_LOCALVERSION=/s/=.+/=\"-${UOS}\"/g" .config
    
    # Build and install kernel and modules
    make all
    sudo make modules_install
    sudo make install
    
    # Update Grub
    sudo sed -ri \
      "/GRUB_DEFAULT/s/=.+/=\"Advanced options for Ubuntu>Ubuntu, with Linux ${BOOT_DEFAULT}\"/g" \
      /etc/default/grub
    sudo update-grub
    
  11. When that completes, poweroff the VM.

    sudo poweroff
    

Run User VM

Now back to the native machine to set up the environment for launching the User VM.

  1. Manually fetch and install the iasl binary to /usr/bin (where ACRN expects it) with a newer version of the than what’s included with Ubuntu 18.04:

    cd /tmp
    wget https://acpica.org/sites/acpica/files/acpica-unix-20191018.tar.gz
    tar zxvf acpica-unix-20191018.tar.gz
    cd acpica-unix-20191018
    make clean && make iasl
    sudo cp ./generate/unix/bin/iasl /usr/sbin/
    
  2. Convert KVM image file format.

    mkdir -p ~/acrn/uosVM
    cd ~/acrn/uosVM
    sudo qemu-img convert -f qcow2 -O raw /var/lib/libvirt/images/ROS2SystemUOS.qcow2 ./ROS2SystemUOS.img
    
  3. Prepare a Launch Script File.

    wget https://raw.githubusercontent.com/Adlink-ROS/ROScube_ACRN_guide/v2.1/scripts/launch_ubuntu_uos.sh
    chmod +x ./launch_ubuntu_uos.sh
    
  4. Set up network and reboot to take effect.

    mkdir -p ~/acrn/tools/
    cd ~/acrn/tools
    wget https://raw.githubusercontent.com/Adlink-ROS/ROScube_ACRN_guide/v2.1/scripts/acrn_bridge.sh
    chmod +x ./acrn_bridge.sh
    ./acrn_bridge.sh
    sudo reboot
    
  5. Reboot to ACRN kernel and now you can launch the VM.

    cd ~/acrn/uosVM
    sudo ./launch_ubuntu_uos.sh
    

Install real-time VM

Copy real-time VM image

Note

Reboot into the native Linux kernel (not the ACRN kernel) and create User VM image.

  1. Clone real-time VM from User VM. (Right-click User VM and then clone)

    ../../_images/rqi-acrn-rtos-clone.png
  2. You’ll see the real-time VM is ready.

    ../../_images/rqi-acrn-rtos-ready.png

Set up real-time VM

Note

The section will show you how to install Xenomai on ROScube-I. If help is needed, contact ADLINK for more information, or ask a question on the ACRN users mailing list

  1. Run the VM and modify your VM hostname.

    hostnamectl set-hostname ros-RTOS
    
  2. Install Xenomai kernel.

    # Install necessary packages
    sudo apt install git build-essential bison flex kernel-package libelf-dev libssl-dev haveged
    
    # Clone code from GitHub
    git clone -b F/4.19.59/base/ipipe/xenomai_3.1 https://github.com/intel/linux-stable-xenomai
    
    # Build
    cd linux-stable-xenomai
    cp arch/x86/configs/xenomai_test_defconfig .config
    make olddefconfig
    sed -i '/CONFIG_GPIO_VIRTIO/c\CONFIG_GPIO_VIRTIO=m' .config
    CONCURRENCY_LEVEL=$(nproc) make-kpkg --rootcmd fakeroot --initrd kernel_image kernel_headers
    
    # Install
    sudo dpkg -i ../linux-headers-4.19.59-xenomai+_4.19.59-xenomai+-10.00.Custom_amd64.deb \
      ../linux-image-4.19.59-xenomai+_4.19.59-xenomai+-10.00.Custom_amd64.deb
    
  3. Install Xenomai library and tools. For more details, refer to Xenomai Official Documentation.

    cd ~
    wget https://xenomai.org/downloads/xenomai/stable/xenomai-3.1.tar.bz2
    tar xf xenomai-3.1.tar.bz2
    cd xenomai-3.1
    ./configure --with-core=cobalt --enable-smp --enable-pshared
    make -j`nproc`
    sudo make install
    
  4. Allow non-root user to run Xenomai.

    sudo addgroup xenomai --gid 1234
    sudo addgroup root xenomai
    sudo usermod -a -G xenomai $USER
    
  5. Update /etc/default/grub.

    GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 4.19.59-xenomai+"
    #GRUB_TIMEOUT_STYLE=hidden
    GRUB_TIMEOUT=5
    ...
    GRUB_CMDLINE_LINUX="xenomai.allowed_group=1234"
    
  6. Update GRUB.

    sudo update-grub
    
  7. Poweroff the VM.

    sudo poweroff
    

Run real-time VM

Now back to the native machine and we’ll set up the environment for launching the real-time VM.

  1. Convert KVM image file format.

    mkdir -p ~/acrn/rtosVM
    cd ~/acrn/rtosVM
    sudo qemu-img convert -f qcow2 \
      -O raw /var/lib/libvirt/images/ROS2SystemRTOS.qcow2 \
      ./ROS2SystemRTOS.img
    
  2. Create a new launch file

    wget https://raw.githubusercontent.com/Adlink-ROS/ROScube_ACRN_guide/v2.1/scripts/launch_ubuntu_rtos.sh
    chmod +x ./launch_ubuntu_rtos.sh
    
  3. Reboot to ACRN kernel and now you can launch the VM.

    cd ~/acrn/rtosVM
    sudo ./launch_ubuntu_rtos.sh
    

Note

Use poweroff instead of reboot in the real-time VM. In ACRN design, rebooting the real-time VM will also reboot the whole system.

Customizing the launch file

The launch file in this tutorial has the following hardware resource allocation.

Resource

Service VM

User VM

Real-time VM

CPU

0

1,2,3

4,5

Memory

Remaining

8 GB

2 GB

Ethernet

Ethernet 1 & 2

Ethernet 3

Ethernet 4

USB

Remaining

1-2

1-1

You can modify the launch file for your own hardware resource allocation. We’ll provide some modification methods below. For more detail, see Device Model Parameters.

CPU

Modify the --cpu-affinity value in the command acrn-dm command. The number should be between 0 and max CPU ID. For example, if you want to run VM with core 1 and 2, use --cpu-affinity 1,2.

Memory

Modify the mem_size in launch file. This variable will be passed to acrn-dm. The possible values are 1024M, 2048M, 4096M, and 8192M.

Ethernet

Run lspci -Dnn | grep "Ethernet controller" to get the ID of Ethernet port.

../../_images/rqi-acrn-ethernet-lspci.png

You’ll see 4 IDs, one for each Ethernet port. Assign the ID of the port you want to passthrough in the launch file. For example, if we want to passthrough Ethernet 3 to the VM:

passthru_vpid=(
["ethernet"]="8086 1539"
)
passthru_bdf=(
["ethernet"]="0000:04:00.0"
)

# Passthrough ETHERNET
echo ${passthru_vpid["ethernet"]} > /sys/bus/pci/drivers/pci-stub/new_id
echo ${passthru_bdf["ethernet"]} > /sys/bus/pci/devices/${passthru_bdf["ethernet"]}/driver/unbind
echo ${passthru_bdf["ethernet"]} > /sys/bus/pci/drivers/pci-stub/bind

acrn-dm
⋮
-s 4,passthru,04/00/0 \

USB

To passthrough USB to VM, we need to know the ID for each USB first.

../../_images/rqi-acrn-usb-port.png

Then modify the launch file and add the USB ID. For example, if you want to passthrough USB 1-2 and 1-4.

acrn-dm
⋮
-s 8,xhci,1-2,1-4 \

GPIO

This is the PIN definition of ROScube-I.

../../_images/rqi-pin-definition.png

To pass GPIO to VM, you need to add the following section.

acrn-dm
⋮
-s X,virtio-gpio,@gpiochip0{<offset>=<alias_name>:<offset>=<alias_name>: ... :} \

The offset and pin mapping is as shown here:

Fn

GPIO Pin

In Chip Offset

DI0

GPIO220

72

DI1

GPIO221

73

DI2

GPIO222

74

DI3

GPIO223

75

DI4

GPIO224

76

DI5

GPIO225

77

DI6

GPIO226

78

DI7

GPIO227

79

DO0

GPIO253

105

DO1

GPIO254

106

DO2

GPIO255

107

DO3

GPIO256

108

DO4

GPIO257

109

DO5

GPIO258

110

DO6

GPIO259

111

DO7

GPIO260

112

For example, if you want to pass DI0 and DO0 to VM:

acrn-dm
⋮
-s X,virtio-gpio,@gpiochip0{72=gpi0:105=gpo0} \