Using CAT on UP2¶

The UP Squared board (UP2) is an x86 maker board based on the Intel Apollo Lake platform and supports Cache Allocation Technology (CAT). With this feature, the usage of cache can be restricted to each VM. ACRN hypervisor can do that by giving each VM a cache mask, so that the VM will not evict the masked cache lines, as shown in :numref: cache-mask-vm:

Figure 73 Cache usage with Cache Mask

CAT Support on ACRN¶

As described at Intel (R) 64 and IA-32 Architectures Software Developer’s Manual, chapter 17.19, volume 3. There are 3 steps to use CAT:

Detect the CAT capability.
Setup cache mask array MSRs, which is referred to as Class of Service (CLOS) array.
Select one of the CLOS array for the CPU, that will be the cache mask of the CPU.

ACRN integrates the usage of CAT into VM configuration. The CLOS array must be defined in the board config source code, and the VM needs to specify which CLOS to use in its VM config data. If the platform supports CAT, the CLOS array will be written to the CLOS MSRs at CPU init time, and the VM will set CLOS for VCPU at VM launch time. The details are shown in Figure 74:

Figure 74 CAT Support on ACRN

Tuning CAT in HV debug shell¶

From the ACRN HV debug shell, you can use cpuid, wrmsr/rdmsr debug commands to enumerate CAT capability and tune CAT parameters. You can use the UP2 board’s serial port for the HV shell (refer to Getting started guide for UP2 board for setup instructions).

Check CAT ability with cupid. First run cpuid 0x10 0x0, the return value ebx[bit 2] reports the L2 CAT is supported. Then run cpuid 0x10 0x2 to query L2 CAT capability, the return value eax[bit 4:0] reports the cache mask has 8 bit, and edx[bit 15:0] reports 4 CLOS are supported, as shown below:
```
ACRN:\>cpuid 0x10 0x0
cpuid leaf: 0x10, subleaf: 0x0, 0x0:0x4:0x0:0x0

ACRN:\>cpuid 0x10 0x2
cpuid leaf: 0x10, subleaf: 0x2, 0x7:0x0:0x0:0x3
```

Check PCPU IDs of each VM, the vcpu_list shows that VM0 is running on PCPU0, and VM1 is running on PCPU1:

ACRN:\>vcpu_list

VM ID    PCPU ID    VCPU ID    VCPU ROLE    VCPU STATE
=====    =======    =======    =========    ==========
  0         0          0        PRIMARY       Running
  1         1          0        PRIMARY       Running

Set CLOS with wrmsr <reg_num> <value>, we want VM1 to use the lower 6 ways of cache, so CLOS0 is set to 0xc0 for the upper 2 ways, and CLOS1 is set to 0x3f for the lower 6 ways:
```
ACRN:\>wrmsr  -p1 0xd10  0xc0
ACRN:\>wrmsr  -p1 0xd11  0x3f
```
Attach COS1 to PCPU1. Because MSR IA32_PQR_ASSOC [bit 63:32], we’ll write 0x100000000 to it to use CLOS1
```
ACRN:\>wrmsr   -p1   0xc8f    0x100000000
```

Configure CAT for VM with VM Configuration¶

CAT on ACRN can be enabled and configured by modifying source code, so the first step is to clone the ACRN source code (if you haven’t already):
```
$ git clone https://github.com/projectacrn/acrn-hypervisor.git
$ cd acrn-hypervisor/
```

The predefined cache masks can be found at hypervisor/arch/x86/configs/$(CONFIG_BOARD)/board.c, for UP2 board, that is hypervisor/arch/x86/configs/apl-up2/board.c, you can change the mask values, but note that the CLOS mask must have continuous bits, or a #GP fault can be triggered.

struct platform_clos_info platform_clos_array[4] = {
        {
                .clos_mask = 0xff,
                .msr_index = MSR_IA32_L2_MASK_0,
        },
        {
                .clos_mask = 0xff,
                .msr_index = MSR_IA32_L2_MASK_1,
        },
        {
                .clos_mask = 0xff,
                .msr_index = MSR_IA32_L2_MASK_2,
        },
        {
                .clos_mask = 0xff,
                .msr_index = MSR_IA32_L2_MASK_3,
        },
};

Set up CLOS in the VM config. If you want a VM to use one of the CLOSs, you need to find its configuration data and modify it. We will take SOS on sharing mode as an example. Its configuration data can be found at hypervisor/arch/x86/configs/vm_config.c

struct acrn_vm_config vm_configs[CONFIG_MAX_VM_NUM] __aligned(PAGE_SIZE) = {
        {
                .type = SOS_VM,
                .name = SOS_VM_CONFIG_NAME,
                .guest_flags = SOS_VM_CONFIG_GUEST_FLAGS | CLOS_REQUIRED,
                .clos = 1,
                .memory = {
                        .start_hpa = 0x0UL,
                        .size = CONFIG_SOS_RAM_SIZE,
                },
                .os_config = {
                        .name = SOS_VM_CONFIG_OS_NAME,
                },
        },
};

Build the ACRN hypervisor and copy the artifact acrn.efi to the /boot/EFI/acrn directory, see Build ACRN from Source for building instructions.

$ make hypervisor BOARD=apl-up2 FIRMWARE=uefi
...

# these operations are done on UP2 board
$ mount /dev/mmcblk0p0 /boot
$ scp <acrn.efi-at-your-compile-PC> /boot/EFI/acrn

Restart the UP2 board