Interrupt is an operation in which an external device sends a request to the operating system to interrupt the task being executed by the CPU and then deal with special events. The device cannot be directly connected to the CPU, but is uniformly connected to the interrupt controller, which manages and distributes device interrupts. In order to simulate a complete operating system, the virtualization layer must also complete the simulation of device interruption. The interrupt controller of the virtual machine is created through VMM, which can inject interrupts into the virtual machine by using the interrupt controller of the virtual machine.

In x86_ Under 64 architecture, interrupt controllers include pic and APIC. PIC controller is cascaded through two Intel 8259 chips and supports 15 interrupts. Limited by the number of PIC interrupt pins and not supporting multiple CPUs, Intel then introduced APIC interrupt controller. The APIC interrupt controller consists of I/O APIC and LAPIC. The external devices are connected to the I/O APIC. There is LAPIC inside each CPU. The I/O APIC and LAPIC are connected through the system bus. When an interrupt is generated, the I/O APIC can distribute the interrupt to the corresponding LAPIC, and then the CPU associated with the LAPIC starts to execute the interrupt processing routine. In addition to the above two interrupt controllers, there are MSI/MSI-x interrupt modes. It bypasses the I/O APIC and writes the interrupt vector number to the LAPIC of the corresponding CPU directly through the system bus. Using MSI/MSI-x interrupt technology will no longer be constrained by the number of pins, support more interrupts and reduce interrupt delay.

Under the aarch64 architecture, the interrupt controller is called GIC (Generic Interrupt Controller). At present, there are v1 ~ v4 versions. Currently StratoVirt only supports GICv3. Similarly, aarch64 also supports MSI/MSI-x interrupt mode.

INTx interrupt mechanism will be used on some traditional old devices. But in fact, in PCIe bus, many devices have been rarely used, and even directly prohibit this function. Therefore, StratoVirt currently does not support INTx interrupt mechanism.

Create interrupt chip

Because the interrupt controller has higher simulation performance in KVM, StratoVirt handed over the specific creation process and interrupt delivery process of interrupt chip to KVM. X86 will be materialized before StratoVirt starts the virtual machine_ 64 or aarch64 virtual motherboard, that is, call the realize() function to complete initialization. At this stage, the interrupt controller is created. The initialization code is as follows.

fn realize(
        vm: &Arc<Mutex<Self>>,
        vm_config: &mut VmConfig,
        is_migrate: bool,
    ) -> MachineResult<()> {
      ...
      locked_vm.init_interrupt_controller(u64::from(vm_config.machine_config.nr_cpus))?;
      ...
    }

StratoVirt provides machineops trail. Whether it is a lightweight motherboard or a standardized motherboard, in x86_ Init is implemented under 64 and aarch64 architectures respectively_ interrupt_ Controller(), initializing the interrupt controller function.

x86_64 architecture

The above calls the initialization interrupt controller function. In its internal execution process, its main function is to call create_irq_chip() function, which is in vm_fd calls ioctl(self, KVM_CREATE_IRQCHIP()) system call to tell the kernel that it needs to simulate the interrupt controller in KVM. Subsequently, the system call enters the KVM module, and the PIC and APIC interrupt chips will be created at the same time, and the default interrupt routing table will be generated.

fn init_interrupt_controller(&mut self, _vcpu_count: u64) -> MachineResult<()> {
  ...
     KVM_FDS
            .load()
            .vm_fd
            .as_ref()
            .unwrap()
            .create_irq_chip()
            .chain_err(|| MachineErrorKind::CrtIrqchipErr)?;
     ...
}

aarch64 architecture

GIC interrupt controller consists of four components: Distributor, CPU Interface, Redistributor and ITS. And x86_64 similarly, you need to create an interrupt controller in KVM. However, the difference is that during the creation process, the KVM module needs to be informed in advance of the address range of GIC components in the memory layout of the virtual machine. Via dist_range，redist_region_ranges，its_range three variables, which pass the memory address of the component to KVM. In addition, VM is still used internally_ FD, vGIC v3 and vGIC ITS interrupt devices are created through system call.

fn init_interrupt_controller(&mut self, vcpu_count: u64) -> Result<()> {
    ...
    let intc_conf = InterruptControllerConfig {
            version: kvm_bindings::kvm_device_type_KVM_DEV_TYPE_ARM_VGIC_V3,
            vcpu_count,
            max_irq: 192,
            msi: true,
            dist_range: MEM_LAYOUT[LayoutEntryType::GicDist as usize],
            redist_region_ranges: vec![
                MEM_LAYOUT[LayoutEntryType::GicRedist as usize],
                MEM_LAYOUT[LayoutEntryType::HighGicRedist as usize],
            ],
            its_range: Some(MEM_LAYOUT[LayoutEntryType::GicIts as usize]),
        };
        let irq_chip = InterruptController::new(&intc_conf)?;
        self.irq_chip = Some(Arc::new(irq_chip));
        self.irq_chip.as_ref().unwrap().realize()?;
        ...
}

Create MSI-x

When designing the Virtio PCI device of StratoVirt, msi-x interrupt mode is used to notify the virtual machine. Therefore, before using the MSI-x device, you need to call init_ in the Vitio PCI device implementation process. Msix(), and perform relevant initialization. The main function of this function is to negotiate MSI related information in the configuration space of PCI device. In addition, assign is provided in the materialization stage_ interrupt_ CB () function, used to encapsulate the interrupt callback function of the device. After the Virtio PCI device processes the I/O request, it will call the interrupt callback and send an interrupt notification to the KVM.

fn realize(mut self) -> PciResult<()> {
  ...
    init_msix(
            VIRTIO_PCI_MSIX_BAR_IDX as usize,
            nvectors as u32,
            &mut self.config,
            self.dev_id.clone(),
        )?;
        self.assign_interrupt_cb();
        ...
}

Manage interrupt routing tables

As mentioned above, when KVM creates an interrupt chip, it will generate a default interrupt routing table. However, some devices (such as pass through devices) need to add additional global interrupt numbers to KVM. At this time, StratoVirt needs to maintain an additional interrupt routing table and synchronize with KVM.

When StratoVirt initializes the interrupt controller, an interrupt routing table is created. Internal unified call init_irq_route_table() function, but the default interrupt routing table information is also different due to different architectures.

In addition to generating the default interrupt routing table, you also need to synchronize with KVM. commit_ irq_ The routing () function provides this function and uses VM internally_ System call IOCTL of FD_ with_ ref(self, KVM_SET_GSI_ROUTING(), irq_ Routing), which will overwrite the interrupt routing table information in the KVM module.

fn init_interrupt_controller(&mut self, vcpu_count: u64) -> Result<()> {
     ...
     KVM_FDS
            .load()
            .irq_route_table
            .lock()
            .unwrap()
            .init_irq_route_table();
        KVM_FDS
            .load()
            .commit_irq_routing()
            .chain_err(|| "Failed to commit irq routing for arm gic")?;
     ...
}

When the device needs to dynamically apply for or release the global interrupt number, StratoVirt provides two functions add_msi_route()，update_msi_route() is used to add or modify interrupt routing table information.

Interrupt process

For simulated virtio devices, the virtual machine exits to KVM by triggering VM Exit. StratoVirt binds the I/O address space and ioeventfd at the initial stage and registers this information with KVM. Therefore, the process of guest OS notifying the device to process I/O will return directly from KVM to StratoVirt loop. StratoVirt then distributes and processes I/O operations. When I/O requests or other events are completed, the virtual machine needs to be notified again to continue to execute, so that the virtual machine can be notified of the event by injecting interrupts.

StratoVirt supports two architectures at the same time: microVM and standardVM. The interrupt methods used in the two architectures are slightly different. In the microVM architecture, an evenetfd is associated with a global interrupt number, and the corresponding relationship is registered with KVM. When an interrupt needs to be sent, StratoVirt only needs to send a signal to the corresponding eventfd of the device, which will cause the corresponding interrupt to be injected into the virtual machine by the KVM module. In the standardVM architecture, msix notify() is used to initiate interrupts. After a series of function calls, finally in vm_fd up regulates IOCTL_ with_ Ref (self, kvm_signal_msi(), & MSI), send interrupt notification to KVM, and finally the KVM module completes the interrupt injection of virtual machine.

Lightweight model

During the virtio device activation phase, the callback function interrupt will be interrupted_ CB, passed in as an input parameter of the activate() function, is saved in the IO handler corresponding to the device. When an interrupt needs to be sent, the interrupt callback function will be called. The activate() function is declared as follows:

fn activate(
        &mut self,
        mem_space: Arc<AddressSpace>,
        interrupt_cb: Arc<VirtioInterrupt>,
        queues: &[Arc<Mutex<Queue>>],
        queue_evts: Vec<EventFd>,
    ) -> Result<()>;

The device under the lightweight model architecture uses the Virtio MMIO protocol. After processing the I/O request, it will call the interrupt callback function to send the interrupt. The specific contents of interrupt callback function are as follows:

let cb = Arc::new(Box::new(
            move |int_type: &VirtioInterruptType, _queue: Option<&Queue>| {
                let status = match int_type {
                    VirtioInterruptType::Config => VIRTIO_MMIO_INT_CONFIG,
                    VirtioInterruptType::Vring => VIRTIO_MMIO_INT_VRING,
                };
                interrupt_status.fetch_or(status as u32, Ordering::SeqCst);
                interrupt_evt
                    .write(1)
                    .chain_err(|| ErrorKind::EventFdWrite)?;

                Ok(())
            },
        ) as VirtioInterrupt);

We mentioned above that the eventfd and interrupt number information have been told to KVM. Interrupt callback through interrupt_ When EVT writes 1, KVM can poll the corresponding event, then find the global interrupt number corresponding to eventfd and inject it into the virtual machine.

Standard model

Different from lightweight models, the devices implemented under the standard model architecture use Virtio PCI protocol. Therefore, the interrupt mode is also changed to MSI-x. The same as above is that the device will save the interrupt callback function in the activation phase. The interrupt callback function corresponding to the standard model is as follows:

let cb = Arc::new(Box::new(
            move |int_type: &VirtioInterruptType, queue: Option<&Queue>| {
                let vector = match int_type {
                    VirtioInterruptType::Config => cloned_common_cfg
                        .lock()
                        .unwrap()
                        .msix_config
                        .load(Ordering::SeqCst),
                    VirtioInterruptType::Vring => {
                        queue.map_or(0, |q| q.vring.get_queue_config().vector)
                    }
                };

                if let Some(msix) = &cloned_msix {
                    msix.lock().unwrap().notify(vector, dev_id);
                } else {
                    bail!("Failed to send interrupt, msix does not exist");
                }
                Ok(())
            },
        ) as VirtioInterrupt);

In the interrupt callback function, obtain the interrupt vector number vector, and then use the notify() function to send the interrupt information to KVM. Get is used internally first_ Message() populates the address and data members of the MSI message structure. Then send the encapsulated message to KVM. Finally, in the kernel KVM module, the corresponding interrupt is injected into the virtual machine according to the interrupt routing table entry.

Pay attention to us

StratoVirt is already open source in the openEuler community. A series of topics will be shared later. If you are interested in the use and implementation of StratoVirt, you are welcome to watch and join us.

Project address

https://gitee.com/openeuler/stratovirt

Project wiki

https://gitee.com/openeuler/stratovirt/wikis

mailing list

https://mailweb.openeuler.org/postorius/lists/virt.openeuler.org/

Submit issue

https://gitee.com/openeuler/stratovirt/issues

Installation guide

https://www.openeuler.org/zh/other/projects/stratovirt/

Group entry

If you are interested in virtualization technology, welcome to join Virt SIG technology exchange group to discuss virtualization related technologies such as StratoVirt, KVM, QEMU and Libvirt. You can add the following wechat assistant to reply StratoVirt to join the group.