Chapter 16
Symmetric Multiprocessing: liboskit_smp.a

16.1 Introduction

This library is designed to simplify the startup and use of multiprocessors. It defines a common interface to multiprocessor machines that is fairly platform independent.

Combined with the spin-locks provided in libkern, it is possible to implement a complete symmetric multiprocessor (SMP) based system using the OSKit code.

There is currently one machine-dependent interface, smp_apic_ack for the x86.

16.2 Supported Systems

Currently, SMP support is only provided for Intel x86 systems conforming to the Intel Multiprocessor Specification.

16.2.1 Intel x86

Systems which fully comply to the Intel MultiProcessing Specification (IMPS) should be supported. Since some of the code is based on Linux 2.0, some features (such as dual I/O APICs) are not fully supported. The APIC (Advanced Programmable Interrupt Controller) is not yet used for general interrupt delivery. Instead, all hardware interrupts are sent to the BootStrap Processor (BSP).

If a machine works with Linux 2.0 it should work with the OSKit; however, testing has been limited to a few dual-processor machines.

The SMP code must be compiled with a compiler that supports .code16 for full functionality. The smp library will compile without it, but it will only support a single processor.

Inter-processor interrupts (IPIs) are implemented. These are currently the only interrupts received by the Application Processors (APs). IPIs allow the client OS to implement TLB-shoot-down and reschedule requests.

It is important to note that if more than one processor wishes to run in “user mode,” that the per-processor data structures in libkern (such as base_tss, base_idt, and base_gdt) will have to be made per-processor.

The OSKit code has not been tested with more than two processors. Success (and failure) reports for systems with three or more processors would be appreciated.

smp_apic_ack mentions a potential pitfall with Intel x86 SMPs. If more than one processor tries to send an IPI to a target processor, or if a processor sends multiple IPIs without waiting for them to be processed, IPIs can get lost. It is up to the programmer to deal with this limitation.

16.2.2 External dependencies

The SMP library assumes that the base environment is usable. It starts up the Application Processors on the kernel support library’s “base” data structures. It is possible (in fact required in many cases) to reload per-processor copies.

The following are symbols from the kernel support library required by the SMP library:

DEPENDENCIES
base_gdt:
 15.7.1
base_idt:
 15.7.4
base_tss_load:
 15.7.8
boot_info:
 15.14.7
phys_mem_va:
 15.6.2

The LMM library is used to allocate pages of memory below 1MB. This requires the symbols:

DEPENDENCIES
lmm_alloc_page:
 25.6.8
malloc_lmm:
 14.5.1

These minimal C library symbols are pulled in by the SMP support code:

DEPENDENCIES
panic:
 14.8.3
printf:
 14.6

This library provides SMP-safe implementations for:

DEPENDENCIES
base_critical_enter:
 15.2.5
base_critical_leave:
 15.2.5

16.3 API reference

16.3.1 smp_init: Initializes the SMP startup code

SYNOPSIS

#include <oskit/smp.h>

int smp_init(void);

DESCRIPTION

This function does the initial setup for the SMP support. It should be called before any other SMP library routines are used. It identifies the processors and gets them ready and waiting in a busy-loop for a “go” from the boot processor.

Note that success does not necessarily mean the system has multiple processors. Rather, failure indicates that the machine does not support multiple processors. smp_get_num_cpus should be used to determine the number of CPUs present.

Don’t call this more than once. . . yet.

RETURNS

It returns 0 on success (SMP-capable system is found). E_SMP_NO_CONFIG is returned on non-IMPS-compliant x86 machines.

16.3.2 smp_find_cur_cpu: Return the processor ID of the current processor.

SYNOPSIS

#include <oskit/smp.h>

int smp_find_cur_cpu(void);

DESCRIPTION

This function returns a unique (per-processor) integer representing the current processor. Note that the numbers are not guaranteed to be sequential or starting from 0, although that may be a common case.

On the x86, these numbers correspond to the processor’s APIC ID, which is set by the hardware. However, these are to be treated as logical processor numbers since the smp library may do a transformation in the future.

RETURNS

The processor’s ID.

16.3.3 smp_find_cpu: Return the next processor ID

SYNOPSIS

#include <oskit/smp.h>

int smp_find_cpu(int first);

DESCRIPTION

Given a number first, it returns the first processor ID such that the ID is greater than or equal to that number.

In order to be assured of finding all the CPUs, the initial call should be made with an argument of 0 and subsequent calls should be made with one more than the previously returned value.

This is designed to be used as an iterator function for the client OS to determine which processor numbers are present.

PARAMETERS
first:
The processor number at which to start searching.
RETURNS

Returns E_SMP_NO_PROC if there are no more processors, otherwise the ID of the next processor.

16.3.4 smp_start_cpu: Starts a processor running a specified function

SYNOPSIS

#include <oskit/smp.h>

void smp_start_cpu(int processor_id, void (*func)(void *data), void *data, void *stack_ptr);

DESCRIPTION

This releases the specified processor to start running a function with the specified stack.

Results are undefined if:

  1. the processor indicated does not exist,
  2. a processor attempts to start itself,
  3. any processor is started more than once, or
  4. any of the parameters is invalid.

smp_find_cur_cpu can be used to prevent calling smp_start_cpu on yourself. This function must be called for each processor started up by smp_init; if the processor is not used, then func should execute the halt instruction immediately.

It is up to the user to verify that the processor is started up correctly.

PARAMETERS
processor_id:
The ID of a processor found by the startup code.
func:
A function pointer to be called by the processor after it has set up its stack.
data:
A pointer to some structure that is placed on that stack before func is called.
stack_ptr:
The stack pointer to be used by the processor. This should point to the top of the stack to be used by the processor, and should be large enough for func’s requirements.

16.3.5 smp_get_num_cpus: Returns the total number of processors

SYNOPSIS

#include <oskit/smp.h>

int smp_get_num_cpus(void);

DESCRIPTION

This returns the number of processors that exist.

RETURNS

The number of processors that have been found. In a non-SMP-capable system, this will always return one.

16.3.6 smp_map_range: Request the OS map physical memory

SYNOPSIS

#include <oskit/smp.h>

oskit_addr_t smp_map_range(oskit_addr_t start, oskit_size_t size);

DESCRIPTION

This function is a hook provided by the host OS to allow the SMP library to request physical memory be mapped into its virtual address space. This is called by smp_init_paging.

Note that this could be implemented using osenv_mem_map_phys.

RETURNS

The virtual address where the physical pages are mapped. Returns zero if unable to map the memory.

16.3.7 smp_init_paging: Tell the SMP code that paging is being enabled

SYNOPSIS

#include <oskit/smp.h>

int smp_init_paging(void);

DESCRIPTION

This routine is called by the OS when it is ready to turn on paging. This call causes the SMP library to make call-backs to the OS to map the regions that are SMP-specific. On Intel x86 processors, this means the APICS.

RETURNS

Zero on success, non-zero on failure.

16.3.8 smp_message_pass: Send an inter-processor interrupt to another CPU

SYNOPSIS

#include <oskit/smp.h>

void smp_message_pass(int cpunum);

DESCRIPTION

This causes the target processor to run its interrupt handler for the IPI vector, if the appropriate entry of smp_message_pass_enable has been set to non-zero by that processor. A processor should only modify its own smp_message_pass_enable entry after it is ready to start receiving IPIs.

This call offers very limited functionality. The expectation is that the OS writer will implement the desired functionality on top of this primitive.

16.3.9 smp_message_pass_enable:

SYNOPSIS

smp_message_pass_enable[CPUID]

DESCRIPTION

This array contains an entry for each processor. If a processor is ready to start receiving inter-processor interrupts, it should set smp_message_pass_enable[smp_find_cur_cpu()] to non-zero. This is used internally by the SMP library to prevent interrupts from being delivered before the processor has set up enough state to receive them.

16.3.10 smp_apic_ack: (X86) acknowledge an inter-processor interrupt

SYNOPSIS

#include <oskit/x86/smp.h>

void smp_apic_ack(void);

DESCRIPTION

This routine ACKs the local APIC. The APIC must be ACKed before returning from the IPI handler. Due to limitations in the APIC design, IPIs can be lost if sent too closely together, as the APIC only handles two outstanding requests.