.\"     $NetBSD: percpu.9,v 1.16 2023/08/02 06:24:46 riastradh Exp $
.\"
.\" Copyright (c) 2010 The NetBSD Foundation, Inc.
.\" All rights reserved.
.\"
.\" This code is derived from software contributed to The NetBSD Foundation
.\" by David Young.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS
.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
.\" PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS
.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
.\" POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd February 7, 2020
.Dt PERCPU 9
.Os
.Sh NAME
.Nm percpu ,
.Nm percpu_alloc ,
.Nm percpu_create ,
.Nm percpu_free ,
.Nm percpu_getref ,
.Nm percpu_putref ,
.Nm percpu_foreach ,
.Nm percpu_foreach_xcall
.Nd per-CPU storage allocator
.Sh SYNOPSIS
.In sys/percpu.h
.Vt typedef void (*percpu_callback_t)(void *, void *, struct cpu_info *);
.Ft percpu_t *
.Fn percpu_alloc "size_t size"
.Ft percpu_t *
.Fn percpu_create "size_t size" "percpu_callback_t ctor" "percpu_callback_t dtor" "void *arg"
.Ft void
.Fn percpu_free "percpu_t *pc" "size_t size"
.Ft void *
.Fn percpu_getref "percpu_t *pc"
.Ft void
.Fn percpu_putref "percpu_t *pc"
.Ft void
.Fn percpu_foreach "percpu_t *pc" "percpu_callback_t cb" "void *arg"
.Ft void
.Fn percpu_foreach_xcall "percpu_t *pc" "u_int xcflags" "percpu_callback_t cb" "void *arg"
.Sh DESCRIPTION
The machine-independent
.Nm
interface provides per-CPU, CPU-local memory reservations to kernel
subsystems.
.Fo percpu_alloc
.Fa size
.Fc
reserves on each CPU an independent memory region of
.Fa size
bytes that is local to that CPU, returning a handle
.Po
.Vt percpu_t
.Pc
to those regions.
A thread may subsequently ask for a pointer,
.Fa p ,
to the region held by the
.Vt percpu_t
on the thread's current CPU.
Until the thread relinquishes the pointer, or voluntarily sleeps,
the thread may read or write the region at
.Fa p
without causing interprocessor memory synchronization.
.Sh FUNCTIONS
.Bl -tag -width compact
.It Fn percpu_alloc "size"
Call this in thread context to allocate
.Fa size
bytes of local storage on each CPU.
The storage is initialized with zeroes.
Treat this as an expensive operation.
.Fn percpu_alloc
returns a handle for the per-CPU storage.
.It Fn percpu_create "size" "ctor" "dtor" "arg"
Like
.Fn percpu_alloc ,
but before any access to the storage on any CPU, arrange to call
.Fn "(*ctor)" p arg ci ,
where
.Fa p
is the pointer to that CPU's storage and
.Fa ci
is the
.Vt "struct cpu_info *"
for that CPU.
This may happen synchronously in
.Fn percpu_create
or when a CPU is attached later.
Further, arrange that
.Fn percpu_free
will do the same with
.Fn "(*dtor)" p arg ci .
.Pp
.Fa ctor
and
.Fa dtor
.Em MAY
sleep, e.g. to allocate memory or to wait for users to drain before
deallocating memory.
Do not rely on any particular order of iteration over the CPUs.
.It Fn percpu_free "pc" "size"
Call this in thread context to
return to the system the per-CPU storage held by
.Fa pc .
.Fa size
should match the
.Fa size
passed to
.Fn percpu_alloc
or
.Fn percpu_create .
When
.Fn percpu_free
returns,
.Fa pc
is undefined.
Treat this as an expensive operation.
.It Fn percpu_getref "pc"
Disable preemption and return a pointer to the storage held by
.Fa pc
on the local CPU.
Use
.Fn percpu_getref
in either thread or interrupt context.
Follow each
.Fn percpu_getref
call with a matching call to
.Fn percpu_putref .
.Pp
Caller
.Em MUST NOT
sleep after
.Fn percpu_getref ,
not even on an adaptive lock, before
.Fn percpu_putref .
.It Fn percpu_putref "pc"
Indicate that the thread is finished
with the pointer returned by the matching
call to
.Fn percpu_getref .
Re-enables preemption.
.It Fn percpu_foreach "pc" "cb" "arg"
For each CPU, with
.Fa ci
being the corresponding
.Vt "struct cpu_info *"
and
.Fa "p"
the CPU-local storage held by
.Fa pc ,
run
.Fo "(*cb)"
.Fa "p"
.Fa "arg"
.Fa "ci"
.Fc .
The call to
.Fa cb
runs in the current thread; use
.Fn percpu_foreach_xcall
to execute the call to
.Fa cb
on the remote CPUs.
.Pp
Must be used in thread context.
.Fa cb
.Em MUST NOT
sleep except on adaptive locks, and should be fast.
Do not rely on any particular order of iteration over the CPUs.
.It Fn percpu_foreach_xcall "pc" "xcflags" "cb" "arg"
Like
.Fn percpu_foreach ,
except the call to
.Fa cb
executes on the remote CPUs.
The
.Fa xcflags
argument specifies how the cross calls are invoked; see
.Xr xc_broadcast 9 .
The same caveats and restrictions that apply to
.Fn percpu_foreach
also apply to
.Fn percpu_foreach_xcall .
.El
.Sh CODE REFERENCES
The
.Nm
interface is implemented within the file
.Pa sys/kern/subr_percpu.c .
.\" .Sh EXAMPLES
.Sh SEE ALSO
.Xr atomic_ops 3 ,
.Xr kmem 9 ,
.Xr pcq 9 ,
.Xr pool_cache 9 ,
.Xr xcall 9
.Sh HISTORY
The
.Nm
interface first appeared in
.Nx 6.0 .
.Sh AUTHORS
.An YAMAMOTO Takashi Aq Mt yamt@NetBSD.org
.\" .Sh CAVEATS
.\" .Sh BUGS
.\" .Sh SECURITY CONSIDERATIONS