A page fault begins when a process touches a virtual address that is not currently backed by physical memory. The CPU raises an exception, and the kernel has to decide whether to allocate a physical page frame, load data from storage, or handle a protection violation. This mechanism supports demand paging, copy-on-write, memory-mapped files, memory protection, and sharing. On AArch64, the path has architecture-specific details: the fault is captured in syndrome registers, routed through the exception vector table, decoded by the kernel, and then passed into the generic memory-management fault machinery.
Exception levels and saved state
AArch64 defines four exception levels, EL0 through EL3. EL0 is user mode, while EL1 is kernel mode. When a page fault occurs, the processor traps from EL0 to EL1 and saves state in several registers:
- ESR_EL1: exception syndrome register, containing the reason for the exception
- FAR_EL1: fault address register, holding the virtual address that triggered the fault
- ELR_EL1: exception link register, holding the return address
- SPSR_EL1: saved program status register
The kernel reads the EC (Exception Class) field in ESR_EL1 to classify the exception. For a data access abort, EC is 0b100101 or 0b100100, i.e. 0x25 or 0x21. For an instruction fetch abort, EC is 0b100000.
Page tables and address translation
AArch64 supports 48-bit or 52-bit virtual address spaces and uses a four-level page table structure: PGD, PUD, PMD, and PTE. Each page table entry is 8 bytes. The architecture supports several page sizes:
- 4KB pages: 4096 bytes per page, with a 12-bit page offset
- 16KB pages: 16384 bytes per page, with a 14-bit page offset
- 64KB pages: 65536 bytes per page, with a 16-bit page offset
Address translation can use two stages. Stage 1 translates a virtual address (VA) to an intermediate physical address (IPA), and Stage 2 translates the IPA to a physical address (PA). In a non-virtualized environment, Stage 1 directly produces the PA.
From the vector table to C code
When the processor takes a data access exception, it jumps to the corresponding entry in the exception vector table. The table is defined in arch/arm64/kernel/entry.S:
// arch/arm64/kernel/entry.S
.align 11
SYM_CODE_START(vectors)
kernel_ventry 1, sync // Synchronous EL1t
kernel_ventry 1, irq // IRQ EL1t
// ...
kernel_ventry 0, sync // Synchronous 64-bit EL0
kernel_ventry 0, irq // IRQ 64-bit EL0
// ...
SYM_CODE_END(vectors)
ARM64 separates exceptions into synchronous and asynchronous classes. Asynchronous exceptions are usually interrupts, while synchronous exceptions include faults. A page fault therefore enters through the synchronous exception vector. For an exception taken at EL1, the handler is el1_sync; in the example below, the code reaches el1h_64_sync_handler:
#define ESR_ELx_EC_SHIFT (26)
#define ESR_ELx_EC_MASK (UL(0x3F) << ESR_ELx_EC_SHIFT)
#define ESR_ELx_EC(esr) (((esr) & ESR_ELx_EC_MASK) >> ESR_ELx_EC_SHIFT) //Shift ESR right by 26 bits and take the low 6 bits: the EC value
asmlinkage void noinstr el1h_64_sync_handler(struct pt_regs *regs)
{
unsigned long esr = read_sysreg(esr_el1);
//printk("---esr:0x%x, at line-%d\n",ESR_ELx_EC(esr),__LINE__);
switch (ESR_ELx_EC(esr)) { ///Read the EC field of esr_el1 to identify the exception type
case ESR_ELx_EC_DABT_CUR: ///0x25: data abort from the current exception level
case ESR_ELx_EC_IABT_CUR:
el1_abort(regs, esr); ///Data abort entry
break;
/*
* We don't handle ESR_ELx_EC_SP_ALIGN, since we will have hit a
* recursive exception when trying to push the initial pt_regs.
*/
case ESR_ELx_EC_PC_ALIGN:
el1_pc(regs, esr);
break;
case ESR_ELx_EC_SYS64:
case ESR_ELx_EC_UNKNOWN:
el1_undef(regs);
break;
case ESR_ELx_EC_BREAKPT_CUR:
case ESR_ELx_EC_SOFTSTP_CUR:
case ESR_ELx_EC_WATCHPT_CUR:
case ESR_ELx_EC_BRK64:
el1_dbg(regs, esr);
break;
case ESR_ELx_EC_FPAC:
el1_fpac(regs, esr);
break;
default:
__panic_unhandled(regs, "64-bit el1h sync", esr);
}
}
With EC equal to 0x25 or 0x21, control reaches el1_abort:
static void noinstr el1_abort(struct pt_regs *regs, unsigned long esr)
{
unsigned long far = read_sysreg(far_el1); ///Read the faulting virtual address from far_el1
enter_from_kernel_mode(regs);
local_daif_inherit(regs);
do_mem_abort(far, esr, regs); ///Exception handling function
local_daif_mask();
exit_to_kernel_mode(regs);
}
From there, the fault enters do_mem_abort.
do_mem_abort: selecting the fault handler
/************************************************************************************
* Page fault handler
* Parameters:
* far: faulting virtual address
* esr: ESR_EL1 value
* regs: stack pointer at exception time
************************************************************************************/
void do_mem_abort(unsigned long far, unsigned int esr, struct pt_regs *regs)
{
const struct fault_info *inf = esr_to_fault_info(esr); ///Look up the fault_info table by DFSC field to get the corresponding handler
unsigned long addr = untagged_addr(far);
if (!inf->fn(far, esr, regs)) ///Run the handler obtained from esr_to_fault_info
return;
if (!user_mode(regs)) {
pr_alert("Unhandled fault at 0x%016lx\n", addr);
mem_abort_decode(esr);
show_pte(addr);
}
/*
* At this point we have an unrecognized fault type whose tag bits may
* have been defined as UNKNOWN. Therefore we only expose the untagged
* address to the signal handler.
*/
arm64_notify_die(inf->name, regs, inf->sig, inf->code, addr, esr); ///If no suitable handler is found, print error information
}
static inline const struct fault_info *esr_to_fault_info(unsigned int esr)
{
return fault_info + (esr & ESR_ELx_FSC); ///Select handler according to ESR_ELx_FSC field
}
esr & ESR_ELx_FSC reads the low 6 bits of ESR and indexes into fault_info.
static const struct fault_info fault_info[] = {
{ do_bad, SIGKILL, SI_KERNEL, "ttbr address size fault" },
{ do_bad, SIGKILL, SI_KERNEL, "level 1 address size fault" },
{ do_bad, SIGKILL, SI_KERNEL, "level 2 address size fault" },
{ do_bad, SIGKILL, SI_KERNEL, "level 3 address size fault" },
{ do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 0 translation fault" },
{ do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 1 translation fault" },
{ do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 2 translation fault" },
{ do_translation_fault, SIGSEGV, SEGV_MAPERR, "level 3 translation fault" },
{ do_bad, SIGKILL, SI_KERNEL, "unknown 8" },
{ do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 access flag fault" },
{ do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 access flag fault" },
{ do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 access flag fault" },
{ do_bad, SIGKILL, SI_KERNEL, "unknown 12" },
{ do_page_fault, SIGSEGV, SEGV_ACCERR, "level 1 permission fault" },
{ do_page_fault, SIGSEGV, SEGV_ACCERR, "level 2 permission fault" },
{ do_page_fault, SIGSEGV, SEGV_ACCERR, "level 3 permission fault" },
{ do_sea, SIGBUS, BUS_OBJERR, "synchronous external abort" },
{ do_tag_check_fault, SIGSEGV, SEGV_MTESERR, "synchronous tag check fault" },
{ do_bad, SIGKILL, SI_KERNEL, "unknown 18" },
{ do_bad, SIGKILL, SI_KERNEL, "unknown 19" },
{ do_sea, SIGKILL, SI_KERNEL, "level 0 (translation table walk)" },
{ do_sea, SIGKILL, SI_KERNEL, "level 1 (translation table walk)" },
//...
}
- do_translation_fault: called for level 0, 1, 2, or 3 translation faults
- do_page_fault: called for level 1, 2, or 3 access-flag or permission faults
- do_bad: other errors
- ...
For example, do_translation_fault is:
static int __kprobes do_translation_fault(unsigned long far,
unsigned int esr,
struct pt_regs *regs)
{
unsigned long addr = untagged_addr(far);
// If addr is a TTBR0 address (i.e., a user-space address)
if (is_ttbr0_addr(addr))
return do_page_fault(far, esr, regs);
// This function handles exceptions related to kernel-space addresses, such as illegal kernel accesses
do_bad_area(far, esr, regs);
return 0;
}
This leads to the core page fault handler, do_page_fault.
do_page_fault: validating the fault context
static int __kprobes do_page_fault(unsigned long far, unsigned int esr,
struct pt_regs *regs)
{
const struct fault_info *inf;
struct mm_struct *mm = current->mm;
vm_fault_t fault;
unsigned long vm_flags;
unsigned int mm_flags = FAULT_FLAG_DEFAULT;
unsigned long addr = untagged_addr(far);
if (kprobe_page_fault(regs, esr))
return 0;
/*
* If we're in an interrupt or have no user context, we must not take
* the fault.
*/
///Check the path at fault time; if conditions are not met, go to no_context for error handling
if (faulthandler_disabled() || !mm) ///Whether page faults are disabled, whether in interrupt context, whether in kernel space; kernel threads' mm is always NULL
goto no_context;
if (user_mode(regs))
mm_flags |= FAULT_FLAG_USER;
/*
* vm_flags tells us what bits we must have in vma->vm_flags
* for the fault to be benign, __do_page_fault() would check
* vma->vm_flags & vm_flags and returns an error if the
* intersection is empty
*/
if (is_el0_instruction_abort(esr)) { ///Determine whether this is an instruction abort from a lower exception level; if so, the address needs execute permission
/* It was exec fault */
vm_flags = VM_EXEC;
mm_flags |= FAULT_FLAG_INSTRUCTION;
} else if (is_write_abort(esr)) { ///Write fault
/* It was write fault */
vm_flags = VM_WRITE;
mm_flags |= FAULT_FLAG_WRITE;
} else {
/* It was read fault */
vm_flags = VM_READ;
/* Write implies read */
vm_flags |= VM_WRITE;
/* If EPAN is absent then exec implies read */
if (!cpus_have_const_cap(ARM64_HAS_EPAN))
vm_flags |= VM_EXEC;
}
///Check for user-space address and EL1 permission fault; if both hold, this is a rare special case and is reported as an error
if (is_ttbr0_addr(addr) && is_el1_permission_fault(addr, esr, regs)) {
if (is_el1_instruction_abort(esr))
///Instruction fault
die_kernel_fault("execution of user memory",
addr, esr, regs);
///Data fault: search exception tables
if (!search_exception_tables(regs->pc))
die_kernel_fault("access to user memory outside uaccess routines",
addr, esr, regs);
}
perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr);
/*
* As per x86, we may deadlock here. However, since the kernel only
* validly references user space from well defined areas of the code,
* we can bug out early if this is from code which shouldn't.
*/
///At this point the fault is not in interrupt context, not in a kernel thread, and not one of the special cases above; next handle faults caused by the address space
if (!mmap_read_trylock(mm)) {
if (!user_mode(regs) && !search_exception_tables(regs->pc))
goto no_context;
retry:
mmap_read_lock(mm); ///Sleep and wait for the lock to be released
} else {
/*
* The above mmap_read_trylock() might have succeeded in which
* case, we'll have missed the might_sleep() from down_read().
*/
might_sleep();
#ifdef CONFIG_DEBUG_VM
if (!user_mode(regs) && !search_exception_tables(regs->pc)) {
mmap_read_unlock(mm);
goto no_context;
}
#endif
}
///Handle the page fault further
fault = __do_page_fault(mm, addr, mm_flags, vm_flags, regs);
///There is a pending fatal signal
/* Quick path to respond to signals */
if (fault_signal_pending(fault, regs)) {
///Kernel space
if (!user_mode(regs))
goto no_context;
return 0;
}
if (fault & VM_FAULT_RETRY) {
if (mm_flags & FAULT_FLAG_ALLOW_RETRY) {
mm_flags |= FAULT_FLAG_TRIED;
goto retry;
}
}
//...
}
This function is abridged above; its main job is to call __do_page_fault:
static vm_fault_t __do_page_fault(struct mm_struct *mm, unsigned long addr,
unsigned int mm_flags, unsigned long vm_flags,
struct pt_regs *regs)
{
struct vm_area_struct *vma = find_vma(mm, addr); ///Find the VMA corresponding to faulting address addr
if (unlikely(!vma)) ///No VMA means addr is not in the process address space; return VM_FAULT_BADMAP
return VM_FAULT_BADMAP; //Indicates the address is invalid or outside the current process's mapping range.
/*
* Ok, we have a good vm_area for this memory access, so we can handle
* it.
*/
if (unlikely(vma->vm_start > addr)) { ///vm_start > addr: special case (stack, grows downward); check whether the VMA can expand to addr, otherwise report an error
if (!(vma->vm_flags & VM_GROWSDOWN))
return VM_FAULT_BADMAP;
if (expand_stack(vma, addr))
return VM_FAULT_BADMAP;
}
/*
* Check that the permissions on the VMA allow for the fault which
* occurred.
*/
if (!(vma->vm_flags & vm_flags)) ///Check VMA flags; if permission is missing, return directly
return VM_FAULT_BADACCESS;
return handle_mm_fault(vma, addr, mm_flags, regs); ///Core page fault handling function
}
After checking that the virtual address is valid, belongs to the process mapping range, and has the required permissions, the kernel calls handle_mm_fault().
handle_mm_fault: entering the generic fault path
vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
unsigned int flags, struct pt_regs *regs)
{
vm_fault_t ret;
// Set current process state to TASK_RUNNING, indicating it is ready to continue
__set_current_state(TASK_RUNNING);
// Count page fault events
count_vm_event(PGFAULT);
count_memcg_event_mm(vma->vm_mm, PGFAULT);
/* do counter updates before entering really critical section. */
check_sync_rss_stat(current);
if (!arch_vma_access_permitted(vma, flags & FAULT_FLAG_WRITE,
flags & FAULT_FLAG_INSTRUCTION,
flags & FAULT_FLAG_REMOTE))
return VM_FAULT_SIGSEGV;
/*
* Enable the memcg OOM handling for faults triggered in user
* space. Kernel faults are handled more gracefully.
*/
if (flags & FAULT_FLAG_USER)
mem_cgroup_enter_user_fault();
// If the VMA uses huge pages, call hugetlb_fault() for this specific page fault.
if (unlikely(is_vm_hugetlb_page(vma)))
ret = hugetlb_fault(vma->vm_mm, vma, address, flags);
else
ret = __handle_mm_fault(vma, address, flags); // Actual page fault handler; allocates physical pages, updates page tables, etc.
// Handle user-space OOM
if (flags & FAULT_FLAG_USER) {
mem_cgroup_exit_user_fault();
/*
* The task may have entered a memcg OOM situation but
* if the allocation error was handled gracefully (no
* VM_FAULT_OOM), there is no need to kill anything.
* Just clean up the OOM state peacefully.
*/
if (task_in_memcg_oom(current) && !(ret & VM_FAULT_OOM))
mem_cgroup_oom_synchronize(false);
}
// Update memory management (MM) statistics
mm_account_fault(regs, address, flags, ret);
return ret;
}
handle_mm_fault() selects different paths for user space, kernel space, huge pages, and access-permission failures. It performs permission checks, memory-cgroup handling, and dispatches huge-page or regular-page faults. The actual repair work is left to lower-level handlers.
__handle_mm_fault: walking PGD, PUD, and PMD
static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
unsigned long address, unsigned int flags)
{
struct vm_fault vmf = { ///Build the vma fault descriptor
.vma = vma,
.address = address & PAGE_MASK,
.flags = flags,
.pgoff = linear_page_index(vma, address),
.gfp_mask = __get_fault_gfp_mask(vma),
};
unsigned int dirty = flags & FAULT_FLAG_WRITE;
struct mm_struct *mm = vma->vm_mm;
pgd_t *pgd;
p4d_t *p4d;
vm_fault_t ret;
pgd = pgd_offset(mm, address); ///Compute PGD entry
p4d = p4d_alloc(mm, pgd, address); ///Compute P4D entry; here it equals PGD
if (!p4d)
return VM_FAULT_OOM;
vmf.pud = pud_alloc(mm, p4d, address); ///Compute PUD entry
if (!vmf.pud)
return VM_FAULT_OOM;
retry_pud:
// Handle transparent huge pages
if (pud_none(*vmf.pud) && __transparent_hugepage_enabled(vma)) {
ret = create_huge_pud(&vmf);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
pud_t orig_pud = *vmf.pud;
barrier();
if (pud_trans_huge(orig_pud) || pud_devmap(orig_pud)) {
/* NUMA case for anonymous PUDs would go here */
if (dirty && !pud_write(orig_pud)) {
ret = wp_huge_pud(&vmf, orig_pud);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
huge_pud_set_accessed(&vmf, orig_pud);
return 0;
}
}
}
vmf.pmd = pmd_alloc(mm, vmf.pud, address); ///Compute PMD entry
if (!vmf.pmd)
return VM_FAULT_OOM;
/* Huge pud page fault raced with pmd_alloc? */
if (pud_trans_unstable(vmf.pud))
goto retry_pud;
if (pmd_none(*vmf.pmd) && __transparent_hugepage_enabled(vma)) {
ret = create_huge_pmd(&vmf);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
vmf.orig_pmd = *vmf.pmd;
barrier();
if (unlikely(is_swap_pmd(vmf.orig_pmd))) {
VM_BUG_ON(thp_migration_supported() &&
!is_pmd_migration_entry(vmf.orig_pmd));
if (is_pmd_migration_entry(vmf.orig_pmd))
pmd_migration_entry_wait(mm, vmf.pmd);
return 0;
}
if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) {
if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma))
return do_huge_pmd_numa_page(&vmf);
if (dirty && !pmd_write(vmf.orig_pmd)) {
ret = wp_huge_pmd(&vmf);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
huge_pmd_set_accessed(&vmf);
return 0;
}
}
}
return handle_pte_fault(&vmf); ///Enter generic PTE handling
}
__handle_mm_fault() is the core of page-table fault handling. It walks the page table level by level (PGD, PUD, PMD). If the fault cannot be resolved at the PUD or PMD level, it proceeds to PTE handling.
handle_pte_fault: the generic PTE decision point
///Generic page fault code
static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
{
pte_t entry;
if (unlikely(pmd_none(*vmf->pmd))) {
/*
* Leave __pte_alloc() until later: because vm_ops->fault may
* want to allocate huge page, and if we expose page table
* for an instant, it will be difficult to retract from
* concurrent faults and from rmap lookups.
*/
vmf->pte = NULL;
} else {
/*
* If a huge pmd materialized under us just retry later. Use
* pmd_trans_unstable() via pmd_devmap_trans_unstable() instead
* of pmd_trans_huge() to ensure the pmd didn't become
* pmd_trans_huge under us and then back to pmd_none, as a
* result of MADV_DONTNEED running immediately after a huge pmd
* fault in a different thread of this mm, in turn leading to a
* misleading pmd_trans_huge() retval. All we have to ensure is
* that it is a regular pmd that we can walk with
* pte_offset_map() and we can do that through an atomic read
* in C, which is what pmd_trans_unstable() provides.
*/
if (pmd_devmap_trans_unstable(vmf->pmd))
return 0;
/*
* A regular pmd is established and it can't morph into a huge
* pmd from under us anymore at this point because we hold the
* mmap_lock read mode and khugepaged takes it in write mode.
* So now it's safe to run pte_offset_map().
*/
vmf->pte = pte_offset_map(vmf->pmd, vmf->address); ///Compute PTE entry
vmf->orig_pte = *vmf->pte; ///Read PTE contents into vmf->orig_pte
/*
* some architectures can have larger ptes than wordsize,
* e.g.ppc44x-defconfig has CONFIG_PTE_64BIT=y and
* CONFIG_32BIT=y, so READ_ONCE cannot guarantee atomic
* accesses. The code below just needs a consistent view
* for the ifs and we later double check anyway with the
* ptl lock held. So here a barrier will do.
*/
barrier(); ///Some CPUs have PTEs larger than a word, so READ_ONCE() cannot guarantee atomicity; add a memory barrier to ensure the PTE contents are read correctly
if (pte_none(vmf->orig_pte)) {
pte_unmap(vmf->pte);
vmf->pte = NULL;
}
}
///PTE is empty
if (!vmf->pte) {
// Check whether vma->vm_ops exists; if not, it is an anonymous mapping
if (vma_is_anonymous(vmf->vma))
return do_anonymous_page(vmf); ///Handle anonymous mapping
else
return do_fault(vmf); ///File mapping
}
///PTE is not empty
if (!pte_present(vmf->orig_pte)) ///PTE exists but page is not in memory; read it back from swap
return do_swap_page(vmf);
if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) ///Handle NUMA scheduling page
return do_numa_page(vmf);
vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
spin_lock(vmf->ptl);
entry = vmf->orig_pte;
if (unlikely(!pte_same(*vmf->pte, entry))) {
update_mmu_tlb(vmf->vma, vmf->address, vmf->pte);
goto unlock;
}
if (vmf->flags & FAULT_FLAG_WRITE) { ///FAULT_FLAG_WRITE is set according to ESR_ELx_WnR
if (!pte_write(entry))
return do_wp_page(vmf); ///VMA is writable but PTE is read-only; this triggers a fault on memory shared by parent and child, i.e. copy-on-write
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry); ///Access flag fault; set PTE_AF
if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry, ///Update PTE and cache
vmf->flags & FAULT_FLAG_WRITE)) {
update_mmu_cache(vmf->vma, vmf->address, vmf->pte);
} else {
/* Skip spurious TLB flush for retried page fault */
if (vmf->flags & FAULT_FLAG_TRIED)
goto unlock;
/*
* This is needed only for protection faults but the arch code
* is not yet telling us if this is a protection fault or not.
* This still avoids useless tlb flushes for .text page faults
* with threads.
*/
if (vmf->flags & FAULT_FLAG_WRITE)
flush_tlb_fix_spurious_fault(vmf->vma, vmf->address);
}
unlock:
pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
The remaining work splits into anonymous-page allocation, file-backed page reading, swap-in, NUMA handling, or copy-on-write. The AArch64 path therefore connects hardware syndrome registers and the exception vector entry to the generic Linux page fault path, which finally resolves the fault at the PTE level.
