Linux 7.0 Halves PostgreSQL Performance: A Kernel Preemption Deep Dive

A recent report from an Amazon/AWS engineer has sent ripples through the Linux and database communities, highlighting a severe performance regression for PostgreSQL on the upcoming Linux 7.0 kernel. The findings indicate that PostgreSQL's throughput could be cut in half compared to previous kernel versions, a significant blow for database-intensive applications. This isn't a minor tweak; it's a fundamental change in kernel behavior that could impact deployments running on the latest Linux environments, including the forthcoming Ubuntu 26.04 LTS.

The Alarming Discovery: PostgreSQL's Throughput Halved

Salvatore Dipietro, an engineer at Amazon/AWS, reported that PostgreSQL performance on a Graviton4 server running a near-final Linux 7.0 development kernel dropped dramatically. Benchmarks showed throughput at approximately 51% of what was observed on earlier kernels. The core of the problem, identified through bisecting, points to a substantial increase in time spent within user-space spinlocks, indicating contention issues directly tied to kernel-level changes.

This isn't just about raw speed; it impacts responsiveness and overall system capacity for any workload heavily relying on PostgreSQL. For cloud providers and enterprises where database performance is paramount, a 50% reduction in throughput can have cascading effects on service availability and operational costs.

The Technical Culprit: Kernel Preemption Mode Changes

At the heart of this performance regression is a specific change introduced in Linux 7.0: a re-evaluation and restriction of available preemption modes. Kernel preemption refers to the ability of the Linux kernel to interrupt a running task (even if it's in kernel mode) to allow a higher-priority task to run. This is crucial for responsiveness and fairness in multi-tasking environments.

Historically, Linux has supported several preemption models, including PREEMPT_NONE (no kernel preemption, tasks run to completion unless they explicitly yield), PREEMPT_VOLUNTARY (tasks yield at specific safe points), and PREEMPT_FULL (full kernel preemption, allowing interruptions almost anywhere). The Linux 7.0 scheduler updates aimed to simplify these modes, with the intention of focusing primarily on PREEMPT_FULL and a 'lazy' preemption model for modern CPU architectures. Critically, PREEMPT_NONE was effectively removed as a default option.

PostgreSQL, like many high-performance database systems, relies heavily on efficient locking mechanisms, often using user-space spinlocks to protect critical data structures. When kernel preemption is less aggressive, or changes unexpectedly, these spinlocks can behave differently. If a thread holding a user-space spinlock is preempted by the kernel for an extended period, other threads waiting on that lock will experience increased latency, leading to the observed performance degradation and increased time spent in these spinlocks.

The Standoff: Revert vs. Application Adaptation

Recognizing the severity of the regression, Salvatore Dipietro submitted a patch to the Linux kernel mailing list, proposing to restore PREEMPT_NONE as the default preemption model. This would effectively revert the problematic change and presumably restore PostgreSQL's performance.

However, Peter Zijlstra, a prominent Linux kernel developer and the author of the original code simplifying the preemption modes, has pushed back against this revert. His response suggests that the solution lies not in rolling back kernel changes, but in PostgreSQL adapting to the new kernel environment. Specifically, he pointed to the Restartable Sequences (RSEQ) time slice extension, which was also upstreamed for Linux 7.0.

RSEQ is a mechanism designed to allow user-space critical sections to operate efficiently without needing full kernel preemption, especially during time-slice extensions. The idea is that applications can declare code sections as restartable, allowing the kernel to preempt and restart them if necessary, without corrupting state or introducing excessive overhead. Zijlstra's argument implies that if PostgreSQL were to integrate RSEQ, it could mitigate the impact of reduced lock holder preemption, thereby making the kernel's simplification of preemption models viable.

Implications and Practical Takeaways for Developers

The current situation presents a dilemma. Linux 7.0 stable is expected in approximately two weeks, and distributions like Ubuntu 26.04 LTS, due later in April, are likely to ship with it. If Peter Zijlstra's stance holds and the PREEMPT_NONE revert is not accepted, PostgreSQL users could face significant performance issues on new Linux 7.0 deployments until the PostgreSQL project itself adopts RSEQ.

For developers and system administrators, this means:

• Monitor Closely: If you are planning to upgrade to Linux 7.0 or a distribution based on it (e.g., Ubuntu 26.04 LTS), rigorously benchmark your PostgreSQL workloads before and after the upgrade. Pay particular attention to throughput and query latency.
• Consider Kernel Pinning: For critical PostgreSQL instances, you might need to pin to an older, proven kernel version (e.g., Linux 6.x) until the situation is resolved, either by a kernel patch or a PostgreSQL update.
• Stay Informed on PostgreSQL Development: Keep an eye on PostgreSQL release notes and development discussions for any mention of RSEQ integration. This will be the long-term solution if the kernel's position remains firm.
• Understand the Trade-offs: This incident highlights the ongoing tension between kernel-level optimizations (simplifying preemption models for broader performance gains) and the specific needs of highly optimized applications like databases. What benefits one aspect of the system might inadvertently penalize another.

This is a critical development that demands attention. While the kernel community pushes for applications to adopt newer, more efficient primitives like RSEQ, the immediate impact on widely used software like PostgreSQL can be substantial. Developers should prepare for potential performance challenges and be ready to adapt their deployment strategies.

FAQ

Q: What is kernel preemption and why is it relevant to PostgreSQL performance?

A: Kernel preemption is the Linux kernel's ability to interrupt a running task, even when it's executing kernel code, to allow a higher-priority task to run. This is crucial for system responsiveness. In Linux 7.0, changes to preemption modes, specifically restricting PREEMPT_NONE as a default, led to less aggressive preemption. For PostgreSQL, this means that a thread holding a user-space spinlock might be preempted for longer, causing other threads waiting on that lock to experience increased latency and reducing overall database throughput.

Q: What are Restartable Sequences (RSEQ) and how do they relate to this fix?

A: Restartable Sequences (RSEQ) are a Linux kernel time slice extension introduced in Linux 7.0. They provide a mechanism for user-space applications to define short, critical code sections that can be safely restarted by the kernel if preemption occurs during their execution. Kernel developers suggest that if PostgreSQL were adapted to use RSEQ, it could more efficiently manage its user-space spinlocks and critical sections in the new kernel preemption environment, thereby mitigating the performance regression without needing to revert kernel-level changes.

Q: How will this affect my PostgreSQL deployment on a new Linux 7.0 system or distributions like Ubuntu 26.04 LTS?

A: Without a fix from either the kernel (restoring PREEMPT_NONE) or PostgreSQL (adopting RSEQ), your PostgreSQL deployment on Linux 7.0 could experience a significant performance degradation, potentially halving its throughput. This impact would be particularly noticeable on database-intensive workloads. You would need to either use an older kernel version, monitor for official fixes from PostgreSQL, or investigate potential workarounds until a resolution is widely adopted.