Skip to content

Commit 2bb138c

Browse files
puranjaymohananakryiko
authored andcommitted
bpf, arm64: Inline bpf_get_current_task/_btf() helpers
On ARM64, the pointer to task_struct is always available in the sp_el0 register and therefore the calls to bpf_get_current_task() and bpf_get_current_task_btf() can be inlined into a single MRS instruction. Here is the difference before and after this change: Before: ; struct task_struct *task = bpf_get_current_task_btf(); 54: mov x10, #0xffffffffffff7978 // #-34440 58: movk x10, #0x802b, lsl gregkh#16 5c: movk x10, #0x8000, lsl #32 60: blr x10 --------------> 0xffff8000802b7978 <+0>: mrs x0, sp_el0 64: add x7, x0, #0x0 <-------------- 0xffff8000802b797c <+4>: ret After: ; struct task_struct *task = bpf_get_current_task_btf(); 54: mrs x7, sp_el0 This shows around 1% performance improvement in artificial microbenchmark. Signed-off-by: Puranjay Mohan <puranjay@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Xu Kuohai <xukuohai@huawei.com> Acked-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20240619131334.4297-1-puranjay@kernel.org
1 parent 2807db7 commit 2bb138c

File tree

1 file changed

+9
-0
lines changed

1 file changed

+9
-0
lines changed

arch/arm64/net/bpf_jit_comp.c

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1244,6 +1244,13 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx,
12441244
break;
12451245
}
12461246

1247+
/* Implement helper call to bpf_get_current_task/_btf() inline */
1248+
if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task ||
1249+
insn->imm == BPF_FUNC_get_current_task_btf)) {
1250+
emit(A64_MRS_SP_EL0(r0), ctx);
1251+
break;
1252+
}
1253+
12471254
ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass,
12481255
&func_addr, &func_addr_fixed);
12491256
if (ret < 0)
@@ -2580,6 +2587,8 @@ bool bpf_jit_inlines_helper_call(s32 imm)
25802587
{
25812588
switch (imm) {
25822589
case BPF_FUNC_get_smp_processor_id:
2590+
case BPF_FUNC_get_current_task:
2591+
case BPF_FUNC_get_current_task_btf:
25832592
return true;
25842593
default:
25852594
return false;

0 commit comments

Comments
 (0)