Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemtap: not to use spin_unlock_wait anymore #21

Closed
gowrishankarm opened this issue Oct 31, 2017 · 1 comment
Closed

systemtap: not to use spin_unlock_wait anymore #21

gowrishankarm opened this issue Oct 31, 2017 · 1 comment

Comments

@gowrishankarm
Copy link

gowrishankarm commented Oct 31, 2017

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160774 </cde:info>

Systemtap (3.1-5.dev.git39b62b4) seems to be broken in current hostos devel (4.14 rc4 kernel) for a reason that it refers to calling spin_unlock_wait() which is no more in kernel as in its commit d3a024a

  locking: Remove spin_unlock_wait() generic definitions 

  There is no agreed-upon definition of spin_unlock_wait()'s semantics,
  and it appears that all callers could do just as well with a lock/unlock
  pair.  This commit therefore removes spin_unlock_wait() and related
  definitions from core code. 

Hence, executing systemtap ends up with below error:

~]# stap -v -e 'probe vfs.read {exit()}' 
Pass 1: parsed user script and 490 library scripts using 174720virt/57280res/7744shr/47296data kb, in 640usr/20sys/653real ms.
Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using 325248virt/209920res/9792shr/197824data kb, in 3170usr/700sys/3885real ms.
Pass 3: translated to C into "/tmp/staptlhTut/stap_83fd83b4a24b54380d462a6bd5886728_2819_src.c" using 325248virt/210112res/9984shr/197824data kb, in 30usr/150sys/187real ms.
In file included from /usr/share/systemtap/runtime/stp_utrace.c:30:0,
                 from /usr/share/systemtap/runtime/linux/task_finder2.c:4,
                 from /usr/share/systemtap/runtime/linux/task_finder.c:17,
                 from /usr/share/systemtap/runtime/linux/runtime.h:222,
                 from /usr/share/systemtap/runtime/runtime.h:26,
                 from /tmp/staptlhTut/stap_83fd83b4a24b54380d462a6bd5886728_2819_src.c:25:
/usr/share/systemtap/runtime/stp_helper_lock.h: In function ‘stp_spin_unlock_wait’:
/usr/share/systemtap/runtime/stp_helper_lock.h:60:1: error: implicit declaration of function ‘spin_unlock_wait’ [-Werror=implicit-function-declaration]
 static inline void stp_spin_unlock_wait(spinlock_t *lock) { spin_unlock_wait(lock); }
 ^
cc1: all warnings being treated as errors
make[1]: *** [/tmp/staptlhTut/stap_83fd83b4a24b54380d462a6bd5886728_2819_src.o] Error 1
make: *** [_module_/tmp/staptlhTut] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_83fd83b4a24b54380d462a6bd5886728_2819.ko" in 14550usr/1330sys/15774real ms.
Pass 4: compilation failed.  [man error::pass4]

Upstream systemtap fix to getrid of spin_unlock_wait:

https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commit;f=runtime/stp_utrace.c;h=0643ca2b7fd8cb6407aa84f41d26a71d2f2f8e90

Applied fix locally With that,

~]# stap -v -e 'probe oneshot { println("hello world") }'
Pass 1: parsed user script and 490 library scripts using 174720virt/57280res/7744shr/47296data kb, in 630usr/20sys/652real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 175488virt/57280res/7744shr/48064data kb, in 10usr/0sys/10real ms.
Pass 3: using cached /root/.systemtap/cache/56/stap_56041bf7d0574051b4814be457798d87_1190.c
Pass 4: using cached /root/.systemtap/cache/56/stap_56041bf7d0574051b4814be457798d87_1190.ko
Pass 5: starting run.
ERROR: module version mismatch (#1 SMP Thu Oct 26 22:52:08 -02 2017 vs #1 SMP Tue Oct 24 00:46:42 UTC 2017), release 4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le
WARNING: /usr/bin/staprun exited with status: 1
Pass 5: run completed in 10usr/50sys/332real ms.
Pass 5: run failed.  [man error::pass5]

Tried wiping out ~/.systemtap/cache/ but did not help. So, matched /usr/src/kernels/.../include/generated/compile.h, for stap and it worked (just a workaround to avoid recompiling stap).

~]# stap -v -e 'probe oneshot { println("hello world") }'
Pass 1: parsed user script and 490 library scripts using 174720virt/57216res/7744shr/47296data kb, in 630usr/20sys/653real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 175488virt/57216res/7744shr/48064data kb, in 0usr/0sys/10real ms.
Pass 3: translated to C into "/tmp/stapG4SOWi/stap_56041bf7d0574051b4814be457798d87_1190_src.c" using 175616virt/59648res/9280shr/48192data kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into "stap_56041bf7d0574051b4814be457798d87_1190.ko" in 2850usr/510sys/3250real ms.
Pass 5: starting run.
hello world
Pass 5: run completed in 20usr/70sys/809real ms.

Please include mentioned patch for systemtap to work.

mopsfelder added a commit to mopsfelder/versions that referenced this issue Oct 31, 2017
This updates systemtap.spec to version 3.2 and dependencies of
open-power-host-os.spec.

Fix open-power-host-os/linux#21
mopsfelder added a commit to mopsfelder/versions that referenced this issue Oct 31, 2017
This updates systemtap.spec to version 3.2 and dependencies of
open-power-host-os.spec.

Fix open-power-host-os/linux#21

(cherry picked from commit 3975da8)
@mopsfelder
Copy link
Member

I confirm that systemtap-3.2-1.dev.git4051c70.el7.centos.ppc64le is working fine with kernel-4.14.0-2.rc8.dev.gitcc4bf22.el7.centos.ppc64le. This issue can be closed.

# stap -v -e 'probe oneshot { println("hello world") }'
Pass 1: parsed user script and 493 library scripts using 175936virt/58432res/7744shr/48448data kb, in 480usr/30sys/577real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 176704virt/58432res/7744shr/49216data kb, in 10usr/0sys/29real ms.
Pass 3: translated to C into "/tmp/stapd8B9HA/stap_af87a4f97e038c002464edfc6be6e231_1194_src.c" using 176832virt/60928res/9344shr/49344data kb, in 0usr/0sys/1real ms.
Pass 4: compiled C into "stap_af87a4f97e038c002464edfc6be6e231_1194.ko" in 16920usr/1440sys/18798real ms.
Pass 5: starting run.
hello world
Pass 5: run completed in 10usr/60sys/783real ms.

# echo $?
0

# rpm -q systemtap kernel
systemtap-3.2-1.dev.git4051c70.el7.centos.ppc64le
kernel-4.14.0-2.rc8.dev.gitcc4bf22.el7.centos.ppc64le

liyi-ibm referenced this issue in liyi-ibm/linux Dec 6, 2018
pid_task() dereferences rcu protected tasks array.
But there is no rcu_read_lock() in shutdown_umh() routine so that
rcu_read_lock() is needed.
get_pid_task() is wrapper function of pid_task. it holds rcu_read_lock()
then calls pid_task(). if task isn't NULL, it increases reference count
of task.

test commands:
   %modprobe bpfilter
   %modprobe -rv bpfilter

splat looks like:
[15102.030932] =============================
[15102.030957] WARNING: suspicious RCU usage
[15102.030985] 4.19.0-rc7+ #21 Not tainted
[15102.031010] -----------------------------
[15102.031038] kernel/pid.c:330 suspicious rcu_dereference_check() usage!
[15102.031063]
	       other info that might help us debug this:

[15102.031332]
	       rcu_scheduler_active = 2, debug_locks = 1
[15102.031363] 1 lock held by modprobe/1570:
[15102.031389]  #0: 00000000580ef2b0 (bpfilter_lock){+.+.}, at: stop_umh+0x13/0x52 [bpfilter]
[15102.031552]
               stack backtrace:
[15102.031583] CPU: 1 PID: 1570 Comm: modprobe Not tainted 4.19.0-rc7+ #21
[15102.031607] Hardware name: To be filled by O.E.M. To be filled by O.E.M./Aptio CRB, BIOS 5.6.5 07/08/2015
[15102.031628] Call Trace:
[15102.031676]  dump_stack+0xc9/0x16b
[15102.031723]  ? show_regs_print_info+0x5/0x5
[15102.031801]  ? lockdep_rcu_suspicious+0x117/0x160
[15102.031855]  pid_task+0x134/0x160
[15102.031900]  ? find_vpid+0xf0/0xf0
[15102.032017]  shutdown_umh.constprop.1+0x1e/0x53 [bpfilter]
[15102.032055]  stop_umh+0x46/0x52 [bpfilter]
[15102.032092]  __x64_sys_delete_module+0x47e/0x570
[ ... ]

Fixes: d2ba09c ("net: add skeleton of bpfilter kernel module")
Signed-off-by: Taehee Yoo <ap420073@gmail.com>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants