Skip to content

Commit

Permalink
Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/lin…
Browse files Browse the repository at this point in the history 
…ux/kernel/git/tip/linux-2.6-tip

* 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip: (31 commits)
  rcu: Make RCU's CPU-stall detector be default
  rcu: Add expedited grace-period support for preemptible RCU
  rcu: Enable fourth level of TREE_RCU hierarchy
  rcu: Rename "quiet" functions
  rcu: Re-arrange code to reduce #ifdef pain
  rcu: Eliminate unneeded function wrapping
  rcu: Fix grace-period-stall bug on large systems with CPU hotplug
  rcu: Eliminate __rcu_pending() false positives
  rcu: Further cleanups of use of lastcomp
  rcu: Simplify association of forced quiescent states with grace periods
  rcu: Accelerate callback processing on CPUs not detecting GP end
  rcu: Mark init-time-only rcu_bootup_announce() as __init
  rcu: Simplify association of quiescent states with grace periods
  rcu: Rename dynticks_completed to completed_fqs
  rcu: Enable synchronize_sched_expedited() fastpath
  rcu: Remove inline from forward-referenced functions
  rcu: Fix note_new_gpnum() uses of ->gpnum
  rcu: Fix synchronization for rcu_process_gp_end() uses of ->completed counter
  rcu: Prepare for synchronization fixes: clean up for non-NO_HZ handling of ->completed counter
  rcu: Cleanup: balance rcu_irq_enter()/rcu_irq_exit() calls
  ...
  • Loading branch information
@torvalds
torvalds committed Dec 5, 2009
2 parents d0b093a + 8bfb2f8 commit 6077817
Show file tree
Hide file tree
Showing 20 changed files with 1,234 additions and 581 deletions.
254 changes: 49 additions & 205 deletions Documentation/RCU/trace.txt
View file
Original file line number Diff line number Diff line change
@@ -1,185 +1,10 @@
CONFIG_RCU_TRACE debugfs Files and Formats


The rcupreempt and rcutree implementations of RCU provide debugfs trace
output that summarizes counters and state. This information is useful for
debugging RCU itself, and can sometimes also help to debug abuses of RCU.
Note that the rcuclassic implementation of RCU does not provide debugfs
trace output.

The following sections describe the debugfs files and formats for
preemptable RCU (rcupreempt) and hierarchical RCU (rcutree).


Preemptable RCU debugfs Files and Formats

This implementation of RCU provides three debugfs files under the
top-level directory RCU: rcu/rcuctrs (which displays the per-CPU
counters used by preemptable RCU) rcu/rcugp (which displays grace-period
counters), and rcu/rcustats (which internal counters for debugging RCU).

The output of "cat rcu/rcuctrs" looks as follows:

CPU last cur F M
0 5 -5 0 0
1 -1 0 0 0
2 0 1 0 0
3 0 1 0 0
4 0 1 0 0
5 0 1 0 0
6 0 2 0 0
7 0 -1 0 0
8 0 1 0 0
ggp = 26226, state = waitzero

The per-CPU fields are as follows:

o "CPU" gives the CPU number. Offline CPUs are not displayed.

o "last" gives the value of the counter that is being decremented
for the current grace period phase. In the example above,
the counters sum to 4, indicating that there are still four
RCU read-side critical sections still running that started
before the last counter flip.

o "cur" gives the value of the counter that is currently being
both incremented (by rcu_read_lock()) and decremented (by
rcu_read_unlock()). In the example above, the counters sum to
1, indicating that there is only one RCU read-side critical section
still running that started after the last counter flip.

o "F" indicates whether RCU is waiting for this CPU to acknowledge
a counter flip. In the above example, RCU is not waiting on any,
which is consistent with the state being "waitzero" rather than
"waitack".

o "M" indicates whether RCU is waiting for this CPU to execute a
memory barrier. In the above example, RCU is not waiting on any,
which is consistent with the state being "waitzero" rather than
"waitmb".

o "ggp" is the global grace-period counter.

o "state" is the RCU state, which can be one of the following:

o "idle": there is no grace period in progress.

o "waitack": RCU just incremented the global grace-period
counter, which has the effect of reversing the roles of
the "last" and "cur" counters above, and is waiting for
all the CPUs to acknowledge the flip. Once the flip has
been acknowledged, CPUs will no longer be incrementing
what are now the "last" counters, so that their sum will
decrease monotonically down to zero.

o "waitzero": RCU is waiting for the sum of the "last" counters
to decrease to zero.

o "waitmb": RCU is waiting for each CPU to execute a memory
barrier, which ensures that instructions from a given CPU's
last RCU read-side critical section cannot be reordered
with instructions following the memory-barrier instruction.

The output of "cat rcu/rcugp" looks as follows:

oldggp=48870 newggp=48873

Note that reading from this file provokes a synchronize_rcu(). The
"oldggp" value is that of "ggp" from rcu/rcuctrs above, taken before
executing the synchronize_rcu(), and the "newggp" value is also the
"ggp" value, but taken after the synchronize_rcu() command returns.


The output of "cat rcu/rcugp" looks as follows:

na=1337955 nl=40 wa=1337915 wl=44 da=1337871 dl=0 dr=1337871 di=1337871
1=50989 e1=6138 i1=49722 ie1=82 g1=49640 a1=315203 ae1=265563 a2=49640
z1=1401244 ze1=1351605 z2=49639 m1=5661253 me1=5611614 m2=49639

These are counters tracking internal preemptable-RCU events, however,
some of them may be useful for debugging algorithms using RCU. In
particular, the "nl", "wl", and "dl" values track the number of RCU
callbacks in various states. The fields are as follows:

o "na" is the total number of RCU callbacks that have been enqueued
since boot.

o "nl" is the number of RCU callbacks waiting for the previous
grace period to end so that they can start waiting on the next
grace period.

o "wa" is the total number of RCU callbacks that have started waiting
for a grace period since boot. "na" should be roughly equal to
"nl" plus "wa".

o "wl" is the number of RCU callbacks currently waiting for their
grace period to end.

o "da" is the total number of RCU callbacks whose grace periods
have completed since boot. "wa" should be roughly equal to
"wl" plus "da".

o "dr" is the total number of RCU callbacks that have been removed
from the list of callbacks ready to invoke. "dr" should be roughly
equal to "da".

o "di" is the total number of RCU callbacks that have been invoked
since boot. "di" should be roughly equal to "da", though some
early versions of preemptable RCU had a bug so that only the
last CPU's count of invocations was displayed, rather than the
sum of all CPU's counts.

o "1" is the number of calls to rcu_try_flip(). This should be
roughly equal to the sum of "e1", "i1", "a1", "z1", and "m1"
described below. In other words, the number of times that
the state machine is visited should be equal to the sum of the
number of times that each state is visited plus the number of
times that the state-machine lock acquisition failed.

o "e1" is the number of times that rcu_try_flip() was unable to
acquire the fliplock.

o "i1" is the number of calls to rcu_try_flip_idle().

o "ie1" is the number of times rcu_try_flip_idle() exited early
due to the calling CPU having no work for RCU.

o "g1" is the number of times that rcu_try_flip_idle() decided
to start a new grace period. "i1" should be roughly equal to
"ie1" plus "g1".

o "a1" is the number of calls to rcu_try_flip_waitack().

o "ae1" is the number of times that rcu_try_flip_waitack() found
that at least one CPU had not yet acknowledge the new grace period
(AKA "counter flip").

o "a2" is the number of time rcu_try_flip_waitack() found that
all CPUs had acknowledged. "a1" should be roughly equal to
"ae1" plus "a2". (This particular output was collected on
a 128-CPU machine, hence the smaller-than-usual fraction of
calls to rcu_try_flip_waitack() finding all CPUs having already
acknowledged.)

o "z1" is the number of calls to rcu_try_flip_waitzero().

o "ze1" is the number of times that rcu_try_flip_waitzero() found
that not all of the old RCU read-side critical sections had
completed.

o "z2" is the number of times that rcu_try_flip_waitzero() finds
the sum of the counters equal to zero, in other words, that
all of the old RCU read-side critical sections had completed.
The value of "z1" should be roughly equal to "ze1" plus
"z2".

o "m1" is the number of calls to rcu_try_flip_waitmb().

o "me1" is the number of times that rcu_try_flip_waitmb() finds
that at least one CPU has not yet executed a memory barrier.

o "m2" is the number of times that rcu_try_flip_waitmb() finds that
all CPUs have executed a memory barrier.
The rcutree implementation of RCU provides debugfs trace output that
summarizes counters and state. This information is useful for debugging
RCU itself, and can sometimes also help to debug abuses of RCU.
The following sections describe the debugfs files and formats.


Hierarchical RCU debugfs Files and Formats
Expand Down Expand Up @@ -210,9 +35,10 @@ rcu_bh:
6 c=-275 g=-275 pq=1 pqc=-275 qp=0 dt=859/1 dn=0 df=15 of=0 ri=0 ql=0 b=10
7 c=-275 g=-275 pq=1 pqc=-275 qp=0 dt=3761/1 dn=0 df=15 of=0 ri=0 ql=0 b=10

The first section lists the rcu_data structures for rcu, the second for
rcu_bh. Each section has one line per CPU, or eight for this 8-CPU system.
The fields are as follows:
The first section lists the rcu_data structures for rcu_sched, the second
for rcu_bh. Note that CONFIG_TREE_PREEMPT_RCU kernels will have an
additional section for rcu_preempt. Each section has one line per CPU,
or eight for this 8-CPU system. The fields are as follows:

o The number at the beginning of each line is the CPU number.
CPUs numbers followed by an exclamation mark are offline,
Expand All @@ -223,9 +49,9 @@ o The number at the beginning of each line is the CPU number.

o "c" is the count of grace periods that this CPU believes have
completed. CPUs in dynticks idle mode may lag quite a ways
behind, for example, CPU 4 under "rcu" above, which has slept
through the past 25 RCU grace periods. It is not unusual to
see CPUs lagging by thousands of grace periods.
behind, for example, CPU 4 under "rcu_sched" above, which has
slept through the past 25 RCU grace periods. It is not unusual
to see CPUs lagging by thousands of grace periods.

o "g" is the count of grace periods that this CPU believes have
started. Again, CPUs in dynticks idle mode may lag behind.
Expand Down Expand Up @@ -308,8 +134,10 @@ The output of "cat rcu/rcugp" looks as follows:
rcu_sched: completed=33062 gpnum=33063
rcu_bh: completed=464 gpnum=464

Again, this output is for both "rcu" and "rcu_bh". The fields are
taken from the rcu_state structure, and are as follows:
Again, this output is for both "rcu_sched" and "rcu_bh". Note that
kernels built with CONFIG_TREE_PREEMPT_RCU will have an additional
"rcu_preempt" line. The fields are taken from the rcu_state structure,
and are as follows:

o "completed" is the number of grace periods that have completed.
It is comparable to the "c" field from rcu/rcudata in that a
Expand All @@ -324,23 +152,24 @@ o "gpnum" is the number of grace periods that have started. It is
If these two fields are equal (as they are for "rcu_bh" above),
then there is no grace period in progress, in other words, RCU
is idle. On the other hand, if the two fields differ (as they
do for "rcu" above), then an RCU grace period is in progress.
do for "rcu_sched" above), then an RCU grace period is in progress.


The output of "cat rcu/rcuhier" looks as follows, with very long lines:

c=6902 g=6903 s=2 jfq=3 j=72c7 nfqs=13142/nfqsng=0(13142) fqlh=6
1/1 0:127 ^0
3/3 0:35 ^0 0/0 36:71 ^1 0/0 72:107 ^2 0/0 108:127 ^3
3/3f 0:5 ^0 2/3 6:11 ^1 0/0 12:17 ^2 0/0 18:23 ^3 0/0 24:29 ^4 0/0 30:35 ^5 0/0 36:41 ^0 0/0 42:47 ^1 0/0 48:53 ^2 0/0 54:59 ^3 0/0 60:65 ^4 0/0 66:71 ^5 0/0 72:77 ^0 0/0 78:83 ^1 0/0 84:89 ^2 0/0 90:95 ^3 0/0 96:101 ^4 0/0 102:107 ^5 0/0 108:113 ^0 0/0 114:119 ^1 0/0 120:125 ^2 0/0 126:127 ^3
c=6902 g=6903 s=2 jfq=3 j=72c7 nfqs=13142/nfqsng=0(13142) fqlh=6 oqlen=0
1/1 .>. 0:127 ^0
3/3 .>. 0:35 ^0 0/0 .>. 36:71 ^1 0/0 .>. 72:107 ^2 0/0 .>. 108:127 ^3
3/3f .>. 0:5 ^0 2/3 .>. 6:11 ^1 0/0 .>. 12:17 ^2 0/0 .>. 18:23 ^3 0/0 .>. 24:29 ^4 0/0 .>. 30:35 ^5 0/0 .>. 36:41 ^0 0/0 .>. 42:47 ^1 0/0 .>. 48:53 ^2 0/0 .>. 54:59 ^3 0/0 .>. 60:65 ^4 0/0 .>. 66:71 ^5 0/0 .>. 72:77 ^0 0/0 .>. 78:83 ^1 0/0 .>. 84:89 ^2 0/0 .>. 90:95 ^3 0/0 .>. 96:101 ^4 0/0 .>. 102:107 ^5 0/0 .>. 108:113 ^0 0/0 .>. 114:119 ^1 0/0 .>. 120:125 ^2 0/0 .>. 126:127 ^3
rcu_bh:
c=-226 g=-226 s=1 jfq=-5701 j=72c7 nfqs=88/nfqsng=0(88) fqlh=0
0/1 0:127 ^0
0/3 0:35 ^0 0/0 36:71 ^1 0/0 72:107 ^2 0/0 108:127 ^3
0/3f 0:5 ^0 0/3 6:11 ^1 0/0 12:17 ^2 0/0 18:23 ^3 0/0 24:29 ^4 0/0 30:35 ^5 0/0 36:41 ^0 0/0 42:47 ^1 0/0 48:53 ^2 0/0 54:59 ^3 0/0 60:65 ^4 0/0 66:71 ^5 0/0 72:77 ^0 0/0 78:83 ^1 0/0 84:89 ^2 0/0 90:95 ^3 0/0 96:101 ^4 0/0 102:107 ^5 0/0 108:113 ^0 0/0 114:119 ^1 0/0 120:125 ^2 0/0 126:127 ^3
c=-226 g=-226 s=1 jfq=-5701 j=72c7 nfqs=88/nfqsng=0(88) fqlh=0 oqlen=0
0/1 .>. 0:127 ^0
0/3 .>. 0:35 ^0 0/0 .>. 36:71 ^1 0/0 .>. 72:107 ^2 0/0 .>. 108:127 ^3
0/3f .>. 0:5 ^0 0/3 .>. 6:11 ^1 0/0 .>. 12:17 ^2 0/0 .>. 18:23 ^3 0/0 .>. 24:29 ^4 0/0 .>. 30:35 ^5 0/0 .>. 36:41 ^0 0/0 .>. 42:47 ^1 0/0 .>. 48:53 ^2 0/0 .>. 54:59 ^3 0/0 .>. 60:65 ^4 0/0 .>. 66:71 ^5 0/0 .>. 72:77 ^0 0/0 .>. 78:83 ^1 0/0 .>. 84:89 ^2 0/0 .>. 90:95 ^3 0/0 .>. 96:101 ^4 0/0 .>. 102:107 ^5 0/0 .>. 108:113 ^0 0/0 .>. 114:119 ^1 0/0 .>. 120:125 ^2 0/0 .>. 126:127 ^3

This is once again split into "rcu" and "rcu_bh" portions. The fields are
as follows:
This is once again split into "rcu_sched" and "rcu_bh" portions,
and CONFIG_TREE_PREEMPT_RCU kernels will again have an additional
"rcu_preempt" section. The fields are as follows:

o "c" is exactly the same as "completed" under rcu/rcugp.

Expand Down Expand Up @@ -372,14 +201,19 @@ o "fqlh" is the number of calls to force_quiescent_state() that
exited immediately (without even being counted in nfqs above)
due to contention on ->fqslock.

o "oqlen" is the number of callbacks on the "orphan" callback
list. RCU callbacks are placed on this list by CPUs going
offline, and are "adopted" either by the CPU helping the outgoing
CPU or by the next rcu_barrier*() call, whichever comes first.

o Each element of the form "1/1 0:127 ^0" represents one struct
rcu_node. Each line represents one level of the hierarchy, from
root to leaves. It is best to think of the rcu_data structures
as forming yet another level after the leaves. Note that there
might be either one, two, or three levels of rcu_node structures,
depending on the relationship between CONFIG_RCU_FANOUT and
CONFIG_NR_CPUS.

o The numbers separated by the "/" are the qsmask followed
by the qsmaskinit. The qsmask will have one bit
set for each entity in the next lower level that
Expand All @@ -389,10 +223,19 @@ o Each element of the form "1/1 0:127 ^0" represents one struct
The value of qsmaskinit is assigned to that of qsmask
at the beginning of each grace period.

For example, for "rcu", the qsmask of the first entry
of the lowest level is 0x14, meaning that we are still
waiting for CPUs 2 and 4 to check in for the current
grace period.
For example, for "rcu_sched", the qsmask of the first
entry of the lowest level is 0x14, meaning that we
are still waiting for CPUs 2 and 4 to check in for the
current grace period.

o The characters separated by the ">" indicate the state
of the blocked-tasks lists. A "T" preceding the ">"
indicates that at least one task blocked in an RCU
read-side critical section blocks the current grace
period, while a "." preceding the ">" indicates otherwise.
The character following the ">" indicates similarly for
the next grace period. A "T" should appear in this
field only for rcu-preempt.

o The numbers separated by the ":" are the range of CPUs
served by this struct rcu_node. This can be helpful
Expand Down Expand Up @@ -431,8 +274,9 @@ rcu_bh:
6 np=120834 qsp=9902 cbr=0 cng=0 gpc=6 gps=3 nf=2 nn=110921
7 np=144888 qsp=26336 cbr=0 cng=0 gpc=8 gps=2 nf=0 nn=118542

As always, this is once again split into "rcu" and "rcu_bh" portions.
The fields are as follows:
As always, this is once again split into "rcu_sched" and "rcu_bh"
portions, with CONFIG_TREE_PREEMPT_RCU kernels having an additional
"rcu_preempt" section. The fields are as follows:

o "np" is the number of times that __rcu_pending() has been invoked
for the corresponding flavor of RCU.
Expand Down
2 changes: 1 addition & 1 deletion Documentation/RCU/whatisRCU.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -830,7 +830,7 @@ sched: Critical sections Grace period Barrier
SRCU: Critical sections Grace period Barrier

srcu_read_lock synchronize_srcu N/A
srcu_read_unlock
srcu_read_unlock synchronize_srcu_expedited

SRCU: Initialization/cleanup
init_srcu_struct
Expand Down
24 changes: 24 additions & 0 deletions include/linux/hardirq.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -139,10 +139,34 @@ static inline void account_system_vtime(struct task_struct *tsk)
#endif

#if defined(CONFIG_NO_HZ)
#if defined(CONFIG_TINY_RCU)
extern void rcu_enter_nohz(void);
extern void rcu_exit_nohz(void);

static inline void rcu_irq_enter(void)
{
rcu_exit_nohz();
}

static inline void rcu_irq_exit(void)
{
rcu_enter_nohz();
}

static inline void rcu_nmi_enter(void)
{
}

static inline void rcu_nmi_exit(void)
{
}

#else
extern void rcu_irq_enter(void);
extern void rcu_irq_exit(void);
extern void rcu_nmi_enter(void);
extern void rcu_nmi_exit(void);
#endif
#else
# define rcu_irq_enter() do { } while (0)
# define rcu_irq_exit() do { } while (0)
Expand Down
10 changes: 2 additions & 8 deletions include/linux/rcupdate.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,6 @@ struct rcu_head {
};

/* Exported common interfaces */
#ifdef CONFIG_TREE_PREEMPT_RCU
extern void synchronize_rcu(void);
#else /* #ifdef CONFIG_TREE_PREEMPT_RCU */
#define synchronize_rcu synchronize_sched
#endif /* #else #ifdef CONFIG_TREE_PREEMPT_RCU */
extern void synchronize_rcu_bh(void);
extern void synchronize_sched(void);
extern void rcu_barrier(void);
Expand All @@ -67,12 +62,11 @@ extern int sched_expedited_torture_stats(char *page);

/* Internal to kernel */
extern void rcu_init(void);
extern void rcu_scheduler_starting(void);
extern int rcu_needs_cpu(int cpu);
extern int rcu_scheduler_active;

#if defined(CONFIG_TREE_RCU) || defined(CONFIG_TREE_PREEMPT_RCU)
#include <linux/rcutree.h>
#elif defined(CONFIG_TINY_RCU)
#include <linux/rcutiny.h>
#else
#error "Unknown RCU implementation specified to kernel configuration"
#endif
Expand Down
Loading

0 comments on commit 6077817

Please sign in to comment.