GCC ARM: Increase develop and release debug level #11059

desowin · 2019-07-16T14:29:49Z

Description

Do not specify the debug level for develop and release profiles. Instead
rely on the compiler to choose sensible default (-g2). Note that -g1 is
minimal debugging information and does not include structure definitions
which quite heavily reduces debugging experience.

For develop and release profiles this results in elf file containing
structure definitions. This does not impact debug profile as it already
did use -g3 which is the highest debug level.

Compatible debuggers (eg. gdb, SEGGER Ozone) can use the extra information
to provide better debugging experience. For example, when compiled .elf is
loaded in gdb, this change makes it trivial to access internal RTX data.

Without this change on develop profile:
(gdb) print osRtxInfo.thread.run
'osRtxInfo' has unknown type; cast it to its declared type

With this change on develop profile:
(gdb) print osRtxInfo.thread.run
$1 = {curr = 0x20014F04, next = 0x20014F04}

Pull request type

[X] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[ ] Test update
[ ] Breaking change

Reviewers

Release Notes

desowin · 2019-07-16T14:40:04Z

This might be platform dependent. I am compiling using the GNU Tools ARM Embedded toolchain on Windows.

ciarmcom · 2019-07-16T15:00:33Z

@desowin, thank you for your changes.
@ARMmbed/mbed-os-tools @ARMmbed/mbed-os-maintainers please review.

kjbracey · 2019-07-16T15:08:11Z

Interesting. So what format is it using by default?

I'd assumed it was defaulting to a sensible version of DWARF.

If specifying, why not -ggdb? (Never really messed with these settings myself)

desowin · 2019-07-17T06:04:53Z

Interesting. So what format is it using by default?

I don't know and neither I do not know how to check for sure... I started inspecting the produced elf files (readelf -S target.elf) and noticed that the one compiled with -gdwarf has one additional section named .debug_loc. I guess this is what makes the difference when debugging this elf file.

Without this patch I get following:

There are 25 section headers, starting at offset 0x3c0a1c:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        08000000 000180 056f94 00  AX  0   0 64
  [ 2] .ARM.exidx        ARM_EXIDX       08056f94 057114 000008 00  AL  1   0  4
  [ 3] .crash_data_ram   NOBITS          200001c8 067238 000100 00  WA  0   0  1
  [ 4] .data             PROGBITS        200002c8 057120 000e60 00  WA  0   0  8
  [ 5] .bss              NOBITS          20001128 065ac0 014b58 00  WA  0   0  8
  [ 6] .heap             PROGBITS        20015c80 1e4da8 039f80 00   W  0   0  1
  [ 7] ExtRAMSection     NOBITS          c0000000 1e4da8 07f800 00  WA  0   0  4
  [ 8] ExtFlashSection   PROGBITS        90000000 067238 17db70 00   A  0   0  4
  [ 9] FontFlashSection  PROGBITS        08057dfc 057f80 00db3c 00   A  0   0  4
  [10] TextFlashSection  PROGBITS        08065938 065abc 00177c 00   A  0   0  4
  [11] .ARM.attributes   ARM_ATTRIBUTES  00000000 21ed28 00002e 00      0   0  1
  [12] .comment          PROGBITS        00000000 21ed56 0000fe 01  MS  0   0  1
  [13] .debug_info       PROGBITS        00000000 21ee54 04cac5 00      0   0  1
  [14] .debug_abbrev     PROGBITS        00000000 26b919 0121d5 00      0   0  1
  [15] .debug_aranges    PROGBITS        00000000 27daf0 00c610 00      0   0  8
  [16] .debug_ranges     PROGBITS        00000000 28a100 017a80 00      0   0  8
  [17] .debug_line       PROGBITS        00000000 2a1b80 048fc0 00      0   0  1
  [18] .debug_str        PROGBITS        00000000 2eab40 04fdfa 01  MS  0   0  1
  [19] .debug_frame      PROGBITS        00000000 33a93c 024d0c 00      0   0  4
  [20] .stab             PROGBITS        00000000 35f648 00003c 0c     21   0  4
  [21] .stabstr          STRTAB          00000000 35f684 000076 00      0   0  1
  [22] .symtab           SYMTAB          00000000 35f6fc 02dcd0 10     23 7679  4
  [23] .strtab           STRTAB          00000000 38d3cc 03353f 00      0   0  1
  [24] .shstrtab         STRTAB          00000000 3c090b 000110 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  y (purecode), p (processor specific)

With this patch, the output is:

There are 26 section headers, starting at offset 0x87d98c:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        08000000 000180 056f94 00  AX  0   0 64
  [ 2] .ARM.exidx        ARM_EXIDX       08056f94 057114 000008 00  AL  1   0  4
  [ 3] .crash_data_ram   NOBITS          200001c8 067238 000100 00  WA  0   0  1
  [ 4] .data             PROGBITS        200002c8 057120 000e60 00  WA  0   0  8
  [ 5] .bss              NOBITS          20001128 065ac0 014b58 00  WA  0   0  8
  [ 6] .heap             PROGBITS        20015c80 1e4da8 039f80 00   W  0   0  1
  [ 7] ExtRAMSection     NOBITS          c0000000 1e4da8 07f800 00  WA  0   0  4
  [ 8] ExtFlashSection   PROGBITS        90000000 067238 17db70 00   A  0   0  4
  [ 9] FontFlashSection  PROGBITS        08057dfc 057f80 00db3c 00   A  0   0  4
  [10] TextFlashSection  PROGBITS        08065938 065abc 00177c 00   A  0   0  4
  [11] .ARM.attributes   ARM_ATTRIBUTES  00000000 21ed28 00002e 00      0   0  1
  [12] .comment          PROGBITS        00000000 21ed56 0000fe 01  MS  0   0  1
  [13] .debug_info       PROGBITS        00000000 21ee54 3a2336 00      0   0  1
  [14] .debug_abbrev     PROGBITS        00000000 5c118a 05d8c9 00      0   0  1
  [15] .debug_loc        PROGBITS        00000000 61ea53 0ae86f 00      0   0  1
  [16] .debug_aranges    PROGBITS        00000000 6cd2c8 00cd00 00      0   0  8
  [17] .debug_ranges     PROGBITS        00000000 6d9fc8 019ba8 00      0   0  8
  [18] .debug_line       PROGBITS        00000000 6f3b70 086a7d 00      0   0  1
  [19] .debug_str        PROGBITS        00000000 77a5ed 07d2a0 01  MS  0   0  1
  [20] .debug_frame      PROGBITS        00000000 7f7890 024d0c 00      0   0  4
  [21] .stab             PROGBITS        00000000 81c59c 00003c 0c     22   0  4
  [22] .stabstr          STRTAB          00000000 81c5d8 000076 00      0   0  1
  [23] .symtab           SYMTAB          00000000 81c650 02dce0 10     24 7680  4
  [24] .strtab           STRTAB          00000000 84a330 03353f 00      0   0  1
  [25] .shstrtab         STRTAB          00000000 87d86f 00011b 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  y (purecode), p (processor specific)

If specifying, why not -ggdb? (Never really messed with these settings myself)

DWARF is debugging standard, that can be used also with tools other than gdb. I don't know how the -ggdb output is different to -gdwarf. I guess that you could find some 3rd-party tools that work fine with elf files containing DWARF symbols but fail at gdb symbols.

kjbracey · 2019-07-17T06:54:12Z

Is this maybe related to this? https://stackoverflow.com/questions/32734222/why-does-gdb-7-4-does-not-show-source-info-on-binaries-generated-with-recent-com

But you're using a gdb and gcc from the same distribution, right, so surely they should understand each other?

arm-none-eabi-readelf --debug-dump=info shows "Version: 4" with -g, suggesting that it is using DWARFv4 by default - maybe -gdwarf is selecting an older version? Docs aren't clear to me whether that's means "v1" or "default version = 4", but if the latter, then why's it changing?

kjbracey · 2019-07-17T06:57:26Z

I don't know how the -ggdb output is different to -gdwarf

From my reading of the docs, it enables gdb extensions to the base format, which should(TM) be compatible.

I think my main point is that we need to understand exactly what this option is changing - what was the format before, what is it after, and why is this change needed and better than any other potential change.

desowin · 2019-07-17T07:29:54Z

Is this maybe related to this? https://stackoverflow.com/questions/32734222/why-does-gdb-7-4-does-not-show-source-info-on-binaries-generated-with-recent-com

That's probably a different issue. The source code information is visible without this change. Adding -gdwarf or -ggdb does result in the structure definitions to be included as well. You can still get some debugging (eg. place breakpoints, do C line stepping) without the structure definitions, but it is seriously impacted in my opinion.

But you're using a gdb and gcc from the same distribution, right, so surely they should understand each other?

Yes, the gdb is from the same GNU Tools ARM Embedded package. I am also using also SEGGER Ozone which is standalone 3rd party debugger - and I observe exactly the same difference as in gdb (only when compiled with -gdwarf or -ggdb, SEGGER Ozone is able to show all the structure variables field names, values, offsets and so on).

arm-none-eabi-readelf --debug-dump=info shows "Version: 4" with -g, suggesting that it is using DWARFv4 by default - maybe -gdwarf is selecting an older version? Docs aren't clear to me whether that's means "v1" or "default version = 4", but if the latter, then why's it changing?

I am seeing "Version: 4" both with and without this change.

kjbracey · 2019-07-17T07:36:24Z

You're right, his symptoms don't match, but his observation about missing debug_loc and the answer that it's responsible for finding variables does.

I believe I have noticed the symptoms you describe - inability to locate global data, but it's never really bothered me enough to track down - I'm usually interested in local variables or members of this.

I think that's what you're seeing, right? It's not seeing structures that's the problem, it's the ability to locate global named variables?

(The symbols are there though, which is how pyOCD thread debugging works - it can find the symbol for osRtxInfo, and show the info using its built-in type layout info).

Can you work through all the options in the manual pin down an explicit option that produces the same output as -g to confirm what on earth that is defaulting to?

desowin · 2019-07-17T07:39:52Z

I think my main point is that we need to understand exactly what this option is changing - what was the format before, what is it after, and why is this change needed and better than any other potential change.

According to gcc docs, -g does "Produce debugging information in the operating system’s native format (stabs, COFF, XCOFF, or DWARF).". Thus essentially it can be something else than DWARF. However, it seems like the debug information is still DWARF without this option atleast on Windows when crosscompiling to Cortex-M7 (atleast I cannot tell the difference by looking at the elf file with my very limited understanding of the format).

The biggest difference I observe, is that adding -gdwarf does result in the structure definitions to be included in the elf file in a way that is understood by both gdb and SEGGER Ozone.

kjbracey · 2019-07-17T07:43:15Z

"the operating system’s native format"

I'd like to assume that doesn't apply for a cross-compiler, or at least would apply to the target operating system, so should be platform independent. The OS in this case is "none", so that default could be anything.

I would like to think there would be some explicit option that produces the same actual binary as '-g' and it isn't producing some particular unique magic.

desowin · 2019-07-17T07:43:17Z

I think that's what you're seeing, right? It's not seeing structures that's the problem, it's the ability to locate global named variables?

Yes, that's the problem.

(The symbols are there though, which is how pyOCD thread debugging works - it can find the symbol for osRtxInfo, and show the info using its built-in type layout info).

If pyOCD does have its own knowledge of how osRtxInfo is structured, then yes, it wouldn't need it in the elf file. Without this change, I know where osRtxInfo is located. I cannot access the inidividual fields without knowing from some other source the offsets. With this change, the elf contains all the offsets.

Can you work through all the options in the manual pin down an explicit option that produces the same output as -g to confirm what on earth that is defaulting to?

I can give it a shot, but I don't think I'll manage to pinpoint it.

desowin · 2019-07-17T14:31:56Z

Can you work through all the options in the manual pin down an explicit option that produces the same output as -g to confirm what on earth that is defaulting to?

I can give it a shot, but I don't think I'll manage to pinpoint it.

It was way too hard to go through the options, so I checked the gcc source code. In gcc/opts.c you can check that the handling of -g and -gdwarf is a bit different. The -g does result in:
set_debug_level (NO_DEBUG, DEFAULT_GDB_EXTENSIONS, arg, opts, opts_set, loc);
while the -gdwarf does result in:
set_debug_level (DWARF2_DEBUG, false, "", opts, opts_set, loc);

Note that optional dwarf version parameter is stored in opts->x_dwarf_version and the DWARF2_DEBUG is common definition used for all supported DWARF versions. Then inside set_debug_level() there is

  /* A debug flag without a level defaults to level 2.
     If off or at level 1, set it to level 2, but if already
     at level 3, don't lower it.  */ 
  if (*arg == '\0')
    {
      if (opts->x_debug_info_level < DINFO_LEVEL_NORMAL)
	opts->x_debug_info_level = DINFO_LEVEL_NORMAL;
    }
  else
    {
      int argval = integral_argument (arg);
      if (argval == -1)
	error_at (loc, "unrecognized debug output level %qs", arg);
      else if (argval > 3)
	error_at (loc, "debug output level %qs is too high", arg);
      else
	opts->x_debug_info_level = (enum debug_info_levels) argval;
    }

The -gdwarf option appearing after -g1 does in change the opts->x_debug_info_level to 2.
Thus the big difference between -gdwarf and settings only the level is the fact that settings only the level results in opts->x_write_symbols being set to PREFERRED_DEBUGGING_TYPE, while specifying the -gdwarf forces it to DWARF2_DEBUG.

The summary is:

if you pass -gdwarf and -g1 to gcc and you want the g1 to take effect, then -gdwarf should appear before -g1
if -g1 is passed then (atleast the osRtxInfo.thread.run) structure information is not stored, even when DWARF debug info is explicitly requested (-gdwarf -g1 on gcc commandline)

I would propose to not use -g1 as it makes debugging harder and it only affects the .elf file. The ARM mbed case is a bit different than the normal applications (eg. Linux) that execute from the elf directly, as it is the .hex or .bin file contents that are flashed to the target device. We could pass only the -gdwarf on to make it explicit that we want DWARF and just rely on the default debugging level which should be enough. ~~For debug, use -gdwarf -g3 as the other order (-g3 -gdwarf does negate the -g3).~~ (edit note: scratched the last sentence as it is not true)

kjbracey · 2019-07-17T14:49:17Z

Thank you for thorough investigation - I'd totally failed to register that we were passing level values to -g. I think that makes sense now.

But what then is the point of actually adding the -gdwarf? You got the benefit because it overrode the -g1 back to -g2. Is it not sufficient to just change -g1 to -g?

If I had to guess why not -g, I'd say it's for CI test farm disk space (and maybe speed?). Not sure if that was a real or theoretical concern though.

Looking at the history, it was @SeppoTakalo who put the -g1 in, but that was actually an upgrade from "no debugging info for release and develop". See #5708 - there was a bit of discussion there.

If this is actually working fine on the debug build, as it has -g3, I think this change is a little debatable now - we need to consider our CI and online compiler load.

desowin · 2019-07-17T14:54:08Z

In #5708 @SeppoTakalo noted:

-g1 is selected because "develop" and "release" builds are optimised and therefore cannot be accurately debugged. Therefore only minimal debug info should be provided.

As my investigation turns out, there is benefit in providing -g2 level for optimized builds as with -g1 you essentially loose the structure informations and local variables information - which is really useful on its own even if the code is optimized and single stepping can be confusing.

desowin · 2019-07-17T14:54:39Z

But what then is the point of actually adding the -gdwarf? You got the benefit because it overrode the -g1 back to -g2. Is it not sufficient to just change -g1 to -g?

The -g would do the trick. The -gdwarf makes it explicit that the symbols will be in DWARF format and not in operating system’s native format which is a bit ambiguous for "none" targets.

0Grit · 2019-07-17T16:55:44Z

@ARMmbed/team-embeddedplanet This conversation is worth it's weight in rhodium.

SeppoTakalo · 2019-07-17T20:48:45Z

@kjbracey-arm Our CI rarely keeps the elf files. Test runs seems to utilise binary files.
Therefore I would not be concerned of minimal increase on .elf file size, as it mainly stays on developers work environment.

Did you guys now conclude whether -gdwarf -g3 makes sense? Should it be just -g on everywhere?

desowin · 2019-07-18T05:24:53Z

Did you guys now conclude whether -gdwarf -g3 makes sense? Should it be just -g on everywhere?

Yes, the -gdwarf -g3 does make sense. It provides some extra debug information over just -gdwarf.

The choice between -gdwarf and -g at the moment is more of a choice/preference/style/verbosity than any real practical reason. This is because even though -g might result in other symbol format than DWARF, it (at least now) does result in DWARF in case of arm-none-eabi.

If you prefer just -g over -gdwarf please let me know and I'll update this pull request.

kjbracey · 2019-07-18T09:53:06Z

I don't see that we have any requirement to explicitly specify DWARF, as opposed to accepting the toolchain's default. The normal assumption should be that the default is correct, and it is.

By specifying DWARF, we'd potentially make the system perform worse if a later toolchain release changes default format, by locking ourselves into the current behaviour.

The only reason I can see for "dwarf" being in the PR is that you found that it fixed the issue without understanding why.

I want to just change the -g1 to -g.

Do not specify the debug level for develop and release profiles. Instead rely on the compiler to choose sensible default (-g2). Note that -g1 is minimal debugging information and does not include structure definitions which quite heavily reduces debugging experience. For develop and release profiles this results in elf file containing structure definitions. This does not impact debug profile as it already did use -g3 which is the highest debug level. Compatible debuggers (eg. gdb, SEGGER Ozone) can use the extra information to provide better debugging experience. For example, when compiled .elf is loaded in gdb, this change makes it trivial to access internal RTX data. Without this change on develop profile: (gdb) print osRtxInfo.thread.run 'osRtxInfo' has unknown type; cast it to its declared type With this change on develop profile: (gdb) print osRtxInfo.thread.run $1 = {curr = 0x20014F04, next = 0x20014F04}

mbed-ci · 2019-07-18T20:30:57Z

Test run: SUCCESS

Summary: 11 of 11 test jobs passed
Build number : 1
Build artifacts

ciarmcom requested review from a team July 16, 2019 15:00

ciarmcom added the needs: review label Jul 16, 2019

desowin force-pushed the dwarf branch from f610be9 to c0f97d4 Compare July 17, 2019 14:40

desowin changed the title ~~GCC ARM: Produce DWARF debugging information~~ GCC ARM: Explicitly request DWARF debug info Jul 17, 2019

desowin force-pushed the dwarf branch from c0f97d4 to 791812e Compare July 17, 2019 14:48

desowin force-pushed the dwarf branch from 791812e to 68e9b3e Compare July 18, 2019 10:02

desowin force-pushed the dwarf branch from 68e9b3e to 4e992cd Compare July 18, 2019 10:03

desowin changed the title ~~GCC ARM: Explicitly request DWARF debug info~~ GCC ARM: Increase develop and release debug level Jul 18, 2019

kjbracey approved these changes Jul 18, 2019

View reviewed changes

SeppoTakalo approved these changes Jul 22, 2019

View reviewed changes

SeppoTakalo added ready for merge release-version: 5.13.2 and removed needs: review labels Jul 22, 2019

SeppoTakalo merged commit a0e88ad into ARMmbed:master Jul 22, 2019

SeppoTakalo removed the ready for merge label Jul 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCC ARM: Increase develop and release debug level #11059

GCC ARM: Increase develop and release debug level #11059

desowin commented Jul 16, 2019 •

edited

Loading

desowin commented Jul 16, 2019

ciarmcom commented Jul 16, 2019

kjbracey commented Jul 16, 2019

desowin commented Jul 17, 2019 •

edited

Loading

kjbracey commented Jul 17, 2019

kjbracey commented Jul 17, 2019

desowin commented Jul 17, 2019

kjbracey commented Jul 17, 2019

desowin commented Jul 17, 2019

kjbracey commented Jul 17, 2019

desowin commented Jul 17, 2019

desowin commented Jul 17, 2019 •

edited

Loading

kjbracey commented Jul 17, 2019

desowin commented Jul 17, 2019 •

edited

Loading

desowin commented Jul 17, 2019 •

edited

Loading

0Grit commented Jul 17, 2019

SeppoTakalo commented Jul 17, 2019

desowin commented Jul 18, 2019

kjbracey commented Jul 18, 2019

mbed-ci commented Jul 18, 2019

GCC ARM: Increase develop and release debug level #11059

GCC ARM: Increase develop and release debug level #11059

Conversation

desowin commented Jul 16, 2019 • edited Loading

Description

Pull request type

Reviewers

Release Notes

desowin commented Jul 16, 2019

ciarmcom commented Jul 16, 2019

kjbracey commented Jul 16, 2019

desowin commented Jul 17, 2019 • edited Loading

kjbracey commented Jul 17, 2019

kjbracey commented Jul 17, 2019

desowin commented Jul 17, 2019

kjbracey commented Jul 17, 2019

desowin commented Jul 17, 2019

kjbracey commented Jul 17, 2019

desowin commented Jul 17, 2019

desowin commented Jul 17, 2019 • edited Loading

kjbracey commented Jul 17, 2019

desowin commented Jul 17, 2019 • edited Loading

desowin commented Jul 17, 2019 • edited Loading

0Grit commented Jul 17, 2019

SeppoTakalo commented Jul 17, 2019

desowin commented Jul 18, 2019

kjbracey commented Jul 18, 2019

mbed-ci commented Jul 18, 2019

Test run: SUCCESS

desowin commented Jul 16, 2019 •

edited

Loading

desowin commented Jul 17, 2019 •

edited

Loading

desowin commented Jul 17, 2019 •

edited

Loading

desowin commented Jul 17, 2019 •

edited

Loading

desowin commented Jul 17, 2019 •

edited

Loading