Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic: Cache disabled but cached memory region accessed in v3.6.6 #794

Closed
kdomanski opened this issue Jan 29, 2023 · 14 comments
Closed

Panic: Cache disabled but cached memory region accessed in v3.6.6 #794

kdomanski opened this issue Jan 29, 2023 · 14 comments

Comments

@kdomanski
Copy link
Contributor

kdomanski commented Jan 29, 2023

Controller Board

FYSETC E4

Machine Description

Modded 3018

Input Circuits

No response

Configuration file

name: "3018 upgraded"
board: "FYSETC E4"
verbose_errors: true
use_line_numbers: true

start:
  must_home: false
  deactivate_parking: true
  check_limits: false

stepping:
  engine: RMT
  idle_ms: 255
  pulse_us: 2
  dir_delay_us: 1
  disable_delay_us: 1

axes:
  shared_stepper_disable_pin: gpio.25

  x:
    steps_per_mm: 800
    max_rate_mm_per_min: 5000
    acceleration_mm_per_sec2: 150
    max_travel_mm: 289
    soft_limits: false
    homing:
      cycle: 2
      allow_single_axis: true
      positive_direction: false
      mpos_mm: 1
      seek_mm_per_min: 1200

    motor0:
      hard_limits: false
      pulloff_mm: 4.000
      limit_all_pin: gpio.34
      tmc_2209:
        uart:
          txd_pin: gpio.22
          rxd_pin: gpio.21
          baud: 115200
          mode: 8N1
        addr: 1
        r_sense_ohms: 0.110
        run_amps: 1.0
        hold_amps: 0.250
        microsteps: 16
        toff_disable: 0
        toff_stealthchop: 5
        toff_coolstep: 3
        run_mode: CoolStep
        homing_mode: CoolStep
        use_enable: false
        step_pin: gpio.27:high
        direction_pin: gpio.26:low

  y:
    steps_per_mm: 800
    max_rate_mm_per_min: 5000
    acceleration_mm_per_sec2: 150
    max_travel_mm: 172
    soft_limits: false
    homing:
      cycle: 2
      allow_single_axis: true
      positive_direction: false
      mpos_mm: 1
      seek_mm_per_min: 1200

    motor0:
      hard_limits: false
      pulloff_mm: 2.000
      limit_all_pin: gpio.35
      tmc_2209:
        addr: 3
        r_sense_ohms: 0.110
        run_amps: 1.0
        hold_amps: 0.250
        microsteps: 16
        toff_disable: 0
        toff_stealthchop: 5
        toff_coolstep: 3
        run_mode: CoolStep
        homing_mode: CoolStep
        use_enable: false
        step_pin: gpio.33:high
        direction_pin: gpio.32:high
    # Optionally E as secondary Y motor
    # motor1:
    #   hard_limits: false
    #   pulloff_mm: 2.000
    #   tmc_2209:
    #     addr: 2
    #     r_sense_ohms: 0.110
    #     run_amps: 1.0
    #     hold_amps: 0.250
    #     microsteps: 4
    #     toff_disable: 0
    #     toff_stealthchop: 5
    #     toff_coolstep: 3
    #     run_mode: CoolStep
    #     homing_mode: CoolStep
    #     use_enable: false
    #     step_pin: gpio.16:high
    #     direction_pin: gpio.17:high

  z:
    steps_per_mm: 800
    max_rate_mm_per_min: 800
    acceleration_mm_per_sec2: 24
    max_travel_mm: 1000
    homing:
      cycle: 1
      allow_single_axis: true
      mpos_mm: 1
      seek_mm_per_min: 500

    motor0:
      hard_limits: false
      pulloff_mm: 2.000
      limit_pos_pin: gpio.36
      tmc_2209:
        addr: 0
        r_sense_ohms: 0.110
        run_amps: 1.0
        hold_amps: 0.250
        microsteps: 16
        toff_disable: 0
        toff_stealthchop: 5
        toff_coolstep: 3
        run_mode: CoolStep
        homing_mode: CoolStep
        use_enable: false
        step_pin: gpio.14:high
        direction_pin: gpio.12:low

spi:
  miso_pin: gpio.19
  mosi_pin: gpio.23
  sck_pin: gpio.18

sdcard:
  cs_pin: gpio.5
  card_detect_pin: NO_PIN

coolant:
  mist_pin: gpio.4
  delay_ms: 50

probe:
  pin: gpio.15:low
  check_mode_start: false

pwm:
  pwm_hz: 5000
  direction_pin: NO_PIN
  output_pin: gpio.2
  enable_pin: NO_PIN
  disable_with_s0: false
  s0_with_disable: true
  spinup_ms: 10
  spindown_ms: 10
  tool_num: 0
  speed_map: 0=0.000% 10000=100.000%
  off_on_alarm: true

Startup Messages

[MSG:INFO: FluidNC v3.6.6]                                                                                                                                                                                                                    
[MSG:INFO: Compiled with ESP32 SDK:v4.4.1-1-gb8050b365e]                                                                                                                                                                                      
[MSG:INFO: Local filesystem type is spiffs]                                                                                                                                                                                                   
[MSG:INFO: Configuration file:spinny_e4.yaml]                                                                                                                                                                                                 
[MSG:INFO: Machine 3018 upgraded]                                                                                                                                                                                                             
[MSG:INFO: Board FYSETC E4]                                                                                                                                                                                                          
[MSG:INFO: SPI SCK:gpio.18 MOSI:gpio.23 MISO:gpio.19]                                                                                                                                                                                         
[MSG:INFO: SD Card cs_pin:gpio.5 detect:NO_PIN]                                                                                                                                                                                               
[MSG:INFO: Stepping:RMT Pulse:2us Dsbl Delay:1us Dir Delay:1us Idle Delay:255ms]                                                                                                                                                              
[MSG:INFO: Axis count 3]                                                                                                                                                                                                                      
[MSG:INFO: Shared stepper disable gpio.25]                                                                                                                                                                                                    
[MSG:INFO: Axis X (1.000,290.000)]                                                                                                                                                                                                            
[MSG:INFO:   Motor0]                                                                                                                                                                                                                          
[MSG:INFO: Trinamic Stepper Uart Tx:gpio.22 Rx:gpio.21 RTS:NO_PIN Baud:115200]                                                                                                                                                                
[MSG:INFO:     tmc_2209 Step:gpio.27 Dir:gpio.26:low Disable:NO_PIN Addr:1 R:0.110]                                                                                                                                                           
[MSG:INFO:  X All Limit gpio.34]                                                                                                                                                                                                              
[MSG:INFO: Axis Y (1.000,173.000)]                                                                                                                                                                                                            
[MSG:INFO:   Motor0]                                                                                                                                                                                                                          
[MSG:INFO:     tmc_2209 Step:gpio.33 Dir:gpio.32 Disable:NO_PIN Addr:3 R:0.110]                                                                                                                                                               
[MSG:INFO:  Y All Limit gpio.35]                                                                                                                                                                                                              
[MSG:INFO: Axis Z (-999.000,1.000)]                                                                                                                                                                                                           
[MSG:INFO:   Motor0]                                                                                                                                                                                                                          
[MSG:INFO:     tmc_2209 Step:gpio.14 Dir:gpio.12:low Disable:NO_PIN Addr:0 R:0.110]                                                                                                                                                           
[MSG:INFO:  Z Pos Limit gpio.39]                                                                                                                                                                                                              
[MSG:INFO: X Axis driver test passed]                                                                                                                                                                                                         
[MSG:INFO: Y Axis driver test passed]                                                                                                                                                                                                         
[MSG:INFO: Z Axis driver test passed]                                                                                                                                                                                                         
[MSG:INFO: Kinematic system: Cartesian]                                                                                                                                                                                                       
[MSG:INFO: PWM Spindle Ena:NO_PIN Out:gpio.2 Dir:NO_PIN Freq:5000Hz Period:8191]                                                                                                                                                              
[MSG:INFO: Using spindle PWM]                                                                                                                                                                                                                 
[MSG:INFO: Mist coolant gpio.4]                                                                                                                                                                                                               
[MSG:INFO: Probe Pin: gpio.15:low]                                                                                                                                                                                                            
[MSG:INFO: AP SSID FluidNC-E4 IP 192.168.0.1 mask 255.255.255.0 channel 1]                                                                                                                                                                    
[MSG:INFO: AP started]                                                                                                                                                                                                                        
[MSG:INFO: WiFi on]                                                                                                                                                                                                                           
[MSG:INFO: Captive Portal Started]                                                                                                                                                                                                            
[MSG:INFO: HTTP started on port 80]                                                                                                                                                                                                           
[MSG:INFO: Telnet started on port 23] 

What happened?

While running gcode, the controller randomly panics:

Guru Meditation Error: Core  1 panic'ed (Cache disabled but cached memory region accessed).

Core  1 register dump:
PC      : 0x40112248  PS      : 0x00060035  A0      : 0x8008a724  A1      : 0x3ffc030c
A2      : 0x3ffb4ad4  A3      : 0x00000000  A4      : 0x3ffbed08  A5      : 0x3ffcd4ac
A6      : 0x3ffca344  A7      : 0x3ffb26fc  A8      : 0x80081e21  A9      : 0x00000002
A10     : 0x0000001a  A11     : 0x00000001  A12     : 0x3ffba630  A13     : 0x3ffba610
A14     : 0x00000004  A15     : 0x3ffcd0d4  SAR     : 0x00000000  EXCCAUSE: 0x00000007
EXCVADDR: 0x00000000  LBEG    : 0x00000000  LEND    : 0x00000000  LCOUNT  : 0x00000000


Backtrace:0x40112245:0x3ffc030c |<-CORRUPTED




ELF file SHA256: 0000000000000000

Rebooting...
ets Jul 29 2019 12:21:46

rst:0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1184
load:0x40078000,len:12812
load:0x40080400,len:3032
entry 0x400805e4

Other Information

No response

@kdomanski
Copy link
Contributor Author

The <-CORRUPTED not at the backtrace is surprising. Any advice how I might try to get a proper backtrace instead?

I looked up the PC address 0x40112248 in the v3.6.6 ELF, but that just points to __digitalWrite.

@bdring
Copy link
Owner

bdring commented Jan 29, 2023

We have some info on using the backtrace here.

http://wiki.fluidnc.com/en/installation#backtrace-decoding-windows

@kdomanski
Copy link
Contributor Author

Thanks, that helped a bit, but I still got nothing that makes sense.

addr2line points me to the framework code itself:

❯ addr2line -a 0x40112245:0x3ffc030c -e /Users/kdomanski/Downloads/wifi-firmware-fluidnc-v3.6.6.elf
0x40112245
/home/runner/.platformio/packages/framework-arduinoespressif32/cores/esp32/esp32-hal-gpio.c:108

My findings so far:

@MitchBradley
Copy link
Collaborator

"Cache disabled but cached memory region accessed" usually means that the ESP32 is trying to fill its instruction cache from FLASH in an interrupt service routine. We try to mark all ISR code with IRAM_ATTR so it runs from instruction RAM and does not need to access FLASH, but the C++ compiler makes that difficult. Both switch statements and virtual methods generate code fragments that land in FLASH, regardless of IRAM_ATTR on their enclosing functions. We have a linker-script workaround for that but sometimes we miss something. The backtrace is often fairly useless for debugging this situation. I will try to repro the problem and see if I can find out what we missed.

@MitchBradley
Copy link
Collaborator

Please supply a GCode file that panics. Also tell us what sender you are using - WebUI, UGS, or whatever. If you are using a serial sender, is WebUI also connected?

How long does it take before the panic happens? I know that it is unlikely to be deterministic, but is it a few minutes, or hours, or what?

@kdomanski
Copy link
Contributor Author

1001.nc.zip

I've been streaming with UGS over serial, with the WebUI opened in background. The panic happens within 5 to 40 minutes.

I was unable to replicate the problem while streaming from SD (command issued in Fluidterm) without the browser UI opened.

@MitchBradley
Copy link
Collaborator

Is WebUI using auto-reporting or polling?
image

@MitchBradley
Copy link
Collaborator

Ouch. __digitalWrite is in FLASH. That is a huge problem because it is used by the stepper interrupt, and it could be displaced from icache and then refilled during an ISR.

@MitchBradley
Copy link
Collaborator

Foiled again by the Arduino Framework:

framework-arduinoespressif32/cores/esp32/esp-hal-gpio.c:

extern void ARDUINO_ISR_ATTR __digitalWrite(uint8_t pin, uint8_t val)
{
   gpio_set_level((gpio_num_t)pin, val);
}

framework-arduinoespressif32/cores/esp32/esp-hal.h:

#if CONFIG_ARDUINO_ISR_IRAM
#define ARDUINO_ISR_ATTR IRAM_ATTR
#define ARDUINO_ISR_FLAG ESP_INTR_FLAG_IRAM
#else
#define ARDUINO_ISR_ATTR
#define ARDUINO_ISR_FLAG (0)
#endif

framework-arduinoespressif32/tools/sdk/esp32/sdkconfig:

# CONFIG_ARDUINO_ISR_IRAM is not set

@kdomanski
Copy link
Contributor Author

Is WebUI using auto-reporting or polling?

Auto-reporting, 50ms.


Food for thought: while researching the issue, I came upon a post that suggested the following:

  1. Turn the complex code that's currently called from ISR into a sleeping task.
  2. The actual ISR only wakes/notifies that task (or posts data to a queue) and exits.
  3. Keep IRAM_ATTR on the performance-critical code to avoid performance penalty.

Would that work?

@MitchBradley
Copy link
Collaborator

MitchBradley commented Jan 29, 2023

I may have fixed it. Re your suggestion: The step timing code in the GPIO case is too critical, by far, to do in a task. In the I2SO case, it already works that way.

@kdomanski
Copy link
Contributor Author

Hey, I just saw the PR. Thanks for the quick response.

Tomorrow I'll try to reproduce with and without that patch.

@kdomanski
Copy link
Contributor Author

For the record, linking to results of the test: #796 (comment)

MitchBradley added a commit that referenced this issue Feb 10, 2023
Fix #794 - change Arduino __digital* to gpio_* in IRAM
@kdomanski
Copy link
Contributor Author

Fixed in Devt by #796

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants