Suspend on a laptop reverts cpu to previous speed

[sfauzia]

pstate-frequency in powersaving mode (via "pstate-frequency -S -p auto" in my script triggered by a udev rule) perfectly sets my CPU to 800 MHz. However, upon resuming from suspend the CPU ramps up to 2.0 GHz, which rapidly depletes battery life, and running the aforementioned pstate-frequency command does not revert the computer to the original, pre-suspend state. All my other powersaving measures post suspend do function normally, so it does not appear to be a problem with the udev rule. (The pstate-frequency command should work from the cli were that the case.)

Prior to suspend, on battery power:
% cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: 0.97 ms.
hardware limits: 800 MHz - 3.30 GHz
available cpufreq governors: performance, powersave
current policy: frequency should be within 800 MHz and 825 MHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency is 800 MHz.
boost state support:
Supported: yes
Active: yes

% pstate-frequency -G r
pstate-frequency 1.2.2[g++]
pstate::CPU_DRIVER -> intel_pstate
pstate::CPU_GOVERNOR -> powersave
pstate::NO_TURBO -> 1 : OFF
pstate::CPU_MIN -> 24% : 800000KHz
pstate::CPU_MAX -> 25% : 825000KHz

After suspend, on battery power:
% cpupower frequency-info
analyzing CPU 0:
driver: intel_pstate
CPUs which run at the same hardware frequency: 0
CPUs which need to have their frequency coordinated by software: 0
maximum transition latency: 0.97 ms.
hardware limits: 800 MHz - 3.30 GHz
available cpufreq governors: performance, powersave
current policy: frequency should be within 800 MHz and 825 MHz.
The governor "powersave" may decide which speed to use
within this range.
current CPU frequency is 2.00 GHz (asserted by call to hardware).
boost state support:
Supported: yes
Active: yes

% pstate-frequency -G r
pstate-frequency 1.2.2[g++]
pstate::CPU_DRIVER -> intel_pstate
pstate::CPU_GOVERNOR -> powersave
pstate::NO_TURBO -> 1 : OFF
pstate::CPU_MIN -> 24% : 800000KHz
pstate::CPU_MAX -> 25% : 825000KH

If there is anything else useful I may provide, please let me know. I am using a Fujjitsu T904 running on the Haswell Core i7 Processor, in ArchLinux 64-bit. Latest kernel 4.0.4-2.

[pyamsoft]

Hello, thank you for your report. Before we dive into this to figure out the problem, I want to clarify two things.

First is the reading that you get from pstate-frequency compared to the reading from cpupower. pstate-frequency reads (and writes) the CPU_MIN and CPU_MAX values from the files nin /sys/devices/system/cpu/cpu#/cpufreq/scaling_{max,min}_freq

The numbers being outputted then, for example 825000 for the MAX should be the same as you see in the files /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq. Can you confirm this? Can you also confirm that because cpupower states that the CPU frequency is 2GHz, the machine does in fact "feel" as if it were running at that speed.

Second is the fact that, despite best efforts, pstate-frequency can not actually take complete and total control over the CPU frequencies. The frequencies are decided in the hardware itself, the pstates can only serve as "suggestions" to the driver to run at specified rates. Should the hardware choose to ignore the request to run at 25% instead of 100%, it shall. As a result, there is no way on modern hardware to guarantee that the intel_pstate driver will be able to control the cpu freqeuncies as fully as it were on the old cpufreq driver.

Also as a side note, are you running Intel's thermald?

[pyamsoft]

I have pushed an update, version 1.2.4, to the dev branch which hopefully resolves this problem. I have added a new systemd rule which upon sleep/resume, will request pstate-frequency to run the automatic plan. This should set, and limit your CPU frequencies to the value decided by the automatic plan

[sfauzia]

Can you confirm this? Can you also confirm that because cpupower states that the CPU frequency is 2GHz, the machine does in fact "feel" as if it were running at that speed.

Sorry for the delay in my reply! It does in fact feel it's running at 2 GHz; the battery life appreciably drops as a result, and the fan kicks in. About to test your latest update presently, many thanks! :)

[sfauzia]

The numbers being outputted then, for example 825000 for the MAX should be the same as you see in the files /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq. Can you confirm this?

$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
825000

$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
800000

Yes, that is correct. Those are the numbers both before and after suspend on battery power.

Also as a side note, are you running Intel's thermald?

Yes, but I commented out the pstate line in the thermal-cpu-cdev-order.xml file as mentioned in the README. I also turned off the thermald daemon entirely then suspending, and the issue still persisted.

I have added a new systemd rule which upon sleep/resume, will request pstate-frequency to run the automatic plan. This should set, and limit your CPU frequencies to the value decided by the automatic plan

Unfortunately I am still having this same issue with the update. (I apologize for not having been able to provide further input sooner.) If there is any other information I can provide, please let me know.

The output from udevadm monitor, by the way (with your update after suspend):

$ udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[1315.418840] change   /devices/platform/regulatory.0 (platform)
KERNEL[1315.418867] remove   /devices/virtual/misc/watchdog (misc)
KERNEL[1315.418883] remove   /devices/virtual/watchdog/watchdog0 (watchdog)
KERNEL[1315.418893] remove   /devices/system/machinecheck/machinecheck1 (machinecheck)
KERNEL[1315.418903] remove   /devices/system/machinecheck/machinecheck2 (machinecheck)
KERNEL[1315.418912] remove   /devices/system/machinecheck/machinecheck3 (machinecheck)
KERNEL[1315.418921] add      /devices/system/machinecheck/machinecheck1 (machinecheck)
KERNEL[1315.418930] add      /devices/system/machinecheck/machinecheck2 (machinecheck)
KERNEL[1315.418941] add      /devices/system/machinecheck/machinecheck3 (machinecheck)
KERNEL[1315.418955] add      /devices/virtual/misc/watchdog (misc)
KERNEL[1315.418969] add      /devices/virtual/watchdog/watchdog0 (watchdog)
KERNEL[1315.418983] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/rfkill2 (rfkill)
KERNEL[1315.418995] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0 (bluetooth)
KERNEL[1315.419010] remove   /devices/platform/i8042/serio2/input/input35/event16 (input)
KERNEL[1315.419025] remove   /devices/platform/i8042/serio2/input/input35/mouse2 (input)
KERNEL[1315.419051] remove   /devices/platform/i8042/serio2/input/input35 (input)
KERNEL[1315.419064] add      /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0 (bluetooth)
KERNEL[1315.419079] add      /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/rfkill3 (rfkill)
KERNEL[1315.419094] change   /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/rfkill3 (rfkill)
UDEV  [1315.425321] remove   /devices/system/machinecheck/machinecheck2 (machinecheck)
UDEV  [1315.425346] remove   /devices/system/machinecheck/machinecheck3 (machinecheck)
UDEV  [1315.425374] remove   /devices/platform/i8042/serio2/input/input35/event16 (input)
UDEV  [1315.425388] add      /devices/system/machinecheck/machinecheck2 (machinecheck)
UDEV  [1315.427321] remove   /devices/system/machinecheck/machinecheck1 (machinecheck)
UDEV  [1315.427345] remove   /devices/virtual/watchdog/watchdog0 (watchdog)
UDEV  [1315.427532] add      /devices/system/machinecheck/machinecheck3 (machinecheck)
UDEV  [1315.428115] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/rfkill2 (rfkill)
UDEV  [1315.428557] remove   /devices/platform/i8042/serio2/input/input35/mouse2 (input)
UDEV  [1315.429006] remove   /devices/platform/i8042/serio2/input/input35 (input)
UDEV  [1315.429769] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0 (bluetooth)
UDEV  [1315.430466] add      /devices/system/machinecheck/machinecheck1 (machinecheck)
UDEV  [1315.430580] remove   /devices/virtual/misc/watchdog (misc)
UDEV  [1315.430897] add      /devices/virtual/watchdog/watchdog0 (watchdog)
UDEV  [1315.431350] change   /devices/platform/regulatory.0 (platform)
UDEV  [1315.432195] add      /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0 (bluetooth)
UDEV  [1315.432434] add      /devices/virtual/misc/watchdog (misc)
UDEV  [1315.433914] add      /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/rfkill3 (rfkill)
UDEV  [1315.435015] change   /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/rfkill3 (rfkill)
KERNEL[1317.767022] add      /devices/platform/i8042/serio2/input/input45 (input)
KERNEL[1317.768868] add      /devices/platform/i8042/serio2/input/input45/event16 (input)
UDEV  [1317.768912] add      /devices/platform/i8042/serio2/input/input45 (input)
KERNEL[1317.768932] add      /devices/platform/i8042/serio2/input/input45/mouse2 (input)
UDEV  [1317.768955] add      /devices/platform/i8042/serio2/input/input45/mouse2 (input)
UDEV  [1317.807584] add      /devices/platform/i8042/serio2/input/input45/event16 (input)
KERNEL[1318.744370] change   /devices/platform/regulatory.0 (platform)
UDEV  [1318.744745] change   /devices/platform/regulatory.0 (platform)
KERNEL[1318.890577] change   /devices/platform/regulatory.0 (platform)
UDEV  [1318.890913] change   /devices/platform/regulatory.0 (platform)
KERNEL[1333.274399] change   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/CMB1 (power_supply)
UDEV  [1333.275007] change   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/CMB1 (power_supply)

[pyamsoft]

Thank you for your input and logs surrounding this bug. I have been unable to reproduce your issues on an idle system, but I am able to get various strange frequency readings when having the system resume with many different tasks and being under reasonable load.

Because of the erratic behavior of the bug, I can only assume that this may be an issue with the fact that the software governor in the intel_pstate driver does not allow exclusive control of the CPU interface. On my system at least, setting the powersave plan after resume will limit usual frequencies to their minimum, but still allows the CPU to enter Turbo frequencies, causing power drain and significant fan activity. Unfortunately there is not much I can do about this, as the goal of pstate-frequency is not to be a monitoring daemon for CPU frequencies, but rather more of a one shot tool to set the frequency of the CPU. pstate-frequency is using only the files and interfaces exposed to it by the system, meaning the sys files relating to cpufreq and the intel_pstate files.

Unfortunately, at the moment I cannot do much more to isolate this behavior. There may be lower level events happening outside of the control of pstate-frequency which I do not seek to modify or hook onto, as that would be out of the scope of the overall project.

[pyamsoft]

If you have x86_enery_perf_policy installed, could I ask the output of

sudo x86_energy_perf_policy -r

along with

pstate-frequency -G -r && sleep 2 && pstate-frequency -G -r

after resume?

[sfauzia]

Thank you for explaining. It is only unfortunate that the prior to suspend behavior is so perfect but thereafter cannot be replicated without rebooting, at least on my laptop, but I understand the limitations. Here's the output you requested (after resuming from suspend):

# x86_energy_perf_policy -r

cpu0: 0x0000000000000000
cpu1: 0x0000000000000006
cpu2: 0x0000000000000006
cpu3: 0x0000000000000006

# pstate-frequency -G -r && sleep 2 && pstate-frequency -G -r

pstate-frequency  1.2.5[g++]
   pstate::CPU[0]  -> 1999.898MHz
   pstate::CPU[1]  -> 1999.898MHz
   pstate::CPU[2]  -> 1999.898MHz
   pstate::CPU[3]  -> 2000.003MHz

pstate-frequency  1.2.5[g++]
   pstate::CPU[0]  -> 2000.003MHz
   pstate::CPU[1]  -> 2000.003MHz
   pstate::CPU[2]  -> 1999.898MHz
   pstate::CPU[3]  -> 2000.003MHz

[pyamsoft]

Thank you for your reports. Please try and verify two things for me.

First, I would like you to cold boot your computer, like you always would.
Upon startup, please run:

sudo x86_energy_perf_policy -r

Please verify that all of your CPU's read 0x0000000000000006.
This is the 'normal' mode.

Please verify that

pstate-frequency -S -p 1 

sets the machine to 'powersave' and your expected minimum frequencies.

Please then put the machine to sleep.

Upon resume, please verify:

sudo x86_energy_perf_policy -r

Verify please that now, your output contains 0x0000000000000000 on CPU0, and 0x0000000000000006 on every other CPU.

0x0000000000000000 is 'performance' mode, which is letting one of your CPU's run at a more powerful frequency. This happens on my machine as well, resulting in the pstate-frequency outputs you are experiencing. This is not an issue with pstate-frequency, as it does not touch those values. Rather it looks to be like something with the kernel itself. The x86_energy_perf_policy binary, can fix this however.

Now, please run:

sudo x86_energy_perf_policy normal

Please verify that all of your CPU's read 0x0000000000000006, and check if pstate-frequency now outputs the proper frequency values.

What I believe the issue to be, is that on boot, ALL cpu's are set the normal mode of 0x06, but on resume, one of the CPU's is left in performance mode, causing this to happen. If you can run on resume:

# x86_energy_perf_policy normal

and verify that the problem has gone away, that will be good.

[sfauzia]

Post cold bolt:

# x86_energy_perf_policy -r
cpu0: 0x0000000000000006
cpu1: 0x0000000000000006
cpu2: 0x0000000000000006
cpu3: 0x0000000000000006

So they are all in the normal mode.

# pstate-frequency -S -p 1

pstate-frequency  1.2.5[g++]
   pstate::CPU_DRIVER     -> intel_pstate
   pstate::CPU_GOVERNOR   -> powersave
   pstate::NO_TURBO       -> 1 : OFF
   pstate::CPU_MIN        -> 24% : 800000KHz
   pstate::CPU_MAX        -> 25% : 825000KHz

# cpupower frequency-info
analyzing CPU 0:
 driver: intel_pstate
 CPUs which run at the same hardware frequency: 0
 CPUs which need to have their frequency coordinated by software: 0
 maximum transition latency: 0.97 ms.
 hardware limits: 800 MHz - 3.30 GHz
 available cpufreq governors: performance, powersave
 current policy: frequency should be within 800 MHz and 825 MHz.
                 The governor "powersave" may decide which speed to use
                 within this range.
 current CPU frequency is 800 MHz (asserted by call to hardware).

As expected.

Post suspend:

# x86_energy_perf_policy -r
cpu0: 0x0000000000000000
cpu1: 0x0000000000000006
cpu2: 0x0000000000000006
cpu3: 0x0000000000000006

# x86_energy_perf_policy normal

# x86_energy_perf_policy -r
cpu0: 0x0000000000000006
cpu1: 0x0000000000000006
cpu2: 0x0000000000000006
cpu3: 0x0000000000000006

However, unless I run pstate-frequency -S -p auto again*, the frequency does not go back down to the originally set limits. It was an issue before that running pstate-frequency again post suspend did not have any effect; I wonder if it was the recent kernel update that fixed it, as you pointed out it is something on the kernel level. My current kernel version is 4.0.5 (on ArchLinux). Running that post suspend does not however fix the x86_energy_perf_policy values, but setting normal mode manually as you suggested fixes that.

I updated my powersaving script to do the above automatically, and now the problem is indeed gone for good! Thank you so much!

*I realize that is why you also supplied the udev rule for pstate-frequency, but it didn't work for me because of an issue I had, outlined and solved here.