-
Notifications
You must be signed in to change notification settings - Fork 7.1k
Performance Tests
- Never run the tests on a Simulator and/or Emulator. Always use real devices
- The older the device the better
- Never run the tests in DEBUG mode. Always use RELEASE mode
- Double check that RELEASE mode is using all the possible optimizations for the C++ code
Devices are multitasking, and many tasks are run in the background and those tasks could affect the performance. In order to minimize that do:
- Turn Airplane mode ON
- Disable Wifi
- Disable Bluetooth
- Kill all running tasks / applications
- Performance Tests must be run before releasing an stable version
- The results must be logged in this spreadsheet
- In order to know the performance of the new release, a comparison must be done with the
previous versions. The comparison must try to use the same testing environment. As an example:
- Same devices (MUST)
- Same device operating system (SHOULD)
- Same compiler and toolchain (DESIRABLE)
Starting from Cocos2d-x v3.3, the "auto run" feature of Sprite Performance Tests logs in the console the output of the tests. This output can be copy & pasted into the spreadsheet
Most of the Cocos2d-x performance tests relies on watching the FPS number. Although this is not a reliable way to test how fast a particular component is, it gives us at least a way to know how fast Cocos2d-x is in general.
Note: The method of taking a look at the FPS MUST NOT be used to measure the performance of a particular algorithm, or function.
Instead Sampling Profiling like Xcode Instruments, or Linux Perf must be used. Intrusive Profiling is also valid method, like using the CC_PROFILER_START
and CC_PROFILER_STOP
macros.
Pros:
- Doesn't require additional services / tools:
- Wifi can be turned off
- Xcode / Visual Studio are not needed
- Can be used to test any kind of tests:
- eg: Sprite Test with 5250 sprites
Cons:
- Like any other manual process, it is error prone and takes a lot of time.
Suggested for:
- Only to test a few tests. Usually to make sure that the Automated test output is correct.
As of v3.3 only the Sprite Performance Test can be automated. The rests of the test must use the "manual" method.
In order to run the automated version of Sprite Performance Test, you have to:
- Launch cpp-tests
- Select the Performance tests test
- Select the Sprite Perf Test subtest
- Press the Auto Test Off button
The automated tests logs the results using the cocos2d::Console::log()
function. That means that the cocos2d::Console
has the output. And it is possible to get the output by connecting the cocos2d::Console
by using telnet (by default cpp-tests
has the the cocos2d::Console
enabled):
$ telnet 10.0.1.16 5678
10.0.1.16 would the IP address of the device.
Once inside the console, you can type help to see all the available commands:
And in order to redirect the log()
messages to the console, you must type:
> debugmsg on
Once the automated test is finished, the output will be in the console, and you will see something like the following:
The result can be copy & pasted into the spreadsheet
Pros:
- Is very reliable, specially if the tests does not use the
log()
function during the tests
Cons:
- Requires WiFi to be ON, violating one of the Basic Rules described above.
- Requires to touch the Phone every now and then to prevent it from going to sleep (this can be easily fixed in the tests, but as of v3.3 it was not fixed)
Suggested for:
- Use this method for automated tests (and fix the "go to sleep bug"!)
It is possible to use Xcode or Visual Studio to redirect the output of the cocos2d::Console
to Xcode's or Visual Studio's.
Although in theory these tools should not affect the performance of the tests, using Xcode's console proved to make the tests slower (Tested on Xcode 6.1 connected to an iPod Touch 4 using iOS 6.1.6). No penalty was found on Visual Studio at the moment.
Pros:
- Can be used with everything turned off (Wifi). They only require to connect to the device using the USB cable.
- They keep the phone awake preventing it from going to sleep mode
Cons:
- Xcode v6.1 + iPod Touch 4 with iOS 6.1.6 makes some tests to run slower
Suggested for:
- Avoid this method since it is not clear why Xcode make the tests run slower.