Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Further optimize memory for Travis jobs #6150

Merged
merged 3 commits into from
Aug 11, 2018
Merged

Conversation

jihoonson
Copy link
Contributor

Fixes #6149.

I tested this PR by triggering several Travis jobs, and could see processing module succeeded 5 times in a row.

In this PR,

  • Added CloseableDefaultBlockingPool and CloseableStupidPool for only unit testing to release buffers as soon as possible. Some tests can be further optimized to release buffers per parameterized run once Junit 4.13 is released which contains @BeforeParam/@AfterParam for Parameterized runner junit-team/junit4#1435. With this change, processing module testing can be run with 1500m off-heap memory.
  • Changed to use different memory configuration for testing processing and others. processing requires small heap and large off-heap memory, but others are opposite. I set the default memory configuration to -Xmx1500m -XX:MaxDirectMemorySize=512m in ${DRUID_HOME}/pom.xml. processing has its own configuration, -Xmx512m -XX:MaxDirectMemorySize=1500m in ${DRUID_HOME}/processing/pom.xml. These configurations can also be set per Travis jobs using _JAVA_OPTIONS, but I think it's better to put them in pom.xml files to use those configurations whenever running tests even in local.
  • Changed all unit tests to be run sequentially to use less memory. Since parallel-test spawns new processes, it's not easy to control the memory being used by each process.

@fjy
Copy link
Contributor

fjy commented Aug 11, 2018

👍

@fjy fjy merged commit ecee3e0 into apache:master Aug 11, 2018
@dclim dclim added this to the 0.13.0 milestone Oct 8, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants