Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MONDRIAN-2279] - When a query is cancelled or times out, Mondrian may continue iterating over large tuple lists #639

Merged
merged 2 commits into from
Jan 28, 2016
Merged

[MONDRIAN-2279] - When a query is cancelled or times out, Mondrian may continue iterating over large tuple lists #639

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ce035a9
[MONDRIAN-2279] - When a query is cancelled or times out, Mondrian ma…
YuryBY Jan 18, 2016
8312435
[MONDRIAN-2279] - Test
YuryBY Jan 21, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[MONDRIAN-2279] - When a query is cancelled or times out, Mondrian ma… 
…y continue iterating over large tuple lists
  • Loading branch information
@YuryBY
YuryBY committed Jan 28, 2016
commit ce035a94621a2510e8df039c399c023ced59b2a6
12 changes: 6 additions & 6 deletions src/main/mondrian/olap/MondrianProperties.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1040,16 +1040,16 @@ allowed when iterating over members to compute aggregates. A value of
<Default>0</Default>
</PropertyDefinition>
<PropertyDefinition>
<Name>CancelPhaseInterval</Name>
<Path>mondrian.rolap.cancelPhaseInterval</Path>
<Name>CheckCancelOrTimeoutInterval</Name>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I just saw this, but we already have another property, CancelPhaseInterval, which does the same. Please standardize on one of the two and make sure to replace all usages of the old property with your new code to handle cancelations as well.

<Path>mondrian.util.checkCancelOrTimeoutInterval</Path>
<Description>
<p>Positive integer property that determines how many rows are read from sql executions between checks for whether the current mdx query has been cancelled.<br/>
Setting the interval too small may result in a performance hit when reading large result sets;
setting it too high can cause a big delay between the query being marked as cancelled and system resources associated to it being released.
<p>Positive integer property that determines loop iterations number between checks for whether the current mdx query has been cancelled or timeout was exceeded.<br/>
Setting the interval too small may result in a performance degradation when reading large result sets;
setting it too high can cause a big delay between the query being marked as cancelled or timeout was exceeded and system resources associated to it being released.
</p>
</Description>
<Type>int</Type>
<Default>10000</Default>
<Default>1000</Default>
</PropertyDefinition>
<PropertyDefinition>
<Name>ExecutionHistorySize</Name>
Expand Down
21 changes: 11 additions & 10 deletions src/main/mondrian/olap/fun/CrossJoinFunDef.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
// You must accept the terms of that agreement to use this software.
//
// Copyright (C) 2002-2005 Julian Hyde
// Copyright (C) 2005-2015 Pentaho and others
// Copyright (C) 2005-2016 Pentaho and others
// All Rights Reserved.
*/
package mondrian.olap.fun;
Expand All @@ -17,6 +17,9 @@
import mondrian.olap.type.*;
import mondrian.resource.MondrianResource;
import mondrian.rolap.RolapEvaluator;
import mondrian.server.Execution;
import mondrian.server.Locus;
import mondrian.util.CancellationChecker;
import mondrian.util.CartesianProductList;

import org.apache.log4j.Logger;
Expand Down Expand Up @@ -565,7 +568,11 @@ private static void cartesianProductRecurse(
final int partialSizeNext = partialSize + tupleList.getArity();
final int iNext = i + 1;
final TupleCursor cursor = tupleList.tupleCursor();
int currentIteration = 0;
Execution execution = Locus.peek().execution;
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
cursor.currentToArray(partialArray, partialSize);
if (i == lists.size() - 1) {
result.addAll(partial);
Expand Down Expand Up @@ -932,9 +939,8 @@ protected TupleList nonEmptyList(
// Measure and non-All Members evaluation is non-null, then
// add it to the result List.
final TupleCursor cursor = list.tupleCursor();
final int checkCancelPeriod =
MondrianProperties.instance().CancelPhaseInterval.get();
int currentIteration = 0;
Execution execution = query.getStatement().getCurrentExecution();
while (cursor.forward()) {
cursor.setContext(evaluator);
for (Member member : memberSet) {
Expand All @@ -948,12 +954,8 @@ protected TupleList nonEmptyList(
// Check if the MDX query was canceled.
// Throws an exception in case of timeout is exceeded
// see MONDRIAN-2425
if (checkCancelPeriod > 0
&& currentIteration % checkCancelPeriod == 0)
{
query.getStatement().getCurrentExecution()
.checkCancelOrTimeout();
}
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
if (checkData(
nonAllMembers,
nonAllMembers.length - 1,
Expand All @@ -962,7 +964,6 @@ protected TupleList nonEmptyList(
{
result.addCurrent(cursor);
}
++currentIteration;
}
return result;
} finally {
Expand Down
51 changes: 34 additions & 17 deletions src/main/mondrian/olap/fun/FilterFunDef.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,22 @@
/*
* This software is subject to the terms of the Eclipse Public License v1.0
* Agreement, available at the following URL:
* http://www.eclipse.org/legal/epl-v10.html.
* You must accept the terms of that agreement to use this software.
*
* Copyright (c) 2002-2015 Pentaho Corporation.. All rights reserved.
// This software is subject to the terms of the Eclipse Public License v1.0
// Agreement, available at the following URL:
// http://www.eclipse.org/legal/epl-v10.html.
// You must accept the terms of that agreement to use this software.
//
// Copyright (C) 2002-2005 Julian Hyde
// Copyright (C) 2005-2016 Pentaho and others
// All Rights Reserved.
*/

package mondrian.olap.fun;

import mondrian.calc.*;
import mondrian.calc.impl.*;
import mondrian.mdx.ResolvedFunCall;
import mondrian.olap.*;
import mondrian.server.Execution;
import mondrian.server.Locus;
import mondrian.util.CancellationChecker;

import java.util.List;

Expand Down Expand Up @@ -155,7 +158,12 @@ protected TupleIterable makeIterable(Evaluator evaluator) {
list.getArity(), list.size() / 2);
evaluator.setNonEmpty(false);
TupleCursor cursor = list.tupleCursor();
int currentIteration = 0;
Execution execution =
evaluator.getQuery().getStatement().getCurrentExecution();
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
cursor.setContext(evaluator);
if (bcalc.evaluateBoolean(evaluator)) {
result.addCurrent(cursor);
Expand Down Expand Up @@ -188,7 +196,12 @@ protected TupleIterable makeIterable(Evaluator evaluator) {
try {
evaluator.setNonEmpty(false);
TupleCursor cursor = members.tupleCursor();
int currentIteration = 0;
Execution execution =
evaluator.getQuery().getStatement().getCurrentExecution();
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
cursor.setContext(evaluator);
if (bcalc.evaluateBoolean(evaluator)) {
result.addCurrent(cursor);
Expand Down Expand Up @@ -221,23 +234,17 @@ protected TupleIterable makeIterable(Evaluator evaluator) {
icalc.evaluateIterable(evaluator);
final Evaluator evaluator2 = evaluator.push();
evaluator2.setNonEmpty(false);
final int checkCancelPeriod =
MondrianProperties.instance().CancelPhaseInterval.get();
return new AbstractTupleIterable(iterable.getArity()) {
public TupleCursor tupleCursor() {
return new AbstractTupleCursor(iterable.getArity()) {
final TupleCursor cursor = iterable.tupleCursor();

public boolean forward() {
int rowCount = -1;
int currentIteration = 0;
Execution execution = Locus.peek().execution;
while (cursor.forward()) {
rowCount++;
if (checkCancelPeriod > 0
&& rowCount % checkCancelPeriod == 0)
{
Locus.peek().execution
.checkCancelOrTimeout();
}
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
cursor.setContext(evaluator2);
if (bcalc.evaluateBoolean(evaluator2)) {
return true;
Expand Down Expand Up @@ -330,7 +337,12 @@ protected TupleList makeList(Evaluator evaluator) {
try {
evaluator.setNonEmpty(false);
final TupleCursor cursor = members0.tupleCursor();
int currentIteration = 0;
Execution execution =
evaluator.getQuery().getStatement().getCurrentExecution();
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
cursor.setContext(evaluator);
if (bcalc.evaluateBoolean(evaluator)) {
result.addCurrent(cursor);
Expand Down Expand Up @@ -364,7 +376,12 @@ protected TupleList makeList(Evaluator evaluator) {
TupleList result = members0.cloneList(members0.size() / 2);
evaluator.setNonEmpty(false);
final TupleCursor cursor = members0.tupleCursor();
int currentIteration = 0;
Execution execution = evaluator.getQuery()
.getStatement().getCurrentExecution();
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
cursor.setContext(evaluator);
if (bcalc.evaluateBoolean(evaluator)) {
result.addCurrent(cursor);
Expand Down
18 changes: 17 additions & 1 deletion src/main/mondrian/olap/fun/FunUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
// You must accept the terms of that agreement to use this software.
//
// Copyright (C) 2002-2005 Julian Hyde
// Copyright (C) 2005-2015 Pentaho and others
// Copyright (C) 2005-2016 Pentaho and others
// All Rights Reserved.
*/
package mondrian.olap.fun;
Expand All @@ -17,6 +17,7 @@
import mondrian.olap.type.*;
import mondrian.resource.MondrianResource;
import mondrian.rolap.*;
import mondrian.server.Execution;
import mondrian.util.*;

import org.apache.commons.collections.ComparatorUtils;
Expand Down Expand Up @@ -634,7 +635,12 @@ public static TupleList sortTuples(
if (tupleList == null) {
tupleArrayList = new ArrayList<List<Member>>();
final TupleCursor cursor = tupleIterable.tupleCursor();
int currentIteration = 0;
Execution execution =
evaluator.getQuery().getStatement().getCurrentExecution();
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
tupleArrayList.add(cursor.current());
}
if (tupleArrayList.size() <= 1) {
Expand Down Expand Up @@ -1591,7 +1597,12 @@ static SetWrapper evaluateSet(
// todo: treat constant exps as evaluateMembers() does
SetWrapper retval = new SetWrapper();
final TupleCursor cursor = members.tupleCursor();
int currentIteration = 0;
Execution execution =
evaluator.getQuery().getStatement().getCurrentExecution();
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
cursor.setContext(evaluator);
Object o = calc.evaluate(evaluator);
if (o == null || o == Util.nullValue) {
Expand Down Expand Up @@ -1635,7 +1646,12 @@ static SetWrapper[] evaluateSet(
retvals[i] = new SetWrapper();
}
final TupleCursor cursor = list.tupleCursor();
int currentIteration = 0;
Execution execution =
evaluator.getQuery().getStatement().getCurrentExecution();
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
cursor.setContext(evaluator);
for (int i = 0; i < calcs.length; i++) {
DoubleCalc calc = calcs[i];
Expand Down
45 changes: 22 additions & 23 deletions src/main/mondrian/olap/fun/GenerateFunDef.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,23 @@
/*
* This software is subject to the terms of the Eclipse Public License v1.0
* Agreement, available at the following URL:
* http://www.eclipse.org/legal/epl-v10.html.
* You must accept the terms of that agreement to use this software.
*
* Copyright (c) 2002-2015 Pentaho Corporation.. All rights reserved.
// This software is subject to the terms of the Eclipse Public License v1.0
// Agreement, available at the following URL:
// http://www.eclipse.org/legal/epl-v10.html.
// You must accept the terms of that agreement to use this software.
//
// Copyright (C) 2002-2005 Julian Hyde
// Copyright (C) 2005-2016 Pentaho and others
// All Rights Reserved.
*/

package mondrian.olap.fun;

import mondrian.calc.*;
import mondrian.calc.impl.*;
import mondrian.mdx.ResolvedFunCall;
import mondrian.olap.*;
import mondrian.olap.type.*;
import mondrian.server.Execution;
import mondrian.server.Locus;
import mondrian.util.CancellationChecker;

import java.util.*;

Expand Down Expand Up @@ -105,24 +108,19 @@ public GenerateListCalcImpl(

public TupleList evaluateList(Evaluator evaluator) {
final int savepoint = evaluator.savepoint();
final int checkCancelPeriod =
MondrianProperties.instance().CancelPhaseInterval.get();
try {
evaluator.setNonEmpty(false);
final TupleIterable iterable1 =
iterCalc1.evaluateIterable(evaluator);
evaluator.restore(savepoint);
TupleList result = TupleCollections.createList(arityOut);
Execution execution = Locus.peek().execution;
if (all) {
final TupleCursor cursor = iterable1.tupleCursor();
int rowCount = -1;
int rowCount = 0;
while (cursor.forward()) {
rowCount++;
if (checkCancelPeriod > 0
&& rowCount % checkCancelPeriod == 0)
{
Locus.peek().execution.checkCancelOrTimeout();
}
CancellationChecker.checkCancelOrTimeout(
rowCount++, execution);
cursor.setContext(evaluator);
final TupleList result2 =
listCalc2.evaluateList(evaluator);
Expand All @@ -133,14 +131,10 @@ public TupleList evaluateList(Evaluator evaluator) {
new HashSet<List<Member>>();
final TupleCursor cursor = iterable1.tupleCursor();

int rowCount = -1;
int rowCount = 0;
while (cursor.forward()) {
rowCount++;
if (checkCancelPeriod > 0
&& rowCount % checkCancelPeriod == 0)
{
Locus.peek().execution.checkCancelOrTimeout();
}
CancellationChecker.checkCancelOrTimeout(
rowCount++, execution);
cursor.setContext(evaluator);
final TupleList result2 =
listCalc2.evaluateList(evaluator);
Expand Down Expand Up @@ -196,7 +190,12 @@ public String evaluateString(Evaluator evaluator) {
final TupleIterable iter11 =
iterCalc.evaluateIterable(evaluator);
final TupleCursor cursor = iter11.tupleCursor();
int currentIteration = 0;
Execution execution =
evaluator.getQuery().getStatement().getCurrentExecution();
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
cursor.setContext(evaluator);
if (k++ > 0) {
String sep = sepCalc.evaluateString(evaluator);
Expand Down
6 changes: 5 additions & 1 deletion src/main/mondrian/rolap/RolapResult.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
// You must accept the terms of that agreement to use this software.
//
// Copyright (C) 2001-2005 Julian Hyde
// Copyright (C) 2005-2015 Pentaho and others
// Copyright (C) 2005-2016 Pentaho and others
// All Rights Reserved.
*/
package mondrian.rolap;
Expand Down Expand Up @@ -1579,7 +1579,11 @@ void mergeTupleList(TupleList list) {
}

private void mergeTupleIter(TupleCursor cursor) {
int currentIteration = 0;
Execution execution = Locus.peek().execution;
while (cursor.forward()) {
CancellationChecker.checkCancelOrTimeout(
currentIteration++, execution);
mergeTuple(cursor);
}
}
Expand Down
Loading