Isolation of backtests

[Froelund]

Hi!

I'm running backtests automated, and my results shows me that when running multiple backtests with completely different strategy-parameters(but the same simulationBalance and daterange), I'm getting the exact same result for every backtest?

My approach is to use the rest api, and send 20 different calls in parallel.

Is every backtest run isolated from other backtests?

Do Gekko support parallel backtests via the rest-api?

[imkane]

I'm working on something similar (#831) but it sounds like you've gotten further.

Are you willing to share your code with the community?

[Froelund]

Hi @imkane .

It's really just a separate nodejs project I put together for testing - So not really shareable.

I could put one together that reproduces these issues?

[imkane]

Ahh gotcha :) If you want to do that I can help test.

[Froelund]

Great 👍

[thegamecat]

It should work.

[askmike]

Is every backtest run isolated from other backtests?

Yes, if you start a server and use the REST api to run backtests, each one runs in an individual child process. See here for the implementation.

I'm getting the exact same result for every backtest?

I have never seen this before, are you able to provide a reproducible example?

[Froelund]

@askmike Thanks for reacting to this. I will create a nodejs project you can run towards a existing gekko server.

[Froelund]

So, I've written down some code that reproduces the error. Sorry for the looong block of code, but I wanted to do it without external deps, so you could run it with the current gekko package.json

If you do try so run it yourself, you will need to have the data imported.

const http = require('http');

function callBacktestApi(strategiConfig) {
  var body = JSON.stringify(strategiConfig);
  var options = {
    host: 'localhost',
    port: 3000,
    path: '/api/backtest',
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Content-Length': Buffer.byteLength(body)
    }
  };
  return new Promise((resolve, reject) => {
    const postRequest = http.request(options, (response) => {
      let str = '';
      response.on('data', chunk => str += chunk);
      response.on('end', () => resolve(JSON.parse(str)));
    });
    postRequest.write(body);
    postRequest.end();
  });
}

function createMacdStragegy(params) {
  return {
    gekkoConfig: {
      watch: {
        exchange: params.exchange || 'poloniex',
        currency: params.currency || 'USDT',
        asset: params.asset || 'BTC',
      },
      paperTrader: {
        fee: 0.25,
        slippage: 0.05,
        riskFreeReturn: 2,
        simulationBalance: {
          asset: 0,
          currency: 3000
        },
        reportRoundtrips: true,
        enabled: true
      },
      backtest: {
        daterange: {
          from: params.fromDate || '2017-04-01T00:00:00Z',
          to: params.toDate || '2017-07-01T00:00:00Z',
        }
      },
      valid: true,
      'MACD': {
        short: params.short,
        long: params.long,
        signal: params.signal,
        thresholds: {
          down: params.down,
          up: params.up,
          persistence: params.persistence,
        }
      },
      'tradingAdvisor': {
        enabled: true,
        method: 'MACD',
        candleSize: params.candleSize,
        historySize: params.historySize,
      },
    },
    data: {
      candleProps: [
      'close',
      'start'
      ],
      indicatorResults: true,
      report: true,
      roundtrips: true,
      trades: true
    }
  }
}

const backtestStrategy1 = createMacdStragegy({
  long: 22,
  short: 18,
  signal: 12,
  down: '-0.025',
  up: '0.025',
  persistence: 1,
  candleSize: 60,
  historySize: 10,
});

const backtestStrategy2 = createMacdStragegy({
  long: 1122,
  short: 1118,
  signal: 1112,
  down: '-0.025',
  up: '0.025',
  persistence: 1,
  candleSize: 60,
  historySize: 10,
});

const backtestStrategy3 = createMacdStragegy({
  long: 2222,
  short: 2218,
  signal: 2212,
  down: '-0.025',
  up: '0.025',
  persistence: 1,
  candleSize: 60,
  historySize: 10,
});

const test1Promise = callBacktestApi(backtestStrategy1);
const test2Promise = callBacktestApi(backtestStrategy2);
const test3Promise = callBacktestApi(backtestStrategy3);

Promise.all([test1Promise, test2Promise, test3Promise]).then((results) => {
  console.log(`test1: Trades: ${results[0].trades.length} Balance: ${results[0].report.balance}`);
  console.log(`test2: Trades: ${results[1].trades.length} Balance: ${results[1].report.balance}`);
  console.log(`test3: Trades: ${results[2].trades.length} Balance: ${results[2].report.balance}`);
});

Gives the output:

test1: Trades: 1 Balance: 5972.589833485438
test2: Trades: 1 Balance: 5972.589833485438
test3: Trades: 1 Balance: 5972.589833485438

[Froelund]

Did you test it @askmike ?

[thegamecat]

I get the same issue:

test1: Trades: 1 Balance: 5131.545330278262
test2: Trades: 1 Balance: 5131.545330278262
test3: Trades: 1 Balance: 5131.545330278262

Used a different currency pair, but worked just fine. I'm not on the latest code base though.

Also, I think this may be a node thing rather than a gekko thing but I could be wrong.

[Froelund]

@thegamecat The node engine runnning the gekko server, or the node running the test?

[thegamecat]

Your code.

[askmike]

I fixed it, was a very subtle bug:

Even though the backtests run in isolation, I build a config file in the shared (UI server) process. When setting up a new backtest I pass this config to the child process (after the process started). But your code hits the backtest API so fast that by the time the child processes are ready to receive the config, I've overwritten the config 3 times and pass the same config to all child processes. Damn hoisted require in combination with _.merge...

[askmike]

@Froelund

My approach is to use the rest api, and send 20 different calls in parallel.

Keep in mind that the current backtester is not very fast (not designed to be), and doing 20 at the same time will probably bring your computer (cpu & IO) to a grind. I would recommand using as the adapter postgresql. But still it won't be very fast..

[thegamecat]

@Froelund did you update your code because the above isnt working and I cant test the fix.

[marcuslopes]

Same here, I am receiving report: false regardless of the interval I pick

[askmike]

@thegamecat @Froelund @marcuslopes Since 0.5.0 the code that calculates performance (and thus is responsible for the report) is not part of the paperTrader anymore, this way it can also be used for the live trader. To fix report false you need to add the new plugin PeformanceAnalyzer to the config you pass, eg. here is an updated createMacdStragegy method:

function createMacdStragegy(params) {
  return {
    gekkoConfig: {
      watch: {
        exchange: params.exchange || 'poloniex',
        currency: params.currency || 'USDT',
        asset: params.asset || 'BTC',
      },
      paperTrader: {
        fee: 0.25,
        slippage: 0.05,
        riskFreeReturn: 2,
        simulationBalance: {
          asset: 0,
          currency: 3000
        },
        reportRoundtrips: true,
        enabled: true
      },
      backtest: {
        daterange: {
          from: '2017-04-01 00:00:00',
          to: '2017-07-01 00:00:00',
        }
      },
      valid: true,
      'MACD': {
        short: params.short,
        long: params.long,
        signal: params.signal,
        thresholds: {
          down: params.down,
          up: params.up,
          persistence: params.persistence,
        }
      },
      'tradingAdvisor': {
        enabled: true,
        method: 'MACD',
        candleSize: params.candleSize,
        historySize: params.historySize,
      },
      performanceAnalyzer: {
        enabled: true,
        riskFreeReturn: 5
      }
    },
    data: {
      candleProps: [ 'close',  'start' ],
      indicatorResults: true,
      report: true,
      roundtrips: true,
      trades: true
    }
  }
}

[thegamecat]

Wonderful, works like a charm.

[marcuslopes]

Thanks Mike! Got it working as well.