Fetching EOD data of multiple symbols via one api call

[robinvalk]

I was wondering if it is possible to fetch the EOD data of multiple symbols via one api call?
One request per symbol is quite slow but I can't find any information that there's a better and faster solution.

Do you have any recommendations or knowledge that there is a way to achieve this?

[DannyBen]

Well - this library is simply a wrapper around Quandl's HTTP API - so since they don't have a common way of doing it (to the best of my knowledge, at least not in standard databases), then PHP Quandl can't do it also.

However, there are a couple of ways you can still create parallel downloads with forking.

Two options I can think of:

1. If you must / prefer to use PHP, then you can use pcntl_fork to execute several processes at once (see example code below).
2. If PHP is not necessarily a requirement, I would probably consider doing it with gondl - the command line Quandl tool I have built. It could be easier to fork processes there from any language you want, and let the entire thing run as a background service.

A simple forking example with PHP:

<?php
require 'quandl.php';

$api_key = "YOUR KEY";
$quandl  = new Quandl($api_key);
$options = ["rows" => 10];

$symbols = ["GOOG/NASDAQ_AAPL", "GOOG/NASDAQ_MSFT"];

foreach ($symbols as $i => $symbol) {
    $pid = pcntl_fork(); 
    if (!$pid) { 
        $symbol = $symbols[$i];
        print "Downloading $symbol\n";
        flush();
        $response = $quandl->getSymbol($symbol, $options);
        $data = $response->data;
        $rows = count($data);
        print "Downloaded $rows rows for $symbol\n";
        exit($i); 
    } 
} 

while (pcntl_waitpid(0, $status) != -1) { 
    $status = pcntl_wexitstatus($status); 
    print "Child $status completed\n"; 
} 

Of course, you will have to limit the number of processes you fork, but that's just a proof of concept.

Does this help?

[robinvalk]

Thanks for the quick response, I'm currently queueing the imports so I can have multiple workers running at the same time. I might need to boot some extra servers to fasten the imports for now.

As for the next step I think I'm gonna check out the possibilities of Gondl. That will probably be way more efficient.

Thx!

[DannyBen]

Sure. Feel free to reopen if you need more help or modifications in the API wrappers.

[DannyBen]

@robinvalk - are you using a premium database, or a free one? If premium - there is a way (in the Quandl API) to get bulk downloads. I am going to make sure the PHP library is working with it, and perhaps add an example later this week.

[robinvalk]

@DannyBen Awesome, nice work! We got in touch with them after waiting for two weeks haha
This is what we received yesterday:

So in 4 months we'll be able to fetch data of multiple tickers in one request.
Great news, but there will probably be loads of changes in the new API so work is on its way ^^

[DannyBen]

I agree the Quandl API does not provide a good endpoint for ongoing bulk download. Still, if you are using a premium database, then the bulk download (partial) endpoint is your best bet I think. You get the zip file with data - not ideally organized, but still.

[robinvalk]

Yes, I'm gonna use the partial bulk download daily on production. Locally, where I only have an X amount of tickers I will probably use the coming API's features