Launch with error 'KeyError: "['specified_funds'] not in index"'

[tiems90]

Describe the bug
Errors at launch. It only affects one of my 3 portfolios on startup, as the format doesn't confirm to the specc here:

pycryptobot/models/CoinbasePro.py

Line 192 in d530ab7

df = resp.copy()[[ 'created_at', 'product_id', 'side', 'type', 'size', 'price', 'status' ]]

To Reproduce
Steps to reproduce the behavior:

1. launch using python3 pycryptobot.py
2. See error

Error Log

Traceback (most recent call last):
  File "pycryptobot.py", line 66, in <module>
    orders = account.getOrders(app.getMarket(), '', 'done')
  File "/home/pi/pycryptobot/BTC/models/TradingAccount.py", line 110, in getOrders
    self.orders = model.getOrders(market, action, status)
  File "/home/pi/pycryptobot/BTC/models/CoinbasePro.py", line 195, in getOrders
    df = resp.copy()[[ 'created_at', 'product_id', 'side', 'type', 'filled_size', 'specified_funds', 'executed_value', 'fill_fees', 'status' ]]
  File "/home/pi/.local/lib/python3.7/site-packages/pandas/core/frame.py", line 3030, in __getitem__
    indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1]
  File "/home/pi/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1265, in _get_listlike_indexer
    self._validate_read_indexer(keyarr, indexer, axis, raise_missing=raise_missing)
  File "/home/pi/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1315, in _validate_read_indexer
    raise KeyError(f"{not_found} not in index")
KeyError: "['specified_funds'] not in index"

I've gone in to actually check the columns of resp, and it only contains the following:

['id', 'price', 'size', 'product_id', 'profile_id', 'side', 'type', 'time_in_force', 'post_only', 'created_at', 'done_at', 'done_reason', 'fill_fees', 'filled_size', 'executed_value', 'status', 'settled']

Desktop (please complete the following information):

• OS: Linux; Debian
• Version: latest git pull as of 2.0.6

Additional context
Config file, aside from base_currency identical to working ones. :

{
     "coinbasepro" : {
         "api_url" : "https://api.pro.coinbase.com",
         "api_key" : "...",
         "api_secret" : "...",
         "api_passphrase" : "...",
         "config" : {
             "base_currency" : "BTC",
             "quote_currency" : "USD",
             "live" : 1,
             "sellatloss" : 0,
             "granularity": 60
         }
     },
     "telegram" : {
         "token" : "...",
         "client_id" : "..."
     }
}

[p9qgtjaliucdtygqapv79h5nm1faov8]

I am also running into this error on Docker. Oddly my bot running BTC-USD runs fine.

Config file:

{
    "api_url" : "https://api.pro.coinbase.com",
    "api_key" : "...",
    "api_secret" : "...",
    "api_pass" : "...",
    "config" : {
        "base_currency" : "EOS",
        "quote_currency" : "USD",
        "live" : 1,
        "verbose" : 0,
        "sellatloss" : 0
    }
}

[whittlem]

Are you still getting this issue after the last update? Run "git pull"

[tiems90]

Hi! Thanks for the quick check-in!
Yes, still running into issues with the definition here:

pycryptobot/models/CoinbasePro.py

Line 195 in d530ab7

df = resp.copy()[[ 'created_at', 'product_id', 'side', 'type', 'filled_size', 'specified_funds', 'executed_value', 'fill_fees', 'status' ]]

Still the same traceback too:

Traceback (most recent call last):
  File "pycryptobot.py", line 66, in <module>
    orders = account.getOrders(app.getMarket(), '', 'done')
  File "/home/pi/pycryptobot/BTC/models/TradingAccount.py", line 110, in getOrders
    self.orders = model.getOrders(market, action, status)
  File "/home/pi/pycryptobot/BTC/models/CoinbasePro.py", line 195, in getOrders
    df = resp.copy()[[ 'created_at', 'product_id', 'side', 'type', 'filled_size', 'specified_funds', 'executed_value', 'fill_fees', 'status' ]]
  File "/home/pi/.local/lib/python3.7/site-packages/pandas/core/frame.py", line 3030, in __getitem__
    indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1]
  File "/home/pi/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1265, in _get_listlike_indexer
    self._validate_read_indexer(keyarr, indexer, axis, raise_missing=raise_missing)
  File "/home/pi/.local/lib/python3.7/site-packages/pandas/core/indexing.py", line 1315, in _validate_read_indexer
    raise KeyError(f"{not_found} not in index")
KeyError: "['specified_funds'] not in index"

The resp dataframe object listed there has the following attributes for me:
['id', 'price', 'size', 'product_id', 'profile_id', 'side', 'type', 'time_in_force', 'post_only', 'created_at', 'done_at', 'done_reason', 'fill_fees', 'filled_size', 'executed_value', 'status', 'settled']

[lesterbx]

Same issue here, apparently the property "specified_funds" is returned only for a single order (GET /orders/) but not for a list of orders (GET /orders). Seen in Coinbase Api Docs

[whittlem]

I will look later and fix it. Thanks.

[anotherlousytag]

I ran into this issue after making a manual partial sell followed by a transfer in one of my bot portfolios, I worked around it by creating a new portfolio.

[gasteazi]

Hello, I have the same error. On the same machine it works correctly with binance.

Traceback (most recent call last):
  File "/app/pycryptobot.py", line 65, in <module>
    orders = account.getOrders(app.getMarket(), '', 'done')
  File "/app/models/TradingAccount.py", line 110, in getOrders
    self.orders = model.getOrders(market, action, status)
  File "/app/models/CoinbasePro.py", line 195, in getOrders
    df = resp.copy()[[ 'created_at', 'product_id', 'side', 'type', 'filled_size', 'specified_funds', 'executed_value', 'fill_fees', 'status' ]]
  File "/usr/local/lib/python3.9/site-packages/pandas/core/frame.py", line 3030, in __getitem__
    indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1]
  File "/usr/local/lib/python3.9/site-packages/pandas/core/indexing.py", line 1266, in _get_listlike_indexer
    self._validate_read_indexer(keyarr, indexer, axis, raise_missing=raise_missing)
  File "/usr/local/lib/python3.9/site-packages/pandas/core/indexing.py", line 1316, in _validate_read_indexer
    raise KeyError(f"{not_found} not in index")
KeyError: "['specified_funds'] not in index"

Version 2.0.6 sha256:63b50c6ea6d30a5d7498c3a77f51332f87ab65c6bb09e429f7c4474dc4e004c3

Configuration:

    "coinbasepro" : {
        "api_url" : "https://api.pro.coinbase.com",
        "api_key" : "",
        "api_secret" : "",
        "api_passphrase" : "",
        "config" : {
            "base_currency" : "ETH",
            "quote_currency" : "EUR",
            "live" : 1,
            "sellatloss" : 0,
            "disablebullonly" : 1,
            "disablebuyobv" : 1,
            "disablebuyelderray" : 1,
            "disablelog" : 1,
            "disabletracker" : 1
        }
    },
    "telegram" : {
        "token" : "",
        "client_id" : ""
    }
}

thank you

[whittlem]

I've not been able to re-create this issue. I can see in the Coinbase Pro documentation that /orders without a status doesn't return the 'specified_funds' column but my code always returns a status.

I've added some debug code...

            ### DEBUG CODE ###
            if 'specified_funds' not in resp:
                print ('If you see this message please include this output in the issue: https://github.com/whittlem/pycryptobot/issues/156')
                print ('---')
                print (f'uri: orders?status={status}')
                print (f'size: {len(resp)}')
                print (resp)
                print ('---')
                return pd.DataFrame()
            ###

If you are still experiencing the issue can you update the Issue with the output above. If you don't want to list your actual orders here can you message it to me on Telegram?

I'm not sure if you know but there is a script-get_orders.py file in the project root. If you run the script it will list your Coinbase Pro and/or Binance orders. This may be helpful in re-creating the issue.

Could it be that you don't have any orders yet for the market?

[tiems90]

Thanks again for checking. This was on a CoinbasePro portfolio that had previously worked fine.
Right now the bot is working again!, but when I run script-get_orders.py it does trigger the exception:

If you see this message please include this output in the issue: https://github.com/whittlem/pycryptobot/issues/156
---
uri: orders?status=done
size: 1
                                     id    price       size product_id                            profile_id side  ... done_reason   fill_fees  filled_size executed_value status settled
0  5247d65b-7409-441c-b075-fdeaf5d89a3e  64090.0  0.0011644    BTC-USD  94ca97a9-a5a0-48e5-ba23-a77b7677f94c  buy  ...      filled  0.37313198    0.0011644      74.626396   done    True

[1 rows x 17 columns]
---
Empty DataFrame
Columns: []
Index: []
Index([], dtype='object')
Traceback (most recent call last):
  File "script-get_orders.py", line 14, in <module>
    app = PyCryptoBot(exchange='binance')
  File "/home/pi/pycryptobot/BTC/models/PyCryptoBot.py", line 1224, in __init__
    raise ValueError('Binance API URL is invalid')
ValueError: Binance API URL is invalid

[tiems90]

In case you wanted the full details of resp this is the full content:

{
    "done_reason": "filled",
    "created_at": "2021-04-14T10:52:57.342779Z",
    "post_only": false,
    "product_id": "BTC-USD",
    "fill_fees": "0.3731319800000000",
    "settled": true,
    "price": "64090.00000000",
    "profile_id": "94ca97a9-a5a0-48e5-ba23-a77b7677f94c",
    "executed_value": "74.6263960000000000",
    "side": "buy",
    "time_in_force": "GTC",
    "status": "done",
    "done_at": "2021-04-14T10:58:15.799Z",
    "filled_size": "0.00116440",
    "type": "limit",
    "id": "5247d65b-7409-441c-b075-fdeaf5d89a3e",
    "size": "0.00116440"
}

[whittlem]

This should be fixed now. Can you confirm?

[gasteazi]

I receive this message:

If you see this message please include this output in the issue: https://github.com/whittlem/pycryptobot/issues/156
---
uri: orders?status=done
size: 1
                                     id       price  ... status settled
0  44ed5c5c-d632-43ad-811c-570800feaa68  0.82450000  ...   done    True

[gasteazi]

I think it's working properly now. Good job. :-)

sudo docker container logs ETH-EUR
If you see this message please include this output in the issue: https://github.com/whittlem/pycryptobot/issues/156
---
uri: orders?status=done
size: 1
                                     id       price  ... status settled
0  44ed5c5c-d632-43ad-811c-570800feaa68  0.82450000  ...   done    True

[1 rows x 18 columns]
---
--------------------------------------------------------------------------------
|                             Python Crypto Bot                                |
--------------------------------------------------------------------------------
|               Release : v2.0.7                                               |
--------------------------------------------------------------------------------
|              Bot Mode : LIVE - live trades using your funds!                 |
|           Bot Started : 2021-05-13 10:49:26.274896                           |
================================================================================
|          Sell At Loss : True  --sellatloss 1                                 |
|    Sell At Resistance : False  --sellatresistance                            |
|       Trade Bull Only : False  --disablebullonly                             |
|         Buy Near High : True  --disablebuynearhigh                           |
|          Use Buy MACD : True  --disablebuymacd                               |
|           Use Buy OBV : False  --disablebuyobv                               |
|     Use Buy Elder-Ray : False  --disablebuyelderray                          |
|    Sell Fibonacci Low : True  --disablefailsafefibonaccilow                  |
|  Candlestick Reversal : True  --disableprofitbankreversal                    |
|              Telegram : True  --disabletelegram                              |
|                   Log : False  --disablelog                                  |
|               Tracker : False  --disabletracker                              |
|      Auto restart Bot : False  --autorestart                                 |
================================================================================
* Candlestick Detected: Inverted Hammer ("Weak - Continuation - Bullish Pattern - Up")

2021-05-13 10:00:00 | ETH-EUR | 3600 | Close: 3048.71 | v EMA12/26: 3233.88 < 3311.52 v | v MACD: -77.64 < -49.34 v | WAIT | Last Action: SELL
2021-05-13 10:51:48 | ETH-EUR | 3600 | Current Price: 3064.67

[cbespin]

I can also confirm the initial issue, but the recent version seems to work. Just leaving my output here as well, in case it helps:

If you see this message please include this output in the issue: https://github.com/whittlem/pycryptobot/issues/156
---
uri: orders?status=done
size: 1
                                     id       price  ... status settled
0  8dfb2d6a-fed5-455e-9635-08ba72c7e08f  0.82800000  ...   done    True

[1 rows x 17 columns]

Thanks for the quick fix, @whittlem !