Fix #370: YAML & CLI (#371)

* Towards #232: Refactoring towards ppss.yaml part 3/3 * move everything in model_eng/ to data_eng/ * Fix #352: [SW eng] High DRY violation in test_predictoor_agent.py <> test_predictoor_agent3.py * Deprecate backend-dev.md (long obsolete), macos.md (obsolete due to vps), and envvars.md (obsolete because of ppss.yaml). * Rename BaseConfig to web3_pp.py and make it yaml-based * Move scripts into util/, incorporate them into pdr cli, some refactoring. * revamp READMEs for cli. And, tighten up text for getting OCEAN & ROSE * Deprecated ADDRESS_FILE and RPC_URL envvars. * deprecate Predictoor approach 2. Pita to maintain Co-authored-by: trizin <25263018+trizin@users.noreply.github.com>
oceanprotocol · Nov 27, 2023 · 4e06884 · 4e06884
1 parent f27b30a
commit 4e06884
Show file tree

Hide file tree

Showing 189 changed files with 4,234 additions and 4,264 deletions.
diff --git a/README.md b/README.md
@@ -10,39 +10,72 @@ SPDX-License-Identifier: Apache-2.0
 - **[Run predictoor bot](READMEs/predictoor.md)** - make predictions, make $
 - **[Run trader bot](READMEs/trader.md)** - consume predictions, trade, make $
 
+
 (If you're a predictoor or trader, you can safely ignore the rest of this README.)
 
+## Settings: PPSS
+
+A "ppss" yaml file, like [`ppss.yaml`](ppss.yaml), holds parameters for all bots and simulation flows.
+- We follow the idiom "pp" = problem setup (what to solve), "ss" = solution strategy (how to solve).
+- `PRIVATE_KEY` is an exception; it's set as an envvar.
+
+When you run a bot from the CLI, you specify your PPSS YAML file.
+
+## CLI
+
+(First, [install pdr-backend](READMEs/predictoor.md#install-pdr-backend-repo) first.)
+
+To see CLI options, in console:
+```console
+pdr
+```
+
+This will output something like:
+```text
+Usage: pdr sim|predictoor|trader|..
+
+Main tools:
+  pdr sim YAML_FILE
+  pdr predictoor APPROACH YAML_FILE NETWORK
+  pdr trader APPROACH YAML_FILE NETWORK
+...
+```
+
 
 ## Atomic READMEs
 
 - [Get tokens](READMEs/get-tokens.md): [testnet faucet](READMEs/testnet-faucet.md), [mainnet ROSE](READMEs/get-rose-on-sapphire.md) & [OCEAN](READMEs/get-ocean-on-sapphire.md)
-- [Envvars](READMEs/envvars.md)
+- [Claim payout for predictoor bot](READMEs/payout.md)
 - [Predictoor subgraph](READMEs/subgraph.md). [Subgraph filters](READMEs/filters.md)
-- [Dynamic model codebase](READMEs/dynamic-model-codebase.md)
-- [Static models in predictoors](READMEs/static-model.md)
+- [Run barge locally](READMEs/barge.md)
 
 ## Flows for core team
 
-- **Backend dev** - for `pdr-backend` itself
-  - [Main backend-dev README](READMEs/backend-dev.md)
+- Backend dev - for `pdr-backend` itself
+  - [Main backend-dev](READMEs/backend-dev.md)
+  - [VPS backend-dev](READMEs/vps.md)
   - [Release process](READMEs/release-process.md)
-  - [Run barge locally](READMEs/barge.md)
-  - [Run barge remotely on VPS](READMEs/vps.md)
-  - [MacOS gotchas](READMEs/macos.md) wrt Docker & ports
-- **[Run dfbuyer bot](READMEs/dfbuyer.md)** - runs Predictoor DF rewards
-- **[Run publisher](READMEs/publisher.md)** - publish new feeds
-- **[Scripts](scripts/)** for performance stats, more
+- [Run dfbuyer bot](READMEs/dfbuyer.md) - runs Predictoor DF rewards
+- [Run publisher](READMEs/publisher.md) - publish new feeds
+- [Run trueval](READMEs/trueval.md) - run trueval bot
 
 ## Repo structure
 
 This repo implements all bots in Predictoor ecosystem.
 
-Each bot has a directory:
-- `predictoor` - submits individual predictions
-- `trader` - buys aggregated predictions, then trades
-- other bots: `trueval` report true values to contract, `dfbuyer` implement Predictoor Data Farming, `publisher` to publish
+Each bot has a directory. Alphabetically:
+- `dfbuyer` - buy feeds on behalf of Predictoor DF
+- `predictoor` - submit individual predictions
+- `publisher` - publish pdr data feeds
+- `trader` - buy aggregated predictions, then trade
+- `trueval` - report true values to contract
 
-Other directories:
-- `util` - tools for use by any agent
-- `models` - classes that wrap Predictoor contracts; for setup (BaseConfig); and for data feeds (Feed)
+Other directories, alphabetically:
+- `accuracy` - calculates % correct, for display in predictoor.ai webapp
+- `data_eng` - data engineering & modeling
+- `models` - class-based data structures, and classes to wrap contracts
+- `payout` - OCEAN & ROSE payout
+- `ppss` - settings
+- `sim` - simulation flow
+- `util` - function-based tools
 
diff --git a/READMEs/backend-dev.md b/READMEs/backend-dev.md
@@ -11,58 +11,62 @@ This is for core devs to improve pdr-backend repo itself.
 
 Follow directions to install pdr-backend in [predictoor.md](predictoor.md)
 
-## Local Network
+## Setup Barge
 
-First, [install barge](barge.md#install-barge).
+**Local barge.** If you're on ubuntu, you can run barge locally.
+- First, [install barge](barge.md#install-barge).
+- Then, run it. In barge console: `./start_ocean.sh --no-provider --no-dashboard --predictoor --with-thegraph`
 
-Then, run barge. In barge console:
-```console
-# Run barge with just predictoor contracts, queryable, but no agents
-./start_ocean.sh --no-provider --no-dashboard --predictoor --with-thegraph
-```
+**Or, remote barge.** If you're on MacOS or Windows, run barge on VPS.
+- Follow the instructions in [vps.md](vps.md)
+
+### Setup dev environment
 
 Open a new "work" console and:
 ```console
 # Setup virtualenv
 cd pdr-backend
 source venv/bin/activate
 
-# Set envvars
+# Set PRIVATE_KEY
 export PRIVATE_KEY="0xc594c6e5def4bab63ac29eed19a134c130388f74f019bc74b8f4389df2837a58"
-export ADDRESS_FILE="${HOME}/.ocean/ocean-contracts/artifacts/address.json"
 
-export RPC_URL=http://127.0.0.1:8545
-export SUBGRAPH_URL="http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph"
-#OR: export SUBGRAPH_URL="http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph"
+# Unit tests default to using "development" network -- a locally-run barge.
+# If you need another network such as barge on VPS, then override via e.g.
+export NETWORK_OVERRIDE=barge-pytest
 ```
 
-([envvars.md](envvars.md) has details.)
+All other settings are in [`ppss.yaml`](../ppss.yaml). Some of these are used in unit tests. Whereas most READMEs make a copy `my_ppss.yaml`, for development we typically want to operate directly on `ppss.yaml`.
 
 ### Local Usage: Testing & linting
 
 In work console, run tests:
 ```console
-#(ensure envvars set as above)
+# (ensure PRIVATE_KEY set as above)
 
-#run a single test
-pytest pdr_backend/util/test/test_constants.py::test_constants1
+# run a single test. The "-s" is for more output.
+# note that pytest does dynamic type-checking too:)
+pytest pdr_backend/util/test_noganache/test_util_constants.py::test_util_constants -s
 
-#run all tests in a file
-pytest pdr_backend/util/test/test_constants.py
+# run all tests in a file
+pytest pdr_backend/util/test_noganache/test_util_constants.py -s
 
-#run all regular tests; see details on pytest markers to select specific suites
+# run a single test that flexes network connection
+pytest pdr_backend/util/test_ganache/test_contract.py::test_get_contract_filename -s
+
+# run all regular tests; see details on pytest markers to select specific suites
 pytest
 ```
 
 In work console, run linting checks:
 ```console
-#run static type-checking. By default, uses config mypy.ini. Note: pytest does dynamic type-checking.
+# mypy does static type-checking and more. Configure it via mypy.ini
 mypy ./
 
-#run linting on code style
+# run linting on code style. Configure it via .pylintrc.
 pylint pdr_backend/*
 
-#auto-fix some pylint complaints
+# auto-fix some pylint complaints like whitespace
 black ./
 ```
 
@@ -82,12 +86,21 @@ In work console:
 ```console
 #(ensure envvars set as above)
 
-# run trader agent
-python pdr_backend/trader/main.py
+# run trader agent, approach 1
+pdr trader 1 ppss.yaml development
+# or
+pdr trader 1 ppss.yaml barge-pytest
 ```
 
 (You can track at finer resolution by writing more logs to the [code](../pdr_backend/predictoor/approach3/predictoor_agent3.py), or [querying Predictoor subgraph](subgraph.md).)
 
 ## Remote Usage
 
-Combine local setup above with remote setup envvars like in [predictoor.md](predictoor.md).
+In the CLI, simply point to a different network:
+```console
+# run on testnet
+pdr trader ppss.yaml sapphire-testnet
+
+# or, run on mainnet
+pdr trader ppss.yaml sapphire-mainnet
+```
diff --git a/READMEs/barge.md b/READMEs/barge.md
@@ -7,6 +7,8 @@ SPDX-License-Identifier: Apache-2.0
 
 Barge is a Docker container to run a local Ganache network having Predictoor contracts and (optionally) local bots. This README describes how to install Barge, and provides reference on running it with various agents.
 
+⚠️ If you're on MacOS or Windows, we recommend using a remotely-run Barge. See [vps flow](vps.md).
+
 ## Contents
 
 Main:
@@ -39,11 +41,7 @@ docker system prune -a --volumes
 ```
 
 **Then, get Docker running.** To run barge, you need the Docker engine running. Here's how:
-- If you're on Linux: you're good, there's nothing extra to do.
-- If you're on MacOS: 
-  - via console: `open -a Docker`
-  - or, via app: open Finder app, find Docker, click to open app. (You don't need to press "play" or anything else. The app being open is enough.)
-  - ⚠️ MacOS may give Docker issues. [Here](macos.md) are workarounds.
+- If you're on Linux: you're good, there's nothing extra to do
 
 Congrats! Barge is installed and ready to be run.
 

diff --git a/READMEs/dfbuyer.md b/READMEs/dfbuyer.md
@@ -27,27 +27,31 @@ Open a new console and:
 cd pdr-backend
 source venv/bin/activate
 
-# Set envvars
+# Set envvar
 export PRIVATE_KEY="0xc594c6e5def4bab63ac29eed19a134c130388f74f019bc74b8f4389df2837a58"
-export ADDRESS_FILE="${HOME}/.ocean/ocean-contracts/artifacts/address.json"
-
-export RPC_URL=http://127.0.0.1:8545
-export SUBGRAPH_URL="http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph"
-#OR: export SUBGRAPH_URL="http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph"
 ```
 
+Copy [`ppss.yaml`](../ppss.yaml) into your own file `my_ppss.yaml` and change parameters as you see fit. The section "dfbuyer_ss" has parameters for this bot.
+
 Then, run dfbuyer bot. In console:
 ```console
-python pdr_backend/dfbuyer/main.py
+pdr dfbuyer my_ppss.yaml development
 ```
 
-There are other environment variables that you might want to set, such as the **weekly spending limit**. To get more information about them check out the [environment variables documentation](./envvars.md).
-
-The bot will consume "WEEKLY_SPENDING_LIMIT" worth of assets each week. This amount is distributed equally among all DF eligible assets.
+The bot will consume "weekly_spending_limit" worth of assets each week. This amount is distributed equally among all DF eligible assets. (This parameter is set in the yaml file.)
 
 ![flow](https://user-images.githubusercontent.com/25263018/269256707-566b9f5d-7e97-4549-b483-2a6700826769.png)
 
+
 ## Remote Usage
 
-Combine local setup above with remote setup envvars like in [predictoor.md](predictoor.md).
+In the CLI, simply point to a different network:
+```console
+# run on testnet
+pdr dfbuyer my_ppss.yaml sapphire-testnet
+
+# or, run on mainnet
+pdr dfbuyer my_ppss.yaml sapphire-mainnet
+```
+
 
diff --git a/READMEs/dynamic-model-codebase.md b/READMEs/dynamic-model-codebase.md