Add integration testing infrastructure, data import, and dev startup scripts by Copilot · Pull Request #2 · zhostev/qlib_t

Copilot · 2026-03-30T09:24:54Z

Wire up missing API routes, add chenditc/investment_data import pipeline, skip email verification for dev, and provide scripts to independently start backend/frontend/worker and run integration tests.

Route registration fixes

Register train and tasks routers in main.py — were defined but never mounted
Add both to app/api/__init__.py as well

Skip email verification

New SKIP_EMAIL_VERIFICATION env var (default False); when True, registration auto-sets email_verified=True
Startup script defaults it to True for local dev

CORS & config cleanup

Extract Settings.get_cors_origins() to deduplicate origin parsing across main.py and train_server.py
Merge env-configured origins with hardcoded production origins, deduplicated
Auto-generate SECRET_KEY in startup script if unset (instead of hardcoded default)

TrainingClient fix

settings.TRAINING_SERVER_URL → settings.training_server_url (attribute casing mismatch)

Historical data import

scripts/import_investment_data.py — downloads CN A-share data via chenditc/investment_data, converts to qlib format at ~/.qlib/qlib_data/cn_data

Startup scripts (`scripts/`)

./scripts/start_backend.sh    # backend on :8000, configures env
./scripts/start_frontend.sh   # frontend on :3001, auto npm install
./scripts/start_worker.sh     # train_worker.py polling loop
./scripts/run_tests.sh        # runs integration_test.py

Integration test suite

scripts/integration_test.py — hits all 12 API modules (auth, experiments, configs, factors, data, models, tasks, train, benchmarks, monitoring, admin) with create/read/update/delete coverage and resource cleanup

…ster train/tasks routes, CORS fix, data import script, automated test script, startup scripts Agent-Logs-Url: https://github.com/zhostev/qlib_t/sessions/510c756a-7ce2-43da-a2b9-b6dcaece6b59 Co-authored-by: zhostev <57177476+zhostev@users.noreply.github.com>

…et key generation, test assertions Agent-Logs-Url: https://github.com/zhostev/qlib_t/sessions/510c756a-7ce2-43da-a2b9-b6dcaece6b59 Co-authored-by: zhostev <57177476+zhostev@users.noreply.github.com>

Copilot

Pull request overview

Adds local dev and integration testing infrastructure while wiring up missing API routes/config paths to support end-to-end backend/frontend/worker workflows.

Changes:

Add dev/startup scripts for backend/frontend/worker plus an integration test runner.
Wire missing train and tasks API routers and fix TrainingClient settings attribute casing.
Add historical data import script via chenditc/investment_data and centralize CORS origin parsing.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
scripts/start_worker.sh	Adds a script to start the training worker with default env vars.
scripts/start_frontend.sh	Adds a script to run the frontend dev server (auto-installs deps).
scripts/start_backend.sh	Adds a script to run backend with local-friendly env defaults and DB init.
scripts/run_tests.sh	Adds a wrapper to run the Python integration test suite.
scripts/integration_test.py	Adds a comprehensive live-API integration test script with CRUD flows + cleanup.
scripts/import_investment_data.py	Adds data import/verification script for QLib CN data via `investment_data`.
backend/train_server.py	Switches CORS origin setup to centralized `settings.get_cors_origins()`.
backend/main.py	Switches CORS origin setup and mounts `train`/`tasks` routers.
backend/app/services/training_client.py	Fixes settings attribute names for training server URL/timeout.
backend/app/config.py	Adds `SKIP_EMAIL_VERIFICATION` and centralized CORS origin parsing helper.
backend/app/api/auth.py	Allows skipping email verification in dev via settings flag.
backend/app/api/init.py	Adds `tasks` router to the API package exports/router.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-30T09:31:58Z

+# Usage: ./scripts/run_tests.sh [--base-url http://localhost:8000]
+
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+
+BASE_URL="${1:-http://localhost:8000}"
+


run_tests.sh’s usage suggests --base-url ..., but the script treats $1 as the URL value. If a user runs ./scripts/run_tests.sh --base-url http://localhost:8000, $1 becomes --base-url and Python receives --base-url --base-url, which breaks. Parse flags properly (e.g., accept --base-url and shift) or change the script to accept only a positional URL and update the usage line accordingly.

Suggested change

# Usage: ./scripts/run_tests.sh [--base-url http://localhost:8000]

set -e

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"

BASE_URL="${1:-http://localhost:8000}"

# Usage: ./scripts/run_tests.sh [--base-url URL] [URL]

set -e

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"

BASE_URL_DEFAULT="http://localhost:8000"

BASE_URL="$BASE_URL_DEFAULT"

if [[ "$1" == "--base-url" ]]; then

if [[ -z "$2" ]]; then

echo "Error: --base-url requires a URL argument." >&2

echo "Usage: ./scripts/run_tests.sh [--base-url URL] [URL]" >&2

exit 1

fi

BASE_URL="$2"

shift 2

elif [[ -n "$1" ]]; then

# Backwards-compatible: treat first positional argument as BASE_URL

BASE_URL="$1"

shift

fi

Copilot · 2026-03-30T09:31:59Z

+
+# Run integration tests
+cd "$PROJECT_ROOT"
+python scripts/integration_test.py --base-url "$BASE_URL"


run_tests.sh’s usage suggests --base-url ..., but the script treats $1 as the URL value. If a user runs ./scripts/run_tests.sh --base-url http://localhost:8000, $1 becomes --base-url and Python receives --base-url --base-url, which breaks. Parse flags properly (e.g., accept --base-url and shift) or change the script to accept only a positional URL and update the usage line accordingly.

Copilot · 2026-03-30T09:31:59Z

+        logger.error("  cd backend && uvicorn main:app --host 0.0.0.0 --port 8000")
+        sys.exit(1)
+
+    tester = IntegrationTest(args.base_url)


The --no-cleanup flag is parsed but never used; run_all() always runs cleanup. Either plumb args.no_cleanup into IntegrationTest (and conditionally skip cleanup()), or remove the flag to avoid misleading behavior.

Suggested change

tester = IntegrationTest(args.base_url)

tester = IntegrationTest(args.base_url)

if args.no_cleanup:

logger.info("Skipping cleanup as --no-cleanup flag is set")

tester.cleanup = lambda: None

Copilot · 2026-03-30T09:31:59Z

+    print("Installing requests...")
+    import subprocess
+    subprocess.check_call([sys.executable, "-m", "pip", "install", "requests"])
+    import requests


Auto-installing dependencies at runtime can cause non-reproducible test runs (network/pip failures, differing versions, polluted environments) and is risky in CI. Prefer failing fast with a clear message (“pip install -r …”) or making requests an explicit repo dependency; if you want convenience, consider a separate bootstrap step/script rather than installing inside the test runner.

Suggested change

print("Installing requests...")

import subprocess

subprocess.check_call([sys.executable, "-m", "pip", "install", "requests"])

import requests

print(

"The 'requests' package is required to run this integration test script.\n"

"Please install it first, for example:\n"

" pip install requests",

file=sys.stderr,

)

raise SystemExit(1)

Copilot · 2026-03-30T09:32:00Z

+
+# Set default environment variables
+export DATABASE_URL="${DATABASE_URL:-sqlite:///./qlib_management.db}"
+export SECRET_KEY="${SECRET_KEY:-dev-secret-key-change-in-production}"


This hardcodes a predictable default SECRET_KEY for the worker. Since start_backend.sh now auto-generates a random key when unset, the worker may end up using a different key than the backend (breaking token/session validation) and also encourages an insecure default. Align behavior with start_backend.sh: either require SECRET_KEY to be set (fail with a message) or auto-generate it in the worker script too so both processes can share the same provided value.

Suggested change

export SECRET_KEY="${SECRET_KEY:-dev-secret-key-change-in-production}"

if [ -z "${SECRET_KEY:-}" ]; then

echo "ERROR: SECRET_KEY is not set. Please set it to the same value used by the backend (see scripts/start_backend.sh)." >&2

exit 1

fi

export SECRET_KEY="$SECRET_KEY"

Copilot · 2026-03-30T09:32:00Z

+    def get_cors_origins(self) -> list:
+        """Parse CORS origins from settings, combining env var and defaults."""
+        origins = self.cors_origins
+        if isinstance(origins, str):
+            parsed = [o.strip() for o in origins.split(",") if o.strip()]
+        elif isinstance(origins, list):
+            parsed = origins
+        else:
+            parsed = []
+        # Merge with defaults, avoiding duplicates
+        for o in self._default_origins:
+            if o not in parsed:
+                parsed.append(o)
+        return parsed


When cors_origins is already a list, parsed = origins means get_cors_origins() mutates the underlying settings list by appending defaults. That can cause surprising side effects across calls. Use a copy for list inputs (e.g., parsed = list(origins)) before appending/deduping.

Copilot · 2026-03-30T09:32:00Z

+        import urllib.request
+        import zipfile
+        import tempfile
+
+        # chenditc/investment_data releases contain pre-built qlib data
+        release_url = "https://github.com/chenditc/investment_data/releases/latest"


urllib.request, zipfile, tempfile, and release_url are currently unused in this fallback path (the function just prints manual instructions and returns False). Either remove these unused imports/variable or implement the intended “download release asset + extract” fallback to match the in-code comment.

Copilot · 2026-03-30T09:32:00Z

 # Include API router without training endpoints
-from app.api import auth, experiments, models, configs, benchmarks, factors, data, monitoring
+from app.api import auth, experiments, models, configs, benchmarks, factors, data, monitoring, train, tasks


The comment says “without training endpoints” but train is now imported and later mounted. Update the comment to reflect the new behavior (or remove it) to avoid misleading future changes.

Copilot · 2026-03-30T09:32:01Z

+export SKIP_EMAIL_VERIFICATION="${SKIP_EMAIL_VERIFICATION:-True}"
+export CORS_ORIGINS="${CORS_ORIGINS:-http://localhost:3001,http://localhost:3000,http://localhost:8000,http://127.0.0.1:3001,http://127.0.0.1:3000}"


Two concrete issues for local reliability/safety: (1) CORS_ORIGINS default here omits http://127.0.0.1:8000 while Settings.cors_origins includes it, which can lead to confusing CORS mismatches depending on how the app is started; align defaults. (2) Defaulting SKIP_EMAIL_VERIFICATION to True is convenient for dev but dangerous if this script is used outside local dev—consider defaulting to False and printing instructions to enable it, or gating it behind an explicit DEV=1-style flag.

Suggested change

export SKIP_EMAIL_VERIFICATION="${SKIP_EMAIL_VERIFICATION:-True}"

export CORS_ORIGINS="${CORS_ORIGINS:-http://localhost:3001,http://localhost:3000,http://localhost:8000,http://127.0.0.1:3001,http://127.0.0.1:3000}"

# Configure email verification behavior:

# - If SKIP_EMAIL_VERIFICATION is set, respect it.

# - Otherwise, if DEV=1, enable skipping (development convenience).

# - Otherwise, require email verification by default.

if [ -z "$SKIP_EMAIL_VERIFICATION" ]; then

if [ "${DEV:-0}" = "1" ]; then

export SKIP_EMAIL_VERIFICATION="True"

echo "INFO: DEV=1 detected, SKIP_EMAIL_VERIFICATION enabled (development mode)."

else

export SKIP_EMAIL_VERIFICATION="False"

echo "INFO: Email verification required by default. To skip in development, set DEV=1 or SKIP_EMAIL_VERIFICATION=True."

fi

fi

export CORS_ORIGINS="${CORS_ORIGINS:-http://localhost:3001,http://localhost:3000,http://localhost:8000,http://127.0.0.1:3001,http://127.0.0.1:3000,http://127.0.0.1:8000}"

Copilot AI and others added 2 commits March 30, 2026 08:55

Address code review: fix email verification default, CORS dedup, secr…

9f1a79a

…et key generation, test assertions Agent-Logs-Url: https://github.com/zhostev/qlib_t/sessions/510c756a-7ce2-43da-a2b9-b6dcaece6b59 Co-authored-by: zhostev <57177476+zhostev@users.noreply.github.com>

Copilot AI assigned Copilot and zhostev Mar 30, 2026

Copilot created this pull request from a session on behalf of zhostev March 30, 2026 09:24 View session

zhostev marked this pull request as ready for review March 30, 2026 09:25

Copilot AI review requested due to automatic review settings March 30, 2026 09:25

zhostev merged commit a347527 into main Mar 30, 2026
1 check passed

Copilot started reviewing on behalf of zhostev March 30, 2026 09:26 View session

Copilot AI reviewed Mar 30, 2026

View reviewed changes

Navigation Menu

Add integration testing infrastructure, data import, and dev startup scripts - #2

Add integration testing infrastructure, data import, and dev startup scripts#2
zhostev merged 2 commits into
mainfrom
copilot/explore-qlib-management-page

Copilot AI commented Mar 30, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 30, 2026

Uh oh!

Copilot AI Mar 30, 2026

Uh oh!

Copilot AI Mar 30, 2026

Uh oh!

Copilot AI Mar 30, 2026

Uh oh!

Copilot AI Mar 30, 2026

Uh oh!

Copilot AI Mar 30, 2026

Uh oh!

Copilot AI Mar 30, 2026

Uh oh!

Copilot AI Mar 30, 2026

Uh oh!

Copilot AI Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-# Usage: ./scripts/run_tests.sh [--base-url http://localhost:8000]
-set -e
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
-BASE_URL="${1:-http://localhost:8000}"
+# Usage: ./scripts/run_tests.sh [--base-url URL] [URL]
+set -e
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+BASE_URL_DEFAULT="http://localhost:8000"
+BASE_URL="$BASE_URL_DEFAULT"
+if [[ "$1" == "--base-url" ]]; then
+    if [[ -z "$2" ]]; then
+        echo "Error: --base-url requires a URL argument." >&2
+        echo "Usage: ./scripts/run_tests.sh [--base-url URL] [URL]" >&2
+        exit 1
+    fi
+    BASE_URL="$2"
+    shift 2
+elif [[ -n "$1" ]]; then
+    # Backwards-compatible: treat first positional argument as BASE_URL
+    BASE_URL="$1"
+    shift
+fi

-    tester = IntegrationTest(args.base_url)
+    tester = IntegrationTest(args.base_url)
+    if args.no_cleanup:
+        logger.info("Skipping cleanup as --no-cleanup flag is set")
+        tester.cleanup = lambda: None

-    print("Installing requests...")
-    import subprocess
-    subprocess.check_call([sys.executable, "-m", "pip", "install", "requests"])
-    import requests
+    print(
+        "The 'requests' package is required to run this integration test script.\n"
+        "Please install it first, for example:\n"
+        "    pip install requests",
+        file=sys.stderr,
+    )
+    raise SystemExit(1)

-export SECRET_KEY="${SECRET_KEY:-dev-secret-key-change-in-production}"
+if [ -z "${SECRET_KEY:-}" ]; then
+  echo "ERROR: SECRET_KEY is not set. Please set it to the same value used by the backend (see scripts/start_backend.sh)." >&2
+  exit 1
+fi
+export SECRET_KEY="$SECRET_KEY"

		export SKIP_EMAIL_VERIFICATION="${SKIP_EMAIL_VERIFICATION:-True}"
		export CORS_ORIGINS="${CORS_ORIGINS:-http://localhost:3001,http://localhost:3000,http://localhost:8000,http://127.0.0.1:3001,http://127.0.0.1:3000}"

-export SKIP_EMAIL_VERIFICATION="${SKIP_EMAIL_VERIFICATION:-True}"
-export CORS_ORIGINS="${CORS_ORIGINS:-http://localhost:3001,http://localhost:3000,http://localhost:8000,http://127.0.0.1:3001,http://127.0.0.1:3000}"
+# Configure email verification behavior:
+# - If SKIP_EMAIL_VERIFICATION is set, respect it.
+# - Otherwise, if DEV=1, enable skipping (development convenience).
+# - Otherwise, require email verification by default.
+if [ -z "$SKIP_EMAIL_VERIFICATION" ]; then
+    if [ "${DEV:-0}" = "1" ]; then
+        export SKIP_EMAIL_VERIFICATION="True"
+        echo "INFO: DEV=1 detected, SKIP_EMAIL_VERIFICATION enabled (development mode)."
+    else
+        export SKIP_EMAIL_VERIFICATION="False"
+        echo "INFO: Email verification required by default. To skip in development, set DEV=1 or SKIP_EMAIL_VERIFICATION=True."
+    fi
+fi
+export CORS_ORIGINS="${CORS_ORIGINS:-http://localhost:3001,http://localhost:3000,http://localhost:8000,http://127.0.0.1:3001,http://127.0.0.1:3000,http://127.0.0.1:8000}"

Conversation

Copilot AI commented Mar 30, 2026

Route registration fixes

Skip email verification

CORS & config cleanup

TrainingClient fix

Historical data import

Startup scripts (scripts/)

Integration test suite

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Startup scripts (`scripts/`)