JGI Flask API

A Flask-based REST API for Johnson Gage and Inspection, Inc. (JGI) that serves as an internal API for quality management operations. The API integrates with Microsoft Entra ID for authentication and the Qualer quality management system for data operations.

🚀 Production API

• Base URL: https://jgiapi.com
• Documentation: https://jgiapi.com/docs (Swagger UI)
• OpenAPI Spec: https://jgiapi.com/openapi.json

📊 Available Endpoints

Endpoint	Description	Purpose
GET /whoami	User authentication info	Returns current user details from JWT token
GET /work-item-details	Work item details	Fetches work item data from Qualer system
GET /pyro-assets	Pyrotechnic assets	Returns assets from specific pyro asset pool
GET /employees	Employee listing	Gets all employees from Qualer
GET /clients	Client companies	Returns all client companies from Qualer
POST /git-pull	Deployment automation	Updates server code (uses separate token auth)

📝 See https://jgiapi.com/docs for complete API documentation and interactive testing

🔄 Background Data Synchronization

The API includes an automated background scheduler that keeps calibration and certification data synchronized with external sources:

Sync Schedule

Data Type	Interval	Source
Wire Set Certificates	Every 24 hours	Qualer API + SharePoint
Wire Offsets	Every 24 hours	SharePoint
DAQBook Offsets	Every 24 hours	SharePoint
Wire Roll Work Items	Every 4 hours	Qualer API

Scheduler Configuration

The background scheduler is enabled by default in production and automatically:

• Checks last sync times on startup
• Runs syncs at configured intervals
• Handles failures gracefully with error logging
• Tracks sync state in the database

Environment Variables:

ENABLE_SYNC_SCHEDULER=true   # Enable/disable scheduler (default: true)
IS_TEST_CONTEXT=false        # Auto-disables scheduler in tests

Manual Sync

You can also trigger syncs manually via the API:

POST /data-sync?init=false
Authorization: Bearer <access_token>

For more details on sync intervals and configuration, see utils/sync/state_manager.py.

🔐 Authentication

This API is protected using Microsoft Entra ID via OAuth 2.0. All routes (except / and /git-pull) require a valid access token issued by the registered Azure application.

Azure App Registration

This application is registered in Azure Active Directory at:

👉 Azure Portal App Registration

• Client ID: 43a01068-983b-41b9-bb61-7ed191bd0e29
• Audience: https://jgiapi.com
• Scope: access_as_user (defined under Expose an API)

Example Usage

GET /whoami
Authorization: Bearer <access_token>

GET /work-item-details?workItemNumber=56561-067667-01
Authorization: Bearer <access_token>

🧪 Local Development

Prerequisites

• Python 3.8+
• Azure AD access for authentication testing
• Qualer API access (for full functionality)

Environment Setup

1. Clone the repository

   git clone <repository-url>
   cd api

2. Install dependencies

   pip install -r requirements.txt

3. 🔐 Environment Configuration

   This project uses a two-tier environment setup for security:

   • config/settings.env: Safe defaults (checked into Git)
   • .env: Secrets (DO NOT COMMIT)

   Create your .env file with secrets:

   # .env (secrets only - DO NOT COMMIT)
   
   # Qualer API Integration
   QUALER_API_KEY=your-qualer-api-key
   
   # Azure Authentication
   AZURE_CLIENT_SECRET=your-azure-client-secret
   
   # Database Connection
   DATABASE_URL=postgresql+psycopg2://user:pass@host:5432/dbname
   
   # Development Settings (optional)
   SKIP_AUTH=false  # Set to 'true' for testing without Azure AD

   Safe configuration is already in config/settings.env:

   • Azure client ID, tenant ID, API audience
   • SharePoint site and drive IDs
   • Other non-sensitive defaults

4. Run the application

   Development mode (Flask dev server):

   python app.py

   Production mode (Waitress WSGI server):

   python wsgi_server.py

   The production server uses Waitress, a production-quality pure-Python WSGI server that:

   • ✅ Supports multi-threading for concurrent requests
   • ✅ Works natively on Windows
   • ✅ Provides better stability and performance than Flask's dev server
   • ✅ Includes built-in request timeouts and connection limits

Development vs Testing Modes

Production Mode (SKIP_AUTH=false):

• Full authentication required with real Azure AD tokens
• All business logic executes against real Qualer API
• Use for integration testing and debugging

Development/CI Mode (SKIP_AUTH=true):

• Mock authentication with fake tokens
• Fast testing without external dependencies
• Ideal for unit testing and CI/CD

Getting Access Tokens

Use the provided helper script to acquire valid Azure AD tokens:

python utils/get_token.py

This will open a browser for interactive Azure AD login and return a valid access token.

🧰 Testing

This project includes a comprehensive testing framework using a mark-based architecture.

Running Tests

# Full test suite
python -m pytest -v

# Optional: with coverage (local only)
python -m pytest --cov=. --cov-report=term-missing

# Bypass auth locally
$env:SKIP_AUTH="true"; python -m pytest

Running Tests

python -m pytest -v

📝 See DEVELOPMENT_GUIDE.md for comprehensive development and testing documentation

🏗️ Project Structure

├── app.py               # Main Flask application
├── config/              # Environment configuration
│   └── settings.
├── requirements.txt     # Python dependencies
├── .env                 # Environment variables (not in git)
├── routes/              # API endpoint blueprints
│   ├── whoami.py        # User authentication info
│   ├── work_item_details.py  # Work item data from Qualer
│   ├── pyro_assets.py   # Pyrotechnic assets
│   ├── employees.py     # Employee listings
│   ├── clients.py       # Client companies
│   └── git_ops.py       # Deployment automation
├── utils/               # Shared utilities
│   ├── auth.py          # Authentication decorators
│   ├── qualer_client.py # Qualer SDK configuration
│   ├── get_token.py     # Azure AD token helper
│   └── schemas.py       # Marshmallow schemas
└── tests/               # Test suite
    ├── conftest.py      # Test fixtures
    └── test_*.py        # Individual test files

🔧 Development Guidelines

Adding New Endpoints

1. Create route file in /routes/ as Flask-Smorest blueprint
2. Add authentication with @require_auth decorator
3. Define schemas for request/response serialization
4. Register blueprint in app.py
5. Create corresponding test file in /tests/
6. Prefer recording HTTP interactions with pytest-vcr and committing cassettes; migrate old mock implementations when convenient

Code Style

• Import Organization: Standard library, third-party, local imports
• Error Handling: Use Flask-Smorest abort() for API errors
• Documentation: Include docstrings for complex business logic
• Type Hints: Use type hints for better code clarity

🚀 Deployment

Production Environment

• URL: https://jgiapi.com
• WSGI Server: Waitress (production-grade, Windows-compatible)
• Deployment: Automated via GitHub Actions
• Proxy: Uses ProxyFix middleware for reverse proxy support
• CORS: Enabled for cross-origin requests from Excel/Power Query

Production Server

The API runs using Waitress WSGI server via wsgi_server.py:

python wsgi_server.py

Why Waitress?

• ✅ Pure Python - works natively on Windows
• ✅ Multi-threaded - handles concurrent requests efficiently
• ✅ Production-stable - no Flask dev server warnings
• ✅ Zero configuration - works out of the box

Deployment Process

1. Code Push: Push to main branch triggers deployment
2. GitHub Action: Calls /git-pull endpoint on production server
3. Server Update: Production server pulls latest code and restarts
4. Health Check: Verify endpoints respond correctly

Log Monitoring

The production server uses Vector (log ingestion agent) to forward application logs to SparkLogs for centralized monitoring. Vector runs as a Windows service (VectorLogAgent) that:

• ✅ Starts automatically on server boot
• ✅ Restarts on failure
• ✅ Forwards logs to SparkLogs without manual intervention
• ✅ Parses Python logging format with multiline support (stack traces, JSON blocks)
• ✅ Enriches logs with metadata (file, line number, logger name, host)

Monitored Logs

Vector ingests logs from:

• app.log – Main Flask application events and errors
• deployment.log – Deployment and sync operation logs

Log Parsing

Vector automatically:

1. Parses Python logging format: YYYY-MM-DD HH:MM:SS,milliseconds LEVEL logger [file:line]: message
2. Reassembles multiline events (stack traces, formatted JSON blocks)
3. Extracts metadata fields: timestamp, level, logger name, source file/line
4. Forwards to SparkLogs with gzip compression

Service Management

# Check service status
Get-Service VectorLogAgent

# Stop the service
Stop-Service -Name VectorLogAgent

# Start the service
Start-Service -Name VectorLogAgent

# View Vector logs (if running interactively)
C:\vector\bin\vector run -c C:\vector\vector.yaml

Configuration

Vector configuration: C:\vector\vector.yaml

• SparkLogs endpoint: https://ingest-us.engine.itlightning.app/ingest/v1
• Data directory: C:\vector\data (stores checkpoints for file position tracking)

📚 Documentation & Resources

API Documentation

• Interactive Docs: /docs - Swagger UI for testing endpoints
• OpenAPI Spec: /openapi.json - Machine-readable API specification

Development Documentation

• Development Guide: DEVELOPMENT_GUIDE.md - Complete development, testing, and architecture guide
• Copilot Instructions: .github/.copilot-instructions.md - GitHub Copilot development guidance
• Domain Logic: .github/DOMAIN_LOGIC.md - Wire calibration business logic and workflows
• Qualer SDK: OpenAPI Specification

External Integrations

• Qualer SDK: Primary integration for quality management data
• Azure AD: Authentication and user management
• Excel/Power Query: Designed for consumption by Microsoft tools

🔒 Security Considerations

• Secret Management: Never commit API keys or secrets to git
• Environment Variables: Use environment variables for all configuration
• Input Validation: All input parameters are validated via Marshmallow schemas
• Token Security: Proper scope checking for Azure AD tokens
• Audit Logging: Security-relevant events are logged for monitoring

📞 Support & Troubleshooting

Common Issues

• Authentication failures: Check environment variables and Azure AD configuration
• Qualer API errors: Verify QUALER_API_KEY permissions and format
• CORS issues: Ensure requests include proper headers for cross-origin access
• Testing failures: See DEVELOPMENT_GUIDE.md for debugging strategies

Development Help

• VS Code: Use the Testing panel for running and debugging individual tests
• Debugging: Set SKIP_AUTH=true for development without Azure AD setup
• Logs: Check Flask application logs for detailed error information

---

🎯 Usage in Excel/Power Query

This API is designed for easy consumption in Microsoft Excel using Power Query:

1. Data → Get Data → From Web
2. URL: https://jgiapi.com/endpoint
3. Authentication: Choose Organizational Account
4. Sign in: Use your JGI Azure AD credentials
5. Load: Data loads automatically with proper authentication headers

Example Power Query M code:

let
    Source = Json.Document(
        Web.Contents(
            "https://jgiapi.com/employees",
            [Headers=[Authorization="Bearer " & AccessToken.Token]]
        )
    )
in
    Source