Adapt assistant rules to new format, and add CLAUDE.md

.claude/rules/creating-new-drivers.md

@@ -0,0 +1 @@
+../../.cursor/rules/creating-new-drivers.mdc

.claude/rules/project-structure.md

@@ -0,0 +1 @@
+../../.cursor/rules/project-structure.mdc

python/.cursor/rules/creating-new-drivers.mdc → .cursor/rules/creating-new-drivers.mdc

@@ -7,7 +7,7 @@ alwaysApply: false
 When asked to create a new driver, follow these steps:
 
 ## 1. Use the Driver Creation Script
-Always use the provided script: `./__templates__/create_driver.sh`
+Always use the provided script: `./python/__templates__/create_driver.sh`
 
 ## 2. Required Information
 Before creating a driver, ask the user for:
@@ -22,24 +22,24 @@ Before creating a driver, ask the user for:
 
 ## 3. Command Format
 ```bash
-./__templates__/create_driver.sh <driver_package> <DriverClass> "<Author Name>" "<author@email.com>"
+./python/__templates__/create_driver.sh <driver_package> <DriverClass> "<Author Name>" "<author@email.com>"
 ```
 Always try to obtain the Author Name and Email from git, by checking the git configuration.
 ## 4. Examples
 ```bash
 # Network driver
-./__templates__/create_driver.sh network TcpNetwork "John Doe" "john@example.com"
+./python/__templates__/create_driver.sh network TcpNetwork "John Doe" "john@example.com"
 
 # Power management driver
-./__templates__/create_driver.sh custom-power CustomPowerDriver "Jane Smith" "jane@example.com"
+./python/__templates__/create_driver.sh custom-power CustomPowerDriver "Jane Smith" "jane@example.com"
 
 # Device control driver
-./__templates__/create_driver.sh device-controller DeviceController "Bob Wilson" "bob@example.com"
+./python/__templates__/create_driver.sh device-controller DeviceController "Bob Wilson" "bob@example.com"
 ```
 
 ## 5. After Creation
 Once the driver is created:
-1. Navigate to the new driver directory: `packages/jumpstarter-driver-<driver_package>/`
+1. Navigate to the new driver directory: `python/packages/jumpstarter-driver-<driver_package>/`
 2. Review the generated files and customize as needed
 3. Implement the driver logic in `driver.py`
 4. Add tests in `driver_test.py`
@@ -59,7 +59,7 @@ Once the driver is created:
 ## 7. Directory Structure
 The script creates:
 ```
-packages/jumpstarter-driver-<driver_package>/
+python/packages/jumpstarter-driver-<driver_package>/
 ├── jumpstarter_driver_<driver_package>/
 │   ├── __init__.py
 │   ├── client.py
@@ -75,9 +75,9 @@ packages/jumpstarter-driver-<driver_package>/
 ## 8. Documentation
 The script automatically creates:
 - A README.md with basic documentation
-- A symlink in `docs/source/reference/package-apis/drivers/` pointing to the README
+- A symlink in `python/docs/source/reference/package-apis/drivers/` pointing to the README
 - Template files for all necessary components
-- A good example of documentation is in `docs/source/reference/package-apis/drivers/gpiod.md` and also `docs/source/reference/package-apis/drivers/pyserial.md`
+- A good example of documentation is in `python/docs/source/reference/package-apis/drivers/gpiod.md` and also `python/docs/source/reference/package-apis/drivers/pyserial.md`
 
 ## Code Style and Testing
 
@@ -99,7 +99,7 @@ The script automatically creates:
 Some drivers implement known classes that provide a CLI interface for the driver, but other
 clients implement their own CLI interface that will appear in the `j` command inside a `jmp shell`.
 
-Good examples can be found in `packages/jumpstarter-driver-shell/jumpstarter_driver_shell/client.py`, `packages/jumpstarter-driver-pyserial/jumpstarter_driver_pyserial/client.py` or `packages/jumpstarter-driver-probe-rs/jumpstarter_driver_probe_rs/client.py`.
+Good examples can be found in `python/packages/jumpstarter-driver-shell/jumpstarter_driver_shell/client.py`, `python/packages/jumpstarter-driver-pyserial/jumpstarter_driver_pyserial/client.py` or `python/packages/jumpstarter-driver-probe-rs/jumpstarter_driver_probe_rs/client.py`.
 
 # Composite drivers
 

python/.cursor/rules/project-structure.mdc → .cursor/rules/project-structure.mdc

@@ -10,20 +10,23 @@ This project follows a monorepo structure with a top-level `pyproject.toml` that
 
 ```
 jumpstarter/
-├── pyproject.toml          # Main workspace configuration
-├── packages/               # All Python packages
-├── examples/               # Example applications
-├── docs/                   # Documentation
-├── __templates__/          # Templates for creating new drivers
+├── python/                 # Python workspace root
+│   ├── pyproject.toml      # Main workspace configuration
+│   ├── packages/           # All Python packages
+│   ├── examples/           # Example applications
+│   ├── docs/               # Documentation
+│   └── __templates__/      # Templates for creating new drivers
+├── controller/             # Controller components
+├── protocol/               # Protocol definitions
 └── .cursor/                # Cursor AI rules
 ```
 
 ## Workspace Configuration
 
 The project uses **UV workspace** for dependency management:
 
-- **Top-level `pyproject.toml`**: Defines the workspace and includes all packages
-- **Workspace members**: All packages in `packages/*` and `examples/*` are included
+- **Top-level `python/pyproject.toml`**: Defines the workspace and includes all packages
+- **Workspace members**: All packages in `python/packages/*` and `python/examples/*` are included
 - **Dependency groups**: Shared development dependencies (docs, dev)
 - **Tool configuration**: Ruff, typos, coverage, pytest settings
 
@@ -62,7 +65,7 @@ All driver packages follow the pattern `jumpstarter-driver-<name>/`:
 Each package follows this structure:
 
 ```
-packages/jumpstarter-driver-<name>/
+python/packages/jumpstarter-driver-<name>/
 ├── jumpstarter_driver_<name>/    # Main Python package
 │   ├── __init__.py
 │   ├── driver.py                 # Driver implementation
@@ -88,7 +91,7 @@ Each package's `pyproject.toml` includes:
 ## Examples Structure
 
 ```
-examples/
+python/examples/
 ├── automotive/                  # Automotive testing example
 │   ├── jumpstarter_example_automotive/
 │   ├── pyproject.toml
@@ -103,7 +106,7 @@ examples/
 
 ### Creating New Packages
 
-1. **Drivers**: Use `./__templates__/create_driver.sh` (see creating-new-drivers.mdc)
+1. **Drivers**: Use `./python/__templates__/create_driver.sh` (see creating-new-drivers.mdc)
 2. **Other packages**: Create manually following existing patterns
 
 ### Package Dependencies
@@ -130,7 +133,7 @@ examples/
 1. **Naming**: Package names use hyphens, module names use underscores
 2. **Entry points**: Drivers register via `jumpstarter.drivers` entry point
 3. **Versioning**: All packages share the same version from VCS
-4. **Documentation**: Each package has its own README.md, and when it's a driver we make a symlink in `docs/source/reference/package-apis/drivers/` pointing to the README.md for the driver.
+4. **Documentation**: Each package has its own README.md, and when it's a driver we make a symlink in `python/docs/source/reference/package-apis/drivers/` pointing to the README.md for the driver.
 5. **Testing**: Comprehensive test coverage required, but always try to focus on starting a server and client to test it e2e. Mock sometimes when there is too much dependency on system tools/services/compatibility issues between MacOs/Linux.
 6. **Dependencies**: Minimal, focused dependencies per package
 

.cursor/rules/working-with-operator.mdc

@@ -0,0 +1,185 @@
+---
+description: when the user is modifying the Kubernetes operator, CRDs, or controller logic
+---
+
+# Working with the Jumpstarter Operator
+
+The Jumpstarter operator is a Kubernetes operator that manages Jumpstarter deployments. It is located in `controller/deploy/operator/`.
+
+## Directory Structure
+
+```
+controller/
+├── deploy/
+│   ├── operator/                    # Operator code
+│   │   ├── api/v1alpha1/           # CRD type definitions
+│   │   │   └── jumpstarter_types.go # Main types file
+│   │   ├── internal/controller/     # Controller logic
+│   │   │   └── jumpstarter/
+│   │   │       ├── jumpstarter_controller.go  # Main reconciler
+│   │   │       ├── endpoints/       # Endpoint handling (routes, ingress, etc.)
+│   │   │       └── rbac.go          # RBAC reconciliation
+│   │   ├── config/                  # Kustomize configurations
+│   │   │   ├── crd/bases/          # Generated CRD YAML
+│   │   │   └── samples/            # Sample CR files
+│   │   ├── bundle/                  # OLM bundle manifests
+│   │   ├── test/e2e/               # E2E tests
+│   │   └── Makefile                # Operator-specific make targets
+│   └── helm/                       # Helm charts (alternative deployment)
+├── api/                            # Jumpstarter core CRDs (Client, Exporter, Lease, etc.)
+├── internal/                       # Controller business logic
+│   └── config/                     # Config structs used by controller
+└── Makefile                        # Top-level controller make targets
+```
+
+## Key Files
+
+- **`api/v1alpha1/jumpstarter_types.go`**: Defines the `Jumpstarter` CRD spec and status
+- **`internal/controller/jumpstarter/jumpstarter_controller.go`**: Main reconciliation logic
+- **`internal/controller/jumpstarter/jumpstarter_controller.go:buildConfig()`**: Translates CRD spec to internal config format for ConfigMaps
+
+## Development Workflow
+
+### After Modifying CRD Types
+
+When you change `jumpstarter_types.go`, you MUST regenerate code:
+
+```bash
+cd controller/deploy/operator
+make manifests generate
+```
+
+This will:
+1. Regenerate CRD YAML in `config/crd/bases/`
+2. Regenerate `DeepCopy` methods in `zz_generated.deepcopy.go`
+
+### Building
+
+```bash
+# Build operator binary
+cd controller/deploy/operator
+make build
+
+# Build container image
+make docker-build
+```
+
+### Running Tests
+
+```bash
+# Unit tests (from operator directory)
+cd controller/deploy/operator
+make test
+
+# E2E tests (from controller directory - sets up kind cluster)
+cd controller
+make test-operator-e2e
+```
+
+### Regenerating OLM Bundle
+
+After CRD changes, regenerate the OLM bundle:
+
+```bash
+cd controller/deploy/operator
+make bundle
+```
+
+## CRD Schema Guidelines
+
+### Adding New Fields
+
+1. Add the field to the appropriate struct in `jumpstarter_types.go`
+2. Use kubebuilder markers for validation and defaults:
+   ```go
+   // +kubebuilder:default="value"
+   // +kubebuilder:validation:Pattern=^[a-z]+$
+   // +kubebuilder:validation:Minimum=1
+   FieldName string `json:"fieldName,omitempty"`
+   ```
+3. Add documentation comments (they become CRD descriptions)
+4. Run `make manifests generate`
+
+### Field Location Conventions
+
+- **Top-level spec fields**: Global settings (baseDomain, useCertManager, authentication)
+- **Controller config**: Controller-specific settings (image, replicas, grpc endpoints)
+- **Router config**: Router-specific settings (image, replicas, topology constraints)
+
+### Example: Authentication Config
+
+Authentication is configured at the **controller level** (not top-level) because it's specific to controller behavior:
+
+```yaml
+spec:
+  controller:
+    authentication:
+      internal:
+        enabled: true
+        prefix: "internal:"
+      k8s:
+        enabled: false
+      jwt: []
+```
+
+## Controller Logic
+
+### The Reconcile Loop
+
+The main reconciler in `jumpstarter_controller.go` follows this order:
+
+1. Fetch the Jumpstarter CR
+2. Apply runtime defaults (via `EndpointReconciler.ApplyDefaults`)
+3. Reconcile RBAC (ServiceAccount, Role, RoleBinding)
+4. Reconcile Controller Deployment
+5. Reconcile Router Deployments (one per replica)
+6. Reconcile Services and networking (Routes, Ingress, NodePort, LoadBalancer)
+7. Reconcile ConfigMaps (includes controller config and router config)
+8. Reconcile Secrets (only created if missing, not updated)
+9. Update status
+
+### The buildConfig Function
+
+`buildConfig()` translates the CRD spec into the internal `config.Config` struct that gets serialized to YAML in the ConfigMap. When adding new CRD fields that affect runtime config:
+
+1. Add the field to `jumpstarter_types.go`
+2. Update `buildConfig()` to read from the new field
+3. Ensure the internal `config.Config` struct (in `controller/internal/config/`) supports the field
+
+## Testing Changes
+
+### Unit Tests
+
+Unit tests use envtest (fake Kubernetes API). They're located alongside the code:
+- `jumpstarter_controller_test.go`
+- `endpoints/*_test.go`
+
+### E2E Tests
+
+E2E tests (`test/e2e/e2e_test.go`) run against a real kind cluster:
+
+1. Deploy the operator
+2. Create Jumpstarter CRs
+3. Verify deployments, services, configmaps are created correctly
+4. Test scaling, endpoint access, etc.
+
+When modifying CRD schema, update the test YAML in `e2e_test.go` to match.
+
+## Common Issues
+
+### CRD Validation Errors
+
+If you get validation errors when applying CRs, check:
+1. kubebuilder markers match your intended constraints
+2. Default values are valid according to validation rules
+3. Run `make manifests` after any type changes
+
+### ConfigMap Not Updated
+
+The controller compares existing vs desired state. If your changes aren't reflected:
+1. Check `buildConfig()` reads from the correct spec field
+2. Verify the comparison logic in `configMapNeedsUpdate()`
+
+### Secrets Not Updated
+
+Secrets are intentionally NOT updated after creation (to preserve keys). They're also NOT owned by the CR (won't be deleted when CR is deleted). This is by design for security.

CLAUDE.md

@@ -0,0 +1,28 @@
+# Claude AI Instructions
+
+This file provides instructions for Claude AI when working with the Jumpstarter project.
+
+## Project Rules and Guidelines
+
+Important project-specific rules and guidelines are located in the `.claude/rules` directory:
+
+- **`.claude/rules/project-structure.md`**: Understanding the monorepo structure, workspace configuration, package organization, and development workflows. Read this to understand how the project is organized and where files should be located.
+
+- **`.claude/rules/creating-new-drivers.md`**: Guidelines for creating new driver packages, including naming conventions, required information, and the driver creation process. Read this when tasked with creating or modifying drivers.
+
+## When to Read These Rules
+
+- **Always**: Read `project-structure.md` when working with files, packages, or understanding the codebase layout
+- **When creating drivers**: Read `creating-new-drivers.md` before creating, improving, or documenting driver packages
+- **When modifying structure**: Consult both files when making changes that affect project organization
+
+## Key Information
+
+- All Python code is located under the `python/` directory
+- The project uses UV workspace for dependency management
+- Driver creation script: `./python/__templates__/create_driver.sh`
+- Testing: Use `make pkg-test-<package_name>` for package-specific tests
+- Linting: Use `make lint-fix` to fix linting issues
+- Type checking: Use `make pkg-ty-<package_name>` for type checking
+
+Please refer to the detailed rules in `.claude/rules/` for comprehensive guidance.

python/__templates__/create_driver.sh

@@ -32,15 +32,19 @@ else
     sed_cmd="sed -i"
 fi
 
+# Get the script directory and navigate to python/ root
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+PYTHON_ROOT="$( cd "${SCRIPT_DIR}/.." && pwd )"
+
 # create the driver directory
-DRIVER_DIRECTORY=packages/jumpstarter-driver-${DRIVER_NAME}
+DRIVER_DIRECTORY=${PYTHON_ROOT}/packages/jumpstarter-driver-${DRIVER_NAME}
 MODULE_DIRECTORY=${DRIVER_DIRECTORY}/jumpstarter_driver_${DRIVER_MODULE_NAME}
 # create the module directories
 mkdir -p ${MODULE_DIRECTORY}
 mkdir -p ${DRIVER_DIRECTORY}/examples
 
 # Define paths
-DOCS_DIRECTORY=docs/source/reference/package-apis/drivers
+DOCS_DIRECTORY=${PYTHON_ROOT}/docs/source/reference/package-apis/drivers
 DOC_FILE=${DOCS_DIRECTORY}/${DRIVER_NAME}.md
 README_FILE=${DRIVER_DIRECTORY}/README.md
 
@@ -87,10 +91,10 @@ echo "Created symlink: ${DOC_FILE} -> ${rel_path}"
 
 for f in __init__.py client.py driver_test.py driver.py; do
     echo "Creating: ${MODULE_DIRECTORY}/${f}"
-    envsubst < __templates__/driver/jumpstarter_driver/${f}.tmpl > ${MODULE_DIRECTORY}/${f}
+    envsubst < ${SCRIPT_DIR}/driver/jumpstarter_driver/${f}.tmpl > ${MODULE_DIRECTORY}/${f}
 done
 
 for f in .gitignore pyproject.toml examples/exporter.yaml; do
     echo "Creating: ${DRIVER_DIRECTORY}/${f}"
-    envsubst < __templates__/driver/${f}.tmpl > ${DRIVER_DIRECTORY}/${f}
+    envsubst < ${SCRIPT_DIR}/driver/${f}.tmpl > ${DRIVER_DIRECTORY}/${f}
 done