Skip to content

NathanWalker/ns-flue

BranchesTags

Repository files navigation

The agent's shell is the phone.

Flue agent harness framework running on NativeScript where the device itself is the agent's sandbox, Apple Intelligence (FoundationModels) is the on-device LLM, and @nstudio/nstreamdown rendering the agent's streaming markdown output natively.

ns-flue-overview.mov 
┌──────────────────────────────────────────────────────────────┐
│  Agnostic UI  (NativeScript)                          │
│   Mission picker · live transcript · animated tool-call cards │
└──────────────┬───────────────────────────────────────────────┘
               │  Stream of FlueEvent
┌──────────────▼───────────────────────────────────────────────┐
│  Flue runtime  (in-tree, NS-native — mirrors @flue/sdk/client)│
│   init() · session.prompt/skill/task · markdown-defined skills│
└──────────────┬───────────────────────────────────────────────┘
               │  SessionEnv interface
┌──────────────▼───────────────────────────────────────────────┐
│  Device Sandbox  (NativeScript)                │
│   fs:    NS Folder / File mapped to app sandbox              │
│   tools: UIKit / AVFoundation / etc.     │
│   model: FoundationModels (iOS 26+) / fallback elsewhere │
└──────────────────────────────────────────────────────────────┘

Two runtimes, side by side

This project ships two Flue runtimes and lets you switch between them at runtime from the Polyfills tab. The switch is persisted across launches.

Runtime A — real — actual @flue/sdk@0.3.x (default at first launch)

The real package, bundled into the NS app via a polyfill layer in webpack.config.js and src/flue/real/. When you switch to this runtime:

  • createFlueContext, FlueAgent, FlueSession, InMemorySessionStore, pi-agent-core's harness, pi-ai's event stream — all the actual upstream bytes execute on your phone.
  • Your NativeSandbox (@nativescript/core Folder/File-backed) is plugged in as Flue's createDefaultEnv and createLocalEnv.
  • An on-device pi-ai provider for Apple FoundationModels is registered via the framework's own registerApiProvider extension point — no fork, no patch, just the public API.

Both runtimes drive the same UI, the same missions, the same skills, the same tools. The Polyfills tab shows you which one is active and lets you flip between them so you can A/B the same prompt under both.

Runtime B — shim

A Flue-API-shaped runtime in src/flue/ (~600 LOC) that mirrors @flue/sdk/client's public surface — same names, same call shapes, same Valibot-style schemas, same skill loading, same RxJS-style events. Boots instantly, has no dependencies, ~5 KB additional bundle.

Why we needed a polyfill layer at all

@flue/sdk@0.3.x is built for Node.js, Cloudflare Workers, and CI runtimes. Its transitive dependency tree pulls in just-bash, @hono/node-server, esbuild, @modelcontextprotocol/sdk, @google/genai, @aws-sdk/client-bedrock-runtime, google-auth-library, undici, ws, proxy-agent, and a long tail of node:fs / node:net / node:tls / node:crypto reach-throughs. NativeScript V8 has none of those.

  1. Replace @mariozechner/pi-ai/dist/providers/register-builtins.js with an in-tree stub that registers no cloud providers. Removes the entire google-genai / openai / anthropic / mistral / bedrock chain (the bulk of the Node-API blockers).
  2. Alias pkce-challenge/dist/index.node.js → pkce-challenge/dist/index.browser.js. The package ships a Web Crypto build; we route MCP HTTP transport through it instead of node:crypto.
  3. Resolve.fallback the remaining Node modules to false (they're only reachable from cloud-provider code paths the on-device demo never exercises).
  4. NormalModuleReplacementPlugin to strip node: URI prefixes so node:crypto falls through to the same crypto fallback as a plain import 'crypto'.

The result: real @flue/sdk runs end-to-end in NativeScript, the Polyfills tab shows every decision live, and the diff between modes is reduced to one AgentService.setRuntime('shim' | 'real') call.

See webpack.config.js and src/flue/real/polyfills/ for the full source.

Prerequisites

Requirement Why
Node 22+ and npm NativeScript CLI install
nativescript (ns command) npm i -g nativescript
Xcode 26+ iOS build, FoundationModels framework available in SDK
iOS 26 simulator or device Required to actually use Apple Intelligence on-device
iPhone 15 Pro / iPhone 16 / 16 Pro Apple Intelligence is gated on these device classes
Android Studio + emulator (opt.) Android target boots, runs the mock LLM, and runs all tools

The app also runs on the iOS simulator and on devices that don't support Apple Intelligence — it falls back to a deterministic mock LLM that emits plausible XML tool-call programs so the full UI flow lights up.

Install & run

git clone https://github.com/NathanWalker/ns-flue
cd ns-flue
npm install

iOS

ns run ios                    # simulator, default booted device
ns run ios --device <udid>    # specific device or simulator

Android

ns run android

Android target uses the mock LLM — Apple Intelligence is iOS-only. The NativeSandbox, all schema-typed tools, the markdown skill loader, the live event timeline, and the entire UI all work on Android.

What you can actually do

The home screen — Mission Control — gives you four canned missions:

  1. Brief me (brief-me skill) — checks battery, location, narrates a one-liner via TTS. The "Phase 0/1 spike" demo.
  2. Catalog this shelf (catalog-shelf skill) — writes a Markdown inventory into the agent's workspace; camera/object-classification tools land in Phase 4.
  3. Find my missing thing (find-my-thing skill) — hotter/colder game with haptics + TTS; BLE RSSI tool lands in Phase 4.
  4. Free prompt — open shell. Send anything to the agent.

While the agent runs, the Live Timeline on the run screen narrates each step: streaming text, tool calls (icon per tool), tool results, errors, nested tasks, final reply. This is the Flue harness narrating itself.

The other tabs:

  • Skills — picker for the bundled *.md skills, with each skill's full instructions visible. Demonstrates the OTA story: skills are content, not code.
  • Architecture — the layered diagram + per-Flue-primitive callouts + phased plan status.
  • Polyfills — the demonstration tab. Live status of every Node API + library surface @flue/sdk pulls in (real / polyfilled / stubbed / not-applicable), runtime selector to flip shim ↔ real, plus a note for NS maintainers on which surfaces would unlock the most SDK integrations.
  • About — the inversion pitch and the three demo-mission descriptions verbatim from the plan.

Markdown rendering — @nstudio/nstreamdown

The agent emits Markdown (skills, replies, streamed turns). We render it natively via @nstudio/nstreamdown's Streamdown component.

How the runtime works (in 60 seconds)

  1. AgentService.ensureReady() (root service) calls init({ tools, skills, model }) once.
  2. init() creates a NativeSandbox (a SessionEnv backed by Documents/agent-workspace/), and an RxJS Subject<FlueEvent> for UI consumption.
  3. The user picks a mission. agent.session('mission-<id>').skill(name, { args }) is called.
  4. Skill markdown is loaded from knownFolders.currentApp()/agent-skills/<name>.md, frontmatter parsed, body rendered with {{ args }} substitution.
  5. The model (FoundationModelsLLM on iOS 26+, MockLLM everywhere else) streams cumulative text via the Swift FlueAI.streamPrompt bridge.
  6. runToolLoop parses any <tool name="…">{json}</tool> blocks the model emits, schema-validates the args via the in-tree valibot-mini, dispatches against the sandbox, then feeds <tool_result for="…">…</tool_result> blocks back into the next turn. Capped at maxSteps (default 8).
  7. Every step is emitted as a FlueEvent (agent_start, thinking, text_delta, tool_start, tool_end, task_*, turn_end, agent_end). The EventTimelineComponent collapses streaming deltas into a "thinking" card and pairs tool_start/tool_end into a single animated tool card.

Extending

Add a new tool

// src/flue/tools/take-photo.ts
import * as v from 'valibot';
import type { ToolDef } from '../types';

export const takePhoto: ToolDef = {
  name: 'take_photo',
  description: 'Capture a photo from the rear camera. Returns the saved path.',
  schema: v.object({
    hint: v.optional(v.string()),
  }),
  async execute({ hint }, ctx) {
    // ...wire to AVCapturePhotoOutput, write to ctx.env, return path...
    return JSON.stringify({ path: 'photos/2026-05-02-001.jpg', hint });
  },
};

Register it in src/flue/tools/index.ts and the agent will discover it through the LLM system prompt automatically.

Add a new skill

<!-- src/agent-skills/my-skill.md -->
---
name: my-skill
description: One-liner shown in the Skills tab.
---

You are doing X.

Steps:
1. Call `tool_a` with these args.
2. If condition, call `tool_b`.
3. Speak a short summary via `speak`.

The webpack copy rule (agent-skills/**/*.md) packages it into the app bundle. loadBundledSkills() discovers it on next cold start.

Add a new mission card

Append a MissionDefinition to the MISSIONS array in src/app/agent.service.ts. It'll show up on the home screen automatically.

Capability probe

The home screen's Capability Probe card surfaces, at first launch:

  • Active platform & SDK version
  • Whether FlueAI.shared.isAvailable() returns true (FoundationModels framework linkable + on-device model present)
  • Active model id (apple/foundationmodels-on-device vs flue/mock-on-device)
  • A human reason if FoundationModels is unavailable
  • Skill / tool counts

Use this as your first triage step when something looks off.

Links

About

Using https://flueframework.com/ with NativeScript where the device itself is the agent's sandbox.

Topics

typescript nativescript applefoundationalmodels

Resources

Readme
Activity

Stars

5 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages