Streaming llm

[BrandonWeng]

No description provided.

[github-actions]

Whisper Sample App Results - Windows x64

Generated: 20250810-210505
Platform: win-x64
Model: whisper-tiny-fp16-ov

Transcription Results

how_are_you_doing_today.wav

OpenVINO.NET Whisper Demo
=========================

Model: D:\a\Fluid.OpenVINO.GenAI\Fluid.OpenVINO.GenAI\Models\whisper-tiny-fp16-ov
Audio: D:\a\Fluid.OpenVINO.GenAI\Fluid.OpenVINO.GenAI\how_are_you_doing_today.wav
Device: CPU

Creating Whisper pipeline...
Setting DLL directory to: D:\a\Fluid.OpenVINO.GenAI\Fluid.OpenVINO.GenAI/build/native/runtimes/win-x64/native
Pipeline created successfully.
Starting transcription...
[WhisperPipeline] Invoking native generate. Handle: 2180636998624, ConfigHandle: 2180760697152
[WhisperPipeline] Native generate returned status: OK, ResultsHandle: 2180760696800
[WhisperPipeline] Extracting results from native handle.
[WhisperPipeline] Extracted 1 results.
Transcription completed. Found 1 result(s).

Transcription:
   How are you doing today?
  Score: 1.0000

[github-actions]

Performance Benchmark Results - Windows x64

Generated: 20250810-210452
Platform: win-x64

Exit Code: 0, Time: 20322ms):
OpenVINO.NET Quick Demo
=======================
Model: FluidInference/qwen3-0.6b-int4-ov-npu
Temperature: 0.7, Max Tokens: 100

✓ Using model from environment variable: D:\a\Fluid.OpenVINO.GenAI\Fluid.OpenVINO.GenAI/samples/QuickDemo/Models/qwen3-0.6b-int4-ov-npu
Device: CPU

Setting DLL directory to: D:\a\Fluid.OpenVINO.GenAI\Fluid.OpenVINO.GenAI/build/native/runtimes/win-x64/native
Prompt 1: "Explain quantum computing in simple terms:"
Response: "<think>
Okay, the user wants a simple explanation of quantum computing. Let me start by breaking down what quantum computing is. It's a field that uses quantum particles, like superposits or qubits, to perform calculations. The key here is that it can solve specific problems faster than classical methods. I should explain the basics without getting too technical. Maybe start with what quantum computing does instead of solving problems. Use a metaphor: solving a puzzle or playing chess. Use a simple example. Use"
Performance: 16.3 tokens/sec, First token: 3022ms, Memory: 0.1MB

Prompt 2: "Write a short poem about artificial intelligence:"
Response: "<think>
Okay, the user wants a short poem about artificial intelligence. Let me start by breaking down the key elements. I need to capture the blend of science and human elements. Maybe use some metaphors or familiar terms to make it relatable. Let's think about the structure: maybe four lines or a stanza. The topic is important here.

First, I should think about the main aspects of AI. It could be about its potential for good or its limitations. Maybe mention both the benefits and"
Performance: 30.9 tokens/sec, First token: 238ms, Memory: 0.0MB

Prompt 3: "What are the benefits of renewable energy?"
Response: "<think>
Okay, the user is asking about the benefits of renewable energy. First, I need to recall what renewable energy means. It's a sustainable source of energy that can help the planet and future generations. The user might be looking for information on the benefits of renewable energy. Let me think about this topic.

The main benefits of renewable energy include its role in reducing greenhouse emissions and combating climate change. It also contributes to reducing long-term costs for energy production. These are three main benefits. The"
Performance: 30.9 tokens/sec, First token: 234ms, Memory: 0.1MB

Prompt 4: "Describe the process of making coffee:"
Response: "<think>
Okay, the user wants a description of the process of making coffee. I need to recall the steps involved. First, beans are roasted and ground. Then, they are usually roasted by a roaster, perhaps in a different way. They need to roast them, maybe in a different way. They need to brew it, maybe in a different way. Then they need to heat it, maybe in a different way. Then they need to clean it, maybe in a different way. Then"
Performance: 31.7 tokens/sec, First token: 231ms, Memory: 0.1MB

Summary Table:
==============
Prompt   | Tokens/sec |  First Token | Tokens |  Time (ms) |  Memory (MB)
----------------------------------------------------------------------
      1  |       16.3 |       3022ms |     98 |       6015 |          0.1
      2  |       30.9 |        238ms |     96 |       3108 |          0.0
      3  |       30.9 |        234ms |     96 |       3103 |          0.1
      4  |       31.7 |        231ms |     98 |       3096 |          0.1
----------------------------------------------------------------------
Average  |       27.4 |        931ms |      - |          - |          0.1