feat: notebooks now run on deepnote kernel

Author: saltenasl

DEEPNOTE_KERNEL_IMPLEMENTATION.md

@@ -19,6 +19,7 @@ This implementation adds automatic kernel selection and startup for `.deepnote`
 - `IDeepnoteToolkitInstaller`: Interface for toolkit installation service
 - `IDeepnoteServerStarter`: Interface for server management
 - `IDeepnoteKernelAutoSelector`: Interface for automatic kernel selection
+- `DeepnoteServerInfo`: Server connection information (URL, port, token)
 - Constants for wheel URL, default port, and notebook type
 
 #### 2. **Deepnote Toolkit Installer** (`src/kernels/deepnote/deepnoteToolkitInstaller.node.ts`)
@@ -47,9 +48,24 @@ This implementation adds automatic kernel selection and startup for `.deepnote`
 - `stopServer(deepnoteFileUri)`: Stops the running server for a specific file
 - `isServerRunning(serverInfo)`: Checks if server is responsive
 
-#### 4. **Deepnote Kernel Auto-Selector** (`src/notebooks/deepnote/deepnoteKernelAutoSelector.node.ts`)
+#### 4. **Deepnote Server Provider** (`src/kernels/deepnote/deepnoteServerProvider.node.ts`)
+- Jupyter server provider that registers and resolves Deepnote toolkit servers
+- Implements `JupyterServerProvider` interface from VSCode Jupyter API
+- Maintains a map of server handles to server connection information
+- Allows the kernel infrastructure to resolve server connections
+
+**Key Methods:**
+- `activate()`: Registers the server provider with the Jupyter server provider registry
+- `registerServer(handle, serverInfo)`: Registers a Deepnote server for a specific handle
+- `provideJupyterServers(token)`: Lists all registered Deepnote servers
+- `resolveJupyterServer(server, token)`: Resolves server connection info by handle
+
+#### 5. **Deepnote Kernel Auto-Selector** (`src/notebooks/deepnote/deepnoteKernelAutoSelector.node.ts`)
 - Activation service that listens for notebook open events
 - Automatically selects Deepnote kernel for `.deepnote` files
+- Queries the Deepnote server for available kernel specs
+- Uses an existing kernel spec from the server (e.g., `python3-venv`)
+- Registers the server with the server provider
 - Creates kernel connection metadata
 - Registers the controller with VSCode
 - Auto-selects the kernel for the notebook
@@ -59,11 +75,12 @@ This implementation adds automatic kernel selection and startup for `.deepnote`
 - `ensureKernelSelected(notebook)`: Main logic for auto-selection
 - `onDidOpenNotebook(notebook)`: Event handler for notebook opens
 
-#### 5. **Service Registry Updates** (`src/notebooks/serviceRegistry.node.ts`)
+#### 6. **Service Registry Updates** (`src/notebooks/serviceRegistry.node.ts`)
 - Registers all new Deepnote kernel services
+- Binds `DeepnoteServerProvider` as an activation service
 - Binds `IDeepnoteKernelAutoSelector` as an activation service
 
-#### 6. **Kernel Types Updates** (`src/kernels/types.ts`)
+#### 7. **Kernel Types Updates** (`src/kernels/types.ts`)
 - Adds `DeepnoteKernelConnectionMetadata` to `RemoteKernelConnectionMetadata` union type
 - Adds deserialization support for `'startUsingDeepnoteKernel'` kind
 
@@ -100,7 +117,13 @@ Start: python -m deepnote_toolkit server --jupyter-port <port>
         ↓
 Wait for server to be ready (poll /api endpoint)
         ↓
-Create DeepnoteKernelConnectionMetadata
+Register server with DeepnoteServerProvider
+        ↓
+Query server for available kernel specs
+        ↓
+Select existing kernel spec (e.g., python3-venv)
+        ↓
+Create DeepnoteKernelConnectionMetadata with server kernel spec
         ↓
 Register controller with IControllerRegistration
         ↓
@@ -112,10 +135,12 @@ User runs cell → Executes on Deepnote kernel
 ## Configuration
 
 ### Hardcoded Values (as requested)
-- **Wheel URL**: `https://deepnote-staging-runtime-artifactory.s3.amazonaws.com/deepnote-toolkit-packages/0.2.30.post19/deepnote_toolkit-0.2.30.post19-py3-none-any.whl`
+- **Wheel URL**: `https://deepnote-staging-runtime-artifactory.s3.amazonaws.com/deepnote-toolkit-packages/0.2.30.post20/deepnote_toolkit-0.2.30.post20-py3-none-any.whl`
 - **Default Port**: `8888` (will find next available if occupied)
 - **Notebook Type**: `deepnote`
 - **Venv Location**: `~/.vscode/extensions/storage/deepnote-venvs/<file-path-hash>/`
+- **Server Provider ID**: `deepnote-server`
+- **Default Kernel**: Uses server's default Python kernel (typically `python3-venv`)
 
 ## Usage
 
@@ -135,6 +160,8 @@ User runs cell → Executes on Deepnote kernel
 - **Multi-file support**: Can run multiple `.deepnote` files with separate servers
 - **Resource efficiency**: Reuses venv and server for notebooks within the same `.deepnote` file
 - **Clean integration**: Uses existing VSCode notebook controller infrastructure
+- **Proper server resolution**: Implements Jupyter server provider for proper kernel connection handling
+- **Compatible kernel specs**: Uses kernel specs that exist on the Deepnote server
 
 ## Future Enhancements
 
@@ -161,19 +188,64 @@ To test the implementation:
 ## Files Modified/Created
 
 ### Created:
-- `src/kernels/deepnote/types.ts`
-- `src/kernels/deepnote/deepnoteToolkitInstaller.node.ts`
-- `src/kernels/deepnote/deepnoteServerStarter.node.ts`
-- `src/notebooks/deepnote/deepnoteKernelAutoSelector.node.ts`
+- `src/kernels/deepnote/types.ts` - Type definitions and interfaces
+- `src/kernels/deepnote/deepnoteToolkitInstaller.node.ts` - Toolkit installation service
+- `src/kernels/deepnote/deepnoteServerStarter.node.ts` - Server lifecycle management
+- `src/kernels/deepnote/deepnoteServerProvider.node.ts` - Jupyter server provider implementation
+- `src/notebooks/deepnote/deepnoteKernelAutoSelector.node.ts` - Automatic kernel selection
 
 ### Modified:
-- `src/kernels/types.ts`
-- `src/notebooks/serviceRegistry.node.ts`
+- `src/kernels/types.ts` - Added DeepnoteKernelConnectionMetadata to union types
+- `src/notebooks/serviceRegistry.node.ts` - Registered new services
 
 ## Dependencies
 
 - `get-port`: For finding available ports
 - Existing VSCode notebook infrastructure
 - Existing kernel controller system
 - Python interpreter service
+- Jupyter server provider registry
+- JupyterLab session management
+
+## Technical Details
+
+### Server Provider Architecture
+
+The implementation uses VSCode's Jupyter server provider API to properly integrate Deepnote servers:
+
+1. **DeepnoteServerProvider** implements the `JupyterServerProvider` interface
+2. It registers with the `IJupyterServerProviderRegistry` during activation
+3. When a Deepnote server is started, it's registered with the provider using a unique handle
+4. The kernel infrastructure can then resolve the server connection through this provider
+5. This allows the kernel session factory to properly connect to the Deepnote server
+
+### Kernel Spec Resolution
+
+Instead of creating custom kernel specs, the implementation:
+
+1. Connects to the running Deepnote server using `JupyterLabHelper`
+2. Queries the server for available kernel specs via `getKernelSpecs()`
+3. Selects the first Python kernel spec (or falls back to `python3-venv`)
+4. Uses this existing spec when creating the kernel connection metadata
+5. This ensures compatibility with the Deepnote server's kernel configuration
+
+## Troubleshooting & Key Fixes
+
+### Issue 1: "Unable to get resolved server information"
+
+**Problem**: The kernel infrastructure couldn't resolve the server connection because the `serverProviderHandle` pointed to a non-existent server provider.
+
+**Solution**: Created `DeepnoteServerProvider` that implements the `JupyterServerProvider` interface and registered it with the `IJupyterServerProviderRegistry`. This allows the kernel session factory to properly resolve server connections.
+
+### Issue 2: "No such kernel named python31211jvsc74a57bd0..."
+
+**Problem**: The extension was creating a custom kernel spec name based on the interpreter hash, but this kernel spec didn't exist on the Deepnote server.
+
+**Solution**: Instead of creating a custom kernel spec, the implementation now:
+- Queries the Deepnote server for available kernel specs
+- Selects an existing Python kernel (typically `python3-venv`)
+- Uses this server-native kernel spec for the connection
 
+These changes ensure that Deepnote notebooks can execute cells properly by:
+1. Providing a valid server provider that can be resolved
+2. Using kernel specs that actually exist on the Deepnote server

src/kernels/deepnote/deepnoteServerProvider.node.ts

@@ -0,0 +1,94 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { inject, injectable } from 'inversify';
+import { CancellationToken, Uri, Event, EventEmitter } from 'vscode';
+import { JupyterServer, JupyterServerProvider } from '../../api';
+import { IExtensionSyncActivationService } from '../../platform/activation/types';
+import { IDisposableRegistry } from '../../platform/common/types';
+import { IJupyterServerProviderRegistry } from '../jupyter/types';
+import { JVSC_EXTENSION_ID } from '../../platform/common/constants';
+import { logger } from '../../platform/logging';
+import { DeepnoteServerInfo } from './types';
+
+/**
+ * Jupyter Server Provider for Deepnote kernels.
+ * This provider resolves server connections for Deepnote kernels.
+ */
+@injectable()
+export class DeepnoteServerProvider implements IExtensionSyncActivationService, JupyterServerProvider {
+    public readonly id = 'deepnote-server';
+    private readonly _onDidChangeServers = new EventEmitter<void>();
+    public readonly onDidChangeServers: Event<void> = this._onDidChangeServers.event;
+
+    // Map of server handles to server info
+    private servers = new Map<string, DeepnoteServerInfo>();
+
+    constructor(
+        @inject(IJupyterServerProviderRegistry)
+        private readonly jupyterServerProviderRegistry: IJupyterServerProviderRegistry,
+        @inject(IDisposableRegistry) private readonly disposables: IDisposableRegistry
+    ) {}
+
+    public activate() {
+        // Register this server provider
+        const collection = this.jupyterServerProviderRegistry.createJupyterServerCollection(
+            JVSC_EXTENSION_ID,
+            this.id,
+            'Deepnote Toolkit Server',
+            this
+        );
+        this.disposables.push(collection);
+        logger.info('Deepnote server provider registered');
+    }
+
+    /**
+     * Register a server for a specific handle.
+     * Called by DeepnoteKernelAutoSelector when a server is started.
+     */
+    public registerServer(handle: string, serverInfo: DeepnoteServerInfo): void {
+        logger.info(`Registering Deepnote server: ${handle} -> ${serverInfo.url}`);
+        this.servers.set(handle, serverInfo);
+        this._onDidChangeServers.fire();
+    }
+
+    /**
+     * Provides the list of available Deepnote servers.
+     */
+    public async provideJupyterServers(_token: CancellationToken): Promise<JupyterServer[]> {
+        const servers: JupyterServer[] = [];
+        for (const [handle, info] of this.servers.entries()) {
+            servers.push({
+                id: handle,
+                label: `Deepnote Toolkit (${info.port})`,
+                connectionInformation: {
+                    baseUrl: Uri.parse(info.url),
+                    token: info.token || ''
+                }
+            });
+        }
+        return servers;
+    }
+
+    /**
+     * Resolves a Jupyter server by its handle.
+     * This is called by the kernel infrastructure when starting a kernel.
+     */
+    public async resolveJupyterServer(server: JupyterServer, _token: CancellationToken): Promise<JupyterServer> {
+        logger.info(`Resolving Deepnote server: ${server.id}`);
+        const serverInfo = this.servers.get(server.id);
+
+        if (!serverInfo) {
+            throw new Error(`Deepnote server not found: ${server.id}`);
+        }
+
+        return {
+            id: server.id,
+            label: server.label,
+            connectionInformation: {
+                baseUrl: Uri.parse(serverInfo.url),
+                token: serverInfo.token || ''
+            }
+        };
+    }
+}

src/kernels/deepnote/types.ts

@@ -20,19 +20,22 @@ export class DeepnoteKernelConnectionMetadata {
     public readonly baseUrl: string;
     public readonly interpreter?: PythonEnvironment;
     public readonly serverProviderHandle: JupyterServerProviderHandle;
+    public readonly serverInfo?: DeepnoteServerInfo; // Store server info for connection
 
     private constructor(options: {
         interpreter?: PythonEnvironment;
         kernelSpec: IJupyterKernelSpec;
         baseUrl: string;
         id: string;
         serverProviderHandle: JupyterServerProviderHandle;
+        serverInfo?: DeepnoteServerInfo;
     }) {
         this.interpreter = options.interpreter;
         this.kernelSpec = options.kernelSpec;
         this.baseUrl = options.baseUrl;
         this.id = options.id;
         this.serverProviderHandle = options.serverProviderHandle;
+        this.serverInfo = options.serverInfo;
     }
 
     public static create(options: {
@@ -41,6 +44,7 @@ export class DeepnoteKernelConnectionMetadata {
         baseUrl: string;
         id: string;
         serverProviderHandle: JupyterServerProviderHandle;
+        serverInfo?: DeepnoteServerInfo;
     }) {
         return new DeepnoteKernelConnectionMetadata(options);
     }

src/notebooks/deepnote/deepnoteKernelAutoSelector.node.ts

@@ -1,7 +1,7 @@
 // Copyright (c) Microsoft Corporation.
 // Licensed under the MIT License.
 
-import { inject, injectable } from 'inversify';
+import { inject, injectable, optional } from 'inversify';
 import { NotebookDocument, workspace, NotebookControllerAffinity } from 'vscode';
 import { IExtensionSyncActivationService } from '../../platform/activation/types';
 import { IDisposableRegistry } from '../../platform/common/types';
@@ -11,15 +11,20 @@ import {
     IDeepnoteKernelAutoSelector,
     IDeepnoteServerStarter,
     IDeepnoteToolkitInstaller,
-    DEEPNOTE_NOTEBOOK_TYPE
+    DEEPNOTE_NOTEBOOK_TYPE,
+    DeepnoteKernelConnectionMetadata
 } from '../../kernels/deepnote/types';
-import { DeepnoteKernelConnectionMetadata } from '../../kernels/deepnote/types';
 import { IControllerRegistration } from '../controllers/types';
 import { JVSC_EXTENSION_ID } from '../../platform/common/constants';
 import { getDisplayPath } from '../../platform/common/platform/fs-paths';
-import { createInterpreterKernelSpec } from '../../kernels/helpers';
 import { JupyterServerProviderHandle } from '../../kernels/jupyter/types';
 import { IPythonExtensionChecker } from '../../platform/api/types';
+import { DeepnoteServerProvider } from '../../kernels/deepnote/deepnoteServerProvider.node';
+import { JupyterLabHelper } from '../../kernels/jupyter/session/jupyterLabHelper';
+import { createJupyterConnectionInfo } from '../../kernels/jupyter/jupyterUtils';
+import { IJupyterRequestCreator, IJupyterRequestAgentCreator } from '../../kernels/jupyter/types';
+import { IConfigurationService } from '../../platform/common/types';
+import { disposeAsync } from '../../platform/common/utils';
 
 /**
  * Automatically selects and starts Deepnote kernel for .deepnote notebooks
@@ -32,7 +37,13 @@ export class DeepnoteKernelAutoSelector implements IDeepnoteKernelAutoSelector,
         @inject(IInterpreterService) private readonly interpreterService: IInterpreterService,
         @inject(IDeepnoteToolkitInstaller) private readonly toolkitInstaller: IDeepnoteToolkitInstaller,
         @inject(IDeepnoteServerStarter) private readonly serverStarter: IDeepnoteServerStarter,
-        @inject(IPythonExtensionChecker) private readonly pythonExtensionChecker: IPythonExtensionChecker
+        @inject(IPythonExtensionChecker) private readonly pythonExtensionChecker: IPythonExtensionChecker,
+        @inject(DeepnoteServerProvider) private readonly serverProvider: DeepnoteServerProvider,
+        @inject(IJupyterRequestCreator) private readonly requestCreator: IJupyterRequestCreator,
+        @inject(IJupyterRequestAgentCreator)
+        @optional()
+        private readonly requestAgentCreator: IJupyterRequestAgentCreator | undefined,
+        @inject(IConfigurationService) private readonly configService: IConfigurationService
     ) {}
 
     public activate() {
@@ -100,20 +111,59 @@ export class DeepnoteKernelAutoSelector implements IDeepnoteKernelAutoSelector,
             const serverInfo = await this.serverStarter.getOrStartServer(venvInterpreter, baseFileUri);
             logger.info(`Deepnote server running at ${serverInfo.url}`);
 
-            // Create kernel connection metadata
-            const kernelSpec = await createInterpreterKernelSpec(interpreter);
+            // Create server provider handle
             const serverProviderHandle: JupyterServerProviderHandle = {
                 extensionId: JVSC_EXTENSION_ID,
                 id: 'deepnote-server',
-                handle: 'deepnote-toolkit-server'
+                handle: `deepnote-toolkit-server-${baseFileUri.fsPath}`
             };
 
+            // Register the server with the provider so it can be resolved
+            this.serverProvider.registerServer(serverProviderHandle.handle, serverInfo);
+
+            // Connect to the server and get available kernel specs
+            const connectionInfo = createJupyterConnectionInfo(
+                serverProviderHandle,
+                {
+                    baseUrl: serverInfo.url,
+                    token: serverInfo.token || '',
+                    displayName: 'Deepnote Server',
+                    authorizationHeader: {}
+                },
+                this.requestCreator,
+                this.requestAgentCreator,
+                this.configService,
+                baseFileUri
+            );
+
+            const sessionManager = JupyterLabHelper.create(connectionInfo.settings);
+            let kernelSpec;
+            try {
+                const kernelSpecs = await sessionManager.getKernelSpecs();
+                logger.info(`Available kernel specs on Deepnote server: ${kernelSpecs.map((s) => s.name).join(', ')}`);
+
+                // Use the first available Python kernel spec, or fall back to 'python3-venv'
+                kernelSpec =
+                    kernelSpecs.find((s) => s.language === 'python') ||
+                    kernelSpecs.find((s) => s.name === 'python3-venv') ||
+                    kernelSpecs[0];
+
+                if (!kernelSpec) {
+                    throw new Error('No kernel specs available on Deepnote server');
+                }
+
+                logger.info(`Using kernel spec: ${kernelSpec.name} (${kernelSpec.display_name})`);
+            } finally {
+                await disposeAsync(sessionManager);
+            }
+
             const connectionMetadata = DeepnoteKernelConnectionMetadata.create({
                 interpreter,
                 kernelSpec,
                 baseUrl: serverInfo.url,
                 id: `deepnote-kernel-${interpreter.id}`,
-                serverProviderHandle
+                serverProviderHandle,
+                serverInfo // Pass the server info so we can use it later
             });
 
             // Register controller for deepnote notebook type