Skip to content

Conversation

@digithree
Copy link
Owner

Summary

Add a new export command to sync Pocket bookmarks to Karakeep via REST API. This provides a complete solution for exporting Pocket data to external bookmark management systems with robust error handling, retry logic, and user-friendly features.

New Export Command Features

🚀 Core Functionality

  • Full CLI Integration - New pocket-to-sqlite export command with comprehensive options
  • Karakeep API Client - Production-ready REST API client with Bearer token authentication
  • Data Mapping - Intelligent mapping from Pocket fields to Karakeep bookmark format
  • Progress Tracking - Real-time progress bars with accurate item counts
  • Resume Capability - Offset-based pagination for large exports and recovery

🛡️ Robust Error Handling

  • Progressive Retry Logic - Up to 5 retries with exponential backoff for network issues
  • Timeout Handling - 30-second request timeouts with retry on failure
  • Rate Limiting - Automatic handling of 429 rate limit responses
  • Client Error Handling - Proper parsing of 400-level errors without unnecessary retries
  • Server Error Recovery - Retry logic for 503/504 server errors

🎯 Advanced Filtering & Options

  • Status Filtering - Export by Pocket status (unread/archived/deleted)
  • Favorite Filtering - Export only favorited items
  • Batching Support - Limit and offset parameters for controlled exports
  • Dry-Run Mode - Preview functionality without making API calls
  • Debug Mode - Comprehensive logging for troubleshooting
  • Silent Mode - Suppress progress output for automated use

🔧 Authentication & Configuration

  • Extended auth.json - Adds karakeep_token and karakeep_base_url to existing auth pattern
  • Configurable Base URL - Support for custom Karakeep instances
  • Validation - Pre-flight checks for required credentials and database tables

Technical Implementation

KarakeepClient API Integration

# Handles actual Karakeep API response formats
{
  "id": "string",
  "createdAt": "string", 
  "title": "string",
  "content": {"type": "link", "url": "string"},
  // ... full Karakeep response structure
}

# Error responses with proper parsing
{
  "code": "VALIDATION_ERROR",
  "message": "Title is required and cannot be empty"
}

Field Mapping

  • resolved_title or given_titletitle
  • excerptsummary
  • resolved_url or given_urlurl
  • type"link" (constant)

Error Recovery Features

  • Network timeouts and connection errors with retry
  • Rate limiting (429) with progressive backoff
  • Server errors (503/504) with retry logic
  • Client errors (400-499) with immediate failure and clear messages
  • Graceful handling of items without URLs (skip with logging)

Usage Examples

Basic Export

# Export all bookmarks
pocket-to-sqlite export pocket.db

# Export with progress and error details
pocket-to-sqlite export pocket.db --debug

Advanced Filtering

# Export only unread items
pocket-to-sqlite export pocket.db --filter-status 0

# Export only favorites
pocket-to-sqlite export pocket.db --filter-favorite

# Combined filters
pocket-to-sqlite export pocket.db --filter-status 1 --filter-favorite

Batching & Resume

# Export first 100 items
pocket-to-sqlite export pocket.db --limit 100

# Resume from item 500  
pocket-to-sqlite export pocket.db --offset 500 --limit 100

Preview & Testing

# Dry-run to preview export
pocket-to-sqlite export pocket.db --dry-run --limit 10

# Silent mode for automation
pocket-to-sqlite export pocket.db --silent

Authentication Setup

Extend existing auth.json file:

{
  "pocket_consumer_key": "...",
  "pocket_access_token": "...",
  "karakeep_token": "your-karakeep-api-token",
  "karakeep_base_url": "https://your-karakeep-instance.com"
}

Testing & Quality

Comprehensive Test Coverage

  • 8 new test cases covering all functionality
  • KarakeepClient tests - API interaction, retry logic, error handling
  • Export function tests - Filtering, batching, error scenarios
  • CLI integration tests - Command validation, dry-run functionality
  • Mock-based testing - Isolated unit tests with proper API response simulation

Real-World Validation

  • Successfully tested against actual Karakeep API
  • Verified authentication and request formatting
  • Confirmed 201 responses with proper bookmark creation
  • Validated error handling with 400-level responses

Files Changed

  • pocket_to_sqlite/cli.py - New export command with full CLI interface
  • pocket_to_sqlite/utils.py - KarakeepClient and export functionality
  • tests/test_save_pocket.py - Comprehensive test coverage
  • README.md - Complete documentation with examples

Breaking Changes

None. This is a purely additive feature that extends existing functionality without modifying current behavior.

Performance & Reliability

  • Efficient API usage - 1-second delays between requests to respect rate limits
  • Memory efficient - Streaming export with generator-based processing
  • Network resilient - Comprehensive retry logic for production reliability
  • Progress tracking - Real-time feedback for long-running exports
  • Resume capability - Recoverable exports for large datasets

Ready for Production

This implementation provides enterprise-grade reliability with:

  • Comprehensive error handling and recovery
  • Production-tested retry logic
  • Clear logging and debugging capabilities
  • User-friendly progress tracking
  • Flexible filtering and batching options
  • Complete documentation and examples

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com

digithree and others added 3 commits May 24, 2025 23:16
Implement complete export system to sync Pocket items to Karakeep:

**New Export Command:**
- `pocket-to-sqlite export` with full CLI interface
- Supports filtering by status (unread/archived/deleted) and favorites
- Includes limit/offset for batching and resume capability
- Dry-run mode for preview without API calls
- Progress bar with accurate tracking
- Comprehensive error handling and logging

**KarakeepClient API Integration:**
- REST API client with Bearer token authentication
- Configurable base URL (defaults to localhost:3000)
- Progressive retry logic for timeouts, rate limits (429), and server errors
- 30-second request timeout to prevent hanging
- Automatic rate limiting with 1-second delays between requests

**Authentication Extension:**
- Extends existing auth.json to include karakeep_token and karakeep_base_url
- Maintains compatibility with existing Pocket authentication
- Validates required Karakeep credentials before export

**Data Mapping & Validation:**
- Maps Pocket fields to Karakeep bookmark format:
  - resolved_title/given_title → title
  - excerpt → summary
  - resolved_url/given_url → url
  - type → "link" (constant)
- Skips items without URLs with proper logging
- Handles missing/null fields gracefully

**Robust Error Handling:**
- Retry logic for network timeouts and server errors
- Graceful handling of malformed data
- Comprehensive error reporting with item-level details
- Resume capability via offset parameter

**Comprehensive Test Coverage:**
- 7 new test cases covering all functionality
- Tests for KarakeepClient retry logic and error handling
- Export function tests with filters and edge cases
- Mock-based testing for isolated unit tests
- Tests for dry-run preview functionality

**CLI Features:**
- `--filter-status [0|1|2]` - Filter by Pocket status
- `--filter-favorite` - Export only favorited items
- `--limit` / `--offset` - Batching and resume support
- `--dry-run` - Preview mode without API calls
- `--silent` - Suppress progress output
- `--debug` - Enable detailed logging
- `--auth FILE` - Custom auth file path

**Example Usage:**
```bash
# Export all items with progress bar
pocket-to-sqlite export pocket.db

# Export only favorites with custom auth
pocket-to-sqlite export pocket.db --filter-favorite --auth myauth.json

# Preview first 10 unread items
pocket-to-sqlite export pocket.db --filter-status 0 --limit 10 --dry-run

# Resume export from offset 1000
pocket-to-sqlite export pocket.db --offset 1000
```

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Improve KarakeepClient to properly handle real Karakeep API responses:

**API Response Handling:**
- Handle 201 success responses with full bookmark object structure
- Parse 400 error responses with {code, message} format
- Add proper error handling for client errors (400-level) without retry
- Log successful bookmark creation with bookmark ID

**Enhanced Error Handling:**
- Client errors (400-499) now show proper error codes and messages
- No retry attempts for validation errors and other client errors
- Maintain retry logic only for server errors and rate limits
- Improved error messages with Karakeep error codes

**Updated Test Coverage:**
- Tests now use actual Karakeep API response structure
- Added test for 400 error response format handling
- Mock responses match real API format with proper fields
- Verified both success and error path handling

**Verified Working Integration:**
- Successfully tested against real Karakeep API
- Confirmed 201 responses with bookmark ID generation
- Debug logging shows proper API interaction flow
- All authentication and request formatting working correctly

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Add detailed section covering Karakeep export functionality:

**New Documentation Sections:**
- Authentication setup for Karakeep integration
- Basic export command usage and examples
- Comprehensive filtering options (status, favorites)
- Batching and resuming capabilities for large exports
- Dry-run preview functionality
- Additional command options (auth file, silent mode, debug)
- Notes about built-in retry logic and error handling

**Clear Examples for:**
- Status filtering (unread/archived/deleted items)
- Favorite filtering
- Batched exports with limit/offset
- Preview mode with dry-run
- Custom authentication file usage
- Debug and silent operation modes

The documentation follows existing README style and provides users
with complete information for using the new export functionality.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@digithree digithree merged commit 6efc935 into main May 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants