Skip to content

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Oct 16, 2025

TAP Server Prototype - Complete Implementation ✅

Successfully created a prototype TAP (Table Access Protocol) server following the IVOA TAP v1.1 specification.

📁 Components

  • tap_server.py - Flask-based TAP server using bin/adql_to_lsdb for query processing
  • example_client.py - Python client demonstrating server usage
  • README.md - Comprehensive documentation with examples
  • requirements.txt - Dependencies (Flask, queryparser-python3, pandas)
  • __init__.py - Package initialization

🎯 Features Implemented

  • ADQL to LSDB Integration - Uses bin/adql_to_lsdb.adql_to_lsdb() function to convert ADQL queries
  • Query Execution - Executes generated LSDB code and converts DataFrame results to VOTable
  • Robust Table Name Extraction - Uses regex with SQL keyword validation for accurate table name parsing
  • Advanced ADQL parsing via queryparser library:
    • SELECT with column list or *
    • FROM with table names (including schema.table)
    • WHERE clause with comparison operators
    • CONTAINS with POINT and CIRCLE for cone searches
    • TOP/LIMIT clauses
  • TAP /sync endpoint (synchronous queries)
  • VOTable XML response format with astronomical UCDs
  • /capabilities endpoint (IVOA VOSI format)
  • /tables endpoint (schema metadata)
  • Error handling with VOTable error responses
  • Sample data generation (for testing without actual LSDB catalogs)

🚀 Usage

Start the server:

cd tap_server
python tap_server.py

Query with curl:

curl -X POST http://localhost:5000/sync \
  -d "REQUEST=doQuery" \
  -d "LANG=ADQL" \
  -d "QUERY=SELECT TOP 10 ra, dec, mag FROM ztf_dr14 WHERE mag < 20"

Cone search query:

curl -X POST http://localhost:5000/sync \
  -d "REQUEST=doQuery" \
  -d "LANG=ADQL" \
  -d "QUERY=SELECT TOP 15 source_id, ra, dec FROM gaia_dr3.gaia WHERE 1 = CONTAINS(POINT('ICRS', 270.0, 23.0), CIRCLE('ICRS', 270.0, 23.0, 0.25))"

✅ Testing Results

All functionality tested and verified:

  • ✓ Integration with bin/adql_to_lsdb.adql_to_lsdb() function
  • ✓ ADQL queries converted to LSDB code
  • ✓ Generated code executed and DataFrame converted to VOTable
  • ✓ Robust table name extraction with regex and keyword validation
  • ✓ Server starts successfully on port 5000
  • ✓ Root endpoint returns HTML documentation
  • /sync endpoint accepts and processes ADQL queries
  • ✓ VOTable responses properly formatted with astronomical metadata
  • /capabilities endpoint returns VOSI format
  • /tables endpoint returns schema metadata
  • ✓ Error handling with invalid parameters
  • ✓ Cone search queries with CONTAINS/POINT/CIRCLE work correctly

📊 TAP Protocol Compliance

Implements IVOA TAP v1.1 specification:

  • ✅ Section 2.1: Basic TAP service structure
  • ✅ Section 2.2: Synchronous query execution via /sync
  • ✅ Section 2.5: ADQL query language support
  • ✅ Section 2.7: VOTable output format
  • ✅ Section 2.8: Error reporting via VOTable INFO elements

🔮 Future Work

  • Actual query execution against LSDB catalogs (currently returns sample data for testing)
  • Asynchronous query support (/async endpoint)
  • Additional output formats (CSV, JSON, FITS)
  • Authentication and authorization
  • Advanced ADQL features (JOINs, subqueries beyond current parser support)

The prototype successfully integrates with bin/adql_to_lsdb and converts ADQL queries to LSDB operations!

Original prompt

Using any simple Python web framework (including SimpleHTTPServer if it can be done), create a prototype TAP server following this spec: https://www.ivoa.net/documents/TAP/20181024/PR-TAP-1.1-20181024.html. The prototype doesn't have to be entirely complete, but it needs to be able to accept ADQL requests. Use the adql_to_lsdb.py script/module in this directory to turn the query into code which can then be executed. It's okay to return sample data instead of real data (the execution of the code can wait for a future task); the idea here is to be able to have a TAP server on which future work can be done.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@Copilot Copilot AI changed the title [WIP] Create prototype TAP server for ADQL requests Add TAP Server Prototype with ADQL Parser and LSDB Integration Oct 16, 2025
@Copilot Copilot AI requested a review from gitosaurus October 16, 2025 13:01
Copilot finished work on behalf of gitosaurus October 16, 2025 13:01
@gitosaurus gitosaurus force-pushed the copilot/create-tap-server-prototype branch from 9c8be76 to 2f15fff Compare October 16, 2025 14:51
@gitosaurus
Copy link
Contributor

@copilot Use the bin/adql_to_lsdb.adql_to_lsdb function to do the ADQL interpretation. That function returns Python code, and that code, when executed, will assign a DataFrame to a variable called result. Convert the contents of the DataFrame to the return payload.

Copy link
Contributor Author

Copilot AI commented Oct 16, 2025

@copilot Use the bin/adql_to_lsdb.adql_to_lsdb function to do the ADQL interpretation. That function returns Python code, and that code, when executed, will assign a DataFrame to a variable called `...

Integrated bin/adql_to_lsdb.adql_to_lsdb() function in commit 0f9a5e8. The TAP server now:

  1. Calls adql_to_lsdb(query) to convert ADQL to Python code
  2. Executes the generated code (currently with mock data for testing)
  3. Converts the resulting DataFrame from the result variable to VOTable format

The server successfully processes both simple queries and complex cone searches with CONTAINS/POINT/CIRCLE.

Copilot finished work on behalf of gitosaurus October 16, 2025 15:13
@gitosaurus gitosaurus force-pushed the copilot/create-tap-server-prototype branch from c62925b to 364dcfa Compare October 16, 2025 18:33
Comment on lines 330 to 335
if 'FROM' in query.upper():
try:
from_part = query.upper().split('FROM')[1].split('WHERE')[0].split('LIMIT')[0]
table_name = from_part.strip().split()[0]
except:
pass
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot This looks a little awkward. Refactor to use a regex to validate and extract all the parts of the string at once, and add comments showing example good and bad cases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored in commit 9b7f96d. The table name extraction now uses a regex pattern with SQL keyword validation. Added comments showing good examples ("FROM ztf_dr14", "FROM gaia_dr3.gaia") and bad examples ("FROMAGE", "FROM WHERE") that properly return 'results' as default.

Copilot AI and others added 2 commits October 16, 2025 15:33
Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com>
Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com>
@gitosaurus gitosaurus force-pushed the copilot/create-tap-server-prototype branch from 364dcfa to 79035f8 Compare October 16, 2025 19:33
Co-authored-by: gitosaurus <6794831+gitosaurus@users.noreply.github.com>
@Copilot Copilot AI requested a review from gitosaurus October 16, 2025 19:38
Copilot finished work on behalf of gitosaurus October 16, 2025 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants