Skip to content
This repository was archived by the owner on May 17, 2024. It is now read-only.

Commit d235360

Browse files
vvkhvvkh
committed
sqeleton vendoring PoC
1 parent bfef4d0 commit d235360

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

81 files changed

+9702
-1
lines changed

‎pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ dsnparse = "*"
2929
click = "^8.1"
3030
rich = "*"
3131
toml = "^0.10.2"
32-
sqeleton = "0.0.8"
32+
sqeleton = { path = "./vendor/sqeleton", develop = true }
3333
mysql-connector-python = {version="8.0.29", optional=true}
3434
psycopg2 = {version="*", optional=true}
3535
snowflake-connector-python = {version="^2.7.2", optional=true}

‎vendor/sqeleton/.gitignore

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
pip-wheel-metadata/
24+
share/python-wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
MANIFEST
29+
30+
# PyInstaller
31+
# Usually these files are written by a python script from a template
32+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
33+
*.manifest
34+
*.spec
35+
36+
# Installer logs
37+
pip-log.txt
38+
pip-delete-this-directory.txt
39+
40+
# Unit test / coverage reports
41+
htmlcov/
42+
.tox/
43+
.nox/
44+
.coverage
45+
.coverage.*
46+
.cache
47+
nosetests.xml
48+
coverage.xml
49+
*.cover
50+
*.py,cover
51+
.hypothesis/
52+
.pytest_cache/
53+
54+
# Translations
55+
*.mo
56+
*.pot
57+
58+
# Django stuff:
59+
*.log
60+
local_settings.py
61+
db.sqlite3
62+
db.sqlite3-journal
63+
64+
# Flask stuff:
65+
instance/
66+
.webassets-cache
67+
68+
# Scrapy stuff:
69+
.scrapy
70+
71+
# Sphinx documentation
72+
docs/_build/
73+
74+
# PyBuilder
75+
target/
76+
77+
# Jupyter Notebook
78+
.ipynb_checkpoints
79+
80+
# IPython
81+
profile_default/
82+
ipython_config.py
83+
84+
# pyenv
85+
.python-version
86+
87+
# pipenv
88+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
89+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
90+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
91+
# install all needed dependencies.
92+
#Pipfile.lock
93+
94+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
95+
__pypackages__/
96+
97+
# Celery stuff
98+
celerybeat-schedule
99+
celerybeat.pid
100+
101+
# SageMath parsed files
102+
*.sage.py
103+
104+
# Environments
105+
.env
106+
.venv
107+
env/
108+
venv/
109+
ENV/
110+
env.bak/
111+
venv.bak/
112+
113+
# Spyder project settings
114+
.spyderproject
115+
.spyproject
116+
117+
# Rope project settings
118+
.ropeproject
119+
120+
# mkdocs documentation
121+
/site
122+
123+
# mypy
124+
.mypy_cache/
125+
.dmypy.json
126+
dmypy.json
127+
128+
# Pyre type checker
129+
.pyre/
130+
131+
# Mac
132+
.DS_Store
133+
134+
# IntelliJ
135+
.idea
136+
137+
# VSCode
138+
.vscode

‎vendor/sqeleton/LICENSE

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2022 Datafold
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎vendor/sqeleton/README.md

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# Sqeleton
2+
3+
**Under construction!**
4+
5+
Sqeleton is a Python library for querying SQL databases.
6+
7+
It consists of -
8+
9+
- A fast and concise query builder, inspired by PyPika and SQLAlchemy
10+
11+
- A modular database interface, with drivers for a long list of SQL databases.
12+
13+
It is comparable to other libraries such as SQLAlchemy or PyPika, in terms of API and intended audience. However there are several notable ways in which it is different.
14+
15+
## Overview
16+
17+
### Built for performance
18+
19+
- Multi-threaded by default -
20+
The same connection object can be used from multiple threads without any additional setup.
21+
22+
- No ORM
23+
ORMs are easy and familiar, but they encourage bad and slow code. Sqeleton is designed to push the compute to SQL.
24+
25+
- Fast query-builder
26+
Sqeleton's query-builder runs about 4 times faster than SQLAlchemy's.
27+
28+
### Type-aware
29+
30+
Sqeleton has a built-in feature to query the schemas of the databases it supports.
31+
32+
This feature can be also used to inform the query-builder, either as an alternative to defining the tables yourself, or to validate that your definitions match the actual schema.
33+
34+
The schema is used for validation when building expressions, making sure the names are correct, and that the data-types align.
35+
36+
(Still WIP)
37+
38+
### Multi-database access
39+
40+
Sqeleton is designed to work with several databases at the same time. Its API abstracts away as many implementation details as possible.
41+
42+
Databases we fully support:
43+
44+
- PostgreSQL >=10
45+
- MySQL
46+
- Snowflake
47+
- BigQuery
48+
- Redshift
49+
- Oracle
50+
- Presto
51+
- Databricks
52+
- Trino
53+
- Clickhouse
54+
- Vertica
55+
- DuckDB >=0.6
56+
- SQLite (coming soon)
57+
58+
## Documentation
59+
60+
[Read the docs!](https://sqeleton.readthedocs.io)
61+
62+
Or jump straight to the [introduction](https://sqeleton.readthedocs.io/en/latest/intro.html).
63+
64+
### Install
65+
66+
Install using pip:
67+
68+
```bash
69+
pip install sqeleton
70+
```
71+
72+
It is recommended to install the driver dependencies using pip's `[]` syntax:
73+
74+
```bash
75+
pip install 'sqeleton[mysql, postgresql]'
76+
```
77+
78+
Read more in [install / getting started.](https://sqeleton.readthedocs.io/en/latest/install.html)
79+
80+
### Basic usage
81+
82+
```python
83+
from sqeleton import connect, table, this
84+
85+
# Create a new database connection
86+
ddb = connect("duckdb://:memory:")
87+
88+
# Define a table with one int column
89+
tbl = table('my_list', schema={'item': int})
90+
91+
# Make a bunch of queries
92+
queries = [
93+
# Create table 'my_list'
94+
tbl.create(),
95+
96+
# Insert 100 numbers
97+
tbl.insert_rows([x] for x in range(100)),
98+
99+
# Get the sum of the numbers
100+
tbl.select(this.item.sum())
101+
]
102+
# Query in order, and return the last result as an int
103+
result = ddb.query(queries, int)
104+
105+
# Prints: Total sum of 0..100 = 4950
106+
print(f"Total sum of 0..100 = {result}")
107+
```
108+
109+
110+
# TODO
111+
112+
- Transactions
113+
114+
- Indexes
115+
116+
- Date/time expressions
117+
118+
- Window functions
119+
120+
## Possible plans for the future (not determined yet)
121+
122+
- Cache the compilation of repetitive queries for even faster query-building
123+
124+
- Compile control flow, functions
125+
126+
- Define tables using type-annotated classes (SQLModel style)

‎vendor/sqeleton/dev/Dockerfile.prestosql.340

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
FROM openjdk:11-jdk-slim-buster
2+
3+
ENV PRESTO_VERSION=340
4+
ENV PRESTO_SERVER_URL=https://repo1.maven.org/maven2/io/prestosql/presto-server/${PRESTO_VERSION}/presto-server-${PRESTO_VERSION}.tar.gz
5+
ENV PRESTO_CLI_URL=https://repo1.maven.org/maven2/io/prestosql/presto-cli/${PRESTO_VERSION}/presto-cli-${PRESTO_VERSION}-executable.jar
6+
ENV PRESTO_HOME=/opt/presto
7+
ENV PATH=${PRESTO_HOME}/bin:${PATH}
8+
9+
WORKDIR $PRESTO_HOME
10+
11+
RUN set -xe \
12+
&& apt-get update \
13+
&& apt-get install -y curl less python \
14+
&& curl -sSL $PRESTO_SERVER_URL | tar xz --strip 1 \
15+
&& curl -sSL $PRESTO_CLI_URL > ./bin/presto \
16+
&& chmod +x ./bin/presto \
17+
&& apt-get remove -y curl \
18+
&& rm -rf /var/lib/apt/lists/*
19+
20+
VOLUME /data
21+
22+
EXPOSE 8080
23+
24+
ENTRYPOINT ["launcher"]
25+
CMD ["run"]

‎vendor/sqeleton/dev/dev.env

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
POSTGRES_USER=postgres
2+
POSTGRES_PASSWORD=Password1
3+
POSTGRES_DB=postgres
4+
5+
MYSQL_DATABASE=mysql
6+
MYSQL_USER=mysql
7+
MYSQL_PASSWORD=Password1
8+
MYSQL_ROOT_PASSWORD=RootPassword1
9+
10+
CLICKHOUSE_USER=clickhouse
11+
CLICKHOUSE_PASSWORD=Password1
12+
CLICKHOUSE_DB=clickhouse
13+
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1
14+
15+
# Vertica credentials
16+
APP_DB_USER=vertica
17+
APP_DB_PASSWORD=Password1
18+
VERTICA_DB_NAME=vertica
19+
20+
# To prevent generating sample demo VMart data (more about it here https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/GettingStartedGuide/IntroducingVMart/IntroducingVMart.htm),
21+
# leave VMART_DIR and VMART_ETL_SCRIPT empty.
22+
VMART_DIR=
23+
VMART_ETL_SCRIPT=

‎vendor/sqeleton/dev/presto-conf/standalone/catalog/jmx.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
connector.name=jmx

‎vendor/sqeleton/dev/presto-conf/standalone/catalog/memory.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
connector.name=memory

‎vendor/sqeleton/dev/presto-conf/standalone/catalog/postgresql.properties

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
connector.name=postgresql
2+
connection-url=jdbc:postgresql://postgres:5432/postgres
3+
connection-user=postgres
4+
connection-password=Password1
5+
allow-drop-table=true

‎vendor/sqeleton/dev/presto-conf/standalone/catalog/tpcds.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
connector.name=tpcds

‎vendor/sqeleton/dev/presto-conf/standalone/catalog/tpch.properties

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
connector.name=tpch

0 commit comments

Comments
 (0)