Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Main candidate test #74

Merged
merged 1,438 commits into from
Jun 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1438 commits
Select commit Hold shift + click to select a range
92b4c38
execution is no longer halted until first fetch, error triggers earli…
Tishj Jun 7, 2024
fb68c2f
Fix for error message
Mytherin Jun 7, 2024
adce365
Merge pull request #12423 from Mytherin/valgrindfix
Mytherin Jun 7, 2024
5244bc2
get rid of calling ByteBuffer::available so much
lnkuiper Jun 7, 2024
ce2e23a
Internal #2186: Swift Nanosecond Test
hawkfish Jun 7, 2024
fcab0a5
Internal #2186: Swift Nanosecond Test
hawkfish Jun 7, 2024
59f3e94
Merge pull request #12420 from lnkuiper/jemalloc_gnu
Mytherin Jun 7, 2024
c2fe86d
tidy fix
Tishj Jun 7, 2024
5349e62
Merge remote-tracking branch 'upstream/feature' into python_enable_re…
Tishj Jun 7, 2024
bdb2047
add back test
Tishj Jun 7, 2024
25c93ae
same as before, error is encountered earlier now
Tishj Jun 7, 2024
2137521
Internal #2186: Swift Nanosecond Test
hawkfish Jun 8, 2024
5304647
Merge pull request #12433 from Mytherin/limitparallel
Mytherin Jun 8, 2024
e6c05de
Merge pull request #12437 from Mytherin/prefetchmetadata
Mytherin Jun 8, 2024
7256088
set max_width to 10k on Jupyter
Tishj Jun 8, 2024
4aa09e4
Merge pull request #12443 from Tishj/jupyter_remove_width_limit
Mytherin Jun 8, 2024
c9f979f
Merge branch 'main' into feature
Mytherin Jun 9, 2024
63167d4
Merge branch 'feature' into rangeinout
Mytherin Jun 9, 2024
7ecd1c4
Fix test
Mytherin Jun 9, 2024
cad3172
Actually fix error message
Mytherin Jun 9, 2024
a087c5c
Merge branch 'rangeinout' of github.com:Mytherin/duckdb into rangeinout
Mytherin Jun 9, 2024
8a26349
Merge branch 'feature' into readonlytransactions
Mytherin Jun 9, 2024
c709bbc
Merge pull request #12202 from Mytherin/readonlytransactions
Mytherin Jun 9, 2024
be8b9d4
Merge pull request #12431 from Mytherin/rangeinout
Mytherin Jun 9, 2024
3284f9d
some std::bitset optimizations
lnkuiper Jun 10, 2024
c014af2
just use an else if
lnkuiper Jun 10, 2024
189f46b
more snappy progress
lnkuiper Jun 10, 2024
6e59955
Merge branch 'feature' into art-not-null
taniabogatsch Jun 10, 2024
34d4af6
Merge branch 'feature' into bs-nightly
taniabogatsch Jun 10, 2024
58f878e
Merge branch 'bs-nightly' into block-size-2.5
taniabogatsch Jun 10, 2024
dd1e442
remove wrapper and improve use of bitset further
lnkuiper Jun 10, 2024
a561a83
more test index of indexing bitset
lnkuiper Jun 10, 2024
a5aaff0
fix locking issue, min_batch should always be accessed behind a lock
Tishj Jun 10, 2024
e3830ad
Merge remote-tracking branch 'upstream/feature' into parallel_streami…
Tishj Jun 10, 2024
4922c6b
minimum_batch_index should only be checked with the BatchMemoryManage…
Tishj Jun 10, 2024
f681792
add original snappy checks
lnkuiper Jun 10, 2024
f1e3143
Merge branch 'feature' into snappy_stuff
lnkuiper Jun 10, 2024
5fb13e5
improve dict read performance
lnkuiper Jun 10, 2024
1852ca1
Merge pull request #12427 from taniabogatsch/bs-nightly
Mytherin Jun 10, 2024
c8a230b
Merge pull request #12428 from taniabogatsch/art-not-null
Mytherin Jun 10, 2024
889dd13
Merge pull request #12434 from Tishj/python_clean_up_execute_code
Mytherin Jun 10, 2024
5c4c3e9
Move collations into a pluggable CollationBinding class
Mytherin Jun 10, 2024
41e024a
Add timetz collation comparison
Mytherin Jun 10, 2024
76d7b82
Unify timetz comparison tests
Mytherin Jun 10, 2024
e4d671f
make test more robust
taniabogatsch Jun 10, 2024
509a919
Fix #11837 - use internal physical type for FIRST/LAST/ANY_VALUE inst…
Mytherin Jun 10, 2024
51ddfdf
Add benchmark to list
Mytherin Jun 10, 2024
4de386d
remove unused variable
Tishj Jun 10, 2024
811f8b9
another test has to be more robust
taniabogatsch Jun 10, 2024
1619a76
20 iterations
taniabogatsch Jun 10, 2024
5be0196
Rework list_value function to no longer use Value for primitive types
Mytherin Jun 10, 2024
6bc1503
Format fix
Mytherin Jun 10, 2024
eb69084
actually enable flags and regrettably revert to older version of snap…
lnkuiper Jun 10, 2024
9ab8df0
Merge branch 'feature' into ns-display
hawkfish Jun 10, 2024
8bedb9e
Merge branch 'feature' into summarize-temporal
Jun 10, 2024
6bd431c
Merge branch 'feature' into decimal-rounding
Jun 10, 2024
fa5777e
Merge pull request #12462 from Mytherin/firstanyperf
Mytherin Jun 10, 2024
741ec55
Internal #2186: Nanosecond Feedback
hawkfish Jun 10, 2024
b00e72c
Issue #12204: Summarize Temporal Quantiles
hawkfish Jun 10, 2024
cded639
Merge pull request #12468 from Mytherin/listvalueperf
Mytherin Jun 11, 2024
5061abb
Merge pull request #12459 from Tishj/remove_dead_code_from_physical_b…
Mytherin Jun 11, 2024
98d3d38
polishing up in preparation of PR
lnkuiper Jun 11, 2024
67ad1c0
Merge branch 'main' into feature
Mytherin Jun 11, 2024
1c11269
small test fixes
taniabogatsch Jun 11, 2024
5862e0f
revert to default for most tests
taniabogatsch Jun 11, 2024
01e8f95
Merge branch 'feature' into block-size-2.5
taniabogatsch Jun 11, 2024
2c5896c
add a condition_variable to notify the main thread that task(s) are r…
Tishj Jun 11, 2024
4908311
attach options
taniabogatsch Jun 11, 2024
86ffa1a
Merge pull request #12482 from taniabogatsch/feature-comp-fix
Mytherin Jun 11, 2024
df95056
Merge branch 'feature' into block-size-2.5
taniabogatsch Jun 11, 2024
e8ccbd7
some low-hanging fruit in the parquet writer
lnkuiper Jun 11, 2024
99f28e1
implement PR feedback
lnkuiper Jun 11, 2024
35a791f
Merge branch 'feature' into snappy_stuff
lnkuiper Jun 11, 2024
c70ad71
revert test
taniabogatsch Jun 11, 2024
bb18a38
fix some bugs
lnkuiper Jun 11, 2024
0f1057a
remove some more block alloc size
taniabogatsch Jun 11, 2024
9c09f4e
remove the 'task_reschedule_lock'
Tishj Jun 11, 2024
c3a30db
Merge pull request #12440 from hawkfish/ns-display
Mytherin Jun 11, 2024
a569707
throw binder error for comment on sys catalog
samansmink Jun 11, 2024
2a66e06
Merge pull request #12425 from Tishj/python_enable_replacements_setting
Mytherin Jun 11, 2024
36962f7
Test fix
Mytherin Jun 11, 2024
d40a6ae
add the number of filtered files to explain
samansmink Jun 11, 2024
25c9459
two issues surfaced: if the streaming query result collector is block…
Tishj Jun 11, 2024
5db917f
format
samansmink Jun 11, 2024
5eefc3f
add more operators the optimizer can skip
Tmonster Jun 11, 2024
e10dd64
only WaitForTask if we have no task or the task we worked on is blocked
Tishj Jun 11, 2024
9b6e7d2
add a wake up after 20 miliseconds for progressbar updating purposes
Tishj Jun 11, 2024
6f0e22d
Merge pull request #12486 from samansmink/throw-binder-error-comment-on
Mytherin Jun 11, 2024
c6c0291
Merge pull request #12478 from lnkuiper/snappy_stuff
Mytherin Jun 11, 2024
2ee44eb
Skip this test in substrait
Mytherin Jun 11, 2024
43332f9
Internal #2186: Nanosecond StrTimeFormat
hawkfish Jun 11, 2024
5fa4e11
Merge pull request #12489 from Tmonster/operators_optimizer_can_skip
Mytherin Jun 11, 2024
a292369
Internal #2186: Nanosecond strftime
hawkfish Jun 11, 2024
4ba5c89
Internal #2186: Nanosecond strptime
hawkfish Jun 11, 2024
39bb21f
Internal #2186: Nanosecond strptime
hawkfish Jun 11, 2024
887da95
format again
samansmink Jun 12, 2024
4647a55
Add DateLookupCache and use that in YEAR() function to speed up perfo…
Mytherin Jun 12, 2024
70bb85c
Format
Mytherin Jun 12, 2024
d02b521
Also use date lookup cache for Month and Day functions
Mytherin Jun 12, 2024
878c1b7
Merge pull request #12481 from taniabogatsch/block-size-2.5
Mytherin Jun 12, 2024
5ee16f4
Add missing includes/compilation fixes
Mytherin Jun 12, 2024
7602320
Generalize the date part cache, which also makes it smaller
Mytherin Jun 12, 2024
8c7a618
Slight code cleanup
Mytherin Jun 12, 2024
68d9042
Reduce allocations
Mytherin Jun 12, 2024
5362a2d
Merge branch 'feature' into block-size-2.6
taniabogatsch Jun 12, 2024
d808a3a
don't fail 'make generate-files' if the python requirements are missi…
Tishj Jun 12, 2024
9ca7007
missed an import
Tishj Jun 12, 2024
fd53120
this is *much* less invasive
Tishj Jun 12, 2024
410fa91
also mention it inside the 'get_cpp_methods.py' script
Tishj Jun 12, 2024
8e2fef6
Use predication in ColumnSegment::FilterSelection
Mytherin Jun 12, 2024
c871899
Merge pull request #12488 from samansmink/file-filter-count
Mytherin Jun 12, 2024
10682a3
skip some tests for now
taniabogatsch Jun 12, 2024
f182b63
Merge branch 'feature' into block-size-2.6
taniabogatsch Jun 12, 2024
899cf22
revert block size
taniabogatsch Jun 12, 2024
98ecbed
Merge pull request #11285 from ZiyaZa/json_map
Mytherin Jun 12, 2024
b7f1dfd
Merge pull request #12483 from Tishj/pending_execute_busy_spin
Mytherin Jun 12, 2024
15ced89
Merge pull request #12492 from Mytherin/pluggablecomparison
Mytherin Jun 12, 2024
34314ca
Intermediate commit - replace minmax implementation for nested types …
Mytherin Jun 12, 2024
1e2da97
WIP sort key decode
Mytherin Jun 12, 2024
cde939d
casting support yes/no strings as boolean values
chrisiou Jun 12, 2024
65824fd
Merge branch 'feature' into summarize-temporal
Jun 12, 2024
26da79b
Merge branch 'summarize-temporal' of github.com:hawkfish/duckdb into …
Jun 12, 2024
1b01a1a
Merge branch 'feature' into decimal-rounding
Jun 12, 2024
c18d87c
Merge pull request #12500 from Tishj/optional_python_code_generation
Mytherin Jun 13, 2024
7d742ed
Merge pull request #12499 from Mytherin/dateopsperf
Mytherin Jun 13, 2024
52c2dc4
WIP varchars and int64 working
Mytherin Jun 13, 2024
f09d89f
Blob support
Mytherin Jun 13, 2024
4390b76
if the dtype is 'string' (not a Numpy type), skip the object analyzer
Tishj Jun 13, 2024
7b7de7b
forgot to remove 'string' here
Tishj Jun 13, 2024
42cb278
List support and more tests
Mytherin Jun 13, 2024
47ecde3
Merge pull request #12502 from taniabogatsch/block-size-2.6
Mytherin Jun 13, 2024
c6ca930
Array support
Mytherin Jun 13, 2024
2ca1a5d
Clean-up byte-swap code, and add support for all signed integers
Mytherin Jun 13, 2024
b2be713
add WaitForTask to the python execution loop
Tishj Jun 13, 2024
c5e3bc8
more block size constant removal
taniabogatsch Jun 13, 2024
6f78a2f
Correctly increment list keys + add test_all_types() test
Mytherin Jun 13, 2024
5d97989
Format fix
Mytherin Jun 13, 2024
98dfd6a
create_sort_key
Mytherin Jun 13, 2024
de66172
fix numpy issues with the 'string' dtype changes
Tishj Jun 13, 2024
9de1463
Use numeric_limits
Mytherin Jun 13, 2024
e07eb36
Format fix
Mytherin Jun 13, 2024
4f4fbc2
Format fix
Mytherin Jun 13, 2024
acae49b
Merge branch 'feature' into sortkeyperf
Mytherin Jun 13, 2024
f0ce22a
Update min/max to cache allocations and prevent unnecessary re-alloca…
Mytherin Jun 13, 2024
e32cd55
missed some constants in FixedSizeBuffer
taniabogatsch Jun 13, 2024
8af8621
first step ci run for android
Jun 13, 2024
2d3307d
baby steps
Jun 13, 2024
272aecb
typo
Jun 13, 2024
f0b2707
explicit platform
Jun 13, 2024
27b7070
extension static build
Jun 13, 2024
29b3a83
more env and ninja
Jun 13, 2024
5572011
add arm64
Jun 13, 2024
4ae6bb4
wrong flag
Jun 13, 2024
b391b3a
extensions, fingers crossed
Jun 13, 2024
c120417
container
Jun 13, 2024
4e31344
using default containers
Jun 13, 2024
6277254
removing it in more places
taniabogatsch Jun 13, 2024
f330776
patch vss
taniabogatsch Jun 13, 2024
7ddee1d
Merge branch 'block-size-1' into block-size-1.1
taniabogatsch Jun 13, 2024
f384fb4
Switch arg_min/arg_max to use sort key instead of vectors
Mytherin Jun 13, 2024
61b184f
Clean up unused functions
Mytherin Jun 13, 2024
8e5ce78
AddStringOrBlob
Mytherin Jun 13, 2024
a99587f
Merge branch 'main' into feature
Mytherin Jun 13, 2024
7db70ef
Skip only built-in optimizers
Mytherin Jun 13, 2024
183eec1
Add support for arg_min(ANY, ANY)
Mytherin Jun 13, 2024
568c184
revert extension patch with optional_idx
taniabogatsch Jun 13, 2024
6e50f03
Merge branch 'block-size-1' into block-size-1.1
taniabogatsch Jun 13, 2024
4a3137b
Merge branch 'feature' into block-size-1
taniabogatsch Jun 13, 2024
960d873
Merge branch 'block-size-1' into block-size-1.1
taniabogatsch Jun 13, 2024
61bc020
Correct count
Mytherin Jun 13, 2024
40e6ed7
Format fix
Mytherin Jun 13, 2024
f32cf6c
Format fix
Mytherin Jun 13, 2024
29a92f8
Switch fallback implementation of FIRST to use sort keys instead of v…
Mytherin Jun 13, 2024
3641078
Merge pull request #12520 from Mytherin/deserializesortkey
Mytherin Jun 13, 2024
1b4c128
Merge pull request #12521 from Mytherin/fewerallocations
Mytherin Jun 13, 2024
bed5d1a
Merge branch 'feature' into sortkeyperf
Mytherin Jun 13, 2024
0f55869
Merge pull request #12522 from Mytherin/dontskipoptimizerextensions
Mytherin Jun 13, 2024
ce23a0a
Merge pull request #12524 from Mytherin/sortkeyperf
Mytherin Jun 14, 2024
bd03af6
Merge pull request #12525 from Mytherin/argminmaxsortkey
Mytherin Jun 14, 2024
3608f3b
Merge pull request #12526 from Mytherin/firstsortkey
Mytherin Jun 14, 2024
b42ffd1
WIP - clean up histogram function, avoid using ListVector::Push
Mytherin Jun 14, 2024
dff4820
Add support for all types to the histogram function
Mytherin Jun 14, 2024
ca1015e
Merge pull request #12333 from carlopi/etag-if-none-match-for-extensi…
Mytherin Jun 14, 2024
5489780
dont WaitOnTask if there are no tasks available
Tishj Jun 14, 2024
b546082
Merge pull request #12514 from taniabogatsch/block-size-1
Mytherin Jun 14, 2024
273e22a
Merge branch 'feature' into block-size-1.1
taniabogatsch Jun 14, 2024
7351971
Rework list_distinct/list_unique to support arbitrary types and to no…
Mytherin Jun 14, 2024
25a5fa1
Format fix
Mytherin Jun 14, 2024
1782777
Merge pull request #12511 from Tishj/python_pandas_string_skip_analyzer
Mytherin Jun 14, 2024
1c47753
Merge pull request #11494 from Tishj/parallel_streaming_query_result_…
Mytherin Jun 14, 2024
bed9d94
Merge branch 'main' into feature
Mytherin Jun 14, 2024
2187970
Merge pull request #12532 from Mytherin/argminmaxany
Mytherin Jun 14, 2024
8cb0749
Use string_t and the arena allocator to allocate strings in the histo…
Mytherin Jun 14, 2024
3baed2d
this is soooo much better
Tishj Jun 14, 2024
50e7e2a
forgot to add new file
Tishj Jun 14, 2024
aed6d16
Merge remote-tracking branch 'upstream/feature' into unpacked_columns
Tishj Jun 14, 2024
78b250f
apply feedback
Tishj Jun 14, 2024
9ef765e
prevent the undefined behaviour of std::tolower() by casting the inpu…
chrisiou Jun 14, 2024
1d6aa06
Binned histograms WIP
Mytherin Jun 14, 2024
23511a1
Merge pull request #12538 from Mytherin/numerichistogram
Mytherin Jun 14, 2024
4c42292
format
Tishj Jun 14, 2024
2ba5cce
fix up test
Tishj Jun 14, 2024
9fbdcbe
More tests
Mytherin Jun 15, 2024
26f9300
Format fix + use string_map_t here
Mytherin Jun 15, 2024
07376ec
Detect duplicate bounds, sort bounds, and allow empty bounds
Mytherin Jun 15, 2024
6e5f7fa
Binned histograms working for all types
Mytherin Jun 15, 2024
2180d9e
Add binned histogram test to test all types
Mytherin Jun 15, 2024
6ae72a8
Merge pull request #12512 from Tishj/python_remove_execution_busy_spin
Mytherin Jun 15, 2024
5b72044
Merge pull request #12537 from taniabogatsch/block-size-1.1
Mytherin Jun 15, 2024
5669683
Merge branch 'histogramstringt' into binnedhistogram
Mytherin Jun 15, 2024
f3da2fa
Unify/clean up histogram and binned histogram
Mytherin Jun 15, 2024
8488363
RequiresExtract
Mytherin Jun 15, 2024
1d1ba39
Add TPC-H tests
Mytherin Jun 15, 2024
d87a1bf
Improve error message
Mytherin Jun 15, 2024
20e8091
Format
Mytherin Jun 15, 2024
8a28852
Add equi-width binning method for integers
Mytherin Jun 15, 2024
e413515
Merge pull request #12545 from Mytherin/histogramstringt
Mytherin Jun 15, 2024
f0439c7
Merge pull request #12548 from Mytherin/binnedhistogram
Mytherin Jun 15, 2024
46e6297
More clean-up and testing, add support for doubles
Mytherin Jun 15, 2024
b48adfc
lets start with this
Jun 15, 2024
997344a
Update .github/workflows/Android.yml
hannes Jun 15, 2024
5e34102
Merge branch 'feature' into summarize-temporal
hawkfish Jun 15, 2024
5510d1a
Merge branch 'feature' into ns-strptime
hawkfish Jun 15, 2024
c2b8894
add missing headers
Tishj Jun 16, 2024
f28d10b
Make equi-width-bins always return the input type
Mytherin Jun 16, 2024
9bbcb1b
format
chrisiou Jun 17, 2024
58f1815
Merge pull request #12550 from hannes/androidbinaries
Mytherin Jun 17, 2024
0bf7991
Add equi-width binning support for dates/timestamps
Mytherin Jun 17, 2024
a16614c
merge into feature
Jun 17, 2024
e4f424b
Merge branch 'parquetbrotli' of github.com:hannes/duckdb into parquet…
Jun 17, 2024
b79abf5
format
Jun 17, 2024
da0d826
Merge pull request #12036 from hawkfish/decimal-rounding
Mytherin Jun 17, 2024
8cba514
Merge pull request #12297 from hawkfish/summarize-temporal
Mytherin Jun 17, 2024
06ef941
Merge pull request #12551 from hawkfish/ns-strptime
Mytherin Jun 17, 2024
4455c46
Make typeof a constant function (removing its argument) so that the o…
Mytherin Jun 17, 2024
37af7b8
Format
Mytherin Jun 17, 2024
f8524e4
Binning test
Mytherin Jun 17, 2024
ab93432
Revert "Make typeof a constant function (removing its argument) so th…
Mytherin Jun 17, 2024
edb58a2
Fix test
Mytherin Jun 17, 2024
d211a7d
Remove duplicate test
Mytherin Jun 17, 2024
f2ed616
Re-generate functions
Mytherin Jun 17, 2024
9306e8f
Use /
Mytherin Jun 17, 2024
e3d1a8a
Merge pull request #12574 from Mytherin/histogrambins
Mytherin Jun 18, 2024
c9995e6
Merge pull request #12501 from chrisiou/cast-function
Mytherin Jun 18, 2024
f2d2167
Merge pull request #12346 from mariotaddeucci/pyspark-spark-session-r…
Mytherin Jun 18, 2024
2ec0a15
Merge pull request #12348 from mariotaddeucci/pyspark-dataframe-drop-…
Mytherin Jun 18, 2024
5560154
Merge pull request #12170 from tiagokepe/icu_noaccent_collation
Mytherin Jun 18, 2024
3085b0e
Merge pull request #12103 from hannes/parquetbrotli
Mytherin Jun 18, 2024
3890262
Merge pull request #11872 from Tishj/unpacked_columns
Mytherin Jun 18, 2024
f9238f4
Merge branch 'main' into feature
Mytherin Jun 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 5 additions & 0 deletions .github/actions/build_extensions/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@ inputs:
unittest_script:
description: 'Script/program to execute the unittests'
default: 'python3 scripts/run_tests_one_by_one.py ./build/release/test/unittest'
cmake_flags:
description: 'Flags to be passed to cmake'
default: ''

runs:
using: "composite"
Expand Down Expand Up @@ -144,6 +147,8 @@ runs:
GEN: ${{ inputs.ninja == 1 && 'ninja' || '' }}
USE_MERGED_VCPKG_MANIFEST: 1
DUCKDB_PLATFORM: ${{ inputs.duckdb_arch }}
CMAKE_VARS_BUILD: ${{ inputs.cmake_flags }}

run: |
ls
mkdir -p ~/.ssh
Expand Down
1 change: 0 additions & 1 deletion .github/config/bundled_extensions.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,4 @@ duckdb_extension_load(autocomplete)
#
## Extensions that are not linked, but we do want to test them as part of the release build
#
duckdb_extension_load(sqlsmith DONT_LINK)
duckdb_extension_load(tpcds DONT_LINK)
1 change: 0 additions & 1 deletion .github/config/in_tree_extensions.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,5 @@ duckdb_extension_load(inet)
duckdb_extension_load(icu)
duckdb_extension_load(json)
duckdb_extension_load(parquet)
duckdb_extension_load(sqlsmith)
duckdb_extension_load(tpcds)
duckdb_extension_load(tpch)
6 changes: 6 additions & 0 deletions .github/config/out_of_tree_extensions.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -100,12 +100,18 @@ duckdb_extension_load(sqlite_scanner
GIT_TAG 50b7870be099186f195bc72bac5e9e11247ee2f9
)

duckdb_extension_load(sqlsmith
GIT_URL https://github.com/duckdb/duckdb_sqlsmith
GIT_TAG 721460ff1f31ce1dc1e4a9c4a55c0faf0b466dcb
)

################# SUBSTRAIT
if (NOT WIN32)
duckdb_extension_load(substrait
LOAD_TESTS DONT_LINK
GIT_URL https://github.com/duckdb/substrait
GIT_TAG 237931391ebc7e6aee7aa81052fa1411f6c4128e
APPLY_PATCHES
)
endif()

Expand Down
40 changes: 40 additions & 0 deletions .github/patches/extensions/substrait/pushdown_semi_anti.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
diff --git a/src/to_substrait.cpp b/src/to_substrait.cpp
index de0ffb3..ece0f5d 100644
--- a/src/to_substrait.cpp
+++ b/src/to_substrait.cpp
@@ -864,7 +864,11 @@ substrait::Rel *DuckDBToSubstrait::TransformComparisonJoin(LogicalOperator &dop)
auto left_col_count = dop.children[0]->types.size();
if (dop.children[0]->type == LogicalOperatorType::LOGICAL_COMPARISON_JOIN) {
auto child_join = (LogicalComparisonJoin *)dop.children[0].get();
- left_col_count = child_join->left_projection_map.size() + child_join->right_projection_map.size();
+ if (child_join->join_type != JoinType::SEMI && child_join->join_type != JoinType::ANTI) {
+ left_col_count = child_join->left_projection_map.size() + child_join->right_projection_map.size();
+ } else {
+ left_col_count = child_join->left_projection_map.size();
+ }
}
sjoin->set_allocated_expression(
CreateConjunction(djoin.conditions, [&](JoinCondition &in) { return TransformJoinCond(in, left_col_count); }));
diff --git a/test/sql/test_substrait_types.test b/test/sql/test_substrait_types.test
index 2bc47da..f3d2c65 100644
--- a/test/sql/test_substrait_types.test
+++ b/test/sql/test_substrait_types.test
@@ -148,6 +148,9 @@ INSERT INTO timestz VALUES ('2021-11-26 10:15:13.123456+00');
statement ok
CALL get_substrait('SELECT s FROM timestz WHERE s > ''2001-11-26 05:02:23.123456+00'' ');

+# not supported because TIMETZ comparisons generate a UBIGINT
+mode skip
+
statement ok
CREATE table times_tz (s TIMETZ);

@@ -157,6 +160,8 @@ INSERT INTO times_tz VALUES ('05:40:00.000001');
statement ok
CALL get_substrait('SELECT s FROM times_tz WHERE s = ''05:40:00.000001'' ');

+mode unskip
+
statement ok
CREATE table times (s TIME);

9 changes: 9 additions & 0 deletions .github/regression/micro_extended.csv
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
benchmark/micro/aggregate/any_value_uuid.benchmark
benchmark/micro/aggregate/bitstring_aggregate.benchmark
benchmark/micro/aggregate/bitwise_aggregate.benchmark
benchmark/micro/aggregate/grouped_distinct.benchmark
Expand Down Expand Up @@ -98,6 +99,8 @@ benchmark/micro/date/extract_multiple_icu.benchmark
benchmark/micro/date/extract_struct.benchmark
benchmark/micro/date/extract_struct_icu.benchmark
benchmark/micro/date/extract_year.benchmark
benchmark/micro/distinct/distinct_on_pushdown.benchmark
benchmark/micro/filter/in_conversion.benchmark
benchmark/micro/filter/parallel_complex_filter.benchmark
benchmark/micro/functions/random.benchmark
benchmark/micro/functions/random_uuid.benchmark
Expand Down Expand Up @@ -127,6 +130,7 @@ benchmark/micro/join/asof_join.benchmark
benchmark/micro/join/asof_join_small_probe.benchmark
benchmark/micro/join/blockwise_nl_join.benchmark
benchmark/micro/join/delim_join_no_blowup.benchmark
benchmark/micro/join/hashjoin_dups_rhs.benchmark
benchmark/micro/join/hashjoin_highcardinality.benchmark
benchmark/micro/join/hashjoin_lhsarithmetic.benchmark
benchmark/micro/join/iejoin_employees.benchmark
Expand Down Expand Up @@ -160,7 +164,9 @@ benchmark/micro/list/column_data_collection_copy/small_lists.benchmark
benchmark/micro/list/column_data_collection_copy/unnest.benchmark
benchmark/micro/list/list_few_rows.benchmark
benchmark/micro/list/list_many_rows.benchmark
benchmark/micro/list/list_min_max.benchmark
benchmark/micro/list/list_order_by.benchmark
benchmark/micro/list/list_value.benchmark
benchmark/micro/list/string_split.benchmark
benchmark/micro/list/string_split_regexp.benchmark
benchmark/micro/list/string_split_unicode.benchmark
Expand All @@ -170,8 +176,10 @@ benchmark/micro/map/map_extract.benchmark
benchmark/micro/map/map_lookup.benchmark
benchmark/micro/nulls/no_nulls_addition.benchmark
benchmark/micro/nulls/null_addition.benchmark
benchmark/micro/optimizer/topn_optimization.benchmark
benchmark/micro/order/orderby.benchmark
benchmark/micro/pushdown/or_pushdown.benchmark
benchmark/micro/pushdown/window_partition_pushdown.benchmark
benchmark/micro/result_collection/batched_stream_query.benchmark
benchmark/micro/simd/auto-vectorization.benchmark
benchmark/micro/string/bitstring.benchmark
Expand All @@ -196,6 +204,7 @@ benchmark/micro/string/string_agg_order_by.benchmark
benchmark/micro/string/string_agg_short.benchmark
benchmark/micro/timestamp/extract_month.benchmark
benchmark/micro/timestamp/extract_year.benchmark
benchmark/micro/update/update_with_join.benchmark
benchmark/micro/view/many_values.benchmark
benchmark/micro/window/window_constant_aggregates.benchmark
benchmark/micro/window/window_constant_count.benchmark
Expand Down
111 changes: 111 additions & 0 deletions .github/workflows/Android.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
name: Android
on:
workflow_call:
inputs:
override_git_describe:
type: string
git_ref:
type: string
skip_tests:
type: string
workflow_dispatch:
inputs:
override_git_describe:
type: string
git_ref:
type: string
skip_tests:
type: string
repository_dispatch:
push:
branches:
- '**'
- '!main'
- '!feature'
paths-ignore:
- '**.md'
- 'tools/**'
- '!tools/odbc/**'
- '!tools/shell/**'
- '.github/patches/duckdb-wasm/**'
- '.github/workflows/**'
- '!.github/workflows/Android.yml'

pull_request:
types: [opened, reopened, ready_for_review]
paths-ignore:
- '**.md'
- 'tools/**'
- '!tools/odbc/**'
- '!tools/shell/**'
- '.github/patches/duckdb-wasm/**'
- '.github/workflows/**'
- '!.github/workflows/Android.yml'


concurrency:
group: android-${{ github.workflow }}-${{ github.ref }}-${{ github.head_ref || '' }}-${{ github.base_ref || '' }}-${{ github.ref != 'refs/heads/main' || github.sha }}-${{ inputs.override_git_describe }}
cancel-in-progress: true

env:
GH_TOKEN: ${{ secrets.GH_TOKEN }}
OVERRIDE_GIT_DESCRIBE: ${{ inputs.override_git_describe }}

jobs:
android:
name: Android
runs-on: ubuntu-latest
container: ubuntu:18.04

strategy:
matrix:
arch: [ armeabi-v7a, arm64-v8a ]

env:
ENABLE_EXTENSION_AUTOLOADING: 1
ENABLE_EXTENSION_AUTOINSTALL: 1
GEN: ninja
EXTENSION_STATIC_BUILD: 1
DUCKDB_PLATFORM: android_${{ matrix.arch}}
DUCKDB_CUSTOM_PLATFORM: android_${{ matrix.arch}}
CMAKE_VARS_BUILD: -DBUILD_UNITTESTS=0 -DBUILD_SHELL=0 -DANDROID_ABI=${{ matrix.arch}} -DCMAKE_TOOLCHAIN_FILE=./android-ndk/build/cmake/android.toolchain.cmake

steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
ref: ${{ inputs.git_ref }}

- uses: ./.github/actions/ubuntu_18_setup
with:
ccache: 1

- name: Checkout (again)
shell: bash
run: git checkout ${{ inputs.git_ref }}

- name: Install Android NDK
shell: bash
run: |
wget https://dl.google.com/android/repository/android-ndk-r26d-linux.zip
unzip android-ndk-r26d-linux.zip
mv android-ndk-r26d android-ndk

- name: Build
shell: bash
run: make

- name: Deploy
shell: bash
env:
AWS_ACCESS_KEY_ID: ${{ secrets.S3_DUCKDB_STAGING_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.S3_DUCKDB_STAGING_KEY }}
run: |
zip -j libduckdb-android_${{matrix.arch}}.zip build/release/src/libduckdb*.* src/include/duckdb.h
./scripts/upload-assets-to-staging.sh github_release libduckdb-android_${{matrix.arch}}.zip

- uses: actions/upload-artifact@v3
with:
name: duckdb-binaries-android-${{matrix.arch}}
path: |
libduckdb-android_${{matrix.arch}}.zip
2 changes: 1 addition & 1 deletion .github/workflows/CodeQuality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:

- name: Install
shell: bash
run: sudo apt-get update -y -qq && sudo apt-get install -y -qq ninja-build clang-format-11 && sudo pip3 install cmake-format black
run: sudo apt-get update -y -qq && sudo apt-get install -y -qq ninja-build clang-format-11 && sudo pip3 install cmake-format black libclang==16.0.6

- name: Format Check
shell: bash
Expand Down
57 changes: 0 additions & 57 deletions .github/workflows/LinuxRelease.yml
Original file line number Diff line number Diff line change
Expand Up @@ -384,63 +384,6 @@ jobs:
shell: bash
run: python3 scripts/exported_symbols_check.py build/release/src/libduckdb*.so

linux-httpfs:
name: Linux HTTPFS
if: ${{ inputs.skip_tests != 'true' }}
runs-on: ubuntu-20.04
needs: linux-release-64
env:
BUILD_HTTPFS: 1
BUILD_TPCH: 1
BUILD_TPCDS: 1
BUILD_PARQUET: 1
BUILD_JSON: 1
S3_TEST_SERVER_AVAILABLE: 1
AWS_DEFAULT_REGION: eu-west-1
AWS_ACCESS_KEY_ID: minio_duckdb_user
AWS_SECRET_ACCESS_KEY: minio_duckdb_user_password
DUCKDB_S3_ENDPOINT: duckdb-minio.com:9000
DUCKDB_S3_USE_SSL: false

GEN: ninja

steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
ref: ${{ inputs.git_ref }}

- uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install Ninja
shell: bash
run: sudo apt-get update -y -qq && sudo apt-get install -y -qq ninja-build

- name: Setup Ccache
uses: hendrikmuhs/ccache-action@main
with:
key: ${{ github.job }}
save: ${{ github.ref == 'refs/heads/main' || github.repository != 'duckdb/duckdb' }}

- name: Build
shell: bash
run: make

- name: Start test server & run tests
shell: bash
run: |
sudo ./scripts/install_s3_test_server.sh
./scripts/generate_presigned_url.sh
source ./scripts/run_s3_test_server.sh
source ./scripts/set_s3_test_server_variables.sh
sleep 60

python3 scripts/get_test_list.py --file-contains 'require httpfs' --list '"*"' > test.list
python3 scripts/run_tests_one_by_one.py ./build/release/test/unittest '-f test.list'
python3 scripts/run_tests_one_by_one.py ./build/release/test/unittest '[secret]'

amalgamation-tests:
name: Amalgamation Tests
if: ${{ inputs.skip_tests != 'true' }}
Expand Down
54 changes: 54 additions & 0 deletions .github/workflows/NightlyTests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -881,3 +881,57 @@ jobs:
pyodide:
uses: ./.github/workflows/Pyodide.yml
secrets: inherit

linux-httpfs:
name: Linux HTTPFS
runs-on: ubuntu-20.04
env:
BUILD_HTTPFS: 1
BUILD_TPCH: 1
BUILD_TPCDS: 1
BUILD_PARQUET: 1
BUILD_JSON: 1
S3_TEST_SERVER_AVAILABLE: 1
AWS_DEFAULT_REGION: eu-west-1
AWS_ACCESS_KEY_ID: minio_duckdb_user
AWS_SECRET_ACCESS_KEY: minio_duckdb_user_password
DUCKDB_S3_ENDPOINT: duckdb-minio.com:9000
DUCKDB_S3_USE_SSL: false

GEN: ninja

steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0

- uses: actions/setup-python@v5
with:
python-version: '3.12'

- name: Install Ninja
shell: bash
run: sudo apt-get update -y -qq && sudo apt-get install -y -qq ninja-build

- name: Setup Ccache
uses: hendrikmuhs/ccache-action@main
with:
key: ${{ github.job }}
save: ${{ github.ref == 'refs/heads/main' || github.repository != 'duckdb/duckdb' }}

- name: Build
shell: bash
run: make

- name: Start test server & run tests
shell: bash
run: |
sudo ./scripts/install_s3_test_server.sh
./scripts/generate_presigned_url.sh
source ./scripts/run_s3_test_server.sh
source ./scripts/set_s3_test_server_variables.sh
sleep 60

python3 scripts/get_test_list.py --file-contains 'require httpfs' --list '"*"' > test.list
python3 scripts/run_tests_one_by_one.py ./build/release/test/unittest '-f test.list'
python3 scripts/run_tests_one_by_one.py ./build/release/test/unittest '[secret]'
Loading
Loading