Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] main from PeerDB-io:main #21

Merged
merged 41 commits into from
Mar 19, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
ff42813
applySchemaDeltas: add missing env wiring (#2676)
Amogh-Bharadwaj Mar 7, 2025
9965679
feat(classifier): mark ch auth failures as recoverable and push out c…
iamKunalGupta Mar 10, 2025
e09f63f
More wiring of Env mapping (#2681)
Amogh-Bharadwaj Mar 10, 2025
8e36edf
feat: upgrade `docker-compose.yml` stable image tags (#2683)
github-actions[bot] Mar 10, 2025
6e46d22
chore(deps): update github-actions dependencies (#2686)
renovate[bot] Mar 10, 2025
c466f07
chore(deps): update dockerfile dependencies (#2685)
renovate[bot] Mar 10, 2025
c50fe8a
chore(deps): update depot/bake-action digest to 58d7160 (#2688)
renovate[bot] Mar 10, 2025
077f8ce
fix(deps): update cargo dependencies (#2689)
renovate[bot] Mar 10, 2025
637b6be
fix(deps): update npm dependencies (#2690)
renovate[bot] Mar 11, 2025
6238b7c
pg: fix timetz with timezone (#2693)
serprex Mar 11, 2025
7b92b70
close connectors after retrieving schema in flow api (#2695)
serprex Mar 11, 2025
b2d5937
feat(maintenance): add check for DropFlow during mirror pause (#2684)
iamKunalGupta Mar 12, 2025
b17171e
qrep: fetched bytes (#2687)
serprex Mar 12, 2025
ed986a2
chore: update dependencies (#2692)
serprex Mar 12, 2025
eeac1ed
Fix MySQL Initial Load Stats (#2682)
Amogh-Bharadwaj Mar 12, 2025
fb3e6ea
mysql: support vector (#2648)
serprex Mar 12, 2025
c61355c
Increase close timeouts for setup flow activities (#2699)
Amogh-Bharadwaj Mar 12, 2025
a81c7ed
PEERDB_SKIP_EXPORT_SNAPSHOT (#2703)
serprex Mar 13, 2025
6a59112
Getting selected columns: use format along with regclass (#2706)
Amogh-Bharadwaj Mar 14, 2025
b946864
ClickHouse: Support for special characters in column names (#2705)
Amogh-Bharadwaj Mar 17, 2025
236ee64
feat: upgrade `docker-compose.yml` stable image tags (#2697)
github-actions[bot] Mar 17, 2025
aa7de99
remove ubicloud (#2708)
serprex Mar 17, 2025
270fe94
Add test for geometry for pg to clickhouse (#2702)
panbhag-ch Mar 17, 2025
f935b29
chore(deps): pin dependencies (#2710)
renovate[bot] Mar 17, 2025
9c96531
chore(deps): update dockerfile dependencies (#2711)
renovate[bot] Mar 17, 2025
3485773
e2e code coverage (#2709)
serprex Mar 17, 2025
be97edb
fix(deps): update npm dependencies (#2713)
renovate[bot] Mar 17, 2025
9531302
fix(deps): update cargo dependencies (#2712)
renovate[bot] Mar 17, 2025
ee0bdf3
chore(deps): pin codecov/codecov-action action to 0565863 (#2714)
renovate[bot] Mar 17, 2025
e642d76
nexus: aws-lc-rs (#2715)
serprex Mar 17, 2025
48b6e7f
fix(deps): update dependency styled-components to v6.1.16 (#2716)
renovate[bot] Mar 17, 2025
82dce88
chore(deps): update golangci/golangci-lint-action digest to 55c2c14 (…
renovate[bot] Mar 18, 2025
328a894
chore(deps): update rust crate rust_decimal to v1.37.0 (#2720)
renovate[bot] Mar 18, 2025
b3e15ef
[postgres] validate hsfeedback on standby (#2718)
heavycrystal Mar 18, 2025
651ab96
only update tableSchema for tables we see deltas for (#2717)
heavycrystal Mar 18, 2025
144838c
added e2e test for cutoff bug in schema change handling (#2723)
heavycrystal Mar 18, 2025
170f8af
initial load: base clone workflow id on source, not destination (#2534)
serprex Mar 18, 2025
e6e145b
Use aggregate table for counts per table (#2696)
iskakaushik Mar 18, 2025
3bd7185
null in uuid[] should be zero uuid (#2725)
serprex Mar 18, 2025
4f5e72a
cast hsfeedback to bool before scan (#2728)
heavycrystal Mar 19, 2025
d9e1bc1
feat: upgrade `docker-compose.yml` stable image tags (#2729)
github-actions[bot] Mar 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/actions/genprotos/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ runs:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4
- name: check cache
id: cache
uses: ubicloud/cache@92361f338d82d2c58a98875f1b5c95cd14cd6b2a # v4
uses: actions/cache@d4323d4df104b026a6aa633fdb11d772146be0bf # v4
with:
path: |
./flow/generated/protos
Expand All @@ -15,7 +15,7 @@ runs:
key: ${{ runner.os }}-build-genprotos-${{ hashFiles('buf.gen.yaml', './protos/peers.proto', './protos/flow.proto', './protos/route.proto') }}

- if: steps.cache.outputs.cache-hit != 'true'
uses: bufbuild/buf-action@b46cbc051dd8fd10a3c2213e784498790d3fe388 # v1
uses: bufbuild/buf-action@1b8e0a0e793562b7850d7e6ff0228b5c0b16111c # v1
with:
setup_only: true
github_token: ${{ github.token }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
json: ${{ secrets.SNOWFLAKE_GH_CI_PKEY }}
dir: "nexus/server/tests/assets/"

- uses: ubicloud/rust-cache@65b3ff06b9bcc69d88c25e212f1ae3d14a0953c3 # v2
- uses: Swatinem/rust-cache@f0deed1e0edfc6a9be95417288c0e1099b1eeec3 # v2
with:
workspaces: nexus

Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/cleanup.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ on:

jobs:
flow_cleanup:
runs-on: ubicloud-standard-2-ubuntu-2204-arm
runs-on: ubuntu-24.04
timeout-minutes: 60
steps:
- name: checkout sources
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- uses: ubicloud/setup-go@6ea6c4fb36acc657571a53f4a7471e75f5fa80ad # v5
- uses: actions/setup-go@f111f3307d8850f501ac008e886eec1fd1932a34 # v5
with:
go-version: '1.24.0'
go-version: '1.24.1'
cache-dependency-path: e2e_cleanup/go.sum

- name: download go modules
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ on:
jobs:
analyze:
name: Analyze (${{ matrix.language }})
runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubicloud-standard-16' }}
runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-24.04' }}
timeout-minutes: ${{ (matrix.language == 'swift' && 120) || 360 }}
permissions:
# required for all workflows
Expand Down Expand Up @@ -47,12 +47,12 @@ jobs:

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@b56ba49b26e50535fa1e7f7db0f4f7b4bf65d80d # v3
uses: github/codeql-action/init@6bb031afdd8eb862ea3fc1848194185e076637e5 # v3
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@b56ba49b26e50535fa1e7f7db0f4f7b4bf65d80d # v3
uses: github/codeql-action/analyze@6bb031afdd8eb862ea3fc1848194185e076637e5 # v3
with:
category: "/language:${{matrix.language}}"
4 changes: 2 additions & 2 deletions .github/workflows/customer-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:
- uses: depot/setup-action@b0b1ea4f69e92ebf5dea3f8713a1b0c37b2126a5 # v1

- name: Login to GitHub Container Registry
uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3
with:
registry: ghcr.io
username: ${{github.actor}}
Expand All @@ -42,7 +42,7 @@ jobs:
echo "branch=$(echo $GITHUB_REF | sed -e 's/.*customer-//')" >> $GITHUB_OUTPUT

- name: Build (optionally publish) PeerDB Images
uses: depot/bake-action@143e50b965398f1f5dc8463be7dde6f62b9e9c21 # v1
uses: depot/bake-action@58d7160c6bfa64eb85e384209e6f2f5ad17948bb # v1
with:
token: ${{ secrets.DEPOT_TOKEN }}
files: ./docker-bake.hcl
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/dev-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
- uses: depot/setup-action@b0b1ea4f69e92ebf5dea3f8713a1b0c37b2126a5 # v1

- name: Login to GitHub Container Registry
uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3
with:
registry: ghcr.io
username: ${{github.actor}}
Expand All @@ -36,7 +36,7 @@ jobs:
run: echo "sha_short=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

- name: Build (optionally publish) PeerDB Images
uses: depot/bake-action@143e50b965398f1f5dc8463be7dde6f62b9e9c21 # v1
uses: depot/bake-action@58d7160c6bfa64eb85e384209e6f2f5ad17948bb # v1
with:
token: ${{ secrets.DEPOT_TOKEN }}
files: ./docker-bake.hcl
Expand Down
31 changes: 21 additions & 10 deletions .github/workflows/flow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
flow_test:
strategy:
matrix:
runner: [ubicloud-standard-16-ubuntu-2204-arm]
runner: [ubuntu-latest-16-cores]
postgres-version: [15, 16, 17]
runs-on: ${{ matrix.runner }}
timeout-minutes: 30
Expand All @@ -25,7 +25,7 @@ jobs:
POSTGRES_DB: postgres
POSTGRES_INITDB_ARGS: --locale=C.UTF-8
mysql:
image: mysql:oracle@sha256:146682692a3aa409eae7b7dc6a30f637c6cb49b6ca901c2cd160becc81127d3b
image: mysql:oracle@sha256:9b9d0aab4860798acff13d2a0ece3bc26639fe18b83fa5cd3e3d0e16b3ed05dd
ports:
- 3306:3306
env:
Expand All @@ -37,20 +37,20 @@ jobs:
# env:
# MARIADB_ROOT_PASSWORD: cipass
redpanda:
image: redpandadata/redpanda@sha256:04baa40ed34edf86028396f762be802f3b6b0acaab65e9de696a3f15d54c550f
image: redpandadata/redpanda@sha256:f3049dcb004ce1ccc0a8b3dc3c7d6ca1f32cb4d6d1da98a47874457dd8cb9907
ports:
- 9092:9092
- 9644:9644
elasticsearch:
image: elasticsearch:8.17.2@sha256:f388389127fe71274ff82493748e26677b88ff599785875f088627e0ca0e4c28
image: elasticsearch:8.17.3@sha256:2cc40b15dff85281ac804e7ff27cbc07b31fdd542e94ea7a2bdb6470e3b6bb68
ports:
- 9200:9200
env:
discovery.type: single-node
xpack.security.enabled: false
xpack.security.enrollment.enabled: false
minio:
image: bitnami/minio:2025.2.28@sha256:ca31735c5a8e6a85de77c63bea09d8780a788ca9bbf7f1d730610c5da84a5796
image: bitnami/minio:2025.3.12@sha256:7c92dd1ba1f48e1009079c5e3f0a98e3c5a34387fc474007f1a887db7643e2c2
ports:
- 9999:9999
env:
Expand All @@ -66,9 +66,9 @@ jobs:
- name: generate or hydrate protos
uses: ./.github/actions/genprotos

- uses: ubicloud/setup-go@6ea6c4fb36acc657571a53f4a7471e75f5fa80ad # v5
- uses: actions/setup-go@f111f3307d8850f501ac008e886eec1fd1932a34 # v5
with:
go-version: '1.24.0'
go-version: '1.24.1'
cache-dependency-path: flow/go.sum

- name: install lib-geos
Expand Down Expand Up @@ -126,7 +126,7 @@ jobs:
env:
PGPASSWORD: postgres

- uses: ubicloud/cache@92361f338d82d2c58a98875f1b5c95cd14cd6b2a # v4
- uses: actions/cache@d4323d4df104b026a6aa633fdb11d772146be0bf # v4
id: cache-clickhouse
with:
path: ./clickhouse
Expand All @@ -147,13 +147,18 @@ jobs:
- name: run tests
run: |
temporal server start-dev --namespace default --headless &
go build -ldflags="-s -w" -o peer-flow
mkdir coverage
go build -cover -ldflags="-s -w" -o peer-flow
temporal operator search-attribute create --name MirrorName --type Text --namespace default
./peer-flow worker &
./peer-flow snapshot-worker &
go test -p 32 ./... -timeout 900s
go test -cover -p 32 ./... -timeout 900s
killall peer-flow
sleep 1
go tool covdata textfmt -i=coverage -o coverage.out
working-directory: ./flow
env:
GOCOVERDIR: coverage
AWS_ENDPOINT_URL_S3: http://localhost:9999
AWS_ACCESS_KEY_ID: minio
AWS_SECRET_ACCESS_KEY: miniosecret
Expand Down Expand Up @@ -191,3 +196,9 @@ jobs:
PEERDB_QUEUE_FORCE_TOPIC_CREATION: "true"
ELASTICSEARCH_TEST_ADDRESS: http://localhost:9200
CI_PG_VERSION: ${{ matrix.postgres-version }}

- name: Upload coverage reports to Codecov
uses: codecov/codecov-action@0565863a31f2c772f9f0395002a31e3f06189574 # v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
working-directory: ./flow
8 changes: 4 additions & 4 deletions .github/workflows/golang-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ permissions:
jobs:
golangci:
name: lint
runs-on: [ubicloud-standard-4-ubuntu-2204-arm]
runs-on: [ubuntu-24.04]
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

Expand All @@ -23,12 +23,12 @@ jobs:
run: |
sudo apt-get update
sudo apt-get install libgeos-dev
- uses: ubicloud/setup-go@6ea6c4fb36acc657571a53f4a7471e75f5fa80ad # v5
- uses: actions/setup-go@f111f3307d8850f501ac008e886eec1fd1932a34 # v5
with:
go-version: '1.24.0'
go-version: '1.24.1'
cache: false
- name: golangci-lint
uses: golangci/golangci-lint-action@2226d7cb06a077cd73e56eedd38eecad18e5d837 # v6
uses: golangci/golangci-lint-action@55c2c1448f86e01eaae002a5a3a9624417608d84 # v6
with:
version: v1.64
working-directory: ./flow
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/stable-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
docker-build:
strategy:
matrix:
runner: [ubicloud-standard-2-ubuntu-2204-arm]
runner: [ubuntu-24.04]
runs-on: ${{ matrix.runner }}
permissions:
contents: read
Expand All @@ -23,14 +23,14 @@ jobs:
- uses: depot/setup-action@b0b1ea4f69e92ebf5dea3f8713a1b0c37b2126a5 # v1

- name: Login to GitHub Container Registry
uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3
with:
registry: ghcr.io
username: ${{github.actor}}
password: ${{secrets.GITHUB_TOKEN}}

- name: Build (optionally publish) PeerDB Images
uses: depot/bake-action@143e50b965398f1f5dc8463be7dde6f62b9e9c21 # v1
uses: depot/bake-action@58d7160c6bfa64eb85e384209e6f2f5ad17948bb # v1
with:
token: ${{ secrets.DEPOT_TOKEN }}
files: ./docker-bake.hcl
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ui-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
name: Build & Test UI
strategy:
matrix:
runner: [ubicloud-standard-2-ubuntu-2204-arm]
runner: [ubuntu-24.04]
runs-on: ${{ matrix.runner }}
steps:
- name: checkout
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ui-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
name: Run UI linters
strategy:
matrix:
runner: [ubicloud-standard-2-ubuntu-2204-arm]
runner: [ubuntu-24.04]
runs-on: ${{ matrix.runner }}
steps:
- name: checkout
Expand Down
10 changes: 5 additions & 5 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ services:

flow-api:
container_name: flow_api
image: ghcr.io/peerdb-io/flow-api:stable-v0.25.6
image: ghcr.io/peerdb-io/flow-api:stable-v0.26.4
restart: unless-stopped
ports:
- 8112:8112
Expand All @@ -128,7 +128,7 @@ services:

flow-snapshot-worker:
container_name: flow-snapshot-worker
image: ghcr.io/peerdb-io/flow-snapshot-worker:stable-v0.25.6
image: ghcr.io/peerdb-io/flow-snapshot-worker:stable-v0.26.4
restart: unless-stopped
environment:
<<: [*catalog-config, *flow-worker-env, *minio-config]
Expand All @@ -138,7 +138,7 @@ services:

flow-worker:
container_name: flow-worker
image: ghcr.io/peerdb-io/flow-worker:stable-v0.25.6
image: ghcr.io/peerdb-io/flow-worker:stable-v0.26.4
restart: unless-stopped
environment:
<<: [*catalog-config, *flow-worker-env, *minio-config]
Expand All @@ -151,7 +151,7 @@ services:
peerdb:
container_name: peerdb-server
stop_signal: SIGINT
image: ghcr.io/peerdb-io/peerdb-server:stable-v0.25.6
image: ghcr.io/peerdb-io/peerdb-server:stable-v0.26.4
restart: unless-stopped
environment:
<<: *catalog-config
Expand All @@ -167,7 +167,7 @@ services:

peerdb-ui:
container_name: peerdb-ui
image: ghcr.io/peerdb-io/peerdb-ui:stable-v0.25.6
image: ghcr.io/peerdb-io/peerdb-ui:stable-v0.26.4
restart: unless-stopped
ports:
- 3000:3000
Expand Down
Loading
Loading