Skip to content

Refactor code and debug performance issues #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 75 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
3ad4fd8
Create only one tls connector for all jetstream sockets
zebreus Jan 8, 2025
ceadafc
Format Cargo.toml
zebreus Feb 20, 2025
f055a3b
Remove outdated information from readme
zebreus Feb 20, 2025
389e8ec
Move the index_repo functionality into a separate module
zebreus Feb 20, 2025
9c68917
Refactor repo indexer
zebreus Feb 20, 2025
7ef01f3
Properly manage the queue of the discovery task
zebreus Feb 20, 2025
d1fae9b
Work on index repo performance
zebreus Feb 21, 2025
e5756d5
Add tokio-console instrumentation
zebreus Feb 24, 2025
5fed4d8
Convert the repo indexer to use a stream
zebreus Feb 24, 2025
78c374f
Add opentelemetry
zebreus Feb 24, 2025
a4cbbf4
Update dependencies
zebreus Feb 24, 2025
d5ccdb8
Add information about opentelemetry to README
zebreus Feb 25, 2025
0465543
Add system metrics reporting
zebreus Feb 25, 2025
c467685
Improve units in metrics reporter
zebreus Feb 25, 2025
d24fd4d
Improve telemetry setup
zebreus Feb 25, 2025
51d2c99
Improve interval accuracy in metrics reporter
zebreus Feb 25, 2025
eaf9429
Refactor repo indexer task
zebreus Feb 25, 2025
ff30972
Moved indexed repos metric into the indexer
zebreus Feb 25, 2025
3e6a1ef
Split main.rs into multiple files
zebreus Feb 25, 2025
6c96c09
Enable LTO for release builds
zebreus Feb 25, 2025
d4aff7a
Document tracing layers
zebreus Feb 25, 2025
42b6786
Instrument on_commit_event_createorupdate
zebreus Feb 25, 2025
6527dec
Instrument index_repo
zebreus Feb 25, 2025
9b0a15d
Add curl to the nix flake
zebreus Feb 26, 2025
6a74979
Add multiplier for the download buffer size
zebreus Feb 26, 2025
7f602ce
Add more pipeline metrics
zebreus Feb 26, 2025
fabb258
Add server setup script
zebreus Feb 26, 2025
fe96360
Changes
zebreus Feb 26, 2025
c6c492b
Adjust queue size
zebreus Mar 2, 2025
8d41b31
Set target-cpu to native
zebreus Mar 2, 2025
6d58208
Reduce build time for release builds
zebreus Mar 2, 2025
5b312d4
Make config global
zebreus Mar 2, 2025
36eac37
Enable incremental release builds
zebreus Mar 2, 2025
454a298
Refactor main
zebreus Mar 2, 2025
396e6ab
Remove threads from jetstream consumer
zebreus Mar 2, 2025
af79ce8
Switch to rustls
zebreus Mar 2, 2025
c945aa0
Adjust boolean CLI flags
zebreus Mar 2, 2025
3873471
Parallelize database writes
zebreus Mar 2, 2025
9cdf606
Add samply to the nix flake
zebreus Mar 2, 2025
18de229
Improve server setup script
zebreus Mar 2, 2025
d0e1cfb
Remove service instance from metrics
zebreus Mar 3, 2025
087125f
Work on performance
zebreus Mar 4, 2025
90bf788
[WIP] Commit dirty worktree for easier testing on hetzner
zebreus Mar 4, 2025
f385f7c
Support multiple database connections
zebreus Mar 4, 2025
f7cffa4
Fix pipeline state metrics
zebreus Mar 4, 2025
f49960f
Bundle all updates into a single transaction
zebreus Mar 4, 2025
9df85f0
Delete unused files
zebreus Mar 5, 2025
366a8e4
Clean up observability initialization
zebreus Mar 5, 2025
4973921
Remove dead code in repo indexer
zebreus Mar 5, 2025
4d43e0f
Fix logging initialization
zebreus Mar 5, 2025
403625c
Rework pipeline using generics
zebreus Mar 5, 2025
d95ba1c
Refactor index_repo
zebreus Mar 5, 2025
9ab7bd0
Add metrics for database updates
zebreus Mar 5, 2025
a0616c2
Add tracing to the pipeline steps
zebreus Mar 5, 2025
6dcacad
Refactor RepoStream
zebreus Mar 5, 2025
a0f92f0
Clean up database/mod.ts
zebreus Mar 5, 2025
f4701b8
Split BigUpdate into its own file
zebreus Mar 5, 2025
8eb1c21
Add surrealDB to the setup script
zebreus Mar 5, 2025
b9b3fdc
Apply automatic clippy fixed
zebreus Mar 5, 2025
f256aa5
Fix remaining clippy issues
zebreus Mar 5, 2025
6161033
Delete some commented out code
zebreus Mar 5, 2025
e9d9e99
Improve cli options
zebreus Mar 5, 2025
9c0f7a3
Add retry mechanism for repo download
zebreus Mar 6, 2025
3b34896
Enable debug info in release builds
zebreus Mar 6, 2025
b6d44c6
Debug database performance
zebreus Mar 6, 2025
a56cf78
Include the cargo lockfile
zebreus Mar 9, 2025
21c3ef8
Include .vscode
zebreus Mar 9, 2025
24abf66
Improve release profile for maximum performance
zebreus Mar 9, 2025
4e2f4f4
Fix warnings
zebreus Mar 9, 2025
2b35f8b
Add option to use a bundled root certificate
zebreus Mar 9, 2025
b80f917
Add a default database server url
zebreus Mar 9, 2025
79d8d18
Document the congestion control mechanism
zebreus Mar 9, 2025
1833798
Add dockerized deployment
zebreus Mar 9, 2025
88f4af9
Adjust deployment docker-compose
zebreus Mar 9, 2025
1325640
Fix grafana collector interval
zebreus Mar 9, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[build]
rustflags = [
"--cfg",
"tokio_unstable",
"-C",
"target-cpu=native",
"--cfg",
"tokio_tracing",
]
2 changes: 2 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
/target
/data
6 changes: 1 addition & 5 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,2 @@
# cargo files
Cargo.lock
/target

# ide files
.vscode
/data
Loading