Skip to content

Sync-Fix #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 39 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
002f90e
change location of secrets files
carykees98 Apr 7, 2025
27b57ec
add secrets directory to gitignore
carykees98 Apr 9, 2025
57a4b92
fixed typo in JobManager
carykees98 Apr 14, 2025
5177906
updating SyncDetails to work with new mirrors.json schema
carykees98 Apr 14, 2025
ca303ea
updated format for SyncScheduer.cpp
carykees98 Apr 14, 2025
a0601c4
updated JobManager to work with mirrors.json change
carykees98 Apr 14, 2025
bed3549
trying to figure out what is causing an error
carykees98 Apr 14, 2025
26f1bc3
more debug
carykees98 Apr 14, 2025
9c9b699
debug again
carykees98 Apr 14, 2025
b373b8f
maybe the fix?
carykees98 Apr 14, 2025
3989f30
more debug (wooo)
carykees98 Apr 14, 2025
784e5f6
moved command logging
carykees98 Apr 14, 2025
5d6ce6b
debug and formatting update
carykees98 Apr 14, 2025
e15aa75
libfmt cant format json, using dump instead
carykees98 Apr 14, 2025
48de63f
adding host, source, and dest shouldn't have been a nested loop
carykees98 Apr 14, 2025
e7425fe
debug print to see if fix worked
carykees98 Apr 14, 2025
20a49bb
log stderr for failing rsync processes
carykees98 Apr 14, 2025
981b588
close stdout and stderr pipes before removing job from active jobs map
carykees98 Apr 14, 2025
635c809
removed extra qualification from write_stderr_to_file
carykees98 Apr 14, 2025
7e9fbf3
write stdout and stderr instead of just stderr
carykees98 Apr 14, 2025
d47be4e
don't use ioctl, just read into a buffer of size bufsiz
carykees98 Apr 14, 2025
98dc3fd
had been registering closed write end of pipe as read end
carykees98 Apr 14, 2025
24fe5d4
last item in argv has to be nullptr
carykees98 Apr 14, 2025
c6a59cc
updated warning for failed sync
carykees98 Apr 14, 2025
fe9ae41
resize stream contents string after clearing it
carykees98 Apr 16, 2025
7068c5f
update compose files
carykees98 Apr 16, 2025
308b28d
fixed bug where all syncs would be interrupted if one timed out
carykees98 Apr 18, 2025
0d0d9ea
added zsh to sync scheduler docker file to run fedora sync
carykees98 Apr 24, 2025
464e2f3
updated how stderr and stdout are written to files
carykees98 Apr 24, 2025
0fa2877
added ability to manual sync all projects
carykees98 Apr 25, 2025
2a52db2
passed the wrong variable to start_sync when syncing all projects
carykees98 Apr 25, 2025
048af47
fixed script commands not being properly composed
carykees98 Apr 25, 2025
9bfa664
tty true for colored logs
carykees98 Apr 25, 2025
52c3d87
log project name instead of "all_projects" when failing to start a ma…
carykees98 May 16, 2025
d7a847d
updated formatting for script command string
carykees98 May 17, 2025
33604fa
treat argument as string instead of auto
carykees98 May 17, 2025
e53ba35
more formatting
carykees98 May 17, 2025
a11d3f6
back to what it was?
carykees98 May 17, 2025
83383bc
add shebang to kicad sync script
carykees98 May 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.vscode
build/*
GeoLite2City
GeoLite2City
secrets
6 changes: 2 additions & 4 deletions docker-compose-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,15 +49,13 @@ services:
- 9281
volumes:
- "./mirror-sync-scheduler/storage:/storage"
- "./mirror-sync-scheduler/error-logs:/mirror/error-logs"
- "./mirror-sync-scheduler/configs:/mirror/configs:ro"
- "./mirror-sync-scheduler/scripts:/mirror/scripts:ro"
- "./mirror-sync-scheduler/secrets:/mirror/secrets:ro"
networks:
- mirror
depends_on:
- log-server
stdin_open: true
tty: true

# --- Torrent handler ---
# torrent-handler:
# build: ./mirror-torrent-handler
Expand Down
5 changes: 2 additions & 3 deletions docker-compose-prod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,12 @@ services:
- configs/sync-scheduler.env
volumes:
- "/storage:/storage"
- "./sync-error-logs:/mirror/error-logs"
- "./configs:/mirror/configs:ro"
- "./scripts:/mirror/scripts:ro"
- "./secrets:/mirror/secrets:ro"
networks:
- mirror
depends_on:
- log-server
stdin_open: true
tty: true

# --- Torrent handler ---
Expand Down
1 change: 1 addition & 0 deletions mirror-sync-scheduler/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
build
*.secret
.idea
secrets
2 changes: 1 addition & 1 deletion mirror-sync-scheduler/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ RUN cmake --build build --target all

# Run
FROM ubuntu:24.04
RUN apt update && apt install -y python3 curl wget libzmq3-dev rsync
RUN apt update && apt install -y python3 curl wget libzmq3-dev rsync zsh
WORKDIR /mirror
COPY --from=builder /mirror/build/src/sync_scheduler .
RUN chmod 744 sync_scheduler
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ class JobManager
public: // Methods
auto start_job(
const std::string& jobName,
std::string command,
std::vector<std::string> command,
const std::filesystem::path& passwordFile
) -> bool;

Expand All @@ -53,13 +53,19 @@ class JobManager
auto kill_all_jobs() -> void;
auto reap_processes() -> std::vector<::pid_t>;
auto deregister_jobs(const std::vector<::pid_t>& completedJobs) -> void;
auto write_streams_to_file(const ::pid_t processID)
-> std::pair<std::string, std::string>;
auto process_reaper(const std::stop_token& stopToken) -> void;

private: // Static Methods
static auto get_child_process_ids(const ::pid_t processID)
-> std::vector<::pid_t>;
static auto interrupt_job(const ::pid_t processID) -> void;
static auto kill_job(const ::pid_t processID) -> void;
static auto write_stream_to_file(
const std::string& logfileName,
const int pipeFileDescriptor
) -> bool;

private: // Members
std::map<::pid_t, SyncJob> m_ActiveJobs;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,26 +40,28 @@ class SyncDetails
}

[[nodiscard]]
auto get_commands() const -> std::vector<std::string>
auto get_commands() const -> std::vector<std::vector<std::string>>
{
return m_Commands;
}

private: // Methods
[[nodiscard]]
static auto compose_rsync_commands(const nlohmann::json& rsyncConfig)
-> std::vector<std::vector<std::string>>;

static auto handle_rsync_options_array(const nlohmann::json& optionsArray)
-> std::vector<std::string>;

auto handle_sync_config(const nlohmann::json& project) -> void;

auto handle_rsync_config(const nlohmann::json& project) -> void;

auto handle_script_config(const nlohmann::json& project) -> void;

private: // Members
std::size_t m_SyncsPerDay;
std::optional<std::filesystem::path> m_PasswordFile;
std::vector<std::string> m_Commands;
std::size_t m_SyncsPerDay;
std::optional<std::filesystem::path> m_PasswordFile;
std::vector<std::vector<std::string>> m_Commands;
};

struct static_project_exception : std::runtime_error
Expand Down
Loading