fix: capture time created from metadata not filename

[de-sh]

Fixes #XXXX.

Description

Currently we use filename to discover files that aren't to be considered for compaction, with this PR we will consider file creation time instead.

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

Summary by CodeRabbit

• Bug Fixes
  • Enhanced file exclusion logic by switching to a more robust, system-aware timestamp comparison.
  • Improved precision when determining file creation times to ensure more reliable file processing.

[coderabbitai]

Walkthrough

The changes update the time handling in file exclusion logic. The parameter type for the exclusion time in the arrow_files_grouped_exclude_time method is changed from NaiveDateTime to SystemTime. This modifies how file creation times are compared, utilizing a new helper function to convert SystemTime into a minute-based timestamp in milliseconds. Additionally, the call from convert_disk_files_to_parquet now directly passes the current system time.

Changes

File(s)	Change Summary
src/parseable/streams.rs	- Changed parameter type in arrow_files_grouped_exclude_time from NaiveDateTime to SystemTime. - Updated time comparison logic to use a new helper function for milliseconds. - Modified call in convert_disk_files_to_parquet to pass current system time.

Sequence Diagram(s)

sequenceDiagram
    participant C as convert_disk_files_to_parquet
    participant S as Stream::arrow_files_grouped_exclude_time
    participant F as Arrow File

    C->>+S: Call arrow_files_grouped_exclude_time(SystemTime, shutdown_signal)
    Note over S, F: For each arrow file, compare creation time in ms
    S-->>-C: Return filtered files based on timestamp

Loading

Poem

I'm a rabbit leaping through code so bright,
With system time, my hops feel just right.
Millis tick in a rhythmic dance,
Guiding each file with precise advance.
In streams of change, I bound with delight!
🐇✨

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e7d8018 and 354dc91.

📒 Files selected for processing (1)

• src/parseable/streams.rs (9 hunks)

🔇 Additional comments (5)

src/parseable/streams.rs (5)

27-27: Added SystemTime import to handle new time comparison approach.

This change is necessary to support the shift from using NaiveDateTime to SystemTime for determining which files to exclude during compaction.

---

76-82: Good addition of helper function for consistent time-based comparisons.

This helper function converts a SystemTime to a minute-based timestamp for consistent time comparisons. It properly handles conversion to milliseconds and then to minutes (dividing by 60000).

Just note that the .expect("Legitimate time") could panic if there's an invalid time, though this is extremely unlikely with SystemTime from file metadata.

---

203-203: API change: parameter type updated from NaiveDateTime to SystemTime.

Changed parameter type in arrow_files_grouped_exclude_time from NaiveDateTime to SystemTime to align with the PR objective of using file creation time from metadata rather than filename.

---

213-219: Improved file filtering using actual file metadata instead of filename parsing.

This change implements the core objective of the PR - using file creation metadata instead of parsing filenames to determine which files to exclude from compaction.

The new approach:

1. Gets the file's creation time from metadata
2. Compares minutes since epoch using the helper function
3. Provides more reliable results than string parsing

This is more reliable and less error-prone than the previous string parsing approach.

---

441-442: Directly using SystemTime instead of converting from chrono time.

Using SystemTime::now() directly is cleaner and more efficient than creating a chrono time and then converting it. This change also ensures consistency with the updated parameter type in arrow_files_grouped_exclude_time.

✨ Finishing Touches

• 📝 Generate Docstrings (Beta)

---

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[nikhilsinhaparseable]

looks good