migration-script to push user data to kafka

[Prajwal17Tunerlabs]

Summary by CodeRabbit

• New Features
  • Introduced a script to push user data, including organizations, roles, and enriched location metadata, to a Kafka topic based on updates within a specified date range.
• Documentation
  • Added a README file with setup instructions, usage details, and workflow explanation for the new data migration script.
• Chores
  • Added the "minimist" dependency for command-line argument parsing.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

A new migration utility for pushing user data to Kafka has been introduced. This includes a standalone Node.js script that queries users updated within a date range, enriches their location metadata, and publishes events to a Kafka topic. Documentation and a new dependency for argument parsing are also added.

Changes

Cohort / File(s)	Change Summary
Migration Script & Logic src/migrations/pushDataToKafka/pushUserDataToKafka.js	Adds a standalone Node.js script that connects to PostgreSQL, fetches users updated in a given date range, enriches user and location data, determines event types, and publishes structured events to a Kafka topic. Handles external HTTP enrichment, error logging, and manages connections. No exported or public entities are introduced.
Documentation src/migrations/pushDataToKafka/README.md	Introduces a detailed README describing the migration script’s prerequisites, usage, workflow, error handling, and configuration. Explains the script’s operation, expected environment, and event structure.
Dependency Addition src/package.json	Adds the "minimist" dependency for command-line argument parsing to support the migration script. No other changes to package configuration.

Sequence Diagram(s)

sequenceDiagram
    participant CLI_User as CLI User
    participant Script as pushUserDataToKafka.js
    participant PG as PostgreSQL
    participant ExtAPI as Location Enrichment Service
    participant Kafka as Kafka Broker

    CLI_User->>Script: Run script with --from and --to dates
    Script->>PG: Connect and query users updated in date range
    loop For each user
        Script->>PG: Fetch user, orgs, roles
        Script->>ExtAPI: Enrich location metadata (HTTP request)
        ExtAPI-->>Script: Return enriched data or error
        Script->>Kafka: Publish user event (create/update/delete)
    end
    Script-->>CLI_User: Log summary and exit

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~15 minutes

Poem

A hop and a skip, new scripts on the run,
Pushing user tales to Kafka—oh what fun!
With minimist in tow and docs to explain,
Enriched locations riding the data train.
PostgreSQL and Kafka, in harmony they sing,
As rabbits log and monitor everything! 🐇✨

✨ Finishing Touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch chore/kafka-data-push-script

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[nevil-mathew]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 7

🧹 Nitpick comments (4)

src/migrations/pushDataToKafka/pushUserDataToKafka.js (2)

27-27: Consider handling all PostgreSQL SSL modes

The current implementation only checks for sslmode=require, but PostgreSQL supports other modes like 'prefer', 'allow', 'disable', 'verify-ca', and 'verify-full'. Consider a more comprehensive SSL configuration.

-		ssl: dbUrl.searchParams.get('sslmode') === 'require',
+		ssl: dbUrl.searchParams.get('sslmode') === 'require' ? { rejectUnauthorized: false } : false,

---

279-302: Consider batch processing and rate limiting

Processing users sequentially might be slow for large datasets. Consider:

1. Processing in batches with controlled concurrency
2. Adding rate limiting for external API calls to avoid overwhelming the entity management service

Would you like me to provide an implementation with batch processing and rate limiting using p-limit (already in your dependencies)?

src/migrations/pushDataToKafka/README.md (2)

32-35: Clarify database URL configuration

The example uses DEV_DATABASE_URL which suggests development use only. Consider clarifying that this should be adjusted for different environments or use a more generic variable name like DATABASE_URL.

-DEV_DATABASE_URL=postgres://user:password@localhost:5432/database
+DATABASE_URL=postgres://user:password@localhost:5432/database  # Adjust for your environment

---

76-76: Add more details about Kafka configuration

The documentation mentions Kafka is configured via ../../configs/kafka/index.js but doesn't specify what Kafka environment variables or configuration are needed.

Consider adding a section about required Kafka configuration (brokers, authentication, etc.) that should be present in the environment or config file.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 011713b and 8c62f57.

📒 Files selected for processing (3)

• src/migrations/pushDataToKafka/README.md (1 hunks)
• src/migrations/pushDataToKafka/pushUserDataToKafka.js (1 hunks)
• src/package.json (1 hunks)

🔇 Additional comments (3)

src/package.json (1)

64-64: No security vulnerabilities found in minimist@^1.2.8

All known prototype-pollution advisories (patched as of versions 0.2.4, 1.2.3, and 1.2.6) affect versions below 1.2.6. Since you’re using ^1.2.8, you’re above the last patched version and free from those vulnerabilities.

src/migrations/pushDataToKafka/pushUserDataToKafka.js (2)

34-84: LGTM!

The SQL query is well-structured with proper parameterization to prevent SQL injection. The multi-join approach efficiently fetches all required data in a single query.

---

126-132: Consider edge cases in event type determination

The logic assumes that when created_at equals updated_at, it's a create event. However, this might not be reliable if the database allows manual timestamp updates or if there are timing precision issues.

Consider adding a more explicit way to determine if this is truly a new record, perhaps by checking if this is the first time the record is being pushed to Kafka.

[Prajwal17Tunerlabs]

@coderabbitai review bf528ac