Skip to content
This repository was archived by the owner on Aug 30, 2025. It is now read-only.

Conversation

@nickzelei
Copy link
Member

No description provided.

@linear
Copy link

linear bot commented Mar 26, 2025

@vercel
Copy link

vercel bot commented Mar 26, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
neosync-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 28, 2025 4:10pm

@nickzelei nickzelei added the enhancement New feature or request label Mar 26, 2025
@github-actions
Copy link

github-actions bot commented Mar 26, 2025

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedMar 28, 2025, 4:10 PM

c.TABLE_SCHEMA AS schema_name,
c.TABLE_NAME AS table_name,
c.COLUMN_NAME AS column_name,
c.COLUMN_TYPE AS data_type,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is kinda weird. the above query we are using c.data_type but this one we are using c.column_type which is different. I didnt want to update it at the expense of potentially causing a bux.

c.TABLE_NAME AS table_name,
c.COLUMN_NAME AS column_name,
c.COLUMN_TYPE AS data_type,
c.COLUMN_TYPE AS column_type, -- same as data_type but includes things like the length, or set/enum information
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alishakawaguchi - This column_type properly tracks mysql set and enum values. So I'm guessing that the mysql schema drift changes won't track if this changes. I created a ticket for this: NEOS-1791

@codecov
Copy link

codecov bot commented Mar 26, 2025

Codecov Report

Attention: Patch coverage is 16.84665% with 385 lines in your changes missing coverage. Please review.

Project coverage is 24.24%. Comparing base (e63cbfc) to head (2266c4f).
Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
...ernal/benthos/benthos-builder/builders/sql-util.go 11.07% 297 Missing ⚠️
backend/sql/postgresql/models/models.go 0.00% 43 Missing ⚠️
...ker/pkg/benthos/transformers/utils/string_utils.go 61.01% 22 Missing and 1 partial ⚠️
internal/job/job.go 0.00% 12 Missing ⚠️
...er/pkg/benthos/transformers/utils/integer_utils.go 38.46% 7 Missing and 1 partial ⚠️
backend/pkg/sqlmanager/mysql/mysql-manager.go 0.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3412      +/-   ##
==========================================
- Coverage   24.32%   24.24%   -0.09%     
==========================================
  Files         417      417              
  Lines       49405    49856     +451     
==========================================
+ Hits        12020    12089      +69     
- Misses      36251    36631     +380     
- Partials     1134     1136       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nickzelei nickzelei marked this pull request as ready for review March 26, 2025 20:58
@github-actions
Copy link

github-actions bot commented Mar 26, 2025

Benchstat Geomean Results

-8.14% sec/op, -4.17% B/op, -0.68% allocs/op

Benchstat results
                                                                 │ main-benchmark.txt │        branch-benchmark.txt         │
                                                                 │       sec/op       │    sec/op     vs base               │
CleanPostgresType-4                                                       42.63n ± 4%   42.43n ±  5%        ~ (p=0.647 n=6)
Runner_Single-4                                                           7.713µ ± 5%   7.641µ ±  3%        ~ (p=0.310 n=6)
GenerateBool-4                                                            11.45n ± 1%   11.60n ±  2%        ~ (p=0.061 n=6)
GenerateBusinessName-4                                                    285.4n ± 0%   328.0n ± 12%  +14.93% (p=0.002 n=6)
GenerateCardNumber-4                                                      28.41n ± 1%   29.14n ±  1%   +2.61% (p=0.002 n=6)
GenerateCategorical-4                                                    104.10n ± 2%   93.28n ±  0%  -10.40% (p=0.002 n=6)
GenerateCity-4                                                            173.7n ± 0%   176.3n ±  2%   +1.50% (p=0.002 n=6)
GenerateCountry-4                                                        105.20n ± 1%   95.44n ±  0%   -9.28% (p=0.002 n=6)
GenerateEmail-4                                                           1.346µ ± 0%   1.312µ ±  1%   -2.53% (p=0.002 n=6)
GenerateFirstName-4                                                       203.9n ± 0%   206.3n ±  4%   +1.18% (p=0.004 n=6)
GenerateFloat64-4                                                         28.73n ± 1%   26.48n ±  0%   -7.85% (p=0.002 n=6)
GenerateFullAddress-4                                                     1.429µ ± 3%   1.419µ ±  0%   -0.73% (p=0.004 n=6)
GenerateFullName-4                                                        1.222µ ± 1%   1.262µ ±  3%   +3.32% (p=0.002 n=6)
GenerateGender-4                                                          41.42n ± 0%   36.97n ±  0%  -10.75% (p=0.002 n=6)
GenerateInt64-4                                                           28.41n ± 2%   27.08n ±  1%   -4.66% (p=0.002 n=6)
GenerateInt64PhoneNumber-4                                                51.46n ± 2%   51.24n ±  3%        ~ (p=0.310 n=6)
GenerateInternationalPhoneNumber-4                                        175.8n ± 1%   174.6n ±  4%        ~ (p=0.121 n=6)
GenerateIpAddress-4                                                       569.3n ± 0%   568.3n ±  0%        ~ (p=0.065 n=6)
GenerateLastName-4                                                        187.4n ± 1%   191.0n ±  1%   +1.92% (p=0.002 n=6)
GenerateRandomString-4                                                    985.6n ± 1%   266.1n ±  1%  -73.00% (p=0.002 n=6)
GenerateSHA256Hash-4                                                      1.016µ ± 0%   1.004µ ±  1%   -1.23% (p=0.002 n=6)
GenerateSSN-4                                                             305.8n ± 1%   265.9n ±  2%  -13.03% (p=0.002 n=6)
GenerateState-4                                                          105.15n ± 2%   95.12n ±  0%   -9.53% (p=0.002 n=6)
GenerateStreetAddress-4                                                   204.0n ± 1%   208.4n ±  1%   +2.18% (p=0.002 n=6)
GenerateStringPhoneNumber-4                                               179.3n ± 0%   170.2n ±  3%   -5.08% (p=0.002 n=6)
GenerateUnixTimestamp-4                                                   77.13n ± 1%   76.59n ±  1%        ~ (p=0.065 n=6)
GenerateUsername-4                                                        351.5n ± 0%   352.3n ±  1%        ~ (p=0.307 n=6)
GenerateUTCTimestamp-4                                                    104.2n ± 1%   105.7n ±  1%   +1.34% (p=0.002 n=6)
GenerateUUID-4                                                            734.0n ± 0%   729.2n ±  1%   -0.65% (p=0.004 n=6)
GenerateZipcode-4                                                        105.15n ± 2%   93.67n ±  0%  -10.91% (p=0.002 n=6)
TransformCharacterScramble-4                                              171.8n ± 1%   161.2n ±  1%   -6.11% (p=0.002 n=6)
TransformE164PhoneNumber-4                                                180.2n ± 3%   171.9n ±  1%   -4.63% (p=0.002 n=6)
TransformEmail-4                                                          1.798µ ± 2%   1.767µ ±  1%   -1.72% (p=0.002 n=6)
TransformFirstName-4                                                      233.9n ± 0%   235.1n ±  1%   +0.49% (p=0.002 n=6)
TransformFloat64-4                                                        110.8n ± 4%   101.5n ±  1%   -8.39% (p=0.002 n=6)
TransformFullName-4                                                       1.235µ ± 2%   1.304µ ±  3%   +5.63% (p=0.002 n=6)
TransformInt64-4                                                          33.74n ± 1%   31.92n ±  0%   -5.39% (p=0.002 n=6)
TransformInt64PhoneNumber-4                                               57.84n ± 2%   58.11n ±  0%        ~ (p=0.056 n=6)
TransformLastName-4                                                       191.6n ± 0%   197.0n ±  1%   +2.84% (p=0.002 n=6)
TransformString-4                                                        1022.5n ± 0%   292.6n ±  1%  -71.39% (p=0.002 n=6)
TransformStringPhoneNumber-4                                              211.0n ± 2%   198.9n ±  0%   -5.71% (p=0.002 n=6)
TransformUuid-4                                                           36.54n ± 1%   32.51n ±  0%  -11.03% (p=0.002 n=6)
CleanMysqlType-4                                                                        54.02n ±  8%
GenerateRandomStringWithInclusiveBounds/SmallRange_1_10-4                               82.67n ±  1%
GenerateRandomStringWithInclusiveBounds/MediumRange_10_100-4                            256.9n ±  1%
GenerateRandomStringWithInclusiveBounds/LargeRange_100_1000-4                           1.871µ ±  1%
GenerateRandomStringWithInclusiveBounds/FixedLength_50_50-4                             198.8n ±  0%
GenerateRandomStringWithInclusiveBounds/ZeroLength_0_0-4                                3.739n ±  0%
GenerateRandomStringWithInclusiveBounds/Text_1_65535-4                                  15.03µ ±  0%
GenerateRandomStringWithInclusiveBounds/AboveLimit_1_100000-4                           15.66µ ±  4%
GenerateRandomStringWithInclusiveBounds/FixedAboveLimit_100000-4                        27.87µ ±  1%
GenerateRandomStringWithInclusiveBounds/MaxTextSize_65535-4                             27.63µ ±  2%
geomean                                                                   194.0n        242.4n         -8.14%

                                                                 │ main-benchmark.txt │         branch-benchmark.txt          │
                                                                 │        B/op        │     B/op      vs base                 │
CleanPostgresType-4                                                      0.000 ± 0%       0.000 ± 0%        ~ (p=1.000 n=6) ¹
Runner_Single-4                                                        2.312Ki ± 1%     2.292Ki ± 1%   -0.89% (p=0.026 n=6)
GenerateBool-4                                                           0.000 ± 0%       0.000 ± 0%        ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                                                   304.0 ± 0%       304.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                                                     8.000 ± 0%       8.000 ± 0%        ~ (p=1.000 n=6) ¹
GenerateCategorical-4                                                    64.00 ± 0%       64.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateCity-4                                                           160.0 ± 0%       160.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateCountry-4                                                        24.00 ± 0%       24.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateEmail-4                                                          385.0 ± 0%       385.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateFirstName-4                                                      208.0 ± 0%       208.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateFloat64-4                                                        8.000 ± 0%       8.000 ± 0%        ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                                                    512.0 ± 0%       512.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateFullName-4                                                       418.0 ± 0%       418.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateGender-4                                                         16.00 ± 0%       16.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateInt64-4                                                          7.000 ± 0%       7.000 ± 0%        ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                                               8.000 ± 0%       8.000 ± 0%        ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4                                       40.00 ± 0%       40.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                                                      31.00 ± 0%       31.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateLastName-4                                                       176.0 ± 0%       176.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateRandomString-4                                                  186.00 ± 0%       72.00 ± 0%  -61.29% (p=0.002 n=6)
GenerateSHA256Hash-4                                                     288.0 ± 0%       288.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateSSN-4                                                            46.00 ± 0%       46.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateState-4                                                          24.00 ± 0%       24.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                                                  240.0 ± 0%       240.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                                              40.00 ± 0%       40.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                                                  8.000 ± 0%       8.000 ± 0%        ~ (p=1.000 n=6) ¹
GenerateUsername-4                                                       205.0 ± 0%       205.0 ± 0%        ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                                                   24.00 ± 0%       24.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateUUID-4                                                           80.00 ± 0%       80.00 ± 0%        ~ (p=1.000 n=6) ¹
GenerateZipcode-4                                                        24.00 ± 0%       24.00 ± 0%        ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4                                             40.00 ± 3%       40.00 ± 0%        ~ (p=0.455 n=6)
TransformE164PhoneNumber-4                                               40.00 ± 0%       40.00 ± 0%        ~ (p=1.000 n=6) ¹
TransformEmail-4                                                         489.0 ± 0%       489.0 ± 0%        ~ (p=1.000 n=6) ¹
TransformFirstName-4                                                     224.0 ± 0%       224.0 ± 0%        ~ (p=1.000 n=6) ¹
TransformFloat64-4                                                       88.00 ± 0%       88.00 ± 0%        ~ (p=1.000 n=6) ¹
TransformFullName-4                                                      418.0 ± 0%       418.0 ± 0%        ~ (p=1.000 n=6) ¹
TransformInt64-4                                                         8.000 ± 0%       8.000 ± 0%        ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                                              8.000 ± 0%       8.000 ± 0%        ~ (p=1.000 n=6) ¹
TransformLastName-4                                                      176.0 ± 0%       176.0 ± 0%        ~ (p=1.000 n=6) ¹
TransformString-4                                                       202.00 ± 0%       88.00 ± 0%  -56.44% (p=0.002 n=6)
TransformStringPhoneNumber-4                                             56.00 ± 0%       56.00 ± 0%        ~ (p=1.000 n=6) ¹
TransformUuid-4                                                          16.00 ± 0%       16.00 ± 0%        ~ (p=1.000 n=6) ¹
CleanMysqlType-4                                                                          0.000 ± 0%
GenerateRandomStringWithInclusiveBounds/SmallRange_1_10-4                                 6.000 ± 0%
GenerateRandomStringWithInclusiveBounds/MediumRange_10_100-4                              61.00 ± 0%
GenerateRandomStringWithInclusiveBounds/LargeRange_100_1000-4                             582.0 ± 0%
GenerateRandomStringWithInclusiveBounds/FixedLength_50_50-4                               64.00 ± 0%
GenerateRandomStringWithInclusiveBounds/ZeroLength_0_0-4                                  0.000 ± 0%
GenerateRandomStringWithInclusiveBounds/Text_1_65535-4                                  34.42Ki ± 0%
GenerateRandomStringWithInclusiveBounds/AboveLimit_1_100000-4                           34.43Ki ± 0%
GenerateRandomStringWithInclusiveBounds/FixedAboveLimit_100000-4                        64.00Ki ± 0%
GenerateRandomStringWithInclusiveBounds/MaxTextSize_65535-4                             64.00Ki ± 0%
geomean                                                                             ²                  -4.17%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                                 │ main-benchmark.txt │         branch-benchmark.txt         │
                                                                 │     allocs/op      │  allocs/op   vs base                 │
CleanPostgresType-4                                                      0.000 ± 0%     0.000 ±  0%        ~ (p=1.000 n=6) ¹
Runner_Single-4                                                          24.00 ± 0%     24.00 ±  0%        ~ (p=1.000 n=6) ¹
GenerateBool-4                                                           0.000 ± 0%     0.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateBusinessName-4                                                   2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateCardNumber-4                                                     1.000 ± 0%     1.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateCategorical-4                                                    2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateCity-4                                                           2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateCountry-4                                                        2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateEmail-4                                                          8.000 ± 0%     8.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateFirstName-4                                                      2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateFloat64-4                                                        1.000 ± 0%     1.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateFullAddress-4                                                    10.00 ± 0%     10.00 ±  0%        ~ (p=1.000 n=6) ¹
GenerateFullName-4                                                       6.000 ± 0%     6.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateGender-4                                                         1.000 ± 0%     1.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateInt64-4                                                          0.000 ± 0%     0.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateInt64PhoneNumber-4                                               1.000 ± 0%     1.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateInternationalPhoneNumber-4                                       3.000 ± 0%     3.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateIpAddress-4                                                      2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateLastName-4                                                       2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateRandomString-4                                                   3.000 ± 0%     2.000 ±  0%  -33.33% (p=0.002 n=6)
GenerateSHA256Hash-4                                                     7.000 ± 0%     7.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateSSN-4                                                            3.000 ± 0%     3.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateState-4                                                          2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateStreetAddress-4                                                  2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateStringPhoneNumber-4                                              3.000 ± 0%     3.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateUnixTimestamp-4                                                  1.000 ± 0%     1.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateUsername-4                                                       6.000 ± 0%     6.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateUTCTimestamp-4                                                   1.000 ± 0%     1.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateUUID-4                                                           3.000 ± 0%     3.000 ±  0%        ~ (p=1.000 n=6) ¹
GenerateZipcode-4                                                        2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
TransformCharacterScramble-4                                             2.000 ± 0%     3.000 ± 33%  +50.00% (p=0.015 n=6)
TransformE164PhoneNumber-4                                               3.000 ± 0%     3.000 ±  0%        ~ (p=1.000 n=6) ¹
TransformEmail-4                                                         14.00 ± 0%     14.00 ±  0%        ~ (p=1.000 n=6) ¹
TransformFirstName-4                                                     3.000 ± 0%     3.000 ±  0%        ~ (p=1.000 n=6) ¹
TransformFloat64-4                                                       3.000 ± 0%     3.000 ±  0%        ~ (p=1.000 n=6) ¹
TransformFullName-4                                                      6.000 ± 0%     6.000 ±  0%        ~ (p=1.000 n=6) ¹
TransformInt64-4                                                         1.000 ± 0%     1.000 ±  0%        ~ (p=1.000 n=6) ¹
TransformInt64PhoneNumber-4                                              1.000 ± 0%     1.000 ±  0%        ~ (p=1.000 n=6) ¹
TransformLastName-4                                                      2.000 ± 0%     2.000 ±  0%        ~ (p=1.000 n=6) ¹
TransformString-4                                                        4.000 ± 0%     3.000 ±  0%  -25.00% (p=0.002 n=6)
TransformStringPhoneNumber-4                                             4.000 ± 0%     4.000 ±  0%        ~ (p=1.000 n=6) ¹
TransformUuid-4                                                          1.000 ± 0%     1.000 ±  0%        ~ (p=1.000 n=6) ¹
CleanMysqlType-4                                                                        0.000 ±  0%
GenerateRandomStringWithInclusiveBounds/SmallRange_1_10-4                               1.000 ±  0%
GenerateRandomStringWithInclusiveBounds/MediumRange_10_100-4                            1.000 ±  0%
GenerateRandomStringWithInclusiveBounds/LargeRange_100_1000-4                           1.000 ±  0%
GenerateRandomStringWithInclusiveBounds/FixedLength_50_50-4                             1.000 ±  0%
GenerateRandomStringWithInclusiveBounds/ZeroLength_0_0-4                                0.000 ±  0%
GenerateRandomStringWithInclusiveBounds/Text_1_65535-4                                  1.000 ±  0%
GenerateRandomStringWithInclusiveBounds/AboveLimit_1_100000-4                           1.000 ±  0%
GenerateRandomStringWithInclusiveBounds/FixedAboveLimit_100000-4                        1.000 ±  0%
GenerateRandomStringWithInclusiveBounds/MaxTextSize_65535-4                             1.000 ±  0%
geomean                                                                             ²                 -0.68%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

@nickzelei nickzelei merged commit 5359c83 into main Mar 28, 2025
24 of 26 checks passed
@nickzelei nickzelei deleted the nick/neos-1787-add-auto-map-support-to-mysql branch March 28, 2025 16:35
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants