Add Postgres commit sha to Postgres version #4603

bayandin · 2023-06-30T13:45:25Z

Problem

Ref https://neondb.slack.com/archives/C036U0GRMRB/p1688122168477729

Summary of changes

Add sha from postgres repo into postgres version string (via --with-extra-version)
Add a test that Postgres version matches expected
Remove build-time hard check and allow only related tests to fail

Checklist before requesting a review

I have performed a self-review of my code.
If it is a core feature, I have added thorough tests.
Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

Do not forget to reformat commit message to not include the above checklist

koivunej

Assuming the tests pass, this is looking great.

github-actions · 2023-06-30T14:03:00Z

2970 tests run: 2843 passed, 0 failed, 127 skipped (full report)

Flaky tests (3)

Postgres 16

test_vm_bit_clear_on_heap_lock: debug

Postgres 15

test_lock_time_tracing: release

Postgres 14

test_partial_evict_tenant[relative_equal]: release

Code coverage* (full report)

functions: 31.3% (6247 of 19976 functions)
lines: 46.7% (46773 of 100097 lines)

* collected from Rust tests only

_{The comment gets automatically updated with the latest test results
e62b7e1 at 2024-05-07T15:26:01.331Z :recycle:}

Makefile

koivunej · 2023-07-12T10:58:00Z

This attempt is blocked on #2592 as it turns out; changing the version string changes the initdb ending up pageserver to resume replication from safekeepers at a wrong LSN.

Slack thread: https://neondb.slack.com/archives/C04KGFVUWUQ/p1689077525203009?thread_ts=1688655397.162369&cid=C04KGFVUWUQ

The error with pg14:

2023-07-08T00:13:36.088096Z ERROR wal_connection_manager{tenant_id=e5628643b3f5d7a1eaf98fa10868a32d timeline_id=0bd5e49271c85f93a41e4c05aaf3afc1}: wal receiver task finished with an error: walreceiver connection handling failure

Caused by:
    invalid xlog page header: xlp_rem_len=1798, expected 21619056 at 0/14A0000

pg15:

2023-07-08T00:13:37.893560Z ERROR wal_connection_manager{tenant_id=c9bea6937d84e2d59a70c065e6575faf timeline_id=c1f8434374ea6448a3dcca43002be31c}: wal receiver task finished with an error: walreceiver connection handling failure

Caused by:
    invalid xlog page header: xlp_rem_len=3746, expected 23675232 at 0/1698000

## Problem Postgres submodule can be changed unintentionally, and these changes are easy to miss during the review. Adding a check that should prevent this from happening, the check fails `build-neon` job with the following message: ``` Expected postgres-v14 rev to be at '1414141414141414141414141414141414141414', but it is at '1144aee1661c79eec65e784a8dad8bd450d9df79' Expected postgres-v15 rev to be at '1515151515151515151515151515151515151515', but it is at '1984832c740a7fa0e468bb720f40c525b652835d' Please update vendors/revisions.json if these changes are intentional. ``` This is an alternative approach to #4603 ## Summary of changes - Add `vendor/revisions.json` file with expected revisions - Add built-time check (to `build-neon` job) that Postgres submodules match revisions from `vendor/revisions.json` - A couple of small improvements for logs from #4603 - Fixed GitHub autocomment for no tests was run case --------- Co-authored-by: Joonas Koivunen <joonas@neon.tech>

Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.7.0 to 3.7.4. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/aio-libs/aiohttp/releases">aiohttp's releases</a>.</em></p> <blockquote> <h2>aiohttp 3.7.3 release</h2> <h2>Features</h2> <ul> <li>Use Brotli instead of brotlipy <code>[neondatabase#3803](aio-libs/aiohttp#3803) <https://github.com/aio-libs/aiohttp/issues/3803></code>_</li> <li>Made exceptions pickleable. Also changed the repr of some exceptions. <code>[neondatabase#4077](aio-libs/aiohttp#4077) <https://github.com/aio-libs/aiohttp/issues/4077></code>_</li> </ul> <h2>Bugfixes</h2> <ul> <li>Raise a ClientResponseError instead of an AssertionError for a blank HTTP Reason Phrase. <code>[neondatabase#3532](aio-libs/aiohttp#3532) <https://github.com/aio-libs/aiohttp/issues/3532></code>_</li> <li>Fix <code>web_middlewares.normalize_path_middleware</code> behavior for patch without slash. <code>[neondatabase#3669](aio-libs/aiohttp#3669) <https://github.com/aio-libs/aiohttp/issues/3669></code>_</li> <li>Fix overshadowing of overlapped sub-applications prefixes. <code>[neondatabase#3701](aio-libs/aiohttp#3701) <https://github.com/aio-libs/aiohttp/issues/3701></code>_</li> <li>Make <code>BaseConnector.close()</code> a coroutine and wait until the client closes all connections. Drop deprecated "with Connector():" syntax. <code>[neondatabase#3736](aio-libs/aiohttp#3736) <https://github.com/aio-libs/aiohttp/issues/3736></code>_</li> <li>Reset the <code>sock_read</code> timeout each time data is received for a <code>aiohttp.client</code> response. <code>[neondatabase#3808](aio-libs/aiohttp#3808) <https://github.com/aio-libs/aiohttp/issues/3808></code>_</li> <li>Fixed type annotation for add_view method of UrlDispatcher to accept any subclass of View <code>[neondatabase#3880](aio-libs/aiohttp#3880) <https://github.com/aio-libs/aiohttp/issues/3880></code>_</li> <li>Fixed querying the address families from DNS that the current host supports. <code>[neondatabase#5156](aio-libs/aiohttp#5156) <https://github.com/aio-libs/aiohttp/issues/5156></code>_</li> <li>Change return type of MultipartReader.<strong>aiter</strong>() and BodyPartReader.<strong>aiter</strong>() to AsyncIterator. <code>[neondatabase#5163](aio-libs/aiohttp#5163) <https://github.com/aio-libs/aiohttp/issues/5163></code>_</li> <li>Provide x86 Windows wheels. <code>[neondatabase#5230](aio-libs/aiohttp#5230) <https://github.com/aio-libs/aiohttp/issues/5230></code>_</li> </ul> <h2>Improved Documentation</h2> <ul> <li>Add documentation for <code>aiohttp.web.FileResponse</code>. <code>[neondatabase#3958](aio-libs/aiohttp#3958) <https://github.com/aio-libs/aiohttp/issues/3958></code>_</li> <li>Removed deprecation warning in tracing example docs <code>[neondatabase#3964](aio-libs/aiohttp#3964) <https://github.com/aio-libs/aiohttp/issues/3964></code>_</li> <li>Fixed wrong "Usage" docstring of <code>aiohttp.client.request</code>. <code>[neondatabase#4603](aio-libs/aiohttp#4603) <https://github.com/aio-libs/aiohttp/issues/4603></code>_</li> <li>Add aiohttp-pydantic to third party libraries <code>[neondatabase#5228](aio-libs/aiohttp#5228) <https://github.com/aio-libs/aiohttp/issues/5228></code>_</li> </ul> <h2>Misc</h2>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst">aiohttp's changelog</a>.</em></p> <blockquote> <h1>3.7.4 (2021-02-25)</h1> <h2>Bugfixes</h2> <ul> <li> <p><strong>(SECURITY BUG)</strong> Started preventing open redirects in the <code>aiohttp.web.normalize_path_middleware</code> middleware. For more details, see <a href="https://github.com/aio-libs/aiohttp/security/advisories/GHSA-v6wp-4m6f-gcjg">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-v6wp-4m6f-gcjg</a>.</p> <p>Thanks to <code>Beast Glatisant <https://github.com/g147></code>__ for finding the first instance of this issue and <code>Jelmer Vernooĳ <https://jelmer.uk/></code>__ for reporting and tracking it down in aiohttp. <code>[neondatabase#5497](aio-libs/aiohttp#5497) <https://github.com/aio-libs/aiohttp/issues/5497></code>_</p> </li> <li> <p>Fix interpretation difference of the pure-Python and the Cython-based HTTP parsers construct a <code>yarl.URL</code> object for HTTP request-target.</p> <p>Before this fix, the Python parser would turn the URI's absolute-path for <code>//some-path</code> into <code>/</code> while the Cython code preserved it as <code>//some-path</code>. Now, both do the latter. <code>[neondatabase#5498](aio-libs/aiohttp#5498) <https://github.com/aio-libs/aiohttp/issues/5498></code>_</p> </li> </ul> <hr /> <h1>3.7.3 (2020-11-18)</h1> <h2>Features</h2> <ul> <li>Use Brotli instead of brotlipy <code>[neondatabase#3803](aio-libs/aiohttp#3803) <https://github.com/aio-libs/aiohttp/issues/3803></code>_</li> <li>Made exceptions pickleable. Also changed the repr of some exceptions. <code>[neondatabase#4077](aio-libs/aiohttp#4077) <https://github.com/aio-libs/aiohttp/issues/4077></code>_</li> </ul> <h2>Bugfixes</h2> <ul> <li>Raise a ClientResponseError instead of an AssertionError for a blank HTTP Reason Phrase. <code>[neondatabase#3532](aio-libs/aiohttp#3532) <https://github.com/aio-libs/aiohttp/issues/3532></code>_</li> <li>Fix <code>web_middlewares.normalize_path_middleware</code> behavior for patch without slash. <code>[neondatabase#3669](aio-libs/aiohttp#3669) <https://github.com/aio-libs/aiohttp/issues/3669></code>_</li> <li>Fix overshadowing of overlapped sub-applications prefixes. <code>[neondatabase#3701](aio-libs/aiohttp#3701) <https://github.com/aio-libs/aiohttp/issues/3701></code>_</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/aio-libs/aiohttp/commit/0a26acc1de9e1b0244456b7881ec16ba8bb64fc3"><code>0a26acc</code></a> Bump aiohttp to v3.7.4 for a security release</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/021c416c18392a111225bc7326063dc4a99a5138"><code>021c416</code></a> Merge branch 'GHSA-v6wp-4m6f-gcjg' into master</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/4ed7c25b537f71c6245bb74d6b20e5867db243ab"><code>4ed7c25</code></a> Bump chardet from 3.0.4 to 4.0.0 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5333">#5333</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/b61f0fdffc887df24244ba7bdfe8567c580240ff"><code>b61f0fd</code></a> Fix how pure-Python HTTP parser interprets <code>//</code></li> <li><a href="https://github.com/aio-libs/aiohttp/commit/5c1efbc32c46820250bd25440bb7ea96cb05abe9"><code>5c1efbc</code></a> Bump pre-commit from 2.9.2 to 2.9.3 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5322">#5322</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/007507580137efcc0a20391a0792f39b337d9c1a"><code>0075075</code></a> Bump pygments from 2.7.2 to 2.7.3 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5318">#5318</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/5085173d947e6cc01b6daf1aa48fe7698834c569"><code>5085173</code></a> Bump multidict from 5.0.2 to 5.1.0 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5308">#5308</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/5d1a75e68d278c641c90021409f4eb5de1810e5e"><code>5d1a75e</code></a> Bump pre-commit from 2.9.0 to 2.9.2 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5290">#5290</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/6724d0e7a944fd7e3a710dc292d785fa8fe424fd"><code>6724d0e</code></a> Bump pre-commit from 2.8.2 to 2.9.0 (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5273">#5273</a>)</li> <li><a href="https://github.com/aio-libs/aiohttp/commit/c688451ce31b914c71b11d2ac6c326b0c87e6d1f"><code>c688451</code></a> Removed duplicate timeout parameter in ClientSession reference docs. (<a href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5262">#5262</a>) ...</li> <li>Additional commits viewable in <a href="https://github.com/aio-libs/aiohttp/compare/v3.7.0...v3.7.4">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aiohttp&package-manager=pip&previous-version=3.7.0&new-version=3.7.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/neondatabase/neon/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Vadim Kharitonov <vadim@neon.tech>

koivunej

The changes look good, but I am unsure how the change in initdb lsn was overcome.

It was commit 0cd49ca.

bayandin · 2024-05-07T15:08:00Z

The changes look good, but I am unsure how the change in initdb lsn was overcome.

The issue with initdb_lsb was fixed within #2592

## Problem Ref https://neondb.slack.com/archives/C036U0GRMRB/p1688122168477729 ## Summary of changes - Add sha from postgres repo into postgres version string (via `--with-extra-version`) - Add a test that Postgres version matches the expected one - Remove build-time hard check and allow only related tests to fail

bayandin requested review from vadim2404 and lubennikovaav June 30, 2023 13:45

koivunej approved these changes Jun 30, 2023

View reviewed changes

bayandin force-pushed the bayandin/add-postgres-version-test branch 5 times, most recently from 75776dd to 278c0bb Compare July 6, 2023 14:22

koivunej requested review from a team as code owners July 6, 2023 14:50

koivunej requested review from skyzh and removed request for a team July 6, 2023 14:50

skyzh reviewed Jul 6, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

bayandin force-pushed the bayandin/add-postgres-version-test branch from 02bba6d to d20b6ce Compare July 8, 2023 00:04

bayandin mentioned this pull request Jul 12, 2023

Prevent unintentional Postgres submodule update #4692

Merged

5 tasks

bayandin marked this pull request as draft July 14, 2023 09:41

koivunej mentioned this pull request Jul 17, 2023

initdb_lsn is not reproducible #2592

Closed

bayandin force-pushed the bayandin/add-postgres-version-test branch from d20b6ce to e7a0fe3 Compare April 17, 2024 15:58

bayandin mentioned this pull request Apr 17, 2024

Add Postgres commit to version string #7408

Closed

5 tasks

bayandin force-pushed the bayandin/add-postgres-version-test branch from e7a0fe3 to 7614af8 Compare April 17, 2024 16:21

bayandin marked this pull request as ready for review April 17, 2024 17:39

bayandin changed the title ~~test_runner: test postgres version~~ Add commit sha to Postgres version Apr 17, 2024

bayandin requested review from koivunej and removed request for vadim2404 and lubennikovaav April 17, 2024 21:51

bayandin changed the title ~~Add commit sha to Postgres version~~ Add Postgres commit sha to Postgres version Apr 18, 2024

bayandin force-pushed the bayandin/add-postgres-version-test branch from 7614af8 to 1fea582 Compare April 18, 2024 16:09

lubennikovaav approved these changes Apr 22, 2024

View reviewed changes

bayandin added 3 commits May 7, 2024 15:35

Add Postgres sha to its version string

235cf28

test_runner: add Postgres version check

d768c7a

CI(build_and_test): remove Postgres version check

e62b7e1

bayandin force-pushed the bayandin/add-postgres-version-test branch from 1fea582 to e62b7e1 Compare May 7, 2024 14:43

bayandin enabled auto-merge (squash) May 7, 2024 14:46

skyzh approved these changes May 7, 2024

View reviewed changes

koivunej approved these changes May 7, 2024

View reviewed changes

bayandin merged commit 51376ef into main May 7, 2024
50 of 51 checks passed

bayandin deleted the bayandin/add-postgres-version-test branch May 7, 2024 15:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Postgres commit sha to Postgres version #4603

Add Postgres commit sha to Postgres version #4603

bayandin commented Jun 30, 2023 •

edited

Loading

koivunej left a comment

github-actions bot commented Jun 30, 2023 •

edited

Loading

Postgres 16

Postgres 15

Postgres 14

koivunej commented Jul 12, 2023 •

edited

Loading

koivunej left a comment •

edited

Loading

bayandin commented May 7, 2024

Add Postgres commit sha to Postgres version #4603

Add Postgres commit sha to Postgres version #4603

Conversation

bayandin commented Jun 30, 2023 • edited Loading

Problem

Summary of changes

Checklist before requesting a review

Checklist before merging

koivunej left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 30, 2023 • edited Loading

2970 tests run: 2843 passed, 0 failed, 127 skipped (full report)

Postgres 16

Postgres 15

Postgres 14

Code coverage* (full report)

koivunej commented Jul 12, 2023 • edited Loading

koivunej left a comment • edited Loading

Choose a reason for hiding this comment

bayandin commented May 7, 2024

bayandin commented Jun 30, 2023 •

edited

Loading

github-actions bot commented Jun 30, 2023 •

edited

Loading

koivunej commented Jul 12, 2023 •

edited

Loading

koivunej left a comment •

edited

Loading