Add PG 18Beta1 compatibility (Build + RuleUtils) - #7981
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7981 +/- ##
==========================================
- Coverage 89.19% 89.19% -0.01%
==========================================
Files 284 284
Lines 61515 61523 +8
Branches 7696 7696
==========================================
+ Hits 54871 54878 +7
- Misses 4440 4443 +3
+ Partials 2204 2202 -2 🚀 New features to boost your workflow:
|
e7a3f1c to
6220ff5
Compare
27ea748 to
5789619
Compare
d4c66aa to
172782a
Compare
b95047d to
6129856
Compare
|
moved to PR desc |
1 similar comment
|
moved to PR desc |
Add support for PostgreSQL version 18 in build and test workflow pipeline isn’t picking up the 18 line because your regex for the full field only allows digits and dots ([0-9.]+), so it skips the "full": "18beta1" entry. Fix image suffix formatting in build parameters
|
moved to PR desc |
2 similar comments
|
moved to PR desc |
|
moved to PR desc |
Naisila Puka (naisila)
left a comment
There was a problem hiding this comment.
The 16 commits in this PR do not correspond to relevant PG18 commit changes, but rather commits fixing your development work (refactoring etc). The main commit that contains a bunch of PG18 related changes all in one place is pg18 compile work cont. Although the changes are minor, it is difficult to review them when they are all in one place, and the comments are scattered throughout the PR description.
I think it is more readable to follow the practice of previous years, and keep the changes separated in commits, while linking the relevant PG18 commit for each change. PG18 comes with a lot of minor changes, and to maintain them we need a tracking system. Links below:
I would suggest you to rearrange your PR similarly, after addressing the review as well. If you have a different idea from the previous years, I'm definitely open to that, as long as it is systematic and readable.
| #if PG_VERSION_NUM >= PG_VERSION_18 | ||
| vac_update_relstats(rel, new_rel_pages, new_live_tuples, | ||
| new_rel_allvisible, /* allvisible */ | ||
| 0, /* all_frozen */ |
There was a problem hiding this comment.
Why all_frozen is 0?
I think it would be more consistent to define it similar to BlockNumber new_rel_allvisible = 0;, together with the explanation why all_frozen is 0.
There was a problem hiding this comment.
I hard-code all_frozen = 0 for columnar storage doesn’t maintain a visibility map or “all-frozen” state the way heap tables do.
I agree on the refactor you suggested. If you are okay on all_frozen = 0, I will update the code.
There was a problem hiding this comment.
Makes sense, you can update the code
There was a problem hiding this comment.
Updated
Implement code changes to enhance functionality and improve performance Revert unwanted edits to configure scripts
5320c98 to
b4ec029
Compare
|
moved to PR desc |
…G18 compatibility
…ner and shard_pruning
Naisila Puka (naisila)
left a comment
There was a problem hiding this comment.
At this point rearranging into separate commits would take some time.
One thing you could do is: gather every change, along with its relevant PG commit, in the PR description. It will be a long PR description, but it is readable and I believe we can understand the reason for every change in the PR.
| #if PG_VERSION_NUM >= PG_VERSION_18 | ||
| vac_update_relstats(rel, new_rel_pages, new_live_tuples, | ||
| new_rel_allvisible, /* allvisible */ | ||
| 0, /* all_frozen */ |
There was a problem hiding this comment.
Makes sense, you can update the code
| * If new object classes are added and none of them are node-wide, then update | ||
| * this assertion check based on latest supported major Postgres version. | ||
| */ | ||
| StaticAssertStmt(PG_MAJORVERSION_NUM <= 17, |
There was a problem hiding this comment.
Actually ObjectClass was removed in PG17, so we need to check for new catalog OIDs. postgres/postgres@89e5ef7
| * If new object classes are added and none of them are node-wide, then update | ||
| * this assertion check based on latest supported major Postgres version. | ||
| */ | ||
| StaticAssertStmt(PG_MAJORVERSION_NUM <= 17, |
There was a problem hiding this comment.
I checked dependency.c file, https://github.com/postgres/postgres/commits/REL_18_STABLE/src/backend/catalog/dependency.c
and objectaddress.c file
https://github.com/postgres/postgres/commits/REL_18_STABLE/src/backend/catalog/objectaddress.c
Seems like there isn't anything new right?
Naisila Puka (naisila)
left a comment
There was a problem hiding this comment.
Reminder to revert changes in Configure script and .yml files, and anywhere else that 18beta1 version is included.
This PR provides successful build against PG18Beta1. RuleUtils PR was reviewed separately: #8010
PG 18Beta1–related changes for building Citus
TupleDesc / Attr layout
What changed in PG: Postgres consolidated the
TupleDescData.attrs[]array into a more compact representation. Direct field access (tupdesc->attrs[i]) was replaced by the newTupleDescAttr()API.Citus adaptation: Everywhere we previously used
tupdesc->attrs[...], we now callTupleDescAttr(tupdesc, idx)(or our ownAttr()macro) under a compatibility guard.General Logic:
Use
Attr(...)in places wherecolumnar_version_compat.his included. This avoids the need to sprinkle#if PG_VERSION_NUMguards around each attribute access.Use
TupleDescAttr(tupdesc, i)when the relevant PostgreSQL header is already included and the additional macro indirection is unnecessary.Collation‐aware
LIKEWhat changed in PG: The
textlikeoperator now requires an explicit collation, to avoid ambiguous‐collation errors. Core code switched fromDirectFunctionCall2(textlike, ...)toDirectFunctionCall2Coll(textlike, DEFAULT_COLLATION_OID, ...).Citus adaptation: In
remote_commands.cand any other LIKE call, we now useDirectFunctionCall2Coll(textlike, DEFAULT_COLLATION_OID, ...)and#include <utils/pg_collation.h>.Columnar storage API
columnar_relation_set_new_filelocator(and related init routines) for PG 18’s revised SMGR and storage-initialization hooks.explain_format.h,columnar_version_compat.h) so the columnar module compiles cleanly against PG 18.heap_modify_tuple + heap_inplace_update only exist on PG < 18; on PG18 the in-place helper was removed upstream
postgres/postgres@a07e03f
OpenSSL / TLS integration
What changed in PG: Moved from the legacy
SSL_library_init()toOPENSSL_init_ssl(OPENSSL_INIT_LOAD_CONFIG, NULL), updated certificate API calls (X509_getm_notBefore,X509_getm_notAfter), and standardized onTLS_method().Citus adaptation: We now
#include <openssl/opensslv.h>and use#if OPENSSL_VERSION_NUMBER >= 0x10100000Lto choose betweenOPENSSL_init_ssl()orSSL_library_init(), and wrapX509_gmtime_adj()calls around the new accessor functions.Adapt
ExtractColumns()to the new PG-18expandRTE()signaturePostgreSQL 18 postgres/postgres@80feb72 added a fourth argument of type
VarReturningTypetoexpandRTE(), so calls that used the old 7-parameter form no longer compile. This patch:expandRTE(...)call in a#if PG_VERSION_NUM >= 180000guard.VAR_RETURNING_DEFAULTargument beforelocation.parser/parse_relation.hforexpandRTEandVarReturningType, andpg_version_constants.hforPG_VERSION_NUM).Adapt
ExecutorStart/ExecutorRunhooks to PG-18’s new signaturesPostgreSQL 18 postgres/postgres@525392d changed the signatures of the executor hooks:
ExecutorStart_hooknow returnsboolinstead ofvoid, andExecutorRun_hookdrops its oldrun_onceargument.This patch preserves Citus’s existing hook logic by:
Adding two adapter functions under
#if PG_VERSION_NUM >= PG_VERSION_18:citus_executor_start_adapter(QueryDesc *queryDesc, int eflags)Calls the old
CitusExecutorStart(queryDesc, eflags)and then returnstrueto satisfy the new hook’sboolreturn type.citus_executor_run_adapter(QueryDesc *queryDesc, ScanDirection direction, uint64 count)Calls the old
CitusExecutorRun(queryDesc, direction, count, true)(passingtruefor the droppedrun_onceargument), and returnsvoid.Installing the adapters in
_PG_init()instead of the original hooks when building against PG 18+:Adapt to PG-18’s removal of the “run_once” flag from ExecutorRun/PortalRun
PostgreSQL commit [3eea7a0](postgres/postgres@3eea7a0) rationalized the executor’s parallelism logic by moving the “execute a plan only once” check into
ExecutePlan()itself and dropping the oldbool run_onceargument from the public APIs:(and similarly for
PortalRun()).To stay compatible across PG 15–18, Citus now:
Updates all internal calls to
ExecutorRun(...)andPortalRun(...):ExecutorRun(qd, dir, count)).ExecutorRun(qd, dir, count, true)) under a#if PG_VERSION_NUM < 180000guard.Guards the dispatcher hooks via the adapter functions (from the earlier patch) so that Citus’s executor hooks continue to work under both the old and new signatures.
Adapt to PG-18’s shortened PortalRun signature
PostgreSQL 18’s refactoring (see commit 3eea7a0) also removed the old run_once and alternate‐dest arguments from the public PortalRun() API. The signature changed from:
To support both versions in Citus, we:
Version-guard each call to
PortalRun():trueforrun_once.Add support for PG-18’s new
plansourceargument inPortalDefineQuery**PostgreSQL 18 extended the
PortalDefineQueryAPI to carry aCachedPlanSource *plansourcepointer so that the portal machinery can track cached‐plan invalidation (as introduced alongside deferred-locking in commit postgres/postgres@525392d. To remain compatible across PG 15–18, Citus now wraps its calls under a version guard:Adapt ExecInitRangeTable() calls to PG-18’s new signature
PostgreSQL commit cbc127917e04a978a788b8bc9d35a70244396d5b overhauled the planner API for range‐table initialization:
PG 18+: added a fourth
Bitmapset *unpruned_relidsargument to support deferred partition pruningIn Citus’s
create_estate_for_relation()(incolumnar_metadata.c), we now wrap the call in a compile‐time guard so that the code compiles correctly on all supported PostgreSQL versions:Adapt
pgstat_report_vacuum()to PG-18’s new timestamp argumentPostgreSQL commit [30a6ed0ce4bb18212ec38cdb537ea4b43bc99b83](postgres/postgres@30a6ed0) extended the
pgstat_report_vacuum()API by adding aTimestampTz start_timeparameter at the end so that the VACUUM statistics collector can record when the operation began:To support both versions, we now wrap the call in
columnar_tableam.cwith a version guard, supplyingGetCurrentTimestamp()for PG-18+:Adapt
ExecuteTaskPlan()to PG-18’s expandedCreateQueryDesc()signaturePostgreSQL 18 changed
CreateQueryDesc()from an eight-argument to a nine-argument call by inserting aCachedPlan *cplanparameter immediately after thePlannedStmt *plannedstmtargument (see commit postgres/postgres@525392d). To remain compatible with PG 15–17, Citus now wraps its invocation inlocal_executor.cwith a version guard:Adapt
RelationGetPrimaryKeyIndex()to PG-18’s new “deferrable_ok” flagPostgreSQL commit postgres/postgres@14e87ff added a new Boolean
deferrable_okparameter toRelationGetPrimaryKeyIndex()so that the lock manager can defer unique‐constraint locks when requested. The API changed from:to:
Because Citus has always taken the lock immediately—just as the old two-arg call did—we pass
falseto keep that same immediate-lock behavior. Passingtruewould switch to deferred locking, which we don’t want.Adapt
ExplainOnePlan()to PG-18’s expanded APIPostgreSQL 18 extended postgres/postgres@525392d the
ExplainOnePlan()function to carry theCachedPlan *andCachedPlanSource *pointers plus an explicitquery_index, letting the EXPLAIN machinery track plan‐source invalidation. The old signature:became, in PG 18:
To compile under both versions, Citus now wraps each call in
multi_explain.cwith:Adapt to the unified “index interpretation” API in PG 18 (commit a8025f544854)
PostgreSQL commit postgres/postgres@a8025f5 generalized the old btree‐specific operator‐interpretation API into a single “index interpretation” interface:
OpBtreeInterpretation→OpIndexInterpretationget_op_btree_interpretation(opno)→get_op_index_interpretation(opno)Each interpretation now carries
cmptypeinstead ofstrategy.To build cleanly on PG 18 while still supporting PG 15–17, Citus’s shard‐pruning code now wraps these changes:
Adapt
create_foreignscan_path()for PG-18’s revised signaturePostgreSQL commit postgres/postgres@e222534 reordered and removed a couple of parameters in the FDW‐path builder:
PG 15–17 signature (11 args)
PG 18+ signature (9 args)
To support both, Citus now defines a compatibility macro in
pg_version_compat.h:Now every call to
create_foreignscan_path_compat(...)—even in tests likefake_fdw.c—automatically picks the correct argument list for PG 15 through PG 18.Drop the obsolete bitmap‐scan hooks on PG 18+
PostgreSQL commit postgres/postgres@c395322 cleaned up the
TableAmRoutineAPI by removing the two bitmap‐scan callback slots:scan_bitmap_next_blockscan_bitmap_next_tupleSince those hook‐slots no longer exist in PG 18, Citus now wraps their NULL‐initialization in a
#if PG_VERSION_NUM < PG_VERSION_18guard. On PG 15–17 we still explicitly set them toNULL(to satisfy the old struct layout), and on PG 18+ we omit them entirely:Adapt
vac_update_relstats()invocation to PG-18’s new “all_frozen” argumentPostgreSQL commit postgres/postgres@99f8f3f extended the
vac_update_relstats()API by inserting anum_all_frozen_pagesparameter between the existingnum_all_visible_pagesandhasindexarguments:To compile cleanly on both PG 15–17 and PG 18+, Citus wraps its call in a version guard and supplies a zero placeholder for the new field:
Why all_frozen = 0?
Columnar storage never embeds transaction IDs in its pages, so it never needs to track “all‐frozen” pages the way a heap does. Setting both allvisible and allfrozen to zero simply tells Postgres “there are no pages with the visibility or frozen‐status bits set,” matching our existing behavior.
This change ensures Citus’s VACUUM‐statistic updates work unmodified across all supported Postgres versions.