Perform unlogged index reinitialization in index_open

[knizhnik]

No description provided.

[hlinnaka]

I think we need to step back and think harder how we can fix unlogged tables. Currently, we're trying to make it OK for the unlogged relation to not exist, and silence errors caused by that. That feels like an unnatural thing to do and an uphill battle, because that's not how unlogged tables work in PostgreSQL.

In PostgreSQL, at server startup we scan the data directory, and initialize every unlogged relation from the "init fork". In other words, we scan the data directory searching for files with the *_init suffix, and copy it over the file without the suffix. If we change the basebackup code in pageserver to include the *_init files, then the PostgreSQL logic to re-initialize unlogged relations work normally. I'd suggest we do that.

[hlinnaka]

Also, we need a test for this

[knizhnik]

I think we need to step back and think harder how we can fix unlogged tables. Currently, we're trying to make it OK for the unlogged relation to not exist, and silence errors caused by that. That feels like an unnatural thing to do and an uphill battle, because that's not how unlogged tables work in PostgreSQL.

In PostgreSQL, at server startup we scan the data directory, and initialize every unlogged relation from the "init fork". In other words, we scan the data directory searching for files with the *_init suffix, and copy it over the file without the suffix. If we change the basebackup code in pageserver to include the *_init files, then the PostgreSQL logic to re-initialize unlogged relations work normally. I'd suggest we do that.

First of all it is not exactly true. Postgres is not resetting unlogged relations on startup: it does it only in case of recovery.
I afraid that scanning all catalog on each start of compute will be too expensive, especially if we are going to deactivate/activate tenants quite frequently.

I think that we need to make even more steps backward and try to understand why do we (actually not we, but Neon users) may want to use unlogged tables at all. Certainly key point is performance: no WAL overhead. In case of Neon performance advantage will be much more bigger, because unlogged table are actually local table and their access time is the same as with standalone postgres. There is one main drawbacks of unlogged tables i Vaniila: them do no survive failures. It leads to 2 ain use cases of unlogged tables in Vanilla Postgres:

1. Use them for non-critical data which can be restored from some other data source. For example them are ideal for OLAP where data is loaded from some external data sources and can be reloaded at any moment.
2. Use them as temporary storage before transformation to normal tables. So data is loaded in unlogged table and once import is completed storage is changed to logged. As far as I understand right now it requires complete copy of all data. So I am not sure is there is actually some sense in such operations. But there are open PRs with fast transformation of unlogged storage in logged.

In case of Neon there are two moe drawbacks of unlogged tables:

1. Them are lost on every compute restart. It is is expected to be performed quite frequently (5 minutes inactivity).
2. Their size is limited for local storage capacity (at stateless node!)

Both items actually mean that it is almost not possible to use unlogged table for more or less large data sources. You can not load data for several hours just to loose them in 5 minutes.

Using unlogged table for efficient transformation into normal table may be have some sense, because we can build indexes without any access to pageserver. But take in account that data has to fit in local storage, it seems to be easier and more efficient just to use this space for local file cache which can help to reach the same goal (eliminate get_page overhead) without this tricks with unlogged->logged transformation.

My option is that unlogged table in current state in eon are almost useless.
As far as I understand @hlinnaka has discussed this question with customers and the m really want to have unlogged tables,
May be I am wrong, but I afraid that them just do not realize all limitations of unlogged tables in Neon. Until now them were not able to load their data in unlogged tables and that it is why them are not faced with this limitations. But once they succeed with importing data, we will get another portion of complaints.

I do not suggest just to ignore unlogged keyword as it was done initially.
I think that we really need to have some fast local tables. But them should not be lost on any compute restart
and better not be limited by local storage size.
One of the possible solutions will be place them in tablespace mounted to some NFS directory.
In this case they can survive compute restart and are not limited by local storage size.
But if access to NFS is significantly faster than getting pages from pageserver, then ... we are doing something wrong.

[hlinnaka]

First of all it is not exactly true. Postgres is not resetting unlogged relations on startup: it does it only in case of recovery. I afraid that scanning all catalog on each start of compute will be too expensive, especially if we are going to deactivate/activate tenants quite frequently.

It should be fine. The whole relation directory is stored as one key-value pair in the pageserver storage. It's quick to scan through. Would you like to write a performance test for that, to verify?

(If you have a large number of relations, the way we store the relation directory as one giant key-value pair might be a performance issue, but that's a separate issue.)

[hlinnaka]

Here's a proof-of-concept of a fix with that approach: https://github.com/neondatabase/neon/pull/new/fix-unlogged-in-basebackup. Need to add tests and #262. But you can use it for performance testing already

[hlinnaka]

Here's a proof-of-concept of a fix with that approach: https://github.com/neondatabase/neon/pull/new/fix-unlogged-in-basebackup. Need to add tests and #262. But you can use it for performance testing already

@arssher just opened a PR for that: neondatabase/neon#3706 :-)