Adapt ES-Archiver to use savedObject types defined in the new platform

[pgayvallet]

Currently, the es-archiver is creating a KibanaMigrator to migrate the kibana index after a load

kibana/src/es_archiver/actions/load.js

Lines 90 to 97 in 8e9a8a8

	// If we affected the Kibana index, we need to ensure it's migrated...
	if (Object.keys(result).some(k => k.startsWith('.kibana'))) {
	await migrateKibanaIndex({ client, log, kibanaPluginIds });
	if (kibanaPluginIds.includes('spaces')) {
	await createDefaultSpace({ client, index: '.kibana' });
	}
	}

However to do that, it manually gathers the uiExports from the legacy plugins to get the SO schemas/mappings/migrations

kibana/src/es_archiver/lib/indices/kibana_index.js

Lines 37 to 49 in c65c28a

	const getUiExports = async kibanaPluginIds => {
	const xpackEnabled = kibanaPluginIds.includes('xpack_main');
	const { spec$ } = await findPluginSpecs({
	plugins: {
	scanDirs: [path.resolve(__dirname, '../../../legacy/core_plugins')],
	paths: xpackEnabled ? [path.resolve(__dirname, '../../../../x-pack')] : [],
	},
	});
	const specs = await spec$.pipe(toArray()).toPromise();
	return collectUiExports(specs);
	};

kibana/src/es_archiver/lib/indices/kibana_index.js

Lines 80 to 81 in c65c28a

	export async function migrateKibanaIndex({ client, log, kibanaPluginIds }) {
	const uiExports = await getUiExports(kibanaPluginIds);

kibana/src/es_archiver/lib/indices/kibana_index.js

Lines 106 to 113 in c65c28a

	savedObjectSchemas: new SavedObjectsSchema(uiExports.savedObjectSchemas),
	savedObjectMappings: convertLegacyMappings(uiExports.savedObjectMappings),
	savedObjectMigrations: uiExports.savedObjectMigrations,
	savedObjectValidations: uiExports.savedObjectValidations,
	callCluster: (path, ...args) => _.get(client, path).call(client, ...args),
	};
	return await new KibanaMigrator(migratorOptions).runMigrations();

After #55825 and the other SO types related issues are done, the SO types will be created from the new platform, and the archiver will not be able to be aware of them for the migration.

We need to find a way to have the archiver be aware of the NP-declared types. However, as the types registration is now done programmatically from the plugins setup phase, I'm really not seeing anything obvious to address this.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[pgayvallet]

One solution is to get back from our programmatic API to a more static declaration (as it's done for the plugin's configuration for example). This way, we would have the archiver scan for the SO definitions as it's currently doing for legacy plugins. But I think we would really like to avoid this if possible.

Other question that could be asked is why does the archiver needs to performs the migration after a load? could there be a workaround to avoid doing this?

[rudolf]

Other question that could be asked is why does the archiver needs to performs the migration after a load? could there be a workaround to avoid doing this?

My guess is that because the functional test runner creates one kibana server for the entire test suite (not for each it() test) we only run migrations once. Most tests load an archive for the entire suite before, but some tests reload the archived data in a beforeEach. If the archive contains unmigrated data this won't be migrated before being persisted in ES which means SavedObjects.find() queries might return the wrong results.

To solve this, the esArchiver could call an internal SavedObjects API to retrieve all registered types, or the esArchiver could call an internal SavedObjects rerunMigrations API. This would also reduce the amount of duplication inside the esArchiver which is brittle to maintain because it's still in JS.

[pgayvallet]

To solve this, the esArchiver could call an internal SavedObjects API to retrieve all registered types

Implementation depends on #55857, but as the esArchiver is not run from inside kibana, and as some SO-related internals are not serializable (thinking mostly of migrations), I don't think this is an option

or the esArchiver could call an internal SavedObjects rerunMigrations API. This would also reduce the amount of duplication inside the esArchiver which is brittle to maintain because it's still in JS.

Just to be sure to understand, as the esArchiver does not run from inside the kibana server during FTR, by internal SavedObjects API you mean an actual internal HTTP endpoint that esarchiver could call?

[rudolf]

Even though the esArchiver runs in a separate process from the Kibana FTR server, it could start it's own Kibana server just for migrations which would give it access to core's setup/start API's. Similar to how we start the server in test_utils/kbn_server.ts

kibana/src/test_utils/kbn_server.ts

Lines 287 to 290 in 205fbce

	const root = createRootWithCorePlugins(kbnSettings);
	await root.setup();
	await root.start();

This internal server would have to use the same configuration as the FTR server (but listen on a different http port). The disadvantage would be that we potentially have a slower startup for test suites but since these two processes can run in parallel it might be negligible...

Another option would be to use IPC between the FTR parent process and the kibana server child process to send a "rerun migrations" message.

I don't know the internals of the FTR very well so there might be some complications I didn't consider. But I think it's worth exploring the options so that we don't design the API around our testing infrastructure.

[pgayvallet]

Even though the esArchiver runs in a separate process from the Kibana FTR server, it could start it's own Kibana server just for migrations which would give it access to core's setup/start API's. Similar to how we start the server in test_utils/kbn_server.ts

If we go in this direction, I think it would be better to have a specific method on our root server (or somewhere else) that would performs only what is required for the migration (doing setup, but only performing the migration and not doing the whole server start lifecycle) to avoid booting up the whole thing (the http server port you mention is an issue, but I think there would be others).

I agree that avoiding exposing test-purposes APIs should be avoided.

I don't really like giving the capability to the es-archiver to boot a whole kibana/core Server, as it means heavy changes in the FTR archiver service instantiation (and we still need to confirm actual doability of this proposition), however I'm not sure I see a better thing to try right now.

ping @spalger @joshdover ^ your insight is welcome.

[pgayvallet]

TIL: we actually already have a SO util for FTR in packages/kbn-dev-utils/src/kbn_client/kbn_client_saved_objects.ts, which is used by the archiver. It's currently only using existing APIs to perform get/create/update calls on saved objects.

However I have the feeling that adding an additional, internal API endpoint to perform a savedObject migration and using the test-util KbnClient to call the endpoint may be the quickest win if we don't find another proper solution...

[joshdover]

However I have the feeling that adding an additional, internal API endpoint to perform a savedObject migration and using the test-util KbnClient to call the endpoint may be the quickest win if we don't find another proper solution...

This was going to be my suggestion. If we go this route, it would be great if this endpoint is only available if some CLI flag is passed to the Kibana server. While it is the "quick-win" solution, I don't think it's necessarily a bad one.

Another option would be to use the SavedObjects REST API to import the documents that EsArchiver needs to create. Since all documents that go through the import API are migrated by the Kibana server, this should solve the problem. This would only be necessary for data that is imported into the .kibana index. I don't know much about the internals of EsArchiver so I can't speak to what kind of effort this would require.

[pgayvallet]

If we go this route, it would be great if this endpoint is only available if some CLI flag is passed to the Kibana server. While it is the "quick-win" solution, I don't think it's necessarily a bad one.

I know some where not big fan of 'test-only' flags, as it means different behavior during FTR (which are supposed to run in 'pure' production mode), and actual prod mode. But I agree that exposing that kind of endpoint in production should (really) be avoided.

Another option would be to use the SavedObjects REST API to import the documents that EsArchiver needs to create.

That would be great. However it seems the es archiver is actually dumping whole indexes instead of only SO documents (see test/functional/fixtures/es_archiver/kibana_sample_data_flights_index_pattern/mappings.json for example), so I'm not sure this is an option.

@elastic/kibana-operations @tylersmalley Can we get some insight / help here ?

[tylersmalley]

My feeling is the proper solution would be to remove the need for migrations to be ran before functional tests. I have long disliked that the es_archive's were gunziped for multiple reasons - if we kept these in plain text, provided tooling to update them when new migrations are added, could we do away with needing to run migrations at all in functional tests?