Bug: cannot open old database (created with 2.9.2) with new version (2.12.0)

[zh217]

Describe the bug

Newer version cannot open database created with older versions of the library.

Steps to reproduce

With the python client version 2.12.0:

usearch.index.Index.restore(idx_path, view=True)

results in

ValueError: Unsupported metric!

where the datafile was created with version 2.9.2 with:

usearch.index.Index(ndim=1024, metric='ip')

Version 2.9.2 can open the datafile without problems.

On further testing, all versions from 2.10.0 onwards fail to open the database.

Expected behavior

Version 2.12.0 should be able to open database created with version 2.9.2, as the version numbers do not indicate any breaking changes.

USearch version

v2.12.0

Operating System

Ubuntu 22.04

Hardware architecture

x86

Which interface are you using?

Python bindings

Contact Details

No response

Are you open to being tagged as a contributor?

• I am open to being mentioned in the project .git history as a contributor

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[ashvardanian]

There were no changes in the file format, but the number of checks and assertions grew. Apparently, one of those checks is hurting us here.

Does it also fail if you create an arbitrary index, and then call .load - reinitializing it with a different file?

[zh217]

It fails with a different error:

RuntimeError: Key type doesn't match, consider rebuilding

triggered by the following code:

idx = usearch.index.Index(ndim=1024, metric='ip')
idx.load(idx_path)

which runs fine if downgraded to 2.9.2.

[ashvardanian]

Interesting. Any chance the file was corrupted somewhere in between?

[zh217]

No. Here is a minimal example that you can test:

# run with usearch-2.9.2 installed

import usearch.index

idx = usearch.index.Index(ndim=1024, metric='ip')
idx.save('index')

# run with usearch-2.12.0 installed

import usearch.index

# will throw an error in usearch-2.12.0
idx = usearch.index.Index.restore('index', view=True)

There's no need to insert anything into the database in order to trigger the error. Should be that the metadata in the old version is messed up.

[zh217]

Update: this works both ways --- old version cannot open databases created by the new version either.

[zh217]

There were no changes in the file format, but the number of checks and assertions grew. Apparently, one of those checks is hurting us here.

In fact the file format changed due to a subtle change in code.

Compare:

usearch/include/usearch/index_plugins.hpp

Lines 122 to 142 in 5ea48c8

	enum class scalar_kind_t : std::uint8_t {
	unknown_k = 0,
	// Custom:
	b1x8_k = 1,
	u40_k = 2,
	uuid_k = 3,
	// Common:
	f64_k = 10,
	f32_k = 11,
	f16_k = 12,
	f8_k = 13,
	// Common Integral:
	u64_k = 14,
	u32_k = 15,
	u16_k = 16,
	u8_k = 17,
	i64_k = 20,
	i32_k = 21,
	i16_k = 22,
	i8_k = 23,
	};

with:

usearch/include/usearch/index_plugins.hpp

Lines 128 to 148 in f79d818

	enum class scalar_kind_t : std::uint8_t {
	unknown_k = 0,
	// Custom:
	b1x8_k,
	u40_k,
	uuid_k,
	// Common:
	f64_k,
	f32_k,
	f16_k,
	f8_k,
	// Common Integral:
	u64_k,
	u32_k,
	u16_k,
	u8_k,
	i64_k,
	i32_k,
	i16_k,
	i8_k,
	};

so different versions interpret enums in the metadata differently.

As the metadata stored on disk also has version information, we can make new version of the library open old databases by mapping the old values to the new values. There seems to be no easy fix for the reverse direction, however.

As this definitely breaks compatibility between versions (affecting all f16, f32, f64 indices and all languages), this should be marked as a breaking change.

[zh217]

We can localize the damage by changing what is returned by this function:

usearch/include/usearch/index_dense.hpp

Lines 176 to 236 in 5ea48c8

	inline index_dense_metadata_result_t index_dense_metadata_from_path(char const* file_path) noexcept {
	index_dense_metadata_result_t result;
	std::unique_ptr<std::FILE, int (*)(std::FILE*)> file(std::fopen(file_path, "rb"), &std::fclose);
	if (!file)
	return result.failed(std::strerror(errno));
	// Read the header
	std::size_t read = std::fread(result.head_buffer, sizeof(index_dense_head_buffer_t), 1, file.get());
	if (!read)
	return result.failed(std::feof(file.get()) ? "End of file reached!" : std::strerror(errno));
	// Check if the file immediately starts with the index, instead of vectors
	result.config.exclude_vectors = true;
	if (std::memcmp(result.head_buffer, default_magic(), std::strlen(default_magic())) == 0)
	return result;
	if (std::fseek(file.get(), 0L, SEEK_END) != 0)
	return result.failed("Can't infer file size");
	// Check if it starts with 32-bit
	std::size_t const file_size = std::ftell(file.get());
	std::uint32_t dimensions_u32[2]{0};
	std::memcpy(dimensions_u32, result.head_buffer, sizeof(dimensions_u32));
	std::size_t offset_if_u32 = std::size_t(dimensions_u32[0]) * dimensions_u32[1] + sizeof(dimensions_u32);
	std::uint64_t dimensions_u64[2]{0};
	std::memcpy(dimensions_u64, result.head_buffer, sizeof(dimensions_u64));
	std::size_t offset_if_u64 = std::size_t(dimensions_u64[0]) * dimensions_u64[1] + sizeof(dimensions_u64);
	// Check if it starts with 32-bit
	if (offset_if_u32 + sizeof(index_dense_head_buffer_t) < file_size) {
	if (std::fseek(file.get(), static_cast<long>(offset_if_u32), SEEK_SET) != 0)
	return result.failed(std::strerror(errno));
	read = std::fread(result.head_buffer, sizeof(index_dense_head_buffer_t), 1, file.get());
	if (!read)
	return result.failed(std::feof(file.get()) ? "End of file reached!" : std::strerror(errno));
	result.config.exclude_vectors = false;
	result.config.use_64_bit_dimensions = false;
	if (std::memcmp(result.head_buffer, default_magic(), std::strlen(default_magic())) == 0)
	return result;
	}
	// Check if it starts with 64-bit
	if (offset_if_u64 + sizeof(index_dense_head_buffer_t) < file_size) {
	if (std::fseek(file.get(), static_cast<long>(offset_if_u64), SEEK_SET) != 0)
	return result.failed(std::strerror(errno));
	read = std::fread(result.head_buffer, sizeof(index_dense_head_buffer_t), 1, file.get());
	if (!read)
	return result.failed(std::feof(file.get()) ? "End of file reached!" : std::strerror(errno));
	// Check if it starts with 64-bit
	result.config.exclude_vectors = false;
	result.config.use_64_bit_dimensions = true;
	if (std::memcmp(result.head_buffer, default_magic(), std::strlen(default_magic())) == 0)
	return result;
	}
	return result.failed("Not a dense USearch index!");
	}

Since the result is returned in various places inside the function, maybe it is best to add a method on index_dense_metadata_result_t to "upgrade" its version to the new enum by mutating its headers appropriately.

I can make a pull request for it if that's OK.

[ashvardanian]

Good catch @zh217! I think a good solution would be a custom function to convert enum to integer and vice-versa, with respect to the file version. Can you add it in index_plugins?

[brittlewis12]

@ashvardanian any update on introducing the backwards compatibility for pre-2.10 indexes in #438?

this would be valuable to me to avoiding recomputing all old indexes, but if that’s not expected to be introduced I will go ahead and do that

[ashvardanian]

Working on it today.