More helpful error when function is missing

[jacobwhall]

Thanks for developing martin, it is a pleasure to use.

I recently spun up martin, asking it to serve a function that did not exist. An error reported that the schema public was not found, with an empty list of possible values:

[2023-10-25T05:27:45Z WARN  martin::pg::utils] Unable to configure source foot because schema 'public' was not found.  Possible values are:

I suggest that a more descriptive/helpful error be returned, indicating that the function itself may be missing.

[sharkAndshark]

I duplicate it with a yaml like this and martin list all the schemas I already created:

  functions:
    function_not_exist:
      schema: not_exist_schema 
      function: function_not_exist

Unable to configure source function_not_exist because schema 'not_exist_schema' was not found.  Possible values are: MixedCase, public

How about update the warning to

Unable to configure source "source1" because the associated function "foo"."bar"  was not found. Please ensure that the schema and function have been created. In case misspelling in the config file, possible schemas values: schema1, schema2

I'm not experienced with logs，any suggestions to improve it? @nyurik @jacobwhall

[nyurik]

I think the reason you got that message is because we do not differentiate between these two cases:

• schema does not exist
• schema exists, but it has no functions that match expected _____(z,x,y) -> blob pattern

To make the message better, we will need to add another PG query to find all available schemas, and then when parsing, use that to make it a bit more helpful. Alternatively, a simpler solution could be to explain that in a message:

Unable to configure source 'src' because function 'schema.func' was not found.
The following schemas with usable function were found: schema1, schema2

@jacobwhall would that have helped in your case?

[jacobwhall]

schema exists, but it has no functions that match expected _____(z,x,y) -> blob pattern

Yes, this is the case I hope to clarify. I think the easiest solution is to suggest that the schema might exist without the specified function. For example:

Unable to configure source 'src' because a schema 'public' with function 'func' was not found.

This would suggest to the user that either the schema or the function could be missing / misspelled / etc.

@nyurik, your suggestion of adding a list of possible options would be even better. If another query is added, it might be best to search for usable functions within the specified schema. Building off your example:

Unable to configure source 'src' because function 'schema.func' was not found.
The following usable functions in schema 'schema' were found: schema.func1, schema.func2

Thank you both for your replies. I hope to contribute more substantially to martin in the future.

[nyurik]

Suggestions is a bit trickier because the user may have autodiscover enabled, in which case all those functions will be added as sources anyway, unless they also filter by schema. Listing all available functions in that case would be useless. I think we should start with simply improving the existing message. The only problem is that it uses a generic function that does all sorts of lookups - i.e. lookups among schemas, among functions, among tables, etc.

Ok, I just looked at the code (below), and it seems we actually have the handling of your case, but it never gets triggered, because an empty array is never returned! So we should simply modify query_available_function to contain all schemas, and not just the ones that have relevant functions, and we should be good!

Initiating code:

martin/martin/src/pg/configurator.rs

Lines 244 to 254 in 94f2c16

	for (id, cfg_inf) in &self.functions {
	let Some(db_funcs) = find_info(&db_funcs_info, &cfg_inf.schema, "schema", id) else {
	continue;
	};
	if db_funcs.is_empty() {
	warn!("No functions found in schema {}. Only functions like (z,x,y) -> bytea and similar are considered. See README.md", cfg_inf.schema);
	continue;
	}
	let Some((pg_sql, _)) = find_info(db_funcs, &cfg_inf.function, "function", id) else {
	continue;
	};

Code that actually prints the message:

martin/martin/src/pg/utils.rs

Lines 104 to 109 in 94f2c16

	match find_kv_ignore_case(map, key) {
	Ok(None) => {
	warn!("Unable to configure source {id} because {info} '{key}' was not found. Possible values are: {}",
	map.keys().map(String::as_str).collect::<Vec<_>>().join(", "));
	None
	}

[nyurik]

Anyone wants to add a simple SQL query to fix this? Should be an easy change to the code:

• Add a new SQL file - martin/src/pg/scripts/query_schemas.sql that simply returns a list of all schemas (as a single column table)
• modify PgBuilder
  • add a new field schemas: Vec<String>
  • in PgBuilder::new run the above query, add store results into schemas field
  • add a new helper function that adds all missing schemas to a hashmap, and call that function inside instantiate_functions and instantiate_tables right after calling query_available_function. This way db_funcs_info will contain empty HashMaps for all schemas if they don't have relevant functions

If you only feel comfortable with writing the SQL query, I can do the rest too, but we are always welcoming new contributors :)

[nyurik]

Did some digging into this:

tests/fixtures/functions/unsupported_function.sql

DROP FUNCTION IF EXISTS other_schema.not_matching_func;

-- missing "zoom" value
CREATE OR REPLACE FUNCTION other_schema.not_matching_func(x integer, y integer) RETURNS bytea AS $$
SELECT ('123' || x || y)::bytea
$$ LANGUAGE SQL;

tests/fixtures/tables/unsupported_table.sql

DROP SCHEMA IF EXISTS other_schema CASCADE;
CREATE SCHEMA other_schema;

CREATE TABLE other_schema.table_without_geo
(
    id   SERIAL PRIMARY KEY,
    name TEXT
);

INSERT INTO other_schema.table_without_geo
values (1, 'foo'),
       (2, 'bar');

myconfig.yaml

postgres:
  connection_string: postgres://postgres:postgres@localhost:5411/db
  tables:
    test_good_schema_bad_table:
      schema: other_schema
      table: table_without_geo
      srid: 4326
      geometry_column: Geom
    test_good_schema_missing_table:
      schema: other_schema
      table: no_such_table
      srid: 4326
      geometry_column: Geom
    test_bad_schema_bad_table:
      schema: invalid_schema
      table: invalid_table
      srid: 4326
      geometry_column: Geom
  functions:
    test_good_schema_bad_func:
      schema: other_schema
      function: not_matching_func
    test_good_schema_missing_func:
      schema: other_schema
      function: missing_func
    test_bad_schema_bad_func:
      schema: invalid_schema
      function: invalid_func

Testing

just start
martin myconfig.yaml

Current Output

########### these are ok, and should not be changed
[... WARN  martin::pg::utils] Unable to configure source test_bad_schema_bad_table because schema 'invalid_schema' was not found.  Possible values are: MixedCase, autodetect, public
[... WARN  martin::pg::utils] Unable to configure source test_bad_schema_bad_func because schema 'invalid_schema' was not found.  Possible values are: MixedCase, public

########### these should be modified
[... WARN  martin::pg::utils] Unable to configure source test_good_schema_bad_table because schema 'other_schema' was not found.  Possible values are: MixedCase, autodetect, public
[... WARN  martin::pg::utils] Unable to configure source test_good_schema_bad_func because schema 'other_schema' was not found.  Possible values are: MixedCase, public
[... WARN  martin::pg::utils] Unable to configure source test_good_schema_missing_table because schema 'other_schema' was not found.  Possible values are: MixedCase, autodetect, public
[... WARN  martin::pg::utils] Unable to configure source test_good_schema_missing_func because schema 'other_schema' was not found.  Possible values are: MixedCase, public

Desired Output

This is not have to be exact, but should give some ideas to implementers

Table exists, but has no geometries

Unable to configure source test_good_schema_bad_table because the table 'table_without_geo' has no geometry columns. Other valid tables are: ...

---or---

Unable to configure source test_good_schema_bad_table because the table 'table_without_geo' has no geometry columns. No valid tables found in schema 'other_schema'.

Table does not exist

Unable to configure source test_good_schema_missing_table because table 'no_such_table' does not exist in schema 'other_schema'. Possible values are: ...

---or---

Unable to configure source test_good_schema_missing_table because table 'no_such_table' does not exist in schema 'other_schema'. No valid tables found in schema 'other_schema'.

Function exists, but does not have recognizable signature

Unable to configure source test_good_schema_bad_func because function 'not_matching_func' does not have the expected signature like (z,x,y) -> bytea. See docs for other supported variants. Other valid functions are: ...

Unable to configure source test_good_schema_missing_func because function 'missing_func' does not exist in the schema 'other_schema'. Possible values are: ...