Add string representation for config and config override

[peitili]

Background

In order to better debug what's inside the config, add a string repr for the Configuration object.

Also in order to provide a way to override certain config values from triggering time, add a few arguments to some tile commands so that those config values can be provided by the trigger.

Test steps

Temporarily modify the tilequeue_main to print out the cfg variable after make_config_from_argparse call. Then in local virtual env do pip install -e .

Then trigger the command by

tilequeue meta-tile-low-zoom --config /Users/pli/config/meta-low-zoom-batch-config.yaml --tile '7/20/44' --run_id=20210426-peiti --postgresql_host '["host.docker.internal"]' --postgresql_dbnames '["gis"]' --postgresql_user gisuser --postgresql_password VHcDuAS0SYx2tlgT --store_name '["my-meta-tiles-us-east-1"]' --store_date_prefix 20210426-peiti --batch_check_metafile_exists false

then I got the config printed out correctly:

yml: {'rawr': {'group-zoom': 10}, 'tiles': {'seed': {'all': {'zoom-until': None, 'zoom-start': None}, 'unique': True, 'n-threads': 50, 'should-add-to-tiles-of-interest': True, 'metro-extract': {'url': None, 'zoom-until': None, 'cities': None, 'zoom-start': None}, 'custom': {'zoom-until': None, 'zoom-start': None, 'bboxes': []}, 'top-tiles': {'url': None, 'zoom-until': None, 'zoom-start': None}}, 'max-zoom-with-changes': 16, 'intersect': {'expired-location': None, 'parent-zoom-until': None}}, 'queue_buffer_size': {'s3': None, 'proc': None, 'sql': None}, 'process': {'reload-templates': False, 'log-queue-sizes': True, 'yaml': {'parse': {'path': '/usr/src/vector-datasource/yaml'}, 'callable': {'dotted-name': ''}, 'type': 'parse'}, 'n-simultaneous-s3-storage': 0, 'formats': ['coanacatl'], 'buffer': {}, 'template-path': '/usr/src/vector-datasource/queries', 'log-queue-sizes-interval-seconds': 10, 'n-simultaneous-query-sets': 0, 'query-config': '/usr/src/vector-datasource/queries.yaml'}, 'use-rawr-tiles': False, 'aws': {'credentials': {'aws_access_key_id': None, 'aws_secret_access_key': None}}, 'toi-store': {'type': None}, 'postgresql': {'user': 'gisuser', 'host': ['host.docker.internal'], 'password': 'VHcDuAS0SYx2tlgT', 'port': 5432, 'dbnames': ['gis']}, 'logging': {'config': '/etc/tilequeue/logging.conf'}, 'toi-prune': {'tile-traffic-log-path': '/tmp/tile-traffic.log'}, 'metatile': {'tile-sizes': [512], 'start-zoom': 0, 'size': 8}, 'redis': {'type': 'redis_client', 'host': 'localhost', 'db': 0, 'port': 6379, 'cache-set-key': 'tilequeue.tiles-of-interest'}, 'batch': {'queue-zoom': 7, 'check-metatile-exists': False}, 'queue': {'timeout-seconds': 20, 'type': 'sqs', 'name': None}, 'store': {'name': ['my-meta-tiles-us-east-1'], 'delete-retry-interval': 60, 'type': 's3', 'tags': {}, 'path': None, 'reduced-redundancy': False, 'object-acl': 'private', 'date-prefix': '20210426-peiti', 'key-format-type': 'hash-prefix'}},
aws_access_key_id: None,
aws_secret_access_key: None,
queue_cfg: {'timeout-seconds': 20, 'type': 'sqs', 'name': None},
store_type: s3,
s3_bucket: ['my-meta-tiles-us-east-1'],
s3_reduced_redundancy: False,
s3_path: None,
s3_date_prefix: 20210426-peiti,
s3_delete_retry_interval: 60,
seed_all_zoom_start: None,
seed_all_zoom_until: None,
seed_n_threads: 50,
seed_metro_extract_url: None,
seed_metro_extract_zoom_start: None,
seed_metro_extract_zoom_until: None,
seed_metro_extract_cities: None,
seed_top_tiles_url: None,
seed_top_tiles_zoom_start: None,
seed_top_tiles_zoom_until: None,
seed_top_tiles_url: None,
toi_store_type: None,
toi_store_s3_bucket: None,
toi_store_s3_key: None,
toi_store_file_name: None,
seed_custom_zoom_start: None,
seed_should_add_to_tiles_of_interest: True,
seed_custom_zoom_until: None,
seed_unique: True,
intersect_expired_tiles_location: None,
intersect_zoom_until: None,
logconfig: /etc/tilequeue/logging.conf,
redis_type: redis_client,
redis_host: localhost,
redis_port: 6379,
redis_db: 0,
redis_cache_set_key: tilequeue.tiles-of-interest,
statsd_host: None,
statsd_port: None,
statsd_prefix: None,
n_simultaneous_query_sets: 0,
n_simultaneous_s3_storage: 0,
log_queue_sizes: True,
log_queue_sizes_interval_seconds: 10,
query_cfg: /usr/src/vector-datasource/queries.yaml,
template_path: /usr/src/vector-datasource/queries,
reload_templates: False,
output_formats: ['coanacatl'],
buffer_cfg: {},
process_yaml_cfg: {'parse': {'path': '/usr/src/vector-datasource/yaml'}, 'callable': {'dotted-name': ''}, 'type': 'parse'},
postgresql_conn_info: {'user': 'gisuser', 'host': ['host.docker.internal'], 'password': 'VHcDuAS0SYx2tlgT', 'port': 5432, 'dbnames': ['gis']},
metatile_size: 8,
metatile_zoom: 3,
metatile_start_zoom: 0,
max_zoom_with_changes: 16,
max_zoom: 13,
sql_queue_buffer_size: None,
proc_queue_buffer_size: None,
s3_queue_buffer_size: None,
tile_traffic_log_path: /tmp/tile-traffic.log,
group_by_zoom: 10,
tile_sizes: [512]

[iandees]

Looks good if you could tweak the arg names across all the commands.

Also, weren't these passed in via environment variables before? Is the goal to not support those anymore?

[iandees]

If this is a list, this param should be called postgresql_hosts?

[peitili]

If this is a list, this param should be called postgresql_hosts?

yup I will change to plural form, I just mimic the existing environment variable naming.

[iandees]

I think we've been trying to change "date_prefix" to "build_id" elsewhere because they don't have to be date-related.

[peitili]

I think we've been trying to change "date_prefix" to "build_id" elsewhere because they don't have to be date-related.

we can probably change that to "build_id" in a separate PR since the renaming may require more changes.

[peitili]

Looks good if you could tweak the arg names across all the commands.

I am not sure whether other command need them, shall we add them on a as-need basis later?

Also, weren't these passed in via environment variables before? Is the goal to not support those anymore?

These were passed in via environment variables and will continue to work. It is just that for the Go service with shelling out to python, we cannot use environment variables since that will have race condition.

[peitili]

If this is a list, this param should be called postgresql_hosts?

yup I will change to plural form, I just mimic the existing environment variable naming.

[peitili]

I think we've been trying to change "date_prefix" to "build_id" elsewhere because they don't have to be date-related.

we can probably change that to "build_id" in a separate PR since the renaming may require more changes.