Closed
Description
Currently, if there is a header schema mismatch (e.g. a FORMAT field is missing from the header), the pipeline will proceed all the way until the very end and only fails when it tries to write to BigQuery. It also does not give a representative error message (complains that a field is missing in BQ JSON schema, see Issue #61). We should catch such errors sooner and give proper error. Depends on Issue #59 as we need to first refactor that part of the code.
Ideally we may be able to 'detect and correct' such errors, but this is more difficult currently as the BQ schema cannot be dynamically created/changed.