split-vep: column type can't be set for custom annotation fields

[sbaross]

Continuing from this post: #1651 about the "Error: cannot use arithmetic operators to compare strings and numbers" error.

After upgrading to v1.16 I am still unable to filter by certain columns, after further testing I have figured out that the error occurs when filtering fields which have been added as custom annotations [see here]. gnomAD allele frequencies are often added by custom annotations as the built-in option in VEP does not use the newer WGS data.

I have used a slightly altered version of the sample data from the bottom of that page as an example here (edited only to add a contig definition): vep_eg.vcf.gz. The ClinVar field is a added via a custom annotation, while the cDNA_position field is a standard VEP field.

bcftools +split-vep -d -c ClinVar:Int,cDNA_position:Int -f "%CHROM\t%POS\t%ClinVar\t%cDNA_position\n" vep_eg.vcf.gz
1	230710048	18068	1018
1	230710048	18068	.

When filtering by a field added as a standard VEP option, such as cDNA_position it works as expected:

bcftools +split-vep -d -c ClinVar:Int,cDNA_position:Int -f "%CHROM\t%POS\t%ClinVar\t%cDNA_position\n" vep_eg.vcf.gz -i "cDNA_position>1000"
1	230710048	18068	1018

but fields from custom annotations such as ClinVar do not:

bcftools +split-vep -d -c ClinVar:Int,cDNA_position:Int -f "%CHROM\t%POS\t%ClinVar\t%cDNA_position\n" vep_eg.vcf.gz -i "ClinVar>1000"
Error: cannot use arithmetic operators to compare strings and numbers

I believe this is due to the fact that custom annotation fields are also defined individually in the VCF header, as well as in the CSQ definition, and are listed as Type=String as default.

##INFO=<ID=CSQ,Number=.,Type=String,Description="Consequence annotations from Ensembl VEP. Format: Allele|Consequence|IMPACT|SYMBOL|Gene|Feature_type|Feature|BIOTYPE|EXON|INTRON|HGVSc|HGVSp|cDNA_position|CDS_position|Protein_position|Amino_acids|Codons|Existing_variation|DISTANCE|STRAND|FLAGS|SYMBOL_SOURCE|HGNC_ID|SOURCE|ClinVar|ClinVar_CLNSIG|ClinVar_CLNREVSTAT|ClinVar_CLNDN">
##INFO=<ID=ClinVar,Number=.,Type=String,Description="/path/to/custom_files/clinvar.vcf.gz (exact)">

If I manually change the type in the VCF header to Type=Integer for the ClinVar field filtering works as expected. I am unsure if it possible to change this when adding custom annotations, but as it is not even mentioned in the custom annotation page, the vast majority of pre-existing files have all custom annotation fields listed as string so it would be beneficial to have the type set in split-vep override the VCF header.

Relatedly, while testing this I have also realised it doesn't seem to be possible to filter for missing values for numeric fields within the VEP CSQ field. This is especially important for annotations like gnomAD allele frequencies as a missing value means the variant has not been observed in the dataset so is essentially equivalent to AF=0. The expressions-i "cDNA_position='.'" or -i "cDNA_position=''" both work with cDNA_position:String but not with cDNA_position:Int

[pd3]

You are right about the cause of the problem: since the tag ClinVar is already defined in the header, the program will not override the existing definition. This is to preserve the integrity of the VCF. Imagine there are ClinVar=SomeTextBlahBlah tags in the VCF body and some of these records would not have the CSQ annotation. Then changing the definition in the header would make some of the records inconsistent with the header. Therefore the recommended way would be to first remove the existing tags with bcftools annotate -x INFO/ClinVar (or bcftools reheader, if you are sure no such tags appear in the body) and then the +split-vep command will work.

[sbaross]

Thanks, reheader worked well. It would be helpful if this was explained on the split-vep plugin page as there's no indication currently that this will be an issue for all non-string fields added as custom annotations by default

[pd3]

I added a note about this in the description, hopefully this will make it clear

-c, --columns [LIST|-][:TYPE]  
    Extract the fields listed either as 0-based indexes or names, "-" to extract all
    fields. See --columns-types for the defaults. Supported types are String/Str,
    Integer/Int and Float/Real. Unlisted fields are set to String. Existing header
    definitions will not be overwritten, remove first with `bcftools annotate -x`