`#`-as-delimiter woes

[kdm9]

Hello Erik,

I'm asking this here as to be honest I'm not sure whether or not this is a bug in Augustus, an oversight in PanSN, or just an unfortunate edge case in both.

We've come across at least one tool that treats # in a contig name as a start-of-inline-comment marker (specifically Augustus's perl code). IMO that's broken code, but I imagine that's not the only piece of software that treats # pretty sloppily.

Would you consider making the default delimiter something like ! or ~, which are less likely to be interpreted as a comment? Ultimately, there's no single printable ASCII char I can think of that every single tool will accept as a valid contig name, but that isn't already used very frequently in either individual or contig names, so I think we're kinda screwed regardless. That said, I don't think ! or ~ are comment markers in any language I know, I've never seen one in a sample name (though I'm sure someone has), and are valid in unix paths.

Best,
Kevin

[colindaven]

I would also rather use anything other than # which is likely a huge problem as seen for perl and python scripts.

! or ~ are better, from my point of view.

I have never, ever seen § used as a special character anywhere in a programming language so that might be useful (just my two cents).

But please, # is not a good choice as a default IMO.

cheers

[kdm9]

Having played around a bunch now, I can recommend not using !, as this triggers bash's history expansion if placed in double quotes. § would be great, but I don't think it's an ASCII 7-bit character (see either man ascii or wiki). I think we've settled on using ~ internally, but we'll see if that causes major issues as we roll this out across the whole pipeline.

As I said above though, this is likely a fool's errand as there'll always be something that breaks...

[ekg]

~ is an interesting solution. So we are between that and #?

…

On Sat, Jul 23, 2022, 10:56 Dr. K. D. Murray ***@***.***> wrote: Having played around a bunch now, I can recommend not using !, as this triggers bash's history expansion if placed in double quotes. § would be great, but I don't think it's an ASCII 7-bit character (see either man ascii or wiki <https://en.wikipedia.org/wiki/ASCII#Character_set>). I *think* we've settled on using ~ internally, but we'll see if that causes major issues as we roll this out across the whole pipeline. As I said above though, this is likely a fool's errand as there'll always be *something* that breaks... — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDQELYGQ3M7U2XJTS2CLDVVOXSHANCNFSM534JVSDA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[kdm9]

@ekg I think so, yes. I don't know of any common situation in which ~ will backfire. I've used ~ for years now to separate wildcards when building paths in Snakemake (e.g. output/bams/{aligner}~{reference}~{sample}.bam to avoid ambiguity when aligner, ref, or sample have e.g. a . or _ in them). Since it's never at the start of a string/path it shouldn't be expanded to $HOME (unless there are some truly daft s/~/$HOME/-style bugs out there).

I've just changed all our references to use ~ and in the coming weeks we'll run a full annotation pipeline, so I'll let you know if there are any particularly hairy edge cases. I've also mostly written some python code that can safely regex backward and forward from e.g. at9900~1~chr1 <-> chr1 in a filetype-aware manner, so that should solve the last edge cases where something just refuses to accept PanSN names. I'll release it publicly when it's done and post it here.

[kdm9]

To follow this up: we gave up on ~ as well, as it needs percent-quoting in GFF files (see the spec). Now, we're using _ and I have code to add and removed pansn names automatically from inputs and outputs.