Allow parenthesis to be used in search strings in boolean filters

[tilmanbergt]

⚠️ Please check that this feature request has NOT been suggested before.

• I searched previous Ideas in Discussions didn't find any similar feature requests.
• I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

Use Case: I create tasks queries programmatically via dataviewjs using the following script:

function getTaskList(includedtasks) {
	const tasklist = dv.array(includedtasks).text.map((t) => "(description includes " + t +")").join(" OR ");
	const query = `
not done
${tasklist}
`;
	let returnvalue = '```tasks\n' + query + '\n```\n';
	return returnvalue;
}

This works pretty fine to return some fragment I can then insert via dv.paragraph anywhere to get fully functioning tasks based on a array of tasks. The only situations in which it currently fails is when the task description contains brackets ("(", ")"), because they interfere with the syntax for creating "OR" queries in tasks.

I did not find any mention of escaping them by any means. Are there any? Otherwise I would propose to allow some escaping.

Please let me know any other approach to this scenario that could help.
Thank you so much for considering this request.

✔️ Solution

e.g. ignore brackets that are prefixed with "" allowing something like .replace("(","\(") to be used.

❓ Alternatives

Another approach would be to allow lines to start with "OR", but this would probably insert too much ambiguity.

📝 Additional Context

As background: I have quite elaborate rules for displaying certain tasks that can not easily be covered by the tasks syntax itself (particularly making sure tasks already shown in some section are not again shown in a different section).

[claremacrae]

Hi @tilmanbergt,

The only thing I can suggest right now is to experiment with double quotes around your search expressions.

In the documentation of boolean queries - https://obsidian-tasks-group.github.io/obsidian-tasks/queries/combining-filters/#syntax - I recommended against using them as equivalents to parentheses. I don't remember what happened in testing, exactly, that prompted me to say that.

However, you might find, for example, that if you had a search string that had () and no double quotes, that surrounding the filter with " may work.

Please could you report back on what you find.

Ohhh... the other thing you could try is to use a regular expression search.
https://obsidian-tasks-group.github.io/obsidian-tasks/queries/regular-expressions/
Although constructing a general regular expression from arbitrary text is non-trivial, I don't think it would confuse the boolean filter code in the same way that () would in plain includes searches.

[claremacrae]

A code-based option is to make the Boolean filter parsing less greedy, so that it ignores parens that are not next to one of the Boolean operators AND OR etc. That might solve this problem without needing to add the ability to escape characters in filters.

[tilmanbergt]

Hi @claremacrae, thanks so much for responding to this. I already experimented a little with quotes it seems to make things rather worse, causing every query to say: Tasks query: do not understand query: "description includes #next work on my demons on this"
Also it would be some awkward code to make this destinction based on what occurs inside the string. Still it was worthwhile to try once more.
I also thought about the regular expression approach already, but feel again that this would bring much more problems, as there are so many things I would need to escape (though in that case at least I could escape everything, I suppose). Also it might slow things done unnecessarily.
So it is not a huge impact, I don't have parentheses so often in my task descriptions and can probably try to avoid them. Perhaps I will just remove those from the tasklist and manually output them without using tasks.

Have now the following approach which works (but is of course a workaround):

function getTaskList(includedtasks) {	
	let returnvalue = "";
	const wrongones = dv.array(includedtasks).text.where(t => t.includes("(") || t.includes(")"));
	if (wrongones.length>0)
	returnvalue += "Following Tasks contains parenthesis: \n" + wrongones.join("\n")+"\n";
	const rightones = dv.array(includedtasks).text.where(s => !s.includes("(") && !s.includes(")"));
	if (rightones.length>0) {
		const tasklist = rightones.map((t) => "(description includes " + t +")").join(" OR ");
	const query = `
not done
${tasklist}
`;
returnvalue += '```tasks\n' + query + '\n```\n';
}
	return returnvalue;
}

[claremacrae]

Thanks for the reply.

For future reference, any information on what you have already considered or tried is always very helpful in feature requests and bug reports, both to add context, and to save spending time writing up things that are already known or already tried.

I doubt very much indeed that using a regular expression would make a perceptible difference to the search speed, so I would encourage you to try that.

just exploring an actual fix, the description of this feature request is currently to allow escape characters in description searches, which is quite a general thing. But I wonder, would actually the ability to use ( or ) in Boolean filters in text searches be sufficient? Am asking as I think it is more specific and maybe likely more doable.

[claremacrae]

In the meantime, another workaround is to split descriptions on ( and ) and have use AND to combine the components….

(a AND b AND C) OR d or E

[tilmanbergt]

yes, sorry for not including that in my initial description.

and also thank you for the second work around idea. I will see if this can be done, looks like a promising approach.

Also I might look into the regular expressions, but somehow have worked not enough with them recently to have a good feeling of side effects.

And to your question: yes: I will rename the title into allowing parenthesis in boolean filters.

[claremacrae]

Hi @tilmanbergt

In case it helps you convert your search to using regular expressions...

Need to escape ( and other characters with special meaning

The issue with taking any arbitrary search text and searching for it using regular expressions is that various ASCII characters that might be present in the search string have special meanings in regular expressions, and so need to be "escaped", to remove their special meaning.

The Tasks docs on regular expressions cover escaping of special characters.

Code from Tasks you could copy

However, there is some code in Tasks that you are welcome to crib, that does this escaping for you, for a given string.

The commit in Tasks that tested and proved (sufficiently) correct a function to do this escaping is as follows - mainly to show you the extent of testing I did to satisfy myself that the code behaved correctly:

fix: Fix unreleased global filter code on unusual characters (#976) · obsidian-tasks-group/obsidian-tasks@427459b · GitHub

You can see escapeRegExp() from the current version of this code:

obsidian-tasks/Task.ts at 04c621db3117b8ccaaa57c10ea212c682e27e716 · obsidian-tasks-group/obsidian-tasks · GitHub

I would suggest copying it as this, to credit the version you copied:

    /**
     * Escape a string so it can be used as part of a RegExp literally.
     * Taken from:
     *     https://github.com/obsidian-tasks-group/obsidian-tasks/blob/04c621db3117b8ccaaa57c10ea212c682e27e716/src/Task.ts#L878:L897
     *     MIT licence
     */

    private escapeRegExp(s: string) {
        return s.replace(/([.*+?^${}()|[\]/\\])/g, '\\$1');
    }

Suggested route

So your steps would be something like:

• get the text you want to search for
• pass it through the above function to make it suitable for use as a regular expression (such as converting ( to \()
• then construct your description filter, based on the instructions in the Tasks docs

I'm on mobile at the moment so not able to try it out.
If you try it and have problems, drop your code in here and I'll look at it on desktop, when I can.

I do appreciate this is all a sticking plaster for now, but I hope it will get you going.

Sharing the results

If you could share what works, I'll look at including it in the Tasks docs, to help others out....

[claremacrae]

(Example code in previous comment updated to include the licence of copied code)

[claremacrae]

This will be fixed by the solving of #1308...

[claremacrae]

This will be fixed by the solving of #1308...

This will be fixed by the solving of #1500 ...

[akatopo]

Hi @tilmanbergt

In case it helps you convert your search to using regular expressions...

Need to escape ( and other characters with special meaning

The issue with taking any arbitrary search text and searching for it using regular expressions is that various ASCII characters that might be present in the search string have special meanings in regular expressions, and so need to be "escaped", to remove their special meaning.

The Tasks docs on regular expressions cover escaping of special characters.

Code from Tasks you could copy

However, there is some code in Tasks that you are welcome to crib, that does this escaping for you, for a given string.

The commit in Tasks that tested and proved (sufficiently) correct a function to do this escaping is as follows - mainly to show you the extent of testing I did to satisfy myself that the code behaved correctly:

fix: Fix unreleased global filter code on unusual characters (#976) · obsidian-tasks-group/obsidian-tasks@427459b · GitHub

You can see escapeRegExp() from the current version of this code:

obsidian-tasks/Task.ts at 04c621db3117b8ccaaa57c10ea212c682e27e716 · obsidian-tasks-group/obsidian-tasks · GitHub

I would suggest copying it as this, to credit the version you copied:

    /**
     * Escape a string so it can be used as part of a RegExp literally.
     * Taken from:
     *     https://github.com/obsidian-tasks-group/obsidian-tasks/blob/04c621db3117b8ccaaa57c10ea212c682e27e716/src/Task.ts#L878:L897
     *     MIT licence
     */

    private escapeRegExp(s: string) {
        return s.replace(/([.*+?^${}()|[\]/\\])/g, '\\$1');
    }

Suggested route

So your steps would be something like:

• get the text you want to search for
• pass it through the above function to make it suitable for use as a regular expression (such as converting ( to \()
• then construct your description filter, based on the instructions in the Tasks docs

I'm on mobile at the moment so not able to try it out. If you try it and have problems, drop your code in here and I'll look at it on desktop, when I can.

I do appreciate this is all a sticking plaster for now, but I hope it will get you going.

Sharing the results

If you could share what works, I'll look at including it in the Tasks docs, to help others out....

Managed to work around a similar issue I had with file paths having parentheses based on the suggestion above:

function escapeRegExp(s) {
  return s
    .replace(/([.*+?^${}|[\]/\\])/g, "\\$1")
    // replace parens w/ hex escapes
    .replaceAll("(", "\\x28")
    .replaceAll(")", "\\x29");
}

[claremacrae]

Hi @akatopo - thank you. Where did you use that code please?
In a custom build of Tasks?
In a custom filter?

[akatopo]

Hi @akatopo - thank you. Where did you use that code please? In a custom build of Tasks? In a custom filter?

Used it along with this recipe to exclude notes with a certain tag from a tasks query via dataview. One of those notes happened to have parens in its path, hence the breakage.

[claremacrae]

Just released in https://github.com/obsidian-tasks-group/obsidian-tasks/releases/tag/7.0.0