add mechanism to filter out known hostnames

[calestyo]

Hey.

It would be nice to have a simple way to filter out known hostnames that bash-completion gathers from files like /etc/hosts or SSH’s various config files.

My use case is that I have some rather complex SSH config with special settings for many single hosts. Typically I don't use/need these hostsnames so I wouldn't want them to show up in completions as they break fast completions of those that I really use all the time, e.g. consider:

• *.rarelyused.example.org
• *.oftenused.example.org
  maybe with pairs belonging together, one being the actual host, the other it's service processor, e.g. host.srv.example.org and host.sp.example.org.

The idea would be something like a regular expression that is stored in some file (which is considered by bash-completion) and that can be used to remove any unwanted names, e.g.
^.+\.rarelyused\.example\.org$
would clear all the unwanted one.

A simple script could then add/remove that regexp and act as a switch,... so when I want to use these host names, I switch them on, otherwise off.

Such file could go into .config/bash-completion/ which may perhaps even be used for further purposes, e.g. a subdir .config/hosts.d/*.conf which contains files with further hostnames (one per line per file),... which shall be completed but do not occur elsewhere.

Cheers,
Chris.

[calestyo]

The following is a (it seems) working proof-of-concept... main problem is that it's terribly slow.

Adding that to _known_hosts_real():

i=${#COMPREPLY[@]}
if [ -f ~/.bcexlusion ]; then
        while [ $i -ge 0 ]; do
                # `grep`’s `-q`-option is not used as it may cause an exit status of `0` even
                # when an error occurred.
                if printf '%s\n' "${COMPREPLY[$i]}" | grep -E -f ~/.bcexlusion >/dev/null 2>/dev/null ; then
                        unset -v COMPREPLY[$i]
                fi
                i="$((i-1))"
        done
fi

• I've used EREs above... BREs in principle don't support \| or \+ ... GNU’s extension do... but POSIX itself doesn’t.
• What didn't seem to work is, if one just empties COMPREPLY matching elements. I tried with bash’s ${parameter/pattern/string} form of parameter expansion, i.e. ${COMPREPLY[@]/filter-string}... but then the completion went crazy.
  I guess that leaves empty elements behind and one has to really delete them (via unset).
  
  Anyway... that form matches with pattern matching notation only... which IMO is anyway too limited to do nice filtering.
• I run the loop backwards, backwards, because I'm not really sure what bash does with the indexes once an element has been unset.
• The outer if that checks for the existence of the configuration is in principle not necessary, because I invoke grep so that it fails gracefully if the file should vanish. It's still good to do the test, so one can completely avoid the loop if there are anyway no patterns.

With a ~/.bcexlusion like:

^.+\.boring\.example\.org$
www

the above would filter out any hostnames that below .boring.example.org (RE is anchored, and the . escaped) and any hostnames containing www (RE is non-anchored).

Possible idea for extension would be to check two config files, one in /etc and one from the user. Or to make the config file(s) configurable via some XDG env vars or so.

Any ideas?

Anyway... in that form it's unusable because it's terribly slow.

I fear that's not only because of the gazillion forks of grep, but also just iterating through the array (I made a test run, where I only checked for one string to match in an if [ ...) seemed to have a noticeable penalty. But grep of course made it really bad.

Any ideas to get that fast?

[calestyo]

Okay I think I found something to speed this up considerably:

if [ -f ~/.bcexlusion ]; then
        COMPREPLY=( $( printf '%s\n' ${COMPREPLY[@]} | grep -E -v -f ~/.bcexlusion 2>/dev/null ) )
fi

using the assumption (is this true?) that the completions may never contain newlines.

No more looping, just printing the whole current array of completions, each hostname in one line, greping the ones to ignore and making it an array again.

Help needed:

• I'm a bit unsure as to which form is the best: COMPREPLY[@] or COMPREPLY[*] and the expansion within quotes or not?
  All but "${COMPREPLY[*]}" seem to work (i.e. print element in one line).
• Do I need to set IFS? I mean it works now,... but I'd assume IFS might be needed for both, the expansion of COMPREPLY and when re-creating the array.

[calestyo]

@scop Perhaps you could have a look at this...

• Would you accept such functionality as a patch?
• How should configuration happen?

For the latter I could imagine to either use e.g. something like:
"${XDG_CONFIG_HOME:-$HOME/.config}/bash_completion/hostname-exclusion-patterns" but you already use .config/bash_completion` as a single file, AFAICS, and I assume you wouldn't want to change that (though I'd think it would be better if bash-completion simply had it's own dir there, as it's "domain")?

Alternatively, I could read the files to be used from an env var (e.g. BASH_COMPLETION_HOSTNAME_EXCLUSION_PATTERN_FILES) which could then be an array and be set in .config/bash_completion (or somewhere else)?

Kinda dislike this, cause ideally it should run more out of the box... and yet another var that would "spoil" the shell env ;-) The good thing about it would however be, that there are no stats whether some file exits or not, unless explicitly configured by the user.

[akinomyoga]

FYI, Ville's reply to the customizability: #539 (comment).

I'm not sure if we should unlimitedly add new configuration variables for these kinds of requests one by one. What if another user requests a configuration for the file that contains the list of the hostnames that should be added to completions? What if another user requests a configuration for the file that contains the list of words that should be excluded from the results of another function? There are many similar helper functions in bash-completion, and it is not realistic to add a bunch of excluding/including pattern files for every relevant helper function. Even if we would finally provide some configuration variables, I feel we should consider solving the problem in a more general way.

Actually, I guess the current proposed way to allow users such detailed control of the behavior is to overwrite the function in the user configuration file at the user's discretion. You can define your version of _known_hosts or _known_hosts_real after sourcing bash-completion. It is not at all elegant but at least it provides the maximal customizability.

If you do not want to maintain such functions, you might instead consider adding -X pattern options in the completion settings of ssh, scp and other relavant commands. Something like __load_completion ssh && eval "$(complete -p ssh) -X \"\$excluded_pattern\""

[calestyo]

FYI, Ville's reply to the customizability: #539 (comment).

Well.. what exactly does that say in the end? It seems his point was, accept such customisations if it makes sense?!

I could of course change my code to work without any config vars and just check each time for some pattern files, but I think that would be much more invasive (checking for those files every time a completion is made).

I guess one shouldn't just throw out new BASH_COMPLETION_* en masse, but I'd say here it makes sense.

I could probably relinquish the part that makes the RE language configurable and simply allow just ERE,... that would already drop one option.

But I guess the other two (pattern from env var + pattern list files from env var) makes sense to keep, cause it gives people the chance to completely work without pattern files (and thus without extra interactions with the filesystem).

Merging the two into one, might possible (e.g. every array element that starts with / or ~ is a file, and everything else is a pattern... but seems a bit hacky.

What if another user requests a configuration for the file that contains the list of the hostnames that should be added to completions?

I had thought about that, too, but I think it's already rather easy to get with current means, just add an empty

Host name name name

with no further rules to your .ssh/config.
That doesn't change any logic (unlike when one would add it to /etc/hosts or so), but make bash-completion pick it up.

What if another user requests a configuration for the file that contains the list of words that should be excluded from the results of another function?

Well... anything might happen. But was there ever such a request or well described use case?

I don't think it makes sense to keep out features which make sense now, because of possible similar things which may never come up in reality.

And if they do, one could still think about some more generalised framework and deprecate the one specifically made for host.

Even if we would finally provide some configuration variables, I feel we should consider solving the problem in a more general way.

Any concrete proposals?

I mean what one could do, is something like a general exclusion helper function that may be called by other helper functions with an array of current completions and a namespace (e.g. the name of the calling function).

The general exclusion helper function could then select it's pattern lists based on the namespace.

But still, that sounds like taking a sledgehammer to crack a nut.

Actually, I guess the current proposed way to allow users such detailed control of the behavior is to overwrite the function in the user configuration file at the user's discretion. You can define your version of _known_hosts or _known_hosts_real after sourcing bash-completion. It is not at all elegant but at least it provides the maximal customizability.

But that's really ugly... and not really maintainable for normal users, I'd say. On every upgrade of bash-completion, people would need to update their own version.

I want something that people can actually use.

If you do not want to maintain such functions, you might instead consider adding -X pattern options in the completion settings of ssh, scp and other relavant commands. Something like __load_completion ssh && eval "$(complete -p ssh) -X "$excluded_pattern""

Which would still require people do override stuff managed by the distribution... and would require them update every command that may complete hosts.

Not really maintainable from the user's side.

[akinomyoga]

It seems his point was, accept such customisations if it makes sense?!

Ah, yes, I think so (though that sounds like a tautology, i.e., accept acceptable customizations). I think Ville wanted to tell something more. As in the first sentence, he thinks we could add customizations only when there are no other ways. At least, we need to think hard whether there are any other solutions before adding extra customizability.

I could of course change my code to work without any config vars and just check each time for some pattern files, but I think that would be much more invasive (checking for those files every time a completion is made).

I don't think the customizability in that discussion is not just the config vars but also includes the config files such as the pattern files that you are suggesting. I mean we should consider first the solution without something like pattern files.

I'm actually not sure what would be the exact situation in your case, but if there are some clear and objective reasons that some part of the hostnames shouldn't be included in completions for general users, then we can actually just explicitly implement the filtering in _known_hosts_real. In this way, other users also take the advantage of the improved _known_hosts. But of course, this is not the case if your needs are purely coming from your personal preferences.

Merging the two into one, might possible (e.g. every array element that starts with / or ~ is a file, and everything else is a pattern... but seems a bit hacky.

I don't either think that is a good solution. Anyway, we are not discussing the number of config variables, but minimizing the resulting freedom of customizability. For example, we do not need to provide the pattern files if we provide a pattern variable and vice versa, so at least one of them is not necessary when the other one is added.

just add an empty

Host name name name

with no further rules to your .ssh/config.

This is just a hack though it works.

Well... anything might happen. But was there ever such a request or well described use case?

No, so far. But I feel that is the same as your suggestion; we have not received such a request so far, and this would be the first one. There is still a possibility that It is not something that many users want as you might think. If we need to accept everything with the first request, we need to accept every future request.

I don't think it makes sense to keep out features which make sense now,

Ville doesn't necessarily request us to reject it but tells us to think about a better solution if it is really useful.

because of possible similar things which may never come up in reality.

I just don't know how many other possible use cases that we don't come up with for now.

And if they do, one could still think about some more generalised framework and deprecate the one specifically made for host.

I think the policy of bash-completion is to think about the possibility from the beginning.

Any concrete proposals?

I think that's something we want to carefully consider. One thing that I had in my mind is to provide a general version of Bash's FIGNORE. FIGNORE only operates on the filenames in Bash, but we might think of preparing a similar configuration variable for the general words generated by the programmable completion. But I wouldn't say this idea is the best one before discussing more for even better solutions.

I mean what one could do, is something like a general exclusion helper function that may be called by other helper functions with an array of current completions and a namespace (e.g. the name of the calling function).
The general exclusion helper function could then select it's pattern lists based on the namespace.

I think if we would provide such detailed customization we can add the general helper function to help implement it, but to begin with, my question is do we really need such structured namespacing for the pattern filtering from the perspective of average users? Here is a dilemma: on one hand we want to keep the user interface simple, and on the other hand we want more flexible customization. I think we need to compromise something, and what would be sacrificed can depend on the policy of the project that Ville determines.

But that's really ugly... and not really maintainable for normal users, I'd say. On every upgrade of bash-completion, people would need to update their own version.

I completely agree that it's ugly, but I don't think we need to update it so frequently as the interface doesn't basically change.
Just to make it clear, what I have suggested in the previous reply is not to directly edit bash_completion but to prepare another file that contains the definition of only the functions that the user wants to replace. But anyway, this solution is the extreme one for those who want the maximal customizability.

[calestyo]

At least, we need to think hard whether there are any other solutions before adding extra customizability.

Well, the more generalised you want such system to have, the more need for customizability you'll get.
I guess there are only two ways, either use variables to tell where to look for the patterns or look directly for the files.
The latter has IMO the disadvantages of:

• this needs to be done on every completion, even if the user doesn't even want to use the feature at all
• requires hard-coded location of where these pattern files should be

I don't think the customizability in that discussion is not just the config vars but also includes the config files such as the pattern files that you are suggesting.

I don't quite see how these should be a problem. Either a user wants to use such feature, then he wants the files - or he doesn't want to, then he doesn't need to have/care about them at all.
It cannot get much less invasive, can it?

I mean we should consider first the solution without something like pattern files.

One cannot really drop the files. Just read the description of my use case before... the idea is that some host names are rarely used - not never. So one doesn't want to see them in day-to-day completions, but still wants a way to enable them if needed.

• Which hostnames these are is obviously completely dependant on the respective user, therefore it makes no sense to implement anything that drops e.g. all with www. for every user in a hardcoded way.
• Storing the exclusion patterns just in a variable, without any files, has the major drawback that one could enable the hosts only per shell (e.g. by unsetting that var), but what's actually desired is something like a switch (e.g. renaming the respective .config/bash-completion/host-exclusion-patterns.d/group.conf to .config/bash-completion/host-exclusion-patterns.d/group.conf.disabled. Can be done via a simple shell script that's added to ~/bin or so, that may even be extended with completion support if one has several groups.).

So the main use case is actually the one with the files,... I've provided the var-based exclusion only for those who want to filter out some names without ever using them and who don't want to read the file on every completion.

Anyway, we are not discussing the number of config variables, but minimizing the resulting freedom of customizability.

Well, it's not my project so I don't decide... but shouldn't it be the opposite what's desired? I.e. if there is some reasonable use case, giving uses the full freedom to customize it without having to effectively patch the code on every release?

And I'm not talking about making just everything configurable, e.g. it might make sense for some people to get a knob to reverse the sort order of results... that'd I'd say is fine... but giving them even more options to do that depending on the current phase of the moons would probably be too much.

For example, we do not need to provide the pattern files if we provide a pattern variable and vice versa, so at least one of them is not necessary when the other one is added.

See above... the files are necessary. If at all, then I could drop the pattern variable. But since it comes at basically zero cost, I don't see much reason why.

This is just a hack though it works.

Yes it is... but I guess we want to keep bash-completion fast. The SSH config files are anyway parsed, while any means of doing this via some extra files/vars would require reading something extra.

And the hack in ssh_config can be made a bit cleaner, by just doing a Include ~/.confg/bash-completion/add-on-hostnames and have these Host name lines there.
That keeps at least ssh_config a bit more tidy.

Ville doesn't necessarily request us to reject it but tells us to think about a better solution if it is really useful.

Well... as said it might very well happen that something like this will requested by other users for further parts.

For example user names. Though in my experience, one completes far more often different hostnames, than user names... and typically there aren't so many completable hostnames on the local system.

Another case could be git... one might want to filter out certain tags or branch names. But that would then probably be per repository, and no general solution would really work.

So as said previously,... I think the only more generalised would be a helper function that selects the pattern-lists based on some namespace, which is then called by _known_hosts_real.
But whether that's really needed now or ever... or whether it just complicates things, is another question.

I think that's something we want to carefully consider. One thing that I had in my mind is to provide a general version of Bash's FIGNORE. FIGNORE only operates on the filenames in Bash, but we might think of preparing a similar configuration variable for the general words generated by the programmable completion. But I wouldn't say this idea is the best one before discussing more for even better solutions.

Variables do not really work for my use case, as they're not shared amongst shell instances. I.e. I cannot call some switch command in one shell, and all other (already open) shells will get it, too.
A major point of the whole idea is to not filter out these hostnames forever, but to have an easy way to get them back if one wants (without closing all shells and starting them again).

The only other thing that I can think of would be some hook functions, that are blindly called by e.g. _known_hosts_real and where users could to their stuff (like filtering).
But this idea seems to be extremely stupid and far more invasive.

I think if we would provide such detailed customization we can add the general helper function to help implement it, but to begin with, my question is do we really need such structured namespacing for the pattern filtering from the perspective of average users? Here is a dilemma: on one hand we want to keep the user interface simple, and on the other hand we want more flexible customization. I think we need to compromise something, and what would be sacrificed can depend on the policy of the project that Ville determines.

Well, as said, I don't see the necessity right now for such functionality beyond hostnames. Though I cannot promise that this won't change in ten years.

But even then, it's IMO unclear whether such needs could be suited with any generalised framework. It may not always be patterns that one wants to filter out.

I'd also say that any customisation knobs don't make the interface more complex for the average user, when they don't change anything when unused (and when well-documented).
bash-completion should work out of the box for everyone, yet be configurable (within reasonable means) for those who need more customised solutions.

And from the developers PoV:

• A generalised framework would likely be more complex.
• bash-completion already ships numerous completions for which one could ask whether they're of any use for the general user (freeciv, hddtemp (which I thought was kinda dead) ... so you can always bring up the question for how many people something is actually useful or not.
• The current proposal is IMO quite isolated and simple. And even if it should ever be replaced with something more general, that would also be based on pattern list files (I mean that's the core of the idea), so the current interface should then be easy to deprecate and replace by the new more general one.

I think we need to compromise something, and what would be sacrificed can depend on the policy of the project that Ville determines.

Well sure... in the end it's his project and he must decide whether he wants it or not... or whether he has some other proposals.

I found it simply quite useful and pretty non-invasive to both the code and the user interface - so I'm actually quite surprised that this opened so many questions.

I completely agree that it's ugly, but I don't think we need to update it so frequently as the interface doesn't basically change.

Not sure whether I understand you correctly.
I though the idea was, that the user overwrites e.g. _known_hosts_real, right?
Sure it's interface may change rarely, but the actual code? Any change in the way SSH files are parsed? Any added sources for hostnames?

The user would always need to check and merge.

[akinomyoga]

After all, @scop will decide what should enter in this project and what not. This is just based on my guess about the policy of this project from my experience, but I feel the direction of this PR seems to be slightly different from the direction of bash-completion.

Well, the more generalised you want such system to have, the more need for customizability you'll get.

Hmm, maybe I miss your point, but I feel I need to clarify that I'm not promoting some generalized complicated mechanism to add much customizability. It's actually opposite. I think the policy of bash-completion is to keep the mechanism simple and the customizability minimum. In general, when we receive some suggestions, we first ask ourselves whether we should add the feature or not, and then think about a customization interface that is as simple as possible from the user's perspective.

I guess there are only two ways, either use variables to tell where to look for the patterns or look directly for the files. The latter has IMO the disadvantages of:

• this needs to be done on every completion, even if the user doesn't even want to use the feature at all
• requires hard-coded location of where these pattern files should be

I don't either think the latter is a good solution. When we would finally support pattern files, that should be specified by some config variables. When I say "minimizing the customizability", I don't just mean reducing the number of configuration variables, but the effective freedom of the customizability. Here, reducing the number of configuration variables by fixing the path of the pattern files does not actually change the range of what can be customized, so that's not what I'm suggesting.

I don't think the customizability in that discussion is not just the config vars but also includes the config files such as the pattern files that you are suggesting.

I don't quite see how these should be a problem. Either a user wants to use such feature, then he wants the files - or he doesn't want to, then he doesn't need to have/care about them at all.
It cannot get much less invasive, can it?

I think there are two separate discussions here. One is whether we provide the freedom of configuring the excluded hostnames, and the other is how we provide such customizability if we decided to provide the customizability. In that part, I was not discussing other solutions to provide the customizability but the solutions without the customizability (though it turned out that it doesn't seem to apply to the present case).

One cannot really drop the files. Just read the description of my use case before... the idea is that some host names are rarely used - not never. So one doesn't want to see them in day-to-day completions, but still wants a way to enable them if needed.

• Which hostnames these are is obviously completely dependant on the respective user, therefore it makes no sense to implement anything that drops e.g. all with www. for every user in a hardcoded way.
• Storing the exclusion patterns just in a variable, without any files, has the major drawback that one could enable the hosts only per shell (e.g. by unsetting that var), but what's actually desired is something like a switch (e.g. renaming the respective .config/bash-completion/host-exclusion-patterns.d/group.conf to .config/bash-completion/host-exclusion-patterns.d/group.conf.disabled. Can be done via a simple shell script that's added to ~/bin or so, that may even be extended with completion support if one has several groups.).

Thank you for explaining the details on your use case. Hmm, unfortunately, after reading it, I now feel that this use case is too specific. I don't think there many users that want to change the list of hostnames dynamically and globally in the system at the same time so frequently. For example, we can easily think about slightly different use case where the user wants to manage the list of excluded hostnames in each Bash session separately where the pattern-file approach is not so useful. To me, the above arguments appears to only exactly matches with your specific use case.

I still think just providing a pattern variable would be simpler and covers many cases even though I understand that that doesn't satisfy your needs. The upstream bash-completion is just a common configuration for general users, but that doesn't stop you using the locally patched version of bash-completion that suits your needs.

Well, it's not my project so I don't decide... but shouldn't it be the opposite what's desired? I.e. if there is some reasonable use case, giving uses the full freedom to customize it without having to effectively patch the code on every release?

And I'm not talking about making just everything configurable, e.g. it might make sense for some people to get a knob to reverse the sort order of results... that'd I'd say is fine... but giving them even more options to do that depending on the current phase of the moons would probably be too much.

From microscopic point of view, adding as many configuration options as possible would be better than not providing them in theory because the user can just keep it default if the user is not interested in them. But the microscopic optimization doesn't necessarily mean the macroscopic improvements. I have seen many arguments about the harm of adding arbitrary customizability that makes sense (excluding the moon-phase option, etc.) even though the customizability is usually useful and powerful for the experience of heavy users. For example, minimizing the interface is the key of encapsulation or making loosely-coupled and maintainable system. Providing too much customizability puts stronger constraints in the future changes of internal implementations (unless we happily drop the backward compatibilities). Also the customizability could lead to dependency hell, [e.g. consider the case external modules B and C assume different configurations X and Y of customizable module A, respectively, etc.]. The larger freedom of the customizability makes it harder to test the code for all the possible cases of the combination of the configuration. The larger freedom of the customizability will confuse average users even if it is fully documented; the users rarely read and understand the details of the entire document of the project from top to the bottom before they start to use it. If the customizability is larger, the user need to take more time to understand what would be the proper way of customizing it to satisfy their needs. My basic stance on that kind of arguments is "that depends"; whether it applies or not is case-by-case. I don't think these are always perfectly correct, but likewise, I don't think accepting every customizability that would make sense is a good idea. Even if we pick up some of these changes, we need to be careful.

And from the developers PoV:

• A generalised framework would likely be more complex.

As I have written above, I'm not promoting a generalized framework. It's actually opposite.

• bash-completion already ships numerous completions for which one could ask whether they're of any use for the general user (freeciv, hddtemp (which I thought was kinda dead) ... so you can always bring up the question for how many people something is actually useful or not.

These are not related to customizability.

• The current proposal is IMO quite isolated and simple. And even if it should ever be replaced with something more general, that would also be based on pattern list files (I mean that's the core of the idea), so the current interface should then be easy to deprecate and replace by the new more general one.

Backward compatibility is considered important in this project.

I found it simply quite useful and pretty non-invasive to both the code and the user interface - so I'm actually quite surprised that this opened so many questions.

I think my above arguments on minimizing customizability answers a part of the reasons.

Not sure whether I understand you correctly. I though the idea was, that the user overwrites e.g. _known_hosts_real, right?

Right.

Sure it's interface may change rarely, but the actual code? Any change in the way SSH files are parsed? Any added sources for hostnames?

The user would always need to check and merge.

The user only needs to check the updates if they want to keep track of the latest features and fixes of _known_hosts/_known_hosts_real. Otherwise, the user can just continue to use the user's version without problems as far as the interface of the function doesn't change. Also, I don't think _known_hosts and _known_hosts_real are so frequently changed.

[calestyo]

After all, @scop will decide what should enter in this project and what not. This is just based on my guess about the policy of this project from my experience, but I feel the direction of this PR seems to be slightly different from the direction of bash-completion.

Sure,... I'd find it a pity if such nice functionality wasn't accepted but life goes on.

I think the policy of bash-completion is to keep the mechanism simple and the customizability minimum. In general, when we receive some suggestions, we first ask ourselves whether we should add the feature or not, and then think about a customization interface that is as simple as possible from the user's perspective.

I guess it's always a trade off. As I've tried to explain before, I agree that it can be overkill to make basically everything configurable.
If I'd have suggest to make the (selected) patterns dependant on e.g. a previously completed username... or maybe by some auto-discovered currently connected network (think about: only showing intranet hostnames, when one has VPN running or so)... than I'd have agreed that this may be overkill.

For my personal taste at least, what I propose is simply non-invasive and yet still quite powerful (e.g. one could implement the connected-network dependent completing outside of bash, by some hooks that add/remove patterns based on e.g. an event by NetworkManager.. or whatever).

But surely it's a matter of personal taste.

I don't just mean reducing the number of configuration variables, but the effective freedom of the customizability.

Well that "policy" I couldn't really follow. I can understand if one doesn't give freedom of customizability when the feature is too complex to maintain... or when one cannot identify any reasonable use case... or when the implementation of it would have too negative consequences even for those who do not even want to use it (e.g. see my first implementation try in the beginning of this ticket, which had really terrible performance).

But just removing customizability for the sake of it... how would that make sense in any way?

That would sound like what many people felt became GNOME philosophy (removing features and selling it as an advantage)... and why things like Cinnamon and friends came into existence.

I now feel that this use case is too specific. I don't think there many users that want to change the list of hostnames dynamically and globally in the system at the same time so frequently.

Uhm, I guess that's difficult to judge without any proper survey or so. I work in research, we're participating in a number of different project... each of them having completely different hosts I need to connect.
Typically I don't do this wildly intermixed but at one time do stuff for one project.
Similarly, people may have different hosts depending on whether they're do some private stuff... or work (for the job).
All our students, for example, get access to a large pool of workstations (physically and remotely),... so in principle, they'd need a SSH key for each of those nodes, whilst typically not connecting to all of them.
Other may simply want to filter out any IPv4 or v6 addresses.

Seem all like not so uncommon cases of where one wants to exclude certain hosts from completions.

For example, we can easily think about slightly different use case where the user wants to manage the list of excluded hostnames in each Bash session separately where the pattern-file approach is not so useful.

Why? It would still work. Just that one would need to set BASH_COMPLETION_HOST_EXCLUSION_PATTERN_FILES to a different value per shell, which can be quite easily controlled via .profile or .bash* configs. Or the same via BASH_COMPLETION_HOST_EXCLUSION_PATTERN.
If one wants to use the former one could simply set up another level in the .config/bash-completion/host-exclusion-patterns.d/<shell-id>, where shell-id could be any group of terminals or even single terminals running a shell.

I still think just providing a pattern variable would be simpler and covers many cases even though I understand that that doesn't satisfy your needs.

Still don't see much advantage of removing the file functionality. If it would be super complex code or so, than sure... but it seems rather simple, so other than saving a line of code and a variable that gets never even set when people don't use the feature... what for?

Of course one could also hack the same functionality in to the system with just having a single pattern variable.
E.g. via bash’s PROMPT_COMMAND, which every time reads the same pattern files that would be now in BASH_COMPLETION_HOST_EXCLUSION_PATTERN_FILES and feeds the result into the single pattern variable.

Sure, works, but makes usage for the end-user just more nuisance, that could would need to be written by the user, it would possibly need to go into both, .profile and .basjrc... and worst of course, it would read the configuration really on every new readline... not just on every completion, quite a performance issue.

From microscopic point of view...

I guess it probably doesn't make much sense to go down the philosophical rabbit hole ;-)
One can surely find arguments for both extremes... some may e.g. the simple POSIX interface is good enough... others might say that Linux kernel has shown it's simply too limited for the real world.

Two things however:

unless we happily drop the backward compatibilities

• This is bash-completion... it's not the kernel were gazillions things break when your remove some interface. This is not meant as an offence against bash-completion, but simply by it's nature it's not so bad when one makes backwards incompatible changes ... should bash-completion merge my feature, and ever decide to remove it, or replace it by some even more general framework... what's the worst that would happen? Well completions wouldn't get filtered anymore (and perhaps people would complain about that - which one can avoid, by making good release notes).
  
  I don't say: break compatibility on purpose... but if at some point in the future, the feature would be removed/replaced,... than it probably wouldn't be the end of the world. Either it's anyway used by only few people - or it's used by many (then bash-completion will likely want to keep it anyway ;-) ).

• No new dependencies are added by my code.

• And just to boringly repeat myself... the code is pretty simply and isolated. I doubt it makes testing or future development any harder.

These are not related to customizability.

Okay than... other example. The current data sources of _known_hosts_real() ... okay SSH obviously makes sense... I can also understand Avahi (though I don't personally like the whole idea behind it)... but ruptime ... didn't even know what that was before I stumbled over it ;-)

I just wanted to say... it's always quite some personal PoV, what gives a big use case and what not.

The user only needs to check the updates if they want to keep track of the latest features and fixes of _known_hosts/_known_hosts_real.

Uhm... sure, yes... but...isn't that what more or less everyone would want unless for some very strong/special reasons?

[akinomyoga]

Sure,... I'd find it a pity if such nice functionality wasn't accepted but life goes on.

Ah, wait. Maybe I have confused you, but the suggested functionality has not yet been rejected by @scop. He's just busy. Next, I have written that I feel the direction is slightly different, but I'm not trying to reject the entire idea, but I thought the PR can be adjusted to better suit the direction of bash-completion to make it accepted (although the resulting changes might not exactly match with your specific case).

For my personal taste at least, what I propose is simply non-invasive and yet still quite powerful

I agree with this point, but this is a microscopic fact (in the sense that I wrote in the previous reply), and that might not necessarily be the sufficient reason that it would be merged. Even if each of the possible microscopical improvements makes sense for some use cases, adding every such feature into one might not be finally so useful. That depends on the project and the maintainer.

From microscopic point of view...

I guess it probably doesn't make much sense to go down the philosophical rabbit hole ;-)

I think this is related to the central point of the policy explained in #539 (comment), so if you avoid discussing this point, actually I don't have something important to discuss here.

unless we happily drop the backward compatibilities

• This is bash-completion... it's not the kernel were gazillions things break when your remove some interface.

The user configurations depend on the interface. External completion scripts depend on the interface (though external completion scripts are irrelevant for the present case). We might write a disclaimer for the breaking changes in release notes to avoid complaints as you say, but that doesn't mean there is zero stress on the users. I wouldn't say making breaking changes is 100% wrong, but also I don't think not making breaking changes is 100% wrong. The maintainer can choose.

Either it's anyway used by only few people - or it's used by many (then bash-completion will likely want to keep it anyway ;-) ).

The latter case is the problem.

• No new dependencies are added by my code.

Although that part about the dependency is rather a general argument, the customizability introducer (i.e., your PR) corresponds to A in that explanation but not B and C at the side that adds the dependencies.

• And just to boringly repeat myself... the code is pretty simply and isolated. I doubt it makes testing or future development any harder.

Right, but that's microscopic. The suggested change as alone wouldn't make any problem, but it's unpredictable when we continue to accept every similar PRs just because they microscopically make sense. It's also related to the amount of resources that the maintainer can provide.

---

I think the current version is already good enough in a microscopic sense. The suggestion of only providing the pattern variable (for general completed words not only for hostnames) is just a naive idea on how to focus on the necessary part of the suggestion while compromising some part of the usefulness, but that might not be something that @scop requires us by the words "if everything else fails, then make it customizable/configurable".

[scop]

There's way too much to read through and completely digest here that I have time for right now, but from the bits I was able to read, I think @akinomyoga is representing my general mindset well. The moments I do find for bash-completion I'll direct towards completing https://github.com/scop/bash-completion/projects/2 (and I haven't found those moments for some time either), pretty much everything else that requires more than a couple of minutes of my time is on hold until that's done as far as I'm concerned, sorry about that.

Take this with a kilogram of salt as it's quite possible I've missed something, but what I was able to gather from here leaves me with a feeling that this is too much complexity for such a specific use case. The ability to switch between BRE/ERE/PCRE most definitely is, but I'm not making any promises I'd accept this even with it removed. Nor unfortunately when I'll have time to revisit here if you still wish to continue to pursue inclusion of this.

[scop]

Just another couple of quick notes:

About the specifity, I have come across cases where I would have liked to filter out some known hosts completions. Those revolved around not getting completions that I know will not work with some commands, but are still useful with others. This implementation would not have helped, because the exclusion in my case would require more context than just a global filter list.

I think a generic and more powerful exclusion mechanism would have a better chance to be accepted than a specific one like this. Yes, it would almost certainly make it harder to use than for example this. But still. (And no promises about that either, except willing to have a look sometime if someone comes up with it.)

[calestyo]

The ability to switch between BRE/ERE/PCRE most definitely is, but I'm not making any promises I'd accept this even with it removed.

I had already put it in a separate commit to be easily skipped. ^^

Anyway... it's a bit strange that one one side you consider the code too complex (whereas it's really just a few lines, that absolutely don't touch anything else or have any side effects, than removing some names).... while on the other hand you wish for some far more generic/powerful exclusion mechanism that takes (arbitrary?) context into account.

Seems a bit contractionary.

Well... I guess it's rather clear that I cannot really invest any time into such more generic method... apart from it being to unspecific it feels that if even this little contribution cannot be accepted, anything bigger is even less likely to ever be and so such contribution would be just another waste of time.

Not sure whether you want to keep the PR here open for anyone who wants it and would then manually patch, cause I've already removed my forked repo, no sure if the code goes away once this is closed here.

Thanks however for looking into this, cheers!

[scop]

It's not "really just a few lines" -- it's 25 lines of code, 3 configuration variables, 60 lines of related documentation. That's just too much for a narrow scoped thing like this in my opinion.

Whereas a generic exclusion mechanism wouldn't necessarily be that hard to implement or use, just some food for thought (which I suggest we discuss in another issue or PR if we want to pursue this further), quite likely needs more work and doesn't account for all things we'd need it to yet:

$ git diff -U2
diff --git a/bash_completion b/bash_completion
index 1d57de3a..7c685f42 100644
--- a/bash_completion
+++ b/bash_completion
@@ -820,4 +820,12 @@ _comp_variable_assignments()
 }
 
+_comp_return_hook()
+{
+    ((${#FUNCNAME[*]} != 2)) && return  # this _will_ need some refinement and thought
+    echo "Hello from return hook for ${FUNCNAME[1]}"
+    echo "words: ${words[@]}"
+    echo "COMPREPLY: ${COMPREPLY[@]}"
+}
+
 # Initialize completion and deal with various general things: do file
 # and variable completion where appropriate, and adjust prev, words,
@@ -839,4 +847,6 @@ _init_completion()
     local exclude="" flag outx errx inx OPTIND=1
 
+    trap _comp_return_hook RETURN
+
     while getopts "n:e:o:i:s" flag "$@"; do
         case $flag in

$ cd helpers
$ foo bar --quux baz <TAB>Hello from return hook for _minimal
words: foo bar --quux baz 
COMPREPLY: Makefile.am perl python Makefile.in Makefile

Makefile     Makefile.am  Makefile.in  perl         python       

Now, we could have an associative array from where users could place their hook functions, keys being for example function or command names, and _comp_return_hook would look them up and run if they exist. That function would have access to COMPREPLY as well as all the context there is (some things such as the words array would need to be standardized probably for this), and could do things, filter from COMPREPLY, add to it, or do something completely different, whatever it likes. The use case at hand could be solved by something like:

_comp_return_hooks[ssh]=_my_ssh_return_hook

_my_ssh_return_hook() {
    for i in "${!COMPREPLY[@]}"; do
          [[ ${COMPREPLY[i]} =~ ^exclusion_regex$ ]] && unset -v 'COMPREPLY[i]'
    done
}