Provide warnings if unknown MCA params used

[ompiteam]

It has long been a problem that users may supply incorrect or unknown MCA parameters and therefore get incorrect or undesired behavior. For example, a user may misspell an MCA parameter name on the mpirun command line and OMPI effectively ignores it because the name that the user provides is effectively never "seen" by the MCA base. That is, there is no error checking in the MCA base to see if there are MCA parameters supplied that do not exist.

While such consistency checking would be extremely helpful to users, it is a fairly difficult problem to solve. Here's a recent mail that I sent on the topic:

---

I think we all agree that this is something that would be Very Good to have. The reason that it hasn't been done is because I'm not sure how to do it. :-( Actually, more specifically, I can think of several complex ways to do it, but they're all quite unattractive.

The problem is that we don't necessarily have global knowledge of all MCA parameters. Consider this example:

mpirun --mca pls_tm_foo 1 --mca btl_openib_foo 1 -np 4 a.out

These MCA params are going to be visible to three types of processes:

• mpirun
• orted
• a.out (assumedly an MPI process)

So how do we tell mpirun and orted that they should ignore the btl_openib MCA parameter, and tell a.out that it should ignore the pls_tm MCA parameter? There are other, similar corner cases (e.g., what if some node doesn't have the openib BTL component, but others do?).

There are a few ways to do this that I can think of:

1. each app registers frameworks that it is and is not interested in -- assuming that all MCA params follow the prefix rule, we can parse out which params in the environment belong to which framework (ugh) and then find a) any that fall outside of that (e.g., mis-typed frameworks), and b) any that are in the frameworks of interest that do not match registered params. This doesn't handle all corner cases, though (e.g., openib on some nodes but not all).
2. some entity (mpirun, most likely) does an ompi_info-like "open all frameworks" and can directly check all MCA params right away. This is an abstraction violation because orterun will be opening frameworks that it should have no knowledge of (e.g., MPI frameworks).
3. some entity (mpirun, most likely) fork/exec's ompi_info in a special mode that checks for invalid MCA params in the environment (because it will inherit the params for mpirun). This is nice because then mpirun doesn't have to open all the frameworks, but it's an abstraction violation because orterun doesn't know about ompi_info (different layers).

So the first one is the only one that is actually viable (i.e., doesn't cause abstraction violation). But it's still klunky, awkward, and doesn't handle all cases. If anyone has any better ideas, I'm all ears...

---

Since writing the above e-mail, I had another idea -- address the common case and provide a workaround for the others. Specifically, do not worry about the case where some nodes have component A and others do not. Hence, in this scenario if a user supplies an MCA param for component A, the processes on some nodes will be ok with it (because they have component A), but others will consider it "unrecognized" (because they do not have component A), and will print a warning/error -- potentially causing the job to fail.

To address this, we can add [yet another] MCA parameter to disable this MCA parameter checking. The default value will be to enable MCA parameter checking, but if a user knows what they're doing, or if they fall into the corner case above, they can disable MCA parameter checking and be "good enough."

It's not perfect and it certainly doesn't cover all cases, but it does cover today's common case (where all nodes are homogeneous) and would probably be a good step forward.

[ompiteam]

Imported from trac issue 174. Created by jsquyres on 2006-07-01T07:42:28, last modified: 2014-04-22T16:33:39

[ompiteam]

Trac comment by brbarret on 2006-07-04 12:44:29:

I had a thought during my ride this morning on this issue... Since we have a naming convention of either (mpi_leave_pinned, mpi_yield_when_idle, etc.) or _ (btl_tcp_if_include, btl_sm_eager_limit, etc.), why not check parameters when their component is loaded (or their project registers params or something for that case). So if you get the framework and component right you get some error checking. If that component isn't loaded, no harm no foul, so no error message.

It's far from perfect, but I think it would catch many of the errors made entering parameters. So might be a good first step...

[ompiteam]

Trac comment by jsquyres on 2008-03-25 13:09:38:

While working on another problem, it occurred to me that this issue could be done with a new downcall into the MCA param base that scans the environment and looks for all OMPI_MCA_ environment parameters and prints out a warning about any that it doesn't have a corresponding registered parameter for.

This downcall can be invoked at the end of ompi_mpi_init().

This is not perfect, either -- it's possible that users will have MCA parameters set for MCA components that aren't loaded, so perhaps there could be different levels/types of warnings:

• warn about anything unrecognized
• warn about parameters that are unrecognized for components/frameworks/levels that are loaded and running
• selectively ignore warnings about specific MCA parameter names (although this option should be used with '''great''' care, because it can hide real errors!)

Just for the heckuvit, I'm moving this to v1.3 to see if it could get done in time.

[ompiteam]

Trac comment by jsquyres on 2008-07-10 07:40:48:

Not going to happen for v1.3.

[ompiteam]

Trac comment by jsquyres on 2009-01-21 17:18:25:

Let's see if we can tackle this for OMPI v1.4; it's been an ongoing user problem.

[ompiteam]

Trac comment by jsquyres on 2010-01-25 17:56:32:

Another idea -- this could be done easily by checking the full list of file and environment variables during MPI_FINALIZE (opal_finalize). It's not as timely as as printing them during MPI_INIT (opal_init), but it avoids the issue of lazily loading things like the MPI one-sided framework, etc.

[ompiteam]

Trac comment by jsquyres on 2013-04-23 11:22:51:

Nathan -- this will be covered by #3347, right?

[ompiteam]

Trac comment by hjelmn on 2013-07-03 12:52:00:

I have been toying around with adding this support. I have a working prototype but there are a caveat: the code will only recognize if an unknown variable to a registered framework or component is specified (ex if btl/openib is registered and btl_openib_foo is specified). This is done to prevent us from printing warnings for components that have not yet been registered.

We can remove this restriction if mpirun is in use by forcing mpirun to register all components (the same path taken by MPI_T and ompi_info) before running the check. We can do a more limited check if the job is direct-launched by aprun, srun, etc.

Thoughts?

[ompiteam]

Trac comment by jsquyres on 2013-07-03 20:00:30:

I can think of multiple failure scenarios:

1. MCA param has unrecognized project name
2. MCA param has unrecognized framework name
3. MCA param has unrecognized component name
4. MCA param has unrecognized param name

!#1 and !#2 are easily detectable at any time because we have the autogen-provided list of projects and frameworks.

!#3 and !#4 are dependent upon registering all component params.

After some thought, I think this is a valuable service to provide to users, so I'm ok with mpirun registering all components (and then closing them out, just to regain that memory back), as long as there is a way to disable this default behavior.

That being said, I reserve the right to reverse my vote if it turns out that mpirun registering all components proves to be too "expensive" (for some definition of "expensive"). :-)

[ompiteam]

Trac comment by jsquyres on 2014-02-07 11:39:56:

Nathan -- is this now done with the new MCA param system?

[ompiteam]

Trac comment by jsquyres on 2014-04-22 16:33:39:

Talked to Nathan in an IM. Just to update this ticket: he did implement this functionality off in a side branch (it's not on the trunk as of Apr 2014). There was resistance from the community to bring this into the trunk. The rationale was that it would have limited usefulness for the use case where users have common shell startup files / environment variables / mca-params.conf files that specify MCA params that exist on some systems and not others.