[sled agent] Compare dladm results with datalink mgmt cache file

[smklein]

This is highly related to #1679 , but doesn't fully resolve the problem.
It's also related to #1682 , which pokes slightly more at the underlying problem.

Instead, this PR focuses on making that problem much more visible.

Under some circumstances, it's possible for datalink devices to appear in the datalink management cache file, but not in the output of dladm. Regardless of how this happens, it results in a scenario where:

• The sled agent isn't easily able to see the "defunct" datalink
• However, if the sled agent tries re-using that datalink name, an error will be returned -- see Instance creation failing during link creation (OPTE ioctl) #1679 .

This PR attempts to compare the datalink cache file with sled agent's expected state, monitoring for devices that exist, even if they should have been deleted.

If this situation is encountered, a fairly verbose error is propagated upwards, with instructions on how-to-fix. If we're confident that we can delete datalinks without resulting in this "zombie" state, we can remove this workaround.

[leftwo]

I like more detailed error messages.

<begin soapbox>
For the Unexpected object in dladm cache, this does seem a little specific, and by adding this in, we are hard coding in the sled agent both the path to the dladm cache file and the format of that file. Since this is all our stuff, we can probably get away with such shenanigans, but their may come a time when this comes back to haunt us.
<end soapbox>

[bnaecker]

Looks good, thanks. I think the error message could be more pedantic about exactly what the developer needs to do to work around this, but otherwise LGTM.

[bnaecker]

Also, it occurs to me that this really should resolve #1679. We delete all "visible" OPTE ports when the sled agent starts. Anything that's "defunct" will be deleted by this change. In that case, I don't see how a datalink with the relevant naming scheme could already exist when we ask to create it. Does that sound right to you?

[smklein]

Also, it occurs to me that this really should resolve #1679. We delete all "visible" OPTE ports when the sled agent starts. Anything that's "defunct" will be deleted by this change. In that case, I don't see how a datalink with the relevant naming scheme could already exist when we ask to create it. Does that sound right to you?

I think it should resolve the symptoms of #1679 , but I do not think it resolves the cause.

I figured we'd want to end up in a state where devices "either are visible in dladm, and can be tracked by the sled agent, or are fully deleted" -- this "half-deleted state" seems like the crux of the issue to me. Furthermore, if we move to a world where Sled Agent does not try to delete VNICs on reboot, but tries to re-construct a meaningful view of the world, I don't really know how to handle these types of defunct devices.

TL;DR: Symptoms should go away short-term, but I want us to fix the underlying datalink issue before marking #1679 resolved, unless we have appropriate tracking elsewhere.

[bnaecker]

@smklein @leftwo Robert brought up the same stability concerns that Alan mentioned.

The basic problem is that this PR relies on private and unstable interfaces and formats. I think a reasonable alternative approach is to just skip any name that results in EEXIST from the attempt to create the link. The sled-agent creates names with an incrementing counter, such as opte0, opte1, etc. It currently deletes all interfaces that appear in dladm output now, which is anything that's not defunct. If the sled-agent tries to create opteN, and gets EEXIST, I think we can be pretty confident we're in this situation of a defunct link. We can log a warning/error message for visibility, but I see no reason the sled-agent cannot just then use the name opte{N+1}. There's an issue of the number of times we allow EEXIST before "truly" failing, but I think we can find a reasonable place to draw the line.

I this alternative is acceptable we have two options:

1. Integrate this, and revert it as soon as possible.
2. Don't integrate it, and instead take an alternative approach instead.

Of course we need to root-cause why these links exist at all. But in the meantime?

[smklein]

@smklein @leftwo Robert brought up the same stability concerns that Alan mentioned.

The basic problem is that this PR relies on private and unstable interfaces and formats. I think a reasonable alternative approach is to just skip any name that results in EEXIST from the attempt to create the link. The sled-agent creates names with an incrementing counter, such as opte0, opte1, etc. It currently deletes all interfaces that appear in dladm output now, which is anything that's not defunct. If the sled-agent tries to create opteN, and gets EEXIST, I think we can be pretty confident we're in this situation of a defunct link. We can log a warning/error message for visibility, but I see no reason the sled-agent cannot just then use the name opte{N+1}. There's an issue of the number of times we allow EEXIST before "truly" failing, but I think we can find a reasonable place to draw the line.

I this alternative is acceptable we have two options:

1. Integrate this, and revert it as soon as possible.
2. Don't integrate it, and instead take an alternative approach instead.

Of course we need to root-cause why these links exist at all. But in the meantime?

Ack; if we're okay punting this to later on in runtime, I can close this PR. The catch-EEXIST-and-retry seems like a viable workaround too.