[Windows] Many templates of Lima are not Windows compatible, when built on Windows

[arixmkii]

Description

Right now templates for QEMU, which have mounts defined are not Windows compatible, because they have Unix absolute path entry "/tmp/lima". They have to be manually changed before they can be used.

Proposed solution:
Provide additional build step which will transform incompatible values into acceptable: something like "/tmp/lima" -> "C:\Temp\lima", which is definitely not imperfect, but somehow better.

So, then one can call command make binaries windows-templates and get templates transformed in _output directory.

[arixmkii]

QEMU on Windows is still work in progress, so, this is probably for backlog, but some discussions to outline the desired way to implement this might be helpful.

[afbjorklund]

We knew this, but it was "good enough" when assuming something like MSYS that would translate paths...

• Add abstraction for the default /tmp/lima directory ? #2339

[arixmkii]

@afbjorklund Thank you! I will read the discussion there. I have some concerns about translation paths in some edge cases, but I need some time with my Windows test box to experiment with it to confirm.

Also, to note here. That cygpath translation could be not consistent across machines, this implementation might not follow the least surprise principle.

[jandubois]

Proposed solution:
Provide additional build step which will transform incompatible values into acceptable

I think we should aim for templates that work on any platform. Otherwise using a template via URL doesn't work; you would have to provide different templates for different platforms, which becomes a maintenance nightmare:

limactl create --name my-vm https://example.com/template.yaml

I've implemented my suggestion from #2339 (comment) in #3318. Please let me know if this would work for you on Windows!

I do have some concerns that this will move the host location on macOS from /tmp to $TMPDIR, but I think this would be the right semantics anyways.

# The /tmp/lima mount point would point to
#   "/tmp/lima"      on a Linux host,
#   "$TMPDIR/lima"   on a macOS host, and
#   "%TEMP%\\lima"   on a Windows host.
mounts:
- location: "{{.Temp}}/lima"
  mountPoint: /tmp/lima

Ok, I just looked it up, and the documentation for os.TempDir() says:

On Unix systems, it returns $TMPDIR if non-empty, else /tmp. On Windows, it uses GetTempPath, returning the first non-empty value from %TMP%, %TEMP%, %USERPROFILE%, or the Windows directory. On Plan 9, it returns /tmp.

[jandubois]

Also: don't we need a similar solution for mounting ~? Because we would want to mount e.g. C:\Users\jan to /c/Users/jan or something similar?

The concept of having mountPoint default to location doesn't really apply on Windows; the paths always need translation. And the translation may be different based on vmType. E.g. for WSL2 the mountPoint would be /mnt/c/Users/jan (by default), and Lima doesn't need to bother about mounting the volume. But for QEMU the mountPoint default may need to be different.

I guess I really should review #3304; it may answer some of these questions already.

[jandubois]

I do have some concerns that this will move the host location on macOS from /tmp to $TMPDIR

One thing that bothers me is that this breaks the symmetry of mounting a host temp file into a container, so this would not work:

docker run --volume $TMPDIR/lima:/mnt/lima ...

It would have to be written like this, even though the data is in $TMPDIR/lima and not /tmp/lima:

docker run --volume /tmp/lima:/mnt/lima ...

This break the magic veil that lets us pretend that the container runs on the host and not in a VM.

So this still needs further thoughts.

[jandubois]

Just brainstorming here:

We could provide the host OS as a template variable and then write:

mounts:
- location: "{{if eq .OS "windows"}}{{.Temp}}{{else}}/tmp/lima{{end}}"
  mountPoint: /tmp/lima

This would only break the veil on Windows, where it is already broken; it requires e.g. a docker proxy to convert the host names to guest names.

[arixmkii]

We could provide the host OS as a template variable and then write:

This sounds like a reasonable idea at least to explore. I had some unpleasant experience with ansible yaml files, but there it was pushed to some extremes. If it is possible to keep this reasonable and UX is good enough could be an option.

Also: don't we need a similar solution for mounting ~? Because we would want to mount e.g. C:\Users\jan to /c/Users/jan or something similar?

I wanted to mention this. Because there is already ~ in the template and it is resolved to Windows path. So it will need special handling for ~ to first convert to Unix like and then to convert back via same procedure like for Unix paths. Not that complex, but feels like sub-optimal.

I think we should aim for templates that work on any platform. Otherwise using a template via URL doesn't work; you would have to provide different templates for different platforms, which becomes a maintenance nightmare:

I didn't think about scenario with remote templates. I was thinking in the domain of delivered with the installation files, where it would be fine to have it platform specific. For this scenario cross platform support is definitely preferred.

Throwing in a random idea - adding to template annotation "supported-platforms" and show error if the template is not supported by the platform. It is partially already that way for machine providers, which are not cross-platform - Lima should print an error that this is not supported, this would just extend definition to template in general instead of VMType only.

I guess I really should review #3304

Please do. I will appreciate it a lot!

the paths always need translation

Yes, but the translation should only be done by Lima, when Lima code decides which tooling is in use and how to convert it. Lima converts a number of windows paths already in the direction Windows -> Unix (user requested to underlying commands), but the reverse conversion is not really a thing.

But for QEMU the mountPoint default may need to be different.

Didn't get what "default" is referencing to here, but I totally agree that there will be platform specific handling for QEMU mounts.

One thing that bothers me is that this breaks the symmetry of mounting a host temp file into a container, so this would not work:

Symmetry on Windows is a far reach unfortunately. Especially if Lima is used from Powershell or CMD shells. For me it is hard to tell if users will choose the visible symmetry with a single supported shell or variety of supported shells and some shenanigans with mount commands for FS mounts into containers.

This break the magic veil that lets us pretend that the container runs on the host and not in a VM.

I believe this was never solved with Podman. How is it implemented in Docker Desktop? I will need to setup fresh VM to test it out, may be someone has tried it.

#3318. Please let me know if this would work for you on Windows!

I will try it and post feedback in the PR.