Allow (or document) Container.push behavior on overwrite

[paulomach]

Issue

Container.push currently does not allow file overwriting, but the behavior is not documented in docs.

Suggestion

1. Add a boolean overwrite flag that when True test if path exists and remove it before pushing.
2. Update docs about overwrite behavior.

[benhoyt]

@paulomach I can't reproduce this locally. I'm running the Pebble server/daemon in one terminal:

$ PEBBLE=~/pebble go run ./cmd/pebble run
...

And then using the test Pebble CLI to execute a push, like so:

$ PEBBLE=~/pebble python3 -m test.pebble_cli push ./setup.py /home/ben/w/pebble/setup.py
wrote ./setup.py to remote file /home/ben/w/pebble/setup.py
$ PEBBLE=~/pebble python3 -m test.pebble_cli push ./setup.py /home/ben/w/pebble/setup.py
wrote ./setup.py to remote file /home/ben/w/pebble/setup.py
# modify ./setup.py
$ PEBBLE=~/pebble python3 -m test.pebble_cli push ./setup.py /home/ben/w/pebble/setup.py
wrote ./setup.py to remote file /home/ben/w/pebble/setup.py
# changes appear in /home/ben/w/pebble/setup.py

I can do the push successfully many times, and when I make changes in the source, the destination updates fine.

Are you able to send a repro for the case you're seeing, and the error messages you get?

[paulomach]

Hi @benhoyt , absolutely.

Since I've observed that on charm context, I've wrote a simple charm that implements replicates the error, available here.

The traceback observed is:

Traceback (most recent call last):
  File "./src/charm.py", line 44, in <module>
    main(TestPebblePushCharm)
  File "/var/lib/juju/agents/unit-test-pebble-push-0/charm/venv/ops/main.py", line 431, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-test-pebble-push-0/charm/venv/ops/main.py", line 142, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-test-pebble-push-0/charm/venv/ops/framework.py", line 283, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-test-pebble-push-0/charm/venv/ops/framework.py", line 743, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-test-pebble-push-0/charm/venv/ops/framework.py", line 790, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 38, in _on_pebble_ready
    container.push("/etc/resolv.conf", "\n".join(content))
  File "/var/lib/juju/agents/unit-test-pebble-push-0/charm/venv/ops/model.py", line 1282, in push
    self._pebble.push(path, source, encoding=encoding, make_dirs=make_dirs,
  File "/var/lib/juju/agents/unit-test-pebble-push-0/charm/venv/ops/pebble.py", line 1726, in push
    self._raise_on_path_error(resp, path)
  File "/var/lib/juju/agents/unit-test-pebble-push-0/charm/venv/ops/pebble.py", line 1683, in _raise_on_path_error
    raise PathError(error['kind'], error['message'])
ops.pebble.PathError: generic-file-error - rename /etc/resolv.conf.8xThkKlBDntr~ /etc/resolv.conf: device or resource busy

[rwcarlsen]

This is because another process has resolv.conf opened - the file is currently in use and can't be removed/overwritten. You can lsof | grep 'resolv.conf' to check what processes are using it.

[paulomach]

Hey @rwcarlsen , that's was my initial thought also, but there's no process owning the file.
Running lsof does not show any process with the file open (link to pastebin)
And this can be reproduced to others containers files, not only resolv.conf.

[rwcarlsen]

Sounds like this might be the issue: https://stackoverflow.com/questions/60549775/device-or-resource-busy-when-i-try-move-etc-resolv-conf-in-ubuntu18-04-how

[paulomach]

That's the issue. There other files of interest that are bind mounts, like /etc/hosts.
Having the push implementation on pebble to copy over the original file can make it work (instead of mv/rename).
But I don't know if there's any drawbacks on doing that.
Should that limitation (overwrite bind mounts) be documented?

[rwcarlsen]

It's probably worth mentioning in docs somewhere that bind mounts for files will interfere with workload-side file operations.

[benhoyt]

Yeah, let's update the push docs to mention this gotcha and perhaps suggest a workaround for locked file / bind mount cases ("use Container.exec for full control" or whatever).