[windows] bosh add-blobs failed

[drnic]

bosh add-blob C:\Projects\mything\bin\mything.zip mything/mything.zip
Tracking blob:
  Moving temp blob to blobs/:
    rename C:\Users\drnic\.bosh\tmp\track-blob603273339 C:\Projects\mything-boshrelease\blobs\mything\mything.zip: The process cannot access the file because it is being used by another process.

[jochenehret]

I have a similar problem when running "bosh create-env" on Windows:

bosh create-env --state state.json -o virtualbox/cpi.yml -o virtualbox/outbound-network.yml -o bosh-lite.yml -o bosh-lite-runc.yml -o jumpbox-user.yml --vars-store ./creds.yml -v director_name="Bosh Lite Director" -v internal_ip=192.168.50.6 -v internal_gw=192.168.50.1 -v internal_cidr=192.168.50.0/24 -v outbound_network_name=NatNetwork bosh.yml
Deployment manifest: 'C:\cf\bosh-deployment\bosh.yml'
Deployment state: 'state.json'

Started validating
  Downloading release 'bosh'... Failed (00:00:38)
Failed validating (00:00:38)

Failed to download from 'https://s3.amazonaws.com/bosh-compiled-release-tarballs/bosh-262.3-ubuntu-trusty-3421.9-20170706-183731-831697577-20170706183736.tgz?versionId=7GmwKfufgb5JwWhJ.cwIWLnejOtm2Hu4':
  Saving downloaded file in cache:
    Failed to save tarball path 'C:\Users\<redacted>\.bosh\installations\772b2932-e002-4b86-66f0-4e4573ecf528\tmp\tarballProvider123779445' in cache:
      rename C:\Users\<redacted>\.bosh\installations\772b2932-e002-4b86-66f0-4e4573ecf528\tmp\tarballProvider123779445 C:\Users\<redacted>\.bosh\downloads\28ba3e644b628db16bade822fe201fb08892b80f-1eae3f06282417e54ebb199656458f9d6c38e2af: The process cannot access the file because it is being used by another process.

Exit code 1

Seems to be a problem in the github.com/cloudfoundry/bosh-utils/fileutil library...

[dpb587-pivotal]

I wonder if we're forgetting to close the original file before attempting to move - Windows seems particularly picky about that.

[Fydon]

I've probably already fixed this issue, but just waiting for a response to my pull requests here and here.

[bingosummer]

@Fydon
I patched the bosh cli with your two PRs and built a private version using GOOS=windows GOARCH=amd64 go build in linux.
But I still got another error when running bosh create-env.

Extracting release 'C:\Users\<redacted>\.bosh\downloads\28ba3e644b628db16bade822fe201fb08892b80f-1eae3f06282417e54ebb199656458f9d6c38e2af':
  Extracting release:
    Shelling out to tar:
      Running command: 'powershell tar --no-same-owner -xzf C:\Users\<redacted>\.bosh\downloads\28ba3e644b628db16bade822fe201fb08892b80f-1eae3f06282417e54ebb199656458f9d6c38e2af -C C:\Users\<redacted>\.bosh\installations\062fbd18-00c4-4d5d-4fb6-564675482ce3\tmp\bosh-release114128398', stdout: '', stderr: 'tar (child): Cannot execute remote shell: No such file or directory
tar (child): C\:\\Users\\<redacted>\\.bosh\\downloads\\28ba3e644b628db16bade822fe201fb08892b80f-1eae3f06282417e54ebb199656458f9d6c38e2af: Cannot open: Input/output error
tar (child): Error is not recoverable: exiting now

gzip: stdin: unexpected end of file
/usr/bin/tar: Child returned status 2
/usr/bin/tar: Error is not recoverable: exiting now
':
        exit status 1

Exit code 1

Did you hit the above error before?

[Fydon]

@bingosummer That error is because the folder containing tar.exe isn't in your PATH environment variable. If you want to use the tar that comes with Git for Windows (usr\bin folder off the installation) then you'll also need also need this pull request, otherwise I'm told that the BSD tar from bosh-agent works without the pull request. I've recommended the use of a golang library, e.g. CloudFoundry's archiver, instead of an external dependency, but nothing has come of that yet.

[bingosummer]

Thanks @Fydon . The BSD tar works for me. I put the BSD tar in my PATH environment. Is it the correct way?

But I got another error about UseIsolatedEnv. Since the value is hardcoded, so I think the bosh create-env is not supported in Windows for now, even if the issues of renaming issues and tar are resolved. How do you think?

The detail error message:

Started installing CPI
  Compiling package 'ruby_azure_cpi/87c15bcd83a9a6011b6001be68e58c59ec1c5dcb'...[CLI] 2017/08/23 16:31:54 ERROR - Panic: UseIsolatedEnv is not supported on Windows
********************
goroutine 1 [running]:
runtime/debug.Stack(0xdfe520, 0xc0422b03f0, 0xc0423536b0)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x80
main.handlePanic()
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/main.go:110 +0x213
panic(0xce27e0, 0xc0423536b0)
        /usr/local/go/src/runtime/panic.go:489 +0x2dd
github.com/cloudfoundry/bosh-cli/cmd.Cmd.Execute.func1(0xc0423c2458)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/cmd/cmd.go:47 +0x9c
panic(0xce27e0, 0xc0423536b0)
        /usr/local/go/src/runtime/panic.go:489 +0x2dd
github.com/cloudfoundry/bosh-cli/vendor/github.com/cloudfoundry/bosh-utils/system.execCmdRunner.buildComplexCommand(0x1397660, 0xc042204330, 0xe69d4e, 0x4, 0xc042264ac0, 0x2, 0x2, 0xc04228c8a0, 0x1, 0xc042249500, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/vendor/github.com/cloudfoundry/bosh-utils/system/exec_cmd_runner.go:86 +0x48f
github.com/cloudfoundry/bosh-cli/vendor/github.com/cloudfoundry/bosh-utils/system.execCmdRunner.RunComplexCommand(0x1397660, 0xc042204330, 0xe69d4e, 0x4, 0xc042264ac0, 0x2, 0x2, 0xc04228c8a0, 0x1, 0xc042249500, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/vendor/github.com/cloudfoundry/bosh-utils/system/exec_cmd_runner.go:22 +0x77
github.com/cloudfoundry/bosh-cli/vendor/github.com/cloudfoundry/bosh-utils/system.(*execCmdRunner).RunComplexCommand(0xc042149930, 0xe69d4e, 0x4, 0xc042264ac0, 0x2, 0x2, 0xc04228c8a0, 0x1, 0xc042249500, 0x56, ...)
        <autogenerated>:8 +0x95
github.com/cloudfoundry/bosh-cli/installation/pkg.(*compiler).Compile(0xc0422fc120, 0x1396040, 0xc0422494a0, 0x0, 0x0, 0x0, 0x0, 0xc042352d00, 0x0, 0x0)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/installation/pkg/compiler.go:109 +0xbd4
github.com/cloudfoundry/bosh-cli/state/job.(*dependencyCompiler).compilePackages.func1(0x13f1718, 0xed12f347a)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/state/job/dependency_compiler.go:115 +0x77
github.com/cloudfoundry/bosh-cli/ui.(*stage).Perform(0xc042322370, 0xc042322460, 0x4b, 0xc0422649a0, 0x2, 0xc042322460)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/ui/stage.go:47 +0x15d
github.com/cloudfoundry/bosh-cli/state/job.(*dependencyCompiler).compilePackages(0xc04228c3f0, 0xc042264940, 0x2, 0x2, 0x138dc80, 0xc042322370, 0x2, 0x0, 0x0, 0xc0422717a0, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/state/job/dependency_compiler.go:133 +0x2ec
github.com/cloudfoundry/bosh-cli/state/job.(*dependencyCompiler).Compile(0xc04228c3f0, 0xc0421b41c0, 0x1, 0x1, 0x138dc80, 0xc042322370, 0x0, 0xc0423a7a58, 0xc0423a7958, 0xc042271770, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/state/job/dependency_compiler.go:50 +0x1ed
github.com/cloudfoundry/bosh-cli/installation.(*packageCompiler).For(0xc0422648c0, 0xc0421b41c0, 0x1, 0x1, 0x138dc80, 0xc042322370, 0xe, 0xc0422fa1b0, 0xc04228d8f0, 0x2f, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/installation/package_compiler.go:38 +0x93
github.com/cloudfoundry/bosh-cli/installation.(*installer).Install(0xc04205c400, 0xc0422923e8, 0x4, 0xc042271770, 0x9, 0xc0422717a0, 0xe, 0xc0422fa1b0, 0xc04228d8f0, 0x2f, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/installation/installer.go:70 +0x433
github.com/cloudfoundry/bosh-cli/cpi/release.CpiInstaller.installCpiRelease.func1(0x138dc80, 0xc042322370, 0x27696528, 0x13d4e60)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/cpi/release/installer.go:37 +0xac
github.com/cloudfoundry/bosh-cli/ui.(*stage).PerformComplex(0xc04230a2d0, 0xe74930, 0xe, 0xc04228c420, 0x1387ec0, 0xc04230c320)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/ui/stage.go:68 +0x2e6
github.com/cloudfoundry/bosh-cli/cpi/release.CpiInstaller.installCpiRelease(0x1fe0760, 0xc0422e7080, 0x1387c00, 0xc042199600, 0x0, 0x138db00, 0xc04205c400, 0xc0422923e8, 0x4, 0xc042271770, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/cpi/release/installer.go:39 +0x1af
github.com/cloudfoundry/bosh-cli/cpi/release.CpiInstaller.WithInstalledCpiRelease(0x1fe0760, 0xc0422e7080, 0x1387c00, 0xc042199600, 0x0, 0xc0422923e8, 0x4, 0xc042271770, 0x9, 0xc0422717a0, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/cpi/release/installer.go:50 +0x153
github.com/cloudfoundry/bosh-cli/cmd.(*DeploymentPreparer).PrepareDeployment(0xc04211a1e0, 0x138dc80, 0xc04230a2d0, 0x0, 0x0)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/cmd/deployment_preparer.go:203 +0xd37
github.com/cloudfoundry/bosh-cli/cmd.(*CreateEnvCmd).Run(0xc0421c0910, 0x138dc80, 0xc04230a2d0, 0x1399a40, 0xc04204bf00, 0xc0422ce000, 0xe13, 0x1e00, 0xc04205fe50, 0x49, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/cmd/create_env.go:25 +0x26c
github.com/cloudfoundry/bosh-cli/cmd.Cmd.Execute(0xea2470, 0xd5ddf3, 0xe, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/cmd/cmd.go:74 +0x10be
main.main()
        /home/<redacted>/workdir/src/github.com/cloudfoundry/bosh-cli/main.go:35 +0x73b

********************
Exit code 1

[Fydon]

@bingosummer Yes I started looking into that problem, but then thought I'd try get bosh-virtualbox-cpi-release working on Windows, as I'm currently using bosh-lite via vagrant, which I understand is no longer maintained. I plan to get back to looking into this soon and will update this if I get further.

[sumkincpp]

same issue on windows with bosh create-env

bosh --version
version 2.0.28-cb77557-2017-07-11T23:04:15Z

Deployment manifest: 'C:\Code\Code\pcf\bosh-deployment\bosh.yml'
Deployment state: 'vbox/state.json'

Started validating
  Downloading release 'bosh'... Failed (00:28:01)
Failed validating (00:28:01)

←[31mFailed to download from 'https://s3.amazonaws.com/bosh-compiled-release-tarballs/bosh-262.3-ubuntu-trusty-3421.9-20170706-183731-83169757
7-20170706183736.tgz?versionId=7GmwKfufgb5JwWhJ.cwIWLnejOtm2Hu4':
  Saving downloaded file in cache:
    Failed to save tarball path '.bosh\installations\2fdaf486-9ae6-4bf0-7963-53e5aeea4156\tmp\tarballProvider628919909' in cache:
      rename .bosh\installations\2fdaf486-9ae6-4bf0-7963-53e5aeea4156\tmp\tarballProvider628919909 .bosh\downloads\28ba3e644b628db16bade822fe2
01fb08892b80f-1eae3f06282417e54ebb199656458f9d6c38e2af: The process cannot access the file because it is being used by another process.←[0m

←[31mExit code 1←[0m

Any workaround or fix?

[Fydon]

@sumkincpp The last release was created 14 days ago, so you either need to built it yourself or wait for the version after 2.0.33 for add-blobs to work. If we extend this issue to include problems with create-env then it is still open, as compilation doesn't work yet, but rename will work with the next release.

[sumkincpp]

There is a tonights artifact for 2.0.34 bosh-cli-2.0.34-windows-amd64.exe, I'll give a try and report back.

[sumkincpp]

So there is an issue with tar after that, tar.exe is present in $PATH

C:\Code\Code\pcf>sh create-vbox-env.sh
Deployment manifest: 'C:\Code\Code\pcf\bosh-deployment\bosh.yml'
Deployment state: 'vbox/state.json'

Started validating
  Downloading release 'bosh'... Finished (00:09:01)
  Validating release 'bosh'... Failed (00:00:05)
Failed validating (00:09:06)

←[31mExtracting release '.bosh\downloads\28ba3e644b628db16bade822fe201fb08892b80f-1eae3f06282417e54ebb199656458f9d6c38e2af':
  Extracting release:
    Shelling out to tar:
      Running command: 'powershell tar --no-same-owner -xzf .bosh\downloads\28ba3e644b628db16bade822fe201fb08892b80f-1eae3f06282417e54ebb19965
6458f9d6c38e2af -C .bosh\installations\2fdaf486-9ae6-4bf0-7963-53e5aeea4156\tmp\bosh-release338794714', stdout: '', stderr: '"C:/MinGW/msys/1.
0/bin/tar.exe": .bosh\\installations\002fdaf486-9ae6-4bf0-7963-53e5aeea4156\tmp\bosh-release338794714: Cannot chdir: No such file or directory

"C:/MinGW/msys/1.0/bin/tar.exe": Error is not recoverable: exiting now
':
        exit status 1←[0m

←[31mExit code 1←[0m

[Fydon]

@sumkincpp Thank you for letting me know about the new build.

I think you need to apply my pull request in a custom build of bosh-cli to get that tar to work, otherwise use BSD tar.

create-env will fail when it reaches the compilation phase, but most other commands should now work. Also I'm not aware of the Virtualbox CPI supporting Windows yet, if that what you are trying.

[Fydon]

@sumkincpp I'm still working on it, but this patch should get you through the compile phase, which is basically using Powershell instead of bash, dropping PATH and Args and setting UseIsolatedEnv to false for Windows builds. You'll need to have the required build tools in your PATH to complete the compile. I'll need to look the change's impact on tests, before I create a pull request.

[Fydon]

Sorry my last suggestion was foolish, however now I have managed to get compiling working. I don't know if it will work for all CPIs, but I can get bosh-virtualbox-cpi to compile, create VMs and attach disks, but currently it is still failing before it completes the create-env command. You can see the changes to bosh-cli on this branch, although I'll split those changes into different branches and create separate pull requests for some of the changes. The reason that I don't think that all the changes should be applied is because I don't think that using bash scripts built for Linux should be used on Windows. There different bash implementations on Windows, e.g. MSYS, Cygwin and Windows Subsystems for Linux, and each of those handling things differently to each other as well as being different to typical Linux distributions. For example they take different approaches as to how to reference the location of a file on a specific drive.

Currently my branch is using the MSYS bash that comes with git for Windows, as I was having problems when using Windows Subsystems for Linux.