File not found in cloud

[rahulguptajss]

Hi,

I am facing strange issue where dvc is unable to pull file from cloud when configured in jenkins. Below are the logs. Not sure, How can i debug this.

aws s3 ls s3://rahul-dvc/data/xyz/
2018-04-01 14:11:55 0
2018-04-03 10:15:53 69474 18\4b9aee2f5df48f1ed2cf22d3c33d07
2018-04-03 10:15:49 128425 22\7fd4cfec8ca7afff7115041d1d23d0
2018-04-03 10:15:50 3304188 41\8721169c96ea6dee8e13709e0fd4c4
2018-04-03 10:15:50 32763 42\4e1fc9ad54266a6246850286b75b45
2018-04-03 10:15:53 97861 5f\b9105c342df2de330913ae8e3767a7

• dvc pull -v
  File '/var/jenkins_home/workspace/dvcbuild/.dvc/cache/22/7fd4cfec8ca7afff7115041d1d23d0' does not exist in the cloud
  File '/var/jenkins_home/workspace/dvcbuild/.dvc/cache/41/8721169c96ea6dee8e13709e0fd4c4' does not exist in the cloud
  File '/var/jenkins_home/workspace/dvcbuild/.dvc/cache/18/4b9aee2f5df48f1ed2cf22d3c33d07' does not exist in the cloud
  File '/var/jenkins_home/workspace/dvcbuild/.dvc/cache/42/4e1fc9ad54266a6246850286b75b45' does not exist in the cloud
  File '/var/jenkins_home/workspace/dvcbuild/.dvc/cache/5f/b9105c342df2de330913ae8e3767a7' does not exist in the cloud

[efiop]

Hi @rahulguptajss !

Could you show us your .dvc/config please?

[rahulguptajss]

[Core]
# Supported clouds: AWS, GCP
Cloud = AWS

# Log levels: Debug, Info, Warning and Error
LogLevel = Info

[AWS]
CredentialPath = ~/.aws/credentials
Profile = default

StoragePath = /rahul-dvc/data/xyz/
Region = us-east-2

[GCP]
StoragePath = 
ProjectName = 

[rahulguptajss]

i tried downloading files directly from s3 but still dvc checkout doesn't identify my files.

• aws s3 ls s3://rahul-dvc/data/xyz/
  2018-04-01 14:11:55 0
  2018-04-03 10:15:53 69474 18\4b9aee2f5df48f1ed2cf22d3c33d07
  2018-04-03 10:15:49 128425 22\7fd4cfec8ca7afff7115041d1d23d0
  2018-04-03 10:15:50 3304188 41\8721169c96ea6dee8e13709e0fd4c4
  2018-04-03 10:15:50 32763 42\4e1fc9ad54266a6246850286b75b45
  2018-04-03 10:15:53 97861 5f\b9105c342df2de330913ae8e3767a7
• aws s3 sync s3://rahul-dvc/data/xyz/ .dvc/cache
  Completed 32.0 KiB/3.5 MiB (249.3 KiB/s) with 5 file(s) remaining
  download: s3://rahul-dvc/data/xyz/42\4e1fc9ad54266a6246850286b75b45 to .dvc/cache/42\4e1fc9ad54266a6246850286b75b45
  Completed 32.0 KiB/3.5 MiB (249.3 KiB/s) with 4 file(s) remaining
  Completed 99.8 KiB/3.5 MiB (677.5 KiB/s) with 4 file(s) remaining
  download: s3://rahul-dvc/data/xyz/18\4b9aee2f5df48f1ed2cf22d3c33d07 to .dvc/cache/18\4b9aee2f5df48f1ed2cf22d3c33d07
  Completed 99.8 KiB/3.5 MiB (677.5 KiB/s) with 3 file(s) remaining
  Completed 195.4 KiB/3.5 MiB (1.1 MiB/s) with 3 file(s) remaining
  Completed 320.8 KiB/3.5 MiB (1.8 MiB/s) with 3 file(s) remaining
  download: s3://rahul-dvc/data/xyz/22\7fd4cfec8ca7afff7115041d1d23d0 to .dvc/cache/22\7fd4cfec8ca7afff7115041d1d23d0
  Completed 320.8 KiB/3.5 MiB (1.8 MiB/s) with 2 file(s) remaining
  download: s3://rahul-dvc/data/xyz/5f\b9105c342df2de330913ae8e3767a7 to .dvc/cache/5f\b9105c342df2de330913ae8e3767a7
  Completed 320.8 KiB/3.5 MiB (1.8 MiB/s) with 1 file(s) remaining
  Completed 576.8 KiB/3.5 MiB (2.9 MiB/s) with 1 file(s) remaining
  Completed 832.8 KiB/3.5 MiB (3.7 MiB/s) with 1 file(s) remaining
  Completed 1.1 MiB/3.5 MiB (4.6 MiB/s) with 1 file(s) remaining
  Completed 1.3 MiB/3.5 MiB (5.4 MiB/s) with 1 file(s) remaining
  Completed 1.6 MiB/3.5 MiB (6.2 MiB/s) with 1 file(s) remaining
  Completed 1.8 MiB/3.5 MiB (7.1 MiB/s) with 1 file(s) remaining
  Completed 2.1 MiB/3.5 MiB (7.8 MiB/s) with 1 file(s) remaining
  Completed 2.3 MiB/3.5 MiB (8.6 MiB/s) with 1 file(s) remaining
  Completed 2.6 MiB/3.5 MiB (9.4 MiB/s) with 1 file(s) remaining
  Completed 2.8 MiB/3.5 MiB (10.1 MiB/s) with 1 file(s) remaining
  Completed 3.1 MiB/3.5 MiB (10.9 MiB/s) with 1 file(s) remaining
  Completed 3.3 MiB/3.5 MiB (11.6 MiB/s) with 1 file(s) remaining
  Completed 3.5 MiB/3.5 MiB (12.1 MiB/s) with 1 file(s) remaining
  download: s3://rahul-dvc/data/xyz/41\8721169c96ea6dee8e13709e0fd4c4 to .dvc/cache/41\8721169c96ea6dee8e13709e0fd4c4
• ls .dvc/cache
  18\4b9aee2f5df48f1ed2cf22d3c33d07
  22\7fd4cfec8ca7afff7115041d1d23d0
  41\8721169c96ea6dee8e13709e0fd4c4
  42\4e1fc9ad54266a6246850286b75b45
  5f\b9105c342df2de330913ae8e3767a7
• ls data/
  CombinedSummary.csv.dvc
  TestSet.csv
• dvc checkout
  '.dvc/cache/22/7fd4cfec8ca7afff7115041d1d23d0(data\feature.p)': cache file not found
  '.dvc/cache/18/4b9aee2f5df48f1ed2cf22d3c33d07(data\train.p)': cache file not found
  '.dvc/cache/42/4e1fc9ad54266a6246850286b75b45(data\vectorizer.pkl)': cache file not found
  '.dvc/cache/41/8721169c96ea6dee8e13709e0fd4c4(data\RandomForest_classifier.pkl)': cache file not found
  '.dvc/cache/5f/b9105c342df2de330913ae8e3767a7(data/CombinedSummary.csv)': cache file not found
• ls data/

[efiop]

Hm... And can you confirm that .dvc/cache files exist after dvc checkout?

[rahulguptajss]

Yes, it is present.

[rahulguptajss]

I did dvc checkout in jenkins docker container as well and its same issue.

[efiop]

@rahulguptajss Hm... One more thing, could you please run dvc checkout -v (notice the -v flag) and show the output?

[rahulguptajss]

Yes attached is the output. Weird that it says it got changed. This works perfectly fine in my windows local machine. Could it be related with windows vs unix file stuff?

[efiop]

And can you also show the contents of your *.dvc files please?

[rahulguptajss]

Attached are all dvc files

dvcfiles.zip

[efiop]

Thank you! I'm not yet able to reproduce this issue =( I have a few more questions:

1. Try running this simple example alongside your project:

mkdir myrepo
cd myrepo
git init
dvc init
echo foo > foo
dvc add -v foo
dvc remove -v foo
dvc checkout -v
cat foo

2. Is this a cygwin you are running with docker image inside?

3. This might be some weird path bug, so could also try replacing windows path's with *nix paths in all of your *.dvc files and run 'dvc checkout' once more to see what happens? E.g. data\feature.p -> data/feature.p

[rahulguptajss]

I have found the issue. It's related with the way windows pushes this cache folder in s3.

In below screenshot, highlighted 18\4b9aee2f5df48f1ed2cf22d3c33d07 should have been a folder structure instead of file name. If you see d5 in screenshot, that's what got pushed from a unix system to s3.

Example: From windows machine a\b is pushed as filename while from unix it is pushed as a (folder)-> b (filename)

[efiop]

@rahulguptajss Great catch! I will fix it ASAP and will release a new version in ~24h. Thank you!

Leaving this issue open util the fixed version is released.