Skip to content

gcs switch to env #1319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 24, 2021
Merged

gcs switch to env #1319

merged 2 commits into from
Mar 24, 2021

Conversation

vnghia
Copy link
Contributor

@vnghia vnghia commented Mar 6, 2021

This PR completes gcs implementation by using env from tensorflow/c

Part of #1203 #1183

@vnghia
Copy link
Contributor Author

vnghia commented Mar 8, 2021

@yongtang Could you help me in the Bazel Windows build ? The error is

2021-03-07T16:47:10.1642526Z Execution platform: @local_config_platform//:host
2021-03-07T16:47:10.1644849Z gs.lo.lib(gcs_helper.obj) : error LNK2019: unresolved external symbol __imp_TF_GetTempFileName referenced in function "class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > __cdecl GCSGetTempFileName(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &)" (?GCSGetTempFileName@@YA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@AEBV12@@Z)
2021-03-07T16:47:10.1650992Z gs.lo.lib(gcs_filesystem.obj) : error LNK2019: unresolved external symbol __imp_TF_NowSeconds referenced in function "class std::unique_ptr<class tensorflow::io::gs::tf_gcs_filesystem::ExpiringLRUCache<struct tensorflow::io::gs::tf_gcs_filesystem::GcsFileSystemStat>,struct std::default_delete<class tensorflow::io::gs::tf_gcs_filesystem::ExpiringLRUCache<struct tensorflow::io::gs::tf_gcs_filesystem::GcsFileSystemStat> > > __cdecl std::make_unique<class tensorflow::io::gs::tf_gcs_filesystem::ExpiringLRUCache<struct tensorflow::io::gs::tf_gcs_filesystem::GcsFileSystemStat>,unsigned __int64 &,unsigned __int64 &,0>(unsigned __int64 &,unsigned __int64 &)" (??$make_unique@V?$ExpiringLRUCache@UGcsFileSystemStat@tf_gcs_filesystem@gs@io@tensorflow@@@tf_gcs_filesystem@gs@io@tensorflow@@AEA_KAEA_K$0A@@std@@YA?AV?$unique_ptr@V?$ExpiringLRUCache@UGcsFileSystemStat@tf_gcs_filesystem@gs@io@tensorflow@@@tf_gcs_filesystem@gs@io@tensorflow@@U?$default_delete@V?$ExpiringLRUCache@UGcsFileSystemStat@tf_gcs_filesystem@gs@io@tensorflow@@@tf_gcs_filesystem@gs@io@tensorflow@@@std@@@0@AEA_K0@Z)
2021-03-07T16:47:10.1658059Z gs.lo.lib(gcs_filesystem.obj) : error LNK2019: unresolved external symbol __imp_TF_DefaultThreadOptions referenced in function "public: __cdecl tensorflow::io::gs::tf_gcs_filesystem::RamFileBlockCache::RamFileBlockCache(unsigned __int64,unsigned __int64,unsigned __int64,class std::function<__int64 __cdecl(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,unsigned __int64,unsigned __int64,char *,struct TF_Status *)>,class std::function<unsigned __int64 __cdecl(void)>)" (??0RamFileBlockCache@tf_gcs_filesystem@gs@io@tensorflow@@QEAA@_K00V?$function@$$A6A_JAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@_K1PEADPEAUTF_Status@@@Z@std@@V?$function@$$A6A_KXZ@6@@Z)
2021-03-07T16:47:10.1663481Z gs.lo.lib(gcs_filesystem.obj) : error LNK2019: unresolved external symbol __imp_TF_StartThread referenced in function "public: __cdecl tensorflow::io::gs::tf_gcs_filesystem::RamFileBlockCache::RamFileBlockCache(unsigned __int64,unsigned __int64,unsigned __int64,class std::function<__int64 __cdecl(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,unsigned __int64,unsigned __int64,char *,struct TF_Status *)>,class std::function<unsigned __int64 __cdecl(void)>)" (??0RamFileBlockCache@tf_gcs_filesystem@gs@io@tensorflow@@QEAA@_K00V?$function@$$A6A_JAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@_K1PEADPEAUTF_Status@@@Z@std@@V?$function@$$A6A_KXZ@6@@Z)
2021-03-07T16:47:10.1667309Z gs.lo.lib(gcs_filesystem.obj) : error LNK2019: unresolved external symbol __imp_TF_JoinThread referenced in function "public: void __cdecl <lambda_cec3005b8c03726e011f3e503838b21d>::operator()(struct TF_Thread *)const " (??R<lambda_cec3005b8c03726e011f3e503838b21d>@@QEBAXPEAUTF_Thread@@@Z)
2021-03-07T16:47:10.1669291Z bazel-out\x64_windows-fastbuild\bin\tensorflow_io\core\python\ops\libtensorflow_io_plugins.so : fatal error LNK1120: 5 unresolved externals

It seems that the functions in env are not exported properly on Windows. But I don't have access to a Windows machine right now so I could not do anything. Thank you !

@vnghia vnghia force-pushed the gcs-improve branch 2 times, most recently from 9ec1bee to 27f8fef Compare March 24, 2021 11:02
@kvignesh1420
Copy link
Member

@vnvo2409 the windows tests are failing due to:

tensorflow.python.framework.errors_impl.AlreadyExistsError: File system for gs already registered

This might be because the TensorFlow team has a separate process of releasing nightly builds for windows.

@vnghia
Copy link
Contributor Author

vnghia commented Mar 24, 2021

@kvignesh1420 Thank you! I haven't known about it before, that might be the reason why this PR failed on the first place. Do you know when the nightly build for Windows will be released ?

@yongtang
Copy link
Member

@vnvo2409 @kvignesh1420 from the latest pypi files it looks like latest windows tf-nightly was released just 40 mins ago:
https://pypi.org/project/tf-nightly/2.5.0.dev20210324/#files

I think re-trigger of the PR may allow the test to pass?

@kvignesh1420
Copy link
Member

Thanks for the update @yongtang. I have restarted the jobs.
cc: @vnvo2409

@vnghia
Copy link
Contributor Author

vnghia commented Mar 24, 2021

@kvignesh1420 It seems that we did the same thing in exactly the same time 😄

@vnghia
Copy link
Contributor Author

vnghia commented Mar 24, 2021

My commit failed but thanks to @kvignesh1420 , we still have one CI passed https://github.com/tensorflow/io/actions/runs/682804240

I think we could merge this PR @yongtang @kvignesh1420.

Copy link
Member

@kvignesh1420 kvignesh1420 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Copy link
Member

@yongtang yongtang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks!

@yongtang yongtang merged commit 653453a into tensorflow:master Mar 24, 2021
@vnghia vnghia deleted the gcs-improve branch March 24, 2021 16:54
michaelbanfield pushed a commit to michaelbanfield/io that referenced this pull request Mar 30, 2021
* switch to env

* switch to gcs on tensorflow-io according to tensorflow/tensorflow#47247
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants