`test_lod_tensor_array_ops` fails in the TeamCity CI build.

[kavyasrinet]

I got an email mentioning that the TeamCity CI failed while executing the test_lod_tensor_array_ops test. Before the PR got merged, all tests passed successfully and my code doesn't touch this test case either.

Has anyone else experienced this issue ?
Following is the stack trace for reference:

206/370 Test #198: test_selected_rows ..............................   Passed    2.89 sec
Start 199: test_lod_tensor_array_ops
207/370 Test #199: test_lod_tensor_array_ops .......................***Failed    2.75 sec
.....E.
[18:31:18]======================================================================
[18:31:18]ERROR: test_lod_tensor_to_array_level_2 (__main__.TestCPULoDTensorArrayOps)
[18:31:18]----------------------------------------------------------------------
[18:31:18]Traceback (most recent call last):
[18:31:18]  File "test_lod_tensor_array_ops.py", line 117, in test_lod_tensor_to_array_level_2
[18:31:18]    expect_max_len=3)
[18:31:18]  File "test_lod_tensor_array_ops.py", line 148, in main
[18:31:18]    exe.run(program, feed={'x': tensor}, scope=scope)
[18:31:18]  File "/paddle/build/python/build/lib-python/paddle/v2/fluid/executor.py", line 290, in run
[18:31:18]    self.executor.run(program.desc, scope, 0, True, True)
[18:31:18]EnforceNotMet: holder_ should not be null
[18:31:18]Tensor not initialized yet when Tensor::place() is called. at [/paddle/paddle/framework/tensor.h:113]
[18:31:18]PaddlePaddle Call Stacks: 
[18:31:18]0       0x7f4c9b6d07ccp paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 572
[18:31:18]1       0x7f4c9b6d1591p paddle::framework::Tensor::place() const + 209
[18:31:18]2       0x7f4c9b7e5986p paddle::framework::Vector<unsigned long>::CopyToCPU() const + 70
[18:31:18]3       0x7f4c9b93b4c0p paddle::framework::Vector<unsigned long>::resize(unsigned long) + 544
[18:31:18]4       0x7f4c9c22a322p paddle::framework::Vector<unsigned long>::Vector(unsigned long, unsigned long const&) + 178
[18:31:18]5       0x7f4c9c22a85ep void std::vector<paddle::framework::Vector<unsigned long>, std::allocator<paddle::framework::Vector<unsigned long> > >::_M_emplace_back_aux<int, int>(int&&, int&&) + 158
[18:31:18]6       0x7f4c9c227d98p paddle::framework::AppendLoD(std::vector<paddle::framework::Vector<unsigned long>, std::allocator<paddle::framework::Vector<unsigned long> > >*, std::vector<paddle::framework::Vector<unsigned long>, std::allocator<paddle::framework::Vector<unsigned long> > > const&) + 2152
[18:31:18]7       0x7f4c9c0d5bb6p paddle::operators::LoDTensorToArrayOp::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 2022
[18:31:18]8       0x7f4c9b7a1b32p paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool) + 1490
[18:31:18]9       0x7f4c9b6ecd23p void pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, paddle::framework::Executor, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(void (paddle::framework::Executor::*)(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool)#1}, void, paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(pybind11::cpp_function::initialize<void, paddle::framework::Executor, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(void (paddle::framework::Executor::*)(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool)#1}&&, void (*)(paddle::framework::Executor*, paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call) + 579
[18:31:18]10      0x7f4c9b6eaa64p pybind11::cpp_function::dispatcher(_object*, _object*, _object*) + 1236
[18:31:18]11            0x4c37edp PyEval_EvalFrameEx + 31165
[18:31:18]12            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]13            0x4c16e7p PyEval_EvalFrameEx + 22711
[18:31:18]14            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]15            0x4c16e7p PyEval_EvalFrameEx + 22711
[18:31:18]16            0x4c136fp PyEval_EvalFrameEx + 21823
[18:31:18]17            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]18            0x4d55f3p
[18:31:18]19            0x4a577ep PyObject_Call + 62
[18:31:18]20            0x4bed3dp PyEval_EvalFrameEx + 12045
[18:31:18]21            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]22            0x4d54b9p
[18:31:18]23            0x4eebeep
[18:31:18]24            0x4a577ep PyObject_Call + 62
[18:31:18]25            0x548253p
[18:31:18]26            0x4c15bfp PyEval_EvalFrameEx + 22415
[18:31:18]27            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]28            0x4d55f3p
[18:31:18]29            0x4a577ep PyObject_Call + 62
[18:31:18]30            0x4bed3dp PyEval_EvalFrameEx + 12045
[18:31:18]31            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]32            0x4d54b9p
[18:31:18]33            0x4eebeep
[18:31:18]34            0x4a577ep PyObject_Call + 62
[18:31:18]35            0x548253p
[18:31:18]36            0x4c15bfp PyEval_EvalFrameEx + 22415
[18:31:18]37            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]38            0x4d55f3p
[18:31:18]39            0x4a577ep PyObject_Call + 62
[18:31:18]40            0x4bed3dp PyEval_EvalFrameEx + 12045
[18:31:18]41            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]42            0x4d54b9p
[18:31:18]43            0x4eebeep
[18:31:18]44            0x4a577ep PyObject_Call + 62
[18:31:18]45            0x548253p
[18:31:18]46            0x4c15bfp PyEval_EvalFrameEx + 22415
[18:31:18]47            0x4c136fp PyEval_EvalFrameEx + 21823
[18:31:18]48            0x4c136fp PyEval_EvalFrameEx + 21823
[18:31:18]49            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]50            0x4d54b9p
[18:31:18]51            0x4eebeep
[18:31:18]52            0x4ee7f6p
[18:31:18]53            0x4aa9abp
[18:31:18]54            0x4c15bfp PyEval_EvalFrameEx + 22415
[18:31:18]55            0x4b9ab6p PyEval_EvalCodeEx + 774
[18:31:18]56            0x4eb30fp
[18:31:18]57            0x4e5422p PyRun_FileExFlags + 130
[18:31:18]58            0x4e3cd6p PyRun_SimpleFileExFlags + 390
[18:31:18]59            0x493ae2p Py_Main + 1554
[18:31:18]60      0x7f4d32440830p __libc_start_main + 240
[18:31:18]61            0x4933e9p _start + 41
[18:31:18]
[18:31:18]
[18:31:18]----------------------------------------------------------------------

[wangkuiyi]

The same problem happened with @wanghaoshuang 's #8334 too.

[dzhwinter]

My bad, this is a bug introduced by the new implementation of Vector. And it has been fixed in #8275. Please merge the latest develop branch.

[wangkuiyi]

And #8358

[sidgoyal78]

@dzhwinter : With the new fix, and latest code, with a gpu build, on my system, when i run ctest -V -R test_lod_tensor_array_ops, the test passes.

However, when i run the same test with logging, GLOG_v=3 ctest -V -R test_lod_tensor_array_ops, i see similar exception:

215: ======================================================================
215: ERROR: test_lod_tensor_to_array_level_1 (__main__.TestCPULoDTensorArrayOps)
215: ----------------------------------------------------------------------
215: Traceback (most recent call last):
215:   File "test_lod_tensor_array_ops.py", line 73, in test_lod_tensor_to_array_level_1
215:     expect_max_len=3)
215:   File "test_lod_tensor_array_ops.py", line 148, in main
215:     exe.run(program, feed={'x': tensor}, scope=scope)
215:   File "/paddle/build/python/build/lib-python/paddle/v2/fluid/executor.py", line 290, in run
215:     self.executor.run(program.desc, scope, 0, True, True)
215: EnforceNotMet: holder_ should not be null
215: Tensor holds no memory. Call Tensor::mutable_data first. at [/paddle/paddle/fluid/framework/tensor_impl.h:63]
215: PaddlePaddle Call Stacks: 
215: 0       0x7febeedaaf0cp paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 572
215: 1       0x7febeedae5e7p paddle::framework::Tensor::check_memory_size() const + 279
215: 2       0x7febeedaf7c8p unsigned long* paddle::framework::Tensor::data<unsigned long>() + 40
215: 3       0x7febef8de548p paddle::framework::operator<<(std::ostream&, std::vector<paddle::framework::Vector<unsigned long>, std::allocator<paddle::framework::Vector<unsigned long> > > const&) + 104

Can you take a look? Thanks.

[sidgoyal78]

I think this is related to: #8368

[dzhwinter]

Could you have a try of #8377 ? the DebugStringEx bug has been fixed several minutes before.