[xray] Turn on flushing to the GCS for the lineage cache #1907
Conversation
|
|
||
| // Try to flush the task to the GCS. | ||
| // TODO(swang): Allow a pluggable policy for when to flush. | ||
| RAY_CHECK_OK(Flush()); |
There was a problem hiding this comment.
So this will result in flushing the lineage cache very frequently, right? E.g., every time a task becomes ready, right?
You already have a TODO, but in the future, we may want to remove this and just call Flush on a timer, right?
|
Test PASSed. |
|
Test FAILed. |
|
retest this please |
|
Test PASSed. |
* master: (56 commits) [xray] Turn on flushing to the GCS for the lineage cache (ray-project#1907) Single Big Object Parallel Transfer. (ray-project#1827) Remove num_threads as a parameter. (ray-project#1891) Adds Valgrind tests for multi-threaded object manager. (ray-project#1890) Pin cython version in docker base dependencies file. (ray-project#1898) Update arrow to efficiently serialize more types of numpy arrays. (ray-project#1889) updates (ray-project#1896) [DataFrame] Inherit documentation from Pandas (ray-project#1727) Update arrow and parquet-cpp. (ray-project#1875) raylet command line resource configuration plumbing (ray-project#1882) use raylet for remote ray nodes (ray-project#1880) [rllib] Propagate dim option to deepmind wrappers (ray-project#1876) [RLLib] DDPG (ray-project#1685) Lint Python files with Yapf (ray-project#1872) [DataFrame] Fixed repr, info, and memory_usage (ray-project#1874) Fix getattr compat (ray-project#1871) check if arrow build dir exists (ray-project#1863) [DataFrame] Encapsulate index and lengths into separate class (ray-project#1849) [DataFrame] Implemented __getattr__ (ray-project#1753) Add better analytics to docs (ray-project#1854) ... # Conflicts: # python/ray/rllib/__init__.py # python/setup.py
What do these changes do?
This flushes the lineage cache every time a task that is ready is added. Ready tasks are ones that have been scheduled for execution on the local node. This also adds an internal cache to the lineage cache that stores tasks that are in this state.