FIX-#3252: Fix GroupBy.__getitem__

[YarShev]

What do these changes do?

@dchigarev writes:

Note: GroupBy.__getitem__ allows to select specific columns of the source frame to apply aggregation functions on.

Usage examples

>>> df = pandas.DataFrame({"a": [1, 2, 2], "b": [3, 4, 5], "c": [7, 8, 9], "d": [9, 8, 7]})
>>> df.groupby("a").sum() # No selection, applying 'sum' for each non-by column
   b   c   d
a
1  3   7   9
2  9  17  15
>>> df.groupby("a")[["b", "c"]].sum() # Selecting "b" and "c" columns to aggregate
   b   c
a
1  3   7
2  9  17
>>> df.groupby("a")[["a", "b"]].sum() # Can even select 'by' columns to be aggregated
   a  b
a
1  1  3
2  4  9

This PR changes the way of handling columns selection for aggregation in GroupBy. Previously the selection was done by masking the source frame with the key columns:

modin/modin/pandas/groupby.py

Lines 341 to 348 in 253ecd9

	return DataFrameGroupBy(
	self._df[key],
	self._by,
	self._axis,
	idx_name=self._idx_name,
	drop=self._drop,
	**kwargs,
	)

this approach does not grab 'by' columns to the resulted source frame which leads to KeyErrors if self._by is presented by a list of column names, it also does not allow to select 'by' columns to be aggregated, because by default pandas don't aggregate 'by' columns and masking can't make it do so.

It was decided to get rid of masking the source frame at __getitem__ and extend GroupBy backend's API by adding selection parameter that caries information about columns to aggregate.

This parameter could help not only for the __getitem__ selection, but to offload backends of mimicking pandas behavior in terms of resolving naming conflicts when as_index=False. Ideally, the following code should be located at the front-end, selection parameter allows to move this to the front and pass the proper columns to aggregate as a selection to the backend:

modin/modin/experimental/engines/omnisci_on_ray/frame/data.py

Lines 444 to 456 in 253ecd9

	if isinstance(agg, dict):
	agg_cols = agg.keys()
	elif not drop:
	# If 'by' data came from a different frame then 'self-aggregation'
	# columns are more prioritized.
	agg_cols = self.columns
	else:
	agg_cols = [col for col in self.columns if col not in groupby_cols]
	# Mimic pandas behaviour: pandas does not allow for aggregation to be empty
	# in case of multi-column 'by'.
	if not as_index and len(agg_cols) == 0 and len(groupby_cols) > 1:
	agg_cols = self.columns

ASV results

Launch command:

MODIN_TEST_DATASET_SIZE=Big MODIN_CPUS=16 asv continuous src/master HEAD --launch-method=spawn --no-only-changed --show-stderr -a repeat=3 -a number=3 -b "^benchmarks.TimeGroupByDefaultAggregations" -b "^benchmarks.TimeGroupByMultiColumn" -b "^benchmarks.TimeGroupByDictionaryAggregation"

Benchmarks were run at this commit which adds groupby benchmarks for the 'selection' parameter

There are some cases for which performance has decreased (ASV marked them by pluses), they're related to the selection for wide frames, reasons for that are described here.

Performance results

       before           after         ratio			benchmark(shape, number_of_groups, number_of_columns_to_select)
     [0bc409d6]       [98198eab]
     <master>       <dev/yigoshev-issue3252>
+      82.6±0.7ms       98.0±0.5ms     1.19  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((5000, 5000), 'huge_amount_groups', 3)
+        82.5±1ms         97.3±1ms     1.18  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((5000, 5000), 100, 3)
+         365±8ms          428±9ms     1.17  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((5000, 5000), 'huge_amount_groups', 'all')
          125±3ms          138±6ms    ~1.10  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((5000, 5000), 'huge_amount_groups', 3, 6)
          108±3ms          117±8ms     1.08  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((5000, 5000), 100, 3, 6)
          166±7ms          178±5ms     1.07  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((5000, 5000), 100, 3, 6)
          271±9ms          288±5ms     1.06  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((5000, 5000), 100, 'all')
        247±0.7ms          261±2ms     1.06  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((1000000, 10), 'huge_amount_groups', 'all')
         101±50ms          107±3ms     1.06  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((1000000, 10), 'huge_amount_groups', 'all')
         400±10ms          423±7ms     1.06  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((5000, 5000), 'huge_amount_groups', None)
          538±6ms         569±10ms     1.06  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((1000000, 10), 100, 'all', 6)
         298±20ms          313±4ms     1.05  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((5000, 5000), 100, 'all')
          236±5ms          248±4ms     1.05  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((5000, 5000), 100, None)
         78.4±4ms       81.9±0.9ms     1.05  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((1000000, 10), 100, None, 'reduction')
        82.7±60ms       85.8±0.4ms     1.04  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((1000000, 10), 100, 'all')
          144±8ms          150±4ms     1.04  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((5000, 5000), 100, None)
         79.2±2ms         82.0±2ms     1.04  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((5000, 5000), 'huge_amount_groups', None, 'reduction')
         109±40ms          112±4ms     1.03  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((1000000, 10), 'huge_amount_groups', 'all')
        88.7±50ms         91.6±2ms     1.03  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((1000000, 10), 100, None)
          231±3ms          238±3ms     1.03  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((1000000, 10), 100, 'all')
       3.33±0.02s       3.43±0.09s     1.03  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((5000, 5000), 'huge_amount_groups', 'all', 6)
         81.5±3ms         83.7±1ms     1.03  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((1000000, 10), 100, None)
         76.7±1ms         78.6±1ms     1.02  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((5000, 5000), 100, None, 'reduction')
          1.22±0s          1.24±0s     1.02  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((1000000, 10), 100, 'all', 6)
         455±40ms          465±9ms     1.02  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((5000, 5000), 'huge_amount_groups', 'all')
          549±2ms          558±8ms     1.02  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((5000, 5000), 'huge_amount_groups', 'all', 6)
          104±3ms          106±1ms     1.02  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((5000, 5000), 'huge_amount_groups', None)
          126±3ms          128±3ms     1.01  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((1000000, 10), 'huge_amount_groups', None)
        97.4±40ms         98.7±2ms     1.01  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((1000000, 10), 'huge_amount_groups', None)
       3.45±0.02s       3.50±0.03s     1.01  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((1000000, 10), 'huge_amount_groups', 'all', 6)
          292±7ms          295±2ms     1.01  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((5000, 5000), 100, 'all', 6)
          402±2ms          405±3ms     1.01  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((1000000, 10), 100, None, 'aggregation')
          229±2ms          231±1ms     1.01  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((1000000, 10), 100, None)
         462±20ms         464±10ms     1.00  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((5000, 5000), 100, 'all', 6)
       3.31±0.01s       3.32±0.01s     1.00  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((5000, 5000), 'huge_amount_groups', None, 6)
         293±10ms          293±9ms     1.00  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((5000, 5000), 100, None)
          405±4ms          405±2ms     1.00  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((1000000, 10), 100, 'all', 'aggregation')
        440±0.8ms          440±1ms     1.00  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((1000000, 10), 'huge_amount_groups', None, 'aggregation')
          103±4ms          103±1ms     1.00  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((1000000, 10), 100, None)
       1.43±0.01s          1.42±0s     1.00  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((1000000, 10), 'huge_amount_groups', None, 6)
          440±2ms          439±2ms     1.00  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((1000000, 10), 'huge_amount_groups', 'all', 'aggregation')
          249±1ms          248±4ms     1.00  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((1000000, 10), 'huge_amount_groups', None)
         96.5±1ms         96.1±4ms     1.00  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((5000, 5000), 100, None, 'aggregation')
       1.42±0.01s       1.42±0.03s     0.99  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((1000000, 10), 'huge_amount_groups', 'all', 6)
         286±10ms          284±9ms     0.99  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((5000, 5000), 100, None, 6)
       3.53±0.03s       3.50±0.01s     0.99  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((1000000, 10), 'huge_amount_groups', None, 6)
         534±10ms         529±10ms     0.99  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((5000, 5000), 'huge_amount_groups', None, 6)
       1.22±0.01s       1.21±0.01s     0.99  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((1000000, 10), 100, None, 6)
         104±60ms          103±5ms     0.99  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((1000000, 10), 100, 3)
        91.9±50ms         90.4±1ms     0.98  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((1000000, 10), 100, 'all')
          540±2ms          531±8ms     0.98  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((1000000, 10), 100, None, 6)
          1.05±0s       1.03±0.01s     0.98  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((1000000, 10), 100, 3, 6)
          127±4ms          125±2ms     0.98  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((1000000, 10), 'huge_amount_groups', 'all')
         111±60ms          109±2ms     0.98  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((1000000, 10), 'huge_amount_groups', None)
         464±10ms         454±10ms     0.98  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((5000, 5000), 100, None, 6)
          154±8ms          151±6ms     0.98  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((5000, 5000), 100, 'all')
         128±50ms        125±0.4ms     0.98  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((1000000, 10), 'huge_amount_groups', 3)
         99.3±2ms         96.6±2ms     0.97  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((5000, 5000), 'huge_amount_groups', None, 'aggregation')
          105±5ms        102±0.6ms     0.97  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((1000000, 10), 100, 'all')
       1.57±0.01s       1.53±0.03s     0.97  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((5000, 5000), 'huge_amount_groups', 3, 6)
          102±4ms         99.2±6ms     0.97  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((5000, 5000), 100, None)
        94.5±50ms         91.3±2ms     0.97  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((1000000, 10), 'huge_amount_groups', 3)
         237±10ms          228±9ms     0.96  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((5000, 5000), 'huge_amount_groups', 'all')
       1.25±0.01s       1.20±0.03s     0.96  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_quan((1000000, 10), 'huge_amount_groups', 3, 6)
          198±3ms          189±3ms     0.95  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((1000000, 10), 'huge_amount_groups', 3)
         236±10ms          225±9ms     0.95  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((5000, 5000), 'huge_amount_groups', None)
         91.8±1ms         86.9±3ms     0.95  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((1000000, 10), 'huge_amount_groups', None, 'reduction')
         92.5±1ms         87.0±3ms     0.94  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((1000000, 10), 'huge_amount_groups', 'all', 'reduction')
       3.56±0.01s       3.34±0.03s     0.94  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((1000000, 10), 'huge_amount_groups', 3, 6)
        81.7±60ms       76.3±0.9ms     0.93  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((5000, 5000), 100, 3)
          183±3ms          170±1ms     0.93  benchmarks.TimeGroupByDefaultAggregations.time_groupby_mean((1000000, 10), 100, 3)
        86.0±50ms       79.5±0.5ms     0.92  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((1000000, 10), 100, 3)
        93.3±60ms         86.3±2ms     0.92  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((1000000, 10), 'huge_amount_groups', 3)
         85.9±2ms         79.3±2ms     0.92  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((1000000, 10), 100, 'all', 'reduction')
         109±50ms          100±2ms     0.92  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((5000, 5000), 100, 3)
         385±30ms          351±3ms     0.91  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((5000, 5000), 'huge_amount_groups', None)
          116±2ms          104±3ms    ~0.90  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((5000, 5000), 100, 'all')
        88.2±50ms         78.3±2ms    ~0.89  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((5000, 5000), 100, 3)
         117±40ms        104±0.9ms    ~0.89  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((5000, 5000), 'huge_amount_groups', 3)
-         118±5ms          103±1ms     0.87  benchmarks.TimeGroupByDefaultAggregations.time_groupby_size((5000, 5000), 'huge_amount_groups', 'all')
-         110±5ms         95.8±2ms     0.87  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((5000, 5000), 'huge_amount_groups', 'all', 'aggregation')
-         112±4ms         97.6±1ms     0.87  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((5000, 5000), 100, 'all', 'aggregation')
        90.8±40ms       79.1±0.8ms    ~0.87  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((5000, 5000), 'huge_amount_groups', 3)
        86.1±50ms         74.3±5ms    ~0.86  benchmarks.TimeGroupByDefaultAggregations.time_groupby_sum((5000, 5000), 'huge_amount_groups', 3)
-        91.9±4ms         78.8±2ms     0.86  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((5000, 5000), 'huge_amount_groups', 'all', 'reduction')
-        89.9±5ms         76.8±1ms     0.85  benchmarks.TimeGroupByDictionaryAggregation.time_groupby_dict_agg((5000, 5000), 100, 'all', 'reduction')
        93.2±40ms         79.1±2ms    ~0.85  benchmarks.TimeGroupByDefaultAggregations.time_groupby_count((1000000, 10), 100, 3)
-         599±8ms          498±3ms     0.83  benchmarks.TimeGroupByMultiColumn.time_groupby_agg_mean((1000000, 10), 100, 3, 6)

TODO list

• commit message follows format outlined here
• passes flake8 modin
• passes black --check modin
• signed commit with git commit -s
• Resolves GroupBy.__getitem__ does not include 'by' columns into resulted object which leads to KeyError #3252, GroupBy and Agg Breaking with Multiple Parameters  #3262
• tests added and passing
• module layout described at docs/developer/architecture.rst is up-to-date

[modin-bot]

TeamCity Python test results bot

Tests PASSed

Tests Logs

ssssssssssssssssssssssssxssssssssssx.ssssssssssss [ 90%]
sssssssssssssssssssxsssssssssssssssss.xssssssssss.ssssssssssssssssssssss [ 90%]
ssxsssssssssssssssxxxs.ssssssssssssssssssssssssssssssssssxssssssssssssss [ 90%]
ssssssx.xsssssssssssssssssssssssssssssssssssssssssssxsssssssssssssssssss [ 90%]
sssssssssssssssssssxxsssssssssssssssssssssssssssssssssssssssssssssssssss [ 91%]
sssssssssssssssxssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 91%]
ssssssssssssssssssssssssssssssssssssxsssssssssssssssxsssssssssssssssssss [ 91%]
ssssssssssssssssssssssssssssssssssssssssssxsssssssssssssssssssssssssssxs [ 91%]
sssssssssssssssxssssssssssssssssssssssssssssssssssssssssssxsssssssssssss [ 91%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 91%]
sssssssssssssssssssssssssssssssssssssssssssssssssxsssssssssssssxssssssss [ 92%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssxss.s [ 92%]
ssssssssssssssssssssssxsssssssssssssssssssssssssssssssssxsssssssssssssss [ 92%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 92%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 92%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 92%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 92%]
sssssssssssssssxssxssssssssssxssss.s..ssssssssssssxsssxxsssssssssssxxs.ss [ 93%]
.sxx..sssssssssssssssssssssssssssssss.xxxx..xsssx.sxssssssssssssxxssssss [ 93%]
sssssssx...xxxx..xxxssssssxxxssx..xs..xxssssssssxxx.xxsxx.xxxssxxxxxxxxx [ 93%]
xssssssxxxxxxxssssxs.xsxxsssssssxxx.xx..xs.xss.xxxssssx.sssxxxxsssxsssxs [ 93%]
ssss.xsxxsssxs.s.sssssssssssssss.x.xsssssssssssx...sssssssssxsssssxx.xxx [ 93%]
x..ssssssx.xs.x.xssssxsxssxxxx.sss.xssssxssxxxssss.xss.ssxssssx..xxxxxxx [ 93%]
s..xs.xsssssssssxxxxssss.sssssssxxssssssssssxx..xsssssx..xsssssssssss.xx [ 93%]
xxxx.sxxxsssssxx..xssssxssssx.sssxx.sssssx.xx..x.xxxxxxsssxxsssxss.xssss [ 94%]
ssssssssssss.xxsssssssss.xxsssssssssssssxsxxsxx.ssssxxx.xx.sxxxxxxx.xx.x [ 94%]
x.xxxx.xxssssssssssxsssssssssssssssxxsxsssxsxssssx.sssxxssssxxxxssssssss [ 94%]
sssxx.xxxssssssssssssss.sssssssssxxxssssxxsssssssssxssss.xxxs.ssssssssss [ 94%]
xsssxxsssxxxxxssssxxssssssssssssxxsssssxssssssssssssxsssssssssssssss.x.s [ 94%]
ssxxsssssssxx.x.sxsssxxsxsxxssssss..sxxssssssssssssssssssx.xsxsssssssss. [ 94%]
ssssssss.ssssssxssssssssssssssssssxsssxssxsssssssssxssxsxssssssssssssxssss [ 94%]
sxssssxxssssxsssssssssxsssssssssssssssssssssssssssssssssssssssssxssssssss [ 95%]
ssssssssxssssssxsssssssxssssssssssssssss.ssssssxsxsssssssssssxssssssssss [ 95%]
ssssssssssssxssssssssssssssssssssssxssssssssssssssssssssss..xsssssssssss [ 95%]
ssssssssssxsssssssssssxsssssssssssssxssssssssssssssssssss.ssssssssssssss [ 95%]
xsxsssssssssssssxsssssssssssssssssssssssssssssssssssssssssssssssxsssssss [ 95%]
sssssssssssxsssssssss.ssssss.sxsssssssssss..sssssssss.sss.s...ssxssss.s. [ 95%]
s.....s..s.sss..ssssssss....xs................x.s.......s....s.........s [ 96%]
..s.s.............s..s......s.....s...........sss.......s...s......s.s.. [ 96%]
.......................................s.......s........................ [ 96%]
..................s..................................................... [ 96%]
......................................................................... [ 96%]
.............sss........................................................ [ 96%]
............................x..........x.........x...................x.. [ 96%]
.....................x........................xssss....................... [ 97%]
........................................................................ [ 97%]
.............................x.......................................... [ 97%]
..............................x.x..............x........................ [ 97%]
....x..........xx.....x.....x..............................x............ [ 97%]
....................x.....................................x............. [ 97%]
........................................................................ [ 97%]
.....................xssss.............................................. [ 98%]
........................................................................ [ 98%]
....................x................................................... [ 98%]
.................x.....................................................x [ 98%]
.......................................................x................ [ 98%]
....................................x................................... [ 98%]
.....................................xsss...s........................... [ 98%]
..x.................xx..x......xx.x...x..xx..x.x..xxxxxssss............. [ 99%]
.x..xx.x..xxxxssss...................xx.x....x....x........sxssss....... [ 99%]
.s....x....x..s..s.....x........x.............sss.ss.sssssssssss.ssss.ss [ 99%]
.s.ssss.ss.sssss..x.....s.xs..s.....s...ss.....s..xssss...s.........s... [ 99%]
......................x.......x......sx..s...sxs......s..x.sxx...s.sxsss [ 99%]
s...........x....x.s..x...x...sx..x..ssss................x...s..x....x... [ 99%]
........x.................s...........................................   [100%]error: object directory /localdisk/tc_agent/system/git/git-CE4319E5.git/objects does not exist; check .git/objects/info/alternates
fatal: bad object HEAD


---------- coverage: platform linux, python 3.8.10-final-0 -----------
Coverage XML written to file coverage.xml

= 8342 passed, 37237 skipped, 3276 xfailed, 20928 warnings in 192.71s (0:03:12) =
PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.
Tests are successful
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /modin, configfile: setup.cfg
plugins: benchmark-3.4.1, cov-2.11.0, forked-1.3.0, xdist-2.3.0
collected 1831 items

modin/pandas/test/test_io.py ........................................... [  2%]
........................................................................ [  6%]
.............................................ssssssssssssssssssss.ss.ss. [ 10%]
ss.ss.ss.ss.ss.ss.ss.ss.ss.ssssssssssssssssssss.ss.ss.ss.ss.ss.ss.ss.ss. [ 14%]
ss.ss.ss.ssssssssssssssssssss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.sssssssss [ 18%]
sssssssssss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ssssssssssssssssssss.ss.ss. [ 22%]
ss.ss.ss.ss.ss.ss.ss.ss.ss.ssssssssssssssssssss.ss.ss.ss.ss.ss.ss.ss.ss. [ 25%]
ss.ss.ss.ssssssssssssssssssss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.sssssssss [ 29%]
sssssssssss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss............................ [ 33%]
........................................................................ [ 37%]
.......................................................................x [ 41%]
x....................................................................... [ 45%]
........................................................................ [ 49%]
........................................................................ [ 53%]
........................................................................ [ 57%]
........................................................................ [ 61%]
........................................................................ [ 65%]
........................................................................ [ 69%]
........................................................................ [ 73%]
........................................................................ [ 77%]
........................................................................ [ 80%]
........................................................................ [ 84%]
........................................................................ [ 88%]
..............................xxxxxxxxxxxxxxxxxxxxxxxx.................. [ 92%]
.............................................................x.......... [ 96%]
...................X........x................xx....ss.......             [100%]

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Coverage XML written to file coverage.xml


= 1462 passed, 338 skipped, 30 xfailed, 1 xpassed, 1734 warnings in 80.50s (0:01:20) =
Closing remaining open files:/modin/modin/pandas/test/io_tests_data/dc4e1fac0b5b11eca6be0242ac110002.hdf...done/modin/modin/pandas/test/io_tests_data/dc4e1dea0b5b11eca6be0242ac110002.hdf...done
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /modin, configfile: setup.cfg
plugins: benchmark-3.4.1, cov-2.11.0, forked-1.3.0, xdist-2.3.0
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I / gw16 I / gw17 I / gw18 I / gw19 I / gw20 I / gw21 I / gw22 I / gw23 I / gw24 I / gw25 I / gw26 I / gw27 I / gw28 I / gw29 I / gw30 I / gw31 I / gw32 I / gw33 I / gw34 I / gw35 I / gw36 I / gw37 I / gw38 I / gw39 I / gw40 I / gw41 I / gw42 I / gw43 I / gw44 I / gw45 I / gw46 I / gw47 I
gw0 [48855] / gw1 [48855] / gw2 [48855] / gw3 [48855] / gw4 [48855] / gw5 [48855] / gw6 [48855] / gw7 [48855] / gw8 [48855] / gw9 [48855] / gw10 [48855] / gw11 [48855] / gw12 [48855] / gw13 [48855] / gw14 [48855] / gw15 [48855] / gw16 [48855] / gw17 [48855] / gw18 [48855] / gw19 [48855] / gw20 [48855] / gw21 [48855] / gw22 [48855] / gw23 [48855] / gw24 [48855] / gw25 [48855] / gw26 [48855] / gw27 [48855] / gw28 [48855] / gw29 [48855] / gw30 [48855] / gw31 [48855] / gw32 [48855] / gw33 [48855] / gw34 [48855] / gw35 [48855] / gw36 [48855] / gw37 [48855] / gw38 [48855] / gw39 [48855] / gw40 [48855] / gw41 [48855] / gw42 [48855] / gw43 [48855] / gw44 [48855] / gw45 [48855] / gw46 [48855] / gw47 [48855]

......................................................................... [  0%]
........................................................................ [  0%]
........................................................................ [  0%]
........................................................................ [  0%]
........................................................................ [  0%]
....................................................x................... [  0%]
.........................x.............................................. [  1%]
......................................x................................. [  1%]
........................................................................ [  1%]
........................................................................ [  1%]
........................................................................ [  1%]
........................................................................ [  1%]
........................................................................ [  1%]
........................................................................ [  2%]
.......................................................................... [  2%]
........................................................................ [  2%]
........................................................................ [  2%]
......................................................................... [  2%]
......................................................................... [  2%]
......................................................................... [  2%]
......................................................................... [  3%]
........................................................................ [  3%]
......................................................................... [  3%]
........................................................................ [  3%]
.......................................................................... [  3%]
.......................................................................... [  3%]
........................................................................ [  3%]
................................................................x....... [  4%]
......................................................................... [  4%]
........................................................................ [  4%]
........................................................................ [  4%]
.....................x.............................................x...... [  4%]
........................................................................ [  4%]
...............x........................................................ [  5%]
............................................................................ [  5%]
......................................................................... [  5%]
........................................................................ [  5%]
.......................................s.........................s...... [  5%]
..................s.............x..................s.................... [  5%]
........s........................s.....................................s. [  5%]
.........................s.......................s...................... [  6%]
.......s...........................s....................s................. [  6%]
......s.............................s.............s............s.......... [  6%]
...s........s.................s........s...............s........s....... [  6%]
..........s.......s..............s......s...............s...........s.... [  6%]
........s....s..................s.....s..............s.x.....s.......... [  6%]
s.........s.x.............s.....s................s......s............s... [  6%]
...s...ss.......s....s.....s.s.....s....x..s....sss..s....s....s..s..s... [  7%]
sss..ss.s.s..s.s..s.....s..s.s.s...sss.s.s.s.s.s.sss..s..ss..s..ss..ss.. [  7%]
.s.s.ss.ssssss..ss.....ss..s.s..s..ssss.s...ssss..sss.ss.ssss..ss.sss.s. [  7%]
.sss.ssssss..ss.sss..s.s.ss.sss.sss.ss.s..sss...s.ss.ssss.sss.ss.ss.ss.. [  7%]
.s.sss...s.sss..ssssssssss.sss..ssss.sss.ssss...s.s.....s.s.ssssssssss.s [  7%]
ssss..sss...ss.ss.ss.ssss.s.s.sss.sssss.ssssssss...sss.sss..ssss.ss.ssss [  7%]
ssss.ss.s.ss..ss.s.ssss.s.sssssssssss.s.ss..ssss..ss.sss.ss.sss.sss.ssss.s [  8%]
s.sss.sssss.sssssssss...s.ssssssssssssssss.ss..sss.s.ssssssssss.ssssssss [  8%]
ssssss..sxssssssss.ssssssssssssssssssssssss.s.sss..sssss.ssssss.ssssssss [  8%]
ssssssssssssssss.ssss.ssssssss.s.sssss.ssssssssssssss..sssssssssssssssss [  8%]
ss.ss.ssss.sssssssssss.ssss.ssss.sssssssssss.s.sssssss.ssssssssssssss.ss [  8%]
ssss.ssssssssssss.s..ssssssssssssssssssssssssssss.sssssssss..sss..sss..s [  8%]
sssssssssssssssssssssssssss...ssssssss.s.ssssssssssssssssssssssssssssss.s.sss.s [  8%]
sssss.sssssss.ssssssssss.s.ssssssssssssssss.ssss..ssssssss.ssssssssssssss [  9%]
ssxssss.ssssssss.sss.ssss.ssssssssssssssssssssssssssssssssssssssss.sssss. [  9%]
sssssssssssss.sssssssssssssssssssss.sssssssss..sssssssssssssss.sssssssss [  9%]
ssssssssssssssssssss.sssssssssssssssssssss.ssssssssssssssssssssss.ssssssss [  9%]
ssssssssssssssssssssss.ssssssssssssssss.ssssssssssssssssssssssssssss.sssssss [  9%]
ssssssssss.ssssssssssssssssssssssssssss.ssssssssssssssssss.ssssssssssssssss [  9%]
sssssssssssssssssssssssssssss.ssssssssssssssssssssssssssssssssssssssssss [  9%]
sss.sssssssssssssssssssssssssssssssssssssssssssssss.ssssssssssssssssssss [ 10%]
ssssssssssssssssssssssssss.ssssssssssssssssssssssssssssssssssssssssssssssss [ 10%]
.sssssssssssssssssssssssssssssssssssssssssssss.sssssssssssssssssssssssssss.s [ 10%]
sssssssssssssssss.ssssssssssssssssssssssssssssssssssssssssssss.ssss.ssss [ 10%]
sssssssssssssssssssssssssssssssssssssssss.ssssss.sssssssssssssssssssssssss [ 10%]
sssssssssssssssss.ssssssssssssssssssssssss.sssssssssssssss.ssssssssssss.ss [ 10%]
sssssssssssssssssssssssssssssssssssss.sssssssssssssssssssssssssss.ssssss [ 10%]
sssssssssssssssssssssssssssssssss.sssssssss.sss.ssssssssssssssssssssssss [ 11%]
ssssssssssssssssssssssssssssss.sssssssssss.sssssssssssssssssssssssssssssss [ 11%]
.sssssssssssssssssssssssssssssssssssssss.sssss.sssssssssssssssssssssssss [ 11%]
sssssssssssssssssssss.ssssss.sss.sssssssssssssssssssssssssssssssssssssss [ 11%]
sssss..ssssssss.ssssssssssssss.ssssssssssssssssssssssss.ssssssssssssssss [ 11%]
sssss..sssssssssssssssssssssssssss.ssssssssssssssssssss.sssssssssssssssss [ 11%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 12%]
ssssssssssssssssssssssssssssssssssssssssssssssss.ssssssssssssssssssssssssss [ 12%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 12%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 12%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 12%]
sssssssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssss [ 12%]
ssssssssssssssssssssssssssssssssssssssss.s.sssssssssssssssssssssssssssssss [ 12%]
sssss.ssssssssssss.sssssssssssssss.ssssssssssssssssssss.ssss.ssss.ssssss [ 13%]
ssssssssssssss.sssssssssssssssssss.sss.sss.sssssssssssssssssss.sssssssss [ 13%]
sssssssssssss.ss.sss.sssss.ssssssss.s.ssssssssss.ssssssssssss.sss.sssssss.s [ 13%]
ssssssssss..ssssssss.sssssssssssssssss.ss.sssss.ssssss..s.sssssssss.sssss [ 13%]
ssssssssssss.s.ssss.sssssssssss.ss.ssssss.ssss.sssssssssss.ssssssssss.s. [ 13%]
ssssss.ss.ssss.ssssss.ssssssssssss.ssssss.sssssssss.sssss.sssssss.sssss. [ 13%]
sssss..sssssssss.sssssssssss.ssss.sssssss.sssssssssssssssss..ssss.sss.ss [ 13%]
.ssss.sssss.ssssssss.sssssss.s.sssssssss..sssssssssss.sss.sss.sssssssssss [ 14%]
..s.sssssssss.s..sssssss.s.sss.sssss.ssssssssssssss...ssss.ss.sssss..sss [ 14%]
ss.ssssss.ss.ss.ssssssss.sss.sssssssssssssss...sss.sssss.ss..sssss..ssss [ 14%]
sssssssss.ssssssssss.s..sssssssss.s.sssssssss..sssssssssss.sssssssss.s...s [ 14%]
sss.ssssss..s.ssssssss.ssssssssss.sssssssssss..s.sssssss.ss.s.sss.ssssssssss. [ 14%]
ssss.ssssssss..sssss.s.ss.sssssssss.ssssss.sss.s.ssssss.s.ssssssssssss.s [ 14%]
s..ssssssss.ssssssss.ss.ssssssssss..ssssssss.s.sssssss.ssssssssssssss.ss [ 15%]
ssss.s..sssss...ss.sss.ssssssssssss.sssss.s..sssssssssss.sssssssss.sss.s [ 15%]
sssssssss.sssssssss.s.sss.ss.ssss.ssss.s.sssssssssssssssss.sssssssss...s [ 15%]
..sssss.ss...ssssss.ssssssssssssss.ss.sssss....ssssssssssss.sss..sssss.s [ 15%]
sssssss.sssss.ssssss.s.s..ssssss.s.ss.ssssssssssssssssss.sssss.s.s..ssss.ss [ 15%]
ss.sss.sssss.ss.sssssssssssss.sssss..s.ssssssssss.ssssssss..ssssssssss.sss [ 15%]
ssssss.ss.s.ssssss.ssssssss.s.ss.ssssssss.sss.sssss.ssssss.ss..ssssssss. [ 15%]
ssss..sssssss..ssss.ssss.ssssssssss.s..s.sssssss.sssssssssss.s.ssss.sssss. [ 16%]
sssss..sss.sssssssssss.sssssssssss...s.ssss.ss.ss.sssssss.ss.s.sssssss...... [ 16%]
ssss...ss.sssssssss.sss.s.ss.ssss..ssssss.....ssssss.sssss..sss.ssssss.ss.s [ 16%]
.ss.ssssssss.sssssssssss...ssss.s..sss.ssss.ssssssss.s.sss.ssssssssssssss [ 16%]
.ss.ssssss.sss.ssssssssss.ssss...s.s.sssssssss.sssssss.sssssss.sssss.s.s [ 16%]
sssss.s.s.s..ssssssssssssss.sssssss..sssss.s.ssssssss.ssssssss...s..ssss [ 16%]
s.sssssss.ssssss.sssssssss.s.ssss.ss.s.s..ss...ssss.sssss.ssssssss.sssss [ 16%]
sss.ss.sssssssss..s.sss..ssss.sss.ss.sssssss.sssssss.sssssssssssss.ss.sss [ 17%]
s.s.ss.sssssssss.sssss.sssss.s.ssss.sssssssssss.ss..s.ssss.ssss..ssss.ss [ 17%]
..sssssssss.sssssss.sssss....sssssss.sssss.ss.sssssss.sssss..ssss.sssss.s. [ 17%]
ss.sssssssss..sssss..sssssss.ssssssss.ssssssssss...ss.s.ssss.ss..ssssss. [ 17%]
ssssssssssssss.ss.sss..s.ssssss.ssssssssss..ssssss.sssss.ssssssssssssss...s [ 17%]
sssss..ss.ssss...s.sssssssssssssssss.sssss...sss.ss.sssssss.ss.s.ssssssss [ 17%]
sssss.ssss.ssssss.ssss.sssssssssssss..sssssssssss.ssssss.s.ssssssssss..ssss [ 18%]
s.sssssss.sssssssss..sss.ss.s..sssss..ssssssssssssssssssssss..ss.ssssssssss [ 18%]
ssssssssssss.s.ssssss.ss.ss.sssss.s.ssssssssssss.sssssssss.s.sssssssssssss [ 18%]
ssss..ssss.....sssssssssssssss.sssss.sssssssssss.ss.sss...ssssssssssssss [ 18%]
ss.ssss.s..ssssss.ssssssss.sssssss.s.s.sss.ssssssss.s..ssss.ssssssssssss [ 18%]
ssss.ss.ss.sss.s.s.ssssssssssss.sssssss.ss.ssssssss.ssssss.s.ssssssssss.s [ 18%]
.ssss.ssss...sssssssssssssssss.ss.ssss.ssssss.ssssssssssssss.sssssssss.ss [ 18%]
s.sssssssss.ss.ssssss.sss.sssssssssss.ssssssssssss.s.sssss..ss.sssssssssss [ 19%]
ssssssssssssssssssss.sssssssssss.sssssssss.sssssssssssssssss.sssssssssss [ 19%]
sssss.sss.sssss.ssssssssssss.sss.ssssssss.ssss.ssssssss.sssss.ssssssss.sssss [ 19%]
sssssssssss..ss.sssss.sssssssssssssss.sssssssss.sss.s.sssssssss.ss.sss.. [ 19%]
sss.sssssssssssssssss...ssssss.s.sssssssss.ss.sss...ssssssssssssssssssssss [ 19%]
s.s.sssssssssssssssssssssssssss.sssss.sssss.s.ssss.sssssssssss.ssssss.ssss. [ 19%]
s.sssssssssssssss.sssss.s.sssssssssssssss..ssss.ssssssssss.sssssssssssssss [ 19%]
sssssssssss.ssss.sssssssssssssss.sss.ssssssssss.sssssss.sss.ssssss.sssss [ 20%]
ssss.sssssssssssssss.sss.sssss..s.sssssssss.ssssssssssssssssssssssssssss [ 20%]
sssssss.sssssssssssssssssss.s.sss.ssssssssssss.s.ss.ssss.sssssssssssssss [ 20%]
ssssss.ssssssssssssssssss.s.sss..ssss.sssssssssssssssss.sss.ssssssssssss [ 20%]
ss.sssss.ssssssssss.ssssssssssss.ssssssssssssssssssssss.ssssssssssssssss [ 20%]
ssss.s.sssssssss.sssssssssss.s.ss.ssssssssssssssssssssssssss.sssssssssss [ 20%]
sssssssssss.sssssssss.sssssssssss.sssssssssssssssssssssssssssssssssssssss [ 21%]
ssssssss.ssssss.sssssssssssssssss.ss.sss.sssssssssssssssssssssssssssssss [ 21%]
ssssss.sssssssssssssssssssssssssss.s.sssssssssssssssssssss.ssssss.ssssss [ 21%]
sssss.ssssssssssssssssssssssssssssss.sssssssss.sssssss.ssssssssssssssssss [ 21%]
.ssssssss.sss..ssssssssssssssssssssssssssss.ssssssssssssss.sssssss.sssss [ 21%]
sssssssssssss.sssssss.ssss.sssssssssssss.sssssssssssssssssssssssssssssss [ 21%]
sssss.sss.sssssssssssss.ssssssssssssss.sssssssss.sssssssssssssssssssssss [ 21%]
ssssssssssssss.ssssss.sssssssss.sssssssssss.sssssssss.ssss.sssss.sssssssss [ 22%]
sssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssss.ssssssss [ 22%]
sss.sssssssssssssssssssssssss.ssss..sssssssssssss.ssssssssssssssssssssss [ 22%]
.ssssssssssssssssssssssssssssssssssssss.sssss..sss.s.s....ss..sssssss.ss [ 22%]
sssss.ssssssssssssssssssssssssssssssssssssss..ssssssssssssssssss....ssss [ 22%]
s.sssss..sssss.....sssssss.sssssssssssssssssssss..ss.ss.s.s..ss..s..s.s. [ 22%]
.s..s...s.s.sssssssssssssssssssss.s...s..s..s.ssssssssssss..s.s.s.s.s.s. [ 22%]
.s..s.sssss.s.s.sssssssss..s.s.ss.s.s.s..s.s.sssssssssssssssssssssssssss [ 23%]
s.s.ssssssssss...ssss.sssssssss.ssssssssss.s..sssssssssssssssssssssss..s [ 23%]
.ssssssssssssssssssssssssssssss.ssssssssssssssssssssssssssssssssssssssss [ 23%]
sssssssssss..ssssss.s.sssss.ssssssssss.s.s.s.s.sssssssssssss.s...sssssss [ 23%]
sssssss.sssssssssssssssssssssss.ssssssssssssssssssssssssssssssssssssssss [ 23%]
ss..s.s.ssssss.sss.sssss.s.sss.s.s.s.sssss.sssssssssssss.ss.ssssssssssss [ 23%]
ssssssssss.sss.ssssssssssssss.sssssss.ss.sssss.sssssssssssssssssssssssss.sss.s [ 23%]
ssss.sssss.ss.ssssss.sssssssssss.ssss..ssssssssssssssssss.sssssssssssssss [ 24%]
sssssssssss..ssssssssssss.sssss.s.s.sssss.ss.ssssss.s.sss.sssssssssss.ss [ 24%]
.sssssss.sssssssssssssssssssssss.ssssssssssssssss.ss.ss.ss.s........sssssss [ 24%]
sssssssssssss.sss.s..s.sss......sssss.ss.ssssssssssssssss.ss.ssss..ssss.... [ 24%]
ssssssssssssssssssssssss.sss.sss..sss.s...sssssssss.ssssss.sssssssssss.s. [ 24%]
ssss..ssss.s.s.s.sssssssssssssssssssssssss.sssss.ssssssss.sss..sssssssss [ 24%]
ssss.sss.ssssssss.sss.ssss.ss.s.ssssssss..sssssssssssssssssss.sssssssssssss [ 25%]
sss..ss.ssssssssssssssss.ss.ssssss.ss.sssss.s.ssssssssssssssssssssss..ss [ 25%]
ssssssssss.ssssssssssssssssss.s.ss.s.ssssss.sss.ssssssss.sssssssss..ss.. [ 25%]
ss.ssssssssssssssssssssss.ss.sss...sssss.sssssssssssssssss.sssssssss..ss [ 25%]
.ssssssssssssssssss.sssssssssssssssss.sss.ssssss..ss.sss.ssssssssssssss.sss [ 25%]
ss.ssssssssssssssssss..sss.ssss.sssssssssss.ssss.sss.sss.ss...ssssssssss [ 25%]
ssssssssss..ssssssssssssss.sss.s.ss.sss.ss.ssssssssssssssssss.sss.ssssssss [ 25%]
ssssss.s.sssssssssssssss.ssssss.ssssssssssss.sss.ssss.ssss.s.sssssssssss [ 26%]
ssssssssssssssss..ssssssssssssss.sssssssssssssssssssssss.sss.s..ssssssss [ 26%]
ssssssssssss.sssss.sssssssss.sssssssss.sssssssss.ssss.sss.ss.ssssssssss. [ 26%]
sssssss.sss.ss.sssssssssssssssssss.ssssssssss..ssss.sssssssssssssssss.sss [ 26%]
sss.sss.sssssssssssssssssssssssssss.ssssssssss.ss..ssssssssss.ssssss.sssss [ 26%]
.ssss.s.ss.sssssssssssssssssssssss.ssssssssssssss..ssssssssssss.s.ss.s.s [ 26%]
sssss.ssssssssssssssss..sss.ssssssssssssss.ssssssss.sss.ss.sssssssss..sss [ 26%]
.ssssssss.ss.ss.sss.ssssssssssssssss.ssssssssssssssss.s.s.ssss.sssssssssss [ 27%]
sssssssssss.ss.sssssssssssssssssssssssssssssssss.ssss.sssssss.sssssssss.. [ 27%]
ss...sss.ssssssssssssssssss.sss.s.ssssssssss.s.ssss.sssssssssssss.ssssss [ 27%]
sssssss.sssssssssssss...ssssssssssssssssssss.ssssssss.ssssss...s.ssss..s [ 27%]
sssssssssssssssssssssssssss.ssssssssssssssssssssssssssssssssssssss.sssss [ 27%]
s.ss.ssssssss.s.s.sssssssss.s.sss.ssssss.sssss.s.sss.ssss.ssssssssssssss [ 27%]
ssss.sssssssss.sssssssss..ssssssssssssssssss.sss.s.ssssss..ssssssssss.ssss [ 28%]
.s.ssssssssssssssssssssss.s.ssssssssssssssssssssssssssssssssssssss.sss.ss [ 28%]
ssssss.sssssss.sssssss.ss.ssssssssssss.ss..s.sssssssssssss.sssssssss.ssss [ 28%]
sssss.s.sssssssssssss.sssssss.ss.s.sssssssssssssssss.sss.sss.ssssssssssssss [ 28%]
.ssssssssssss.s.s.ssss.sssssss.sssssssssssssssssssssssssssssssssssssss.sss [ 28%]
sssssssssss.sssss.ss..ssss.s.sssssssss..ss.sssssssssssssss.ssssss.ssssss [ 28%]
.ssssss.s.ssssssssssssss..ss.ssssssssssssssss.sssssssssss.ssssssssss.ssss [ 28%]
ss.ssssss.sss.ssss.ss..ssssssssssssssssssssssssssssssssssssssssssss.ssss [ 29%]
ss.ssssssssssssssssssssss.s.ssssssss.ssssss.sssssssssssssssssssss..ss..ss [ 29%]
ss.sssssss.ssssss.s..ssssss.ssssssssssssssssss.ssssssssssssssss.ssssssss [ 29%]
sssssssss.sssss.s..ss.ssssss.sssssss.ss.s.sssssssssssssssssssssssssssssss [ 29%]
ssssss.sssssssss.sssssssssssss.sssssssssssssssssssssssssssssssssss.sss.s [ 29%]
ssss.sssssssssssssssssssssssss.s..s.ss.s.ss.sssssssss..ssssssssssssssssss [ 29%]
sssssssssssssssssss.s.sssss.sssssssss.s.ssssssssssssssssssssss.s.s.sssss [ 29%]
ssssssssssssss.ssssssssssssssss.s.ssssssssssssssssssssssss.sssssssssssss [ 30%]
sssssssssssssssss.ssssss.ssssssss.sssssssssssssssssssss.s.s.s.s.s.s.s.ss [ 30%]
sssssssssssss.s.s.sss.ssssssssssssssssss.sssssss.ssssssssss.s.s..s.s.ss. [ 30%]
s.ssssssssssssssssssssss..s.sssssssssssssssssssssss.s.ssssssssssssssssss [ 30%]
sss.s.ssssssssssssssssssssssssss.s.ssssssssssssss.ssssssssssssss.ss.s.s. [ 30%]
...sss.s.s.ssssssssssssss..ssssssssssssssssssssssss.sssssss.ssssssssss.ss [ 30%]
sssssssss.ssssss..sssssss..s.sssssssssssss.ssssssssssssssssssssss.sssss. [ 31%]
sssssssssssssssssssss.sssssssss.sssssssssssssssssss.ssss.ssssssssssssssssss [ 31%]
.ssssssssssssssss.sss.ss...ss.sss.sssssssssss.sssssssssssssssssssssssssss. [ 31%]
ssssss.sssssssssssssssssss.ssssss.sssssssssss....ssssssssssss.ssssssssss [ 31%]
ssssssssssssssssssssssssssssss.sssss..ssssssssssssssssssss.ssssssssss.ss [ 31%]
.sssssssssss.ssssssssssssssssssss.ss.sssssssssss...sssss.sssssss.ssssssss [ 31%]
.sssssssssssssssssssss.ss.sssssssss.ssssssssssssssssss.sssss.sssssssssss [ 31%]
sss.ss..ssssssssssssssss.sssssssssssssssss.ssssss.ssssssss.ssssssssssss. [ 32%]
sssssssssss.sssssss.ssssssssssssssssssssss.sssssssssss.ssssss.ssssssssss [ 32%]
sssssssssss.s..s.sssssssss.sssssssssss.ssssssssssss.s.ssssssss.ssssss.ss [ 32%]
ssssssssssss.sssssssssssssssssss...ssssssssssssssssssssssss.ssssssssssss [ 32%]
sss.ssssssss.s..ssssssssssssssssss.ssssssssssss.ssss.sssssssssssssssssss [ 32%]
sss.ssssssssssss.sss.sssssssssssss.ssss.sss.ssssssssssss.sssssssss.ss.ssss [ 32%]
sss.ssssssssssssssssss.ssss.sss.sssss.ss.sssssssssssss.ss.ssssss.sssssss [ 32%]
ssssssssssssssssss.ssssssssss.ssssssss.ssssssssssssssssss.ss.s.....sssssss [ 33%]
ss.ssssss.ssssssss.sssss.ssssss.ssss.....ssssssssssss.sssss.sssssssssssss [ 33%]
ss.sssssss.ssssssss....sssssss.sssssssssss.ssssssss.sssssssssss.sss....s [ 33%]
ssssssssss.s.sss.sssssssssssssss.ss.s.ssssssss.sssssssssssssssss.ss.ssss [ 33%]
ss.sssssssssss.sssssssssssssssssssss.sssssssss.sssss..ssssssssssssssss.sssss [ 33%]
sssssssssss..ssssss.s.sssss.ssssssssssssssssssssssss.sssssss.s.sssssssss [ 33%]
ss.s.sssssssssssssssssss.ss.ssssssss..sssssssssssss.ssssssssssss.s.ssssss [ 33%]
sss..ss.sssss.sssssssssssss.ssssssssssss..sss.ssssssssssssssssss.sss.sss [ 34%]
.ss....sss.s.sssssssssssssssss.sssssss.s..s..ssssssssss.ssssssss.ssssss. [ 34%]
.sss..sssssssssssssssssss.sss.ssssssssssssssss.sssssss.sssssss.ssssssssss..s [ 34%]
sss.sssssssssssssssssssssssssssssssssss..ssssssssssssssssssss.sssssss.ssss [ 34%]
ssss.sssssssssssssssssssssss.s.sssssssssssss.ssssss.ssssssss.ssssssss.sss [ 34%]
ssssss.ssssssssssssssssssss.sssssssss.ssssssssssssssssss.ssssssssss.s.ss [ 34%]
.ssssssssss.ssssss.ssssssssssssss.ssssssssssssssssssssss.sssssssssssssss [ 35%]
ss.ssssssssssssssssssssssssssssssssss.sssssssssssssssssss.sss.sss.sss.ss [ 35%]
ssssssssssss.s.sss.sssssssssssssssssssssssssss.sssssssssssssssss.sssssss [ 35%]
ssssssssssssss.ssssssssss.sssssss.sssssss.sssssssssssssssssssssssss.ssss [ 35%]
ssssssssssssssssssssssssss.sss.ssssssssssssssssssssssssssssssss.ssssssss [ 35%]
ss.sssssssss.s.ssssssssssssssssssssssssssss.sssssssssss.ssssssssssssssss [ 35%]
ssssss.sssssssssssssssss.s.ssssssssssssssssssssssssssssssssss.ssssssssss [ 35%]
sssssssssssssssssssssssss.sssssssssssss.sss..sssssssssss.sssssssssssssss [ 36%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss. [ 36%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.sssss [ 36%]
ssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 36%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 36%]
ssssssssssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssss [ 36%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 36%]
ssssssssssssssssssssssssssssssssss.sssssssssssssssssssssssss.sssssssssss [ 37%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 37%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 37%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 37%]
ssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssss [ 37%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 37%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 37%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 38%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 38%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 38%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 38%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 38%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 38%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 39%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 40%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 40%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 40%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 40%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 40%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssss.ssssssssssssssss [ 40%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssss.sssssssssssssssss [ 40%]
ssssssssssssssssssss.ssssssssssssssssssssssssssssssssssssssss.ssssssssss [ 41%]
ssssssssssssssssssssssssssssssssssssssssssssssssss...sssssssssssssssssss [ 41%]
sssssssssssssss..ssssss.sssssssssssssssss...ssss.ssssssss.s.ssssssssssss [ 41%]
sss..ssssssssssssss..ssssssssssssssssssssssssssssssssssssssssssssssss.ss [ 41%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 41%]
ssssssssssssssssssssss.....ssssssssssssss.ssssssssss.ssssss.s........sss [ 41%]
ssssssssss.sssss..ssssssssssssssssssssssssssssssssssssssssssssssssssssss. [ 41%]
sssssssssss...sssssssssssssssssssssssssssssssssssssssssssss.s.ssssssssss [ 42%]
ssssssssssssssssssssssssssssssssssssssss.sssssssssss.sssssssssssssssssss [ 42%]
ssssssssssssssssss.sssssssssssssssssssssss.sssssssssssssssssssssssssssss [ 42%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.s.sss.ss [ 42%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 42%]
ssssssssssssssssssssssssssssssssssssssssssssssss.sssssssssssss.sssssssss [ 42%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss..sssssss [ 42%]
sssssss.sssssssssssssssssssssssssssssssss.ssssssssssssssssssssssssssssss [ 43%]
ssssss.sssssssssss..s.ssssx....s.sss..ssss...sssssssssssssssssssssssssss [ 43%]
s..ssssssssssss.sssssssssss...sxssssssssssssssssssssssssss.xssssssssssss [ 43%]
sssssssssssx.ss.sxsssssssssssssssssssssssssssssssssssssssss.ssssssssssss [ 43%]
ssssssssssss..sss.sxsssssssssssssssssssss.x.sssssssssss.sxssssssssssssss [ 43%]
sssssssssx.sssssssssssss.sssssssssssssxxssssssssssssssssssssss.sssssssss [ 43%]
ssx.s.ssssssssssssssssssxxxxs.xsssssssssssss.ssssssss.sxsssssssxsssss.xs [ 44%]
sssssssssssxsssssssssssssssssssssssssssssssssx.sssssssssssssssssssssssss [ 44%]
s.sxssssssssssssssssssssssssssssssxssssssssssssssssssssxssssssss.sssssss [ 44%]
sssssssssssssssssx..sssssssssssssssssssssss.x.ssssssssxsssssssssssssssss [ 44%]
sssssssssssssssssssssssssxssss.ss.ssssss.ssssssssss.sssssssxssssssssssss. [ 44%]
sssssssssssssssssssssssssx..xssssxssssssssssssssssssssssssssssx.sxssss.s [ 44%]
sssssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssxssss.s [ 44%]
sssssxssssssssssssssss.ssssssssssssssss...ssssssssssssssssssssssssssssss [ 45%]
sssssss..s.sssss..sxsssssssssssssssssssssssssssssssssss.ssssssxsssssssss [ 45%]
ssssssssssssssxsss.sssssssssssssssssssss.sssssssssssssss.x.s.sssssssssss [ 45%]
sssssssssssxssssssssssssssss.ssssssssssssxssssssssssssssssssssssssssssss [ 45%]
sssssssssssssssssssssssssssssssssssssxxx.ssssssssssssssssssssssssssssssx [ 45%]
sssss.sssss.ssssssssssssssssssssssssssssss.xsssss.xsssssssssssssssxsssss [ 45%]
ssssssssssxssssssssssssssssssssssss.s.sssssssssxssss.sssss..xx.x.xxxssssssss [ 45%]
..sxxxxxssss.s..ssssxxxssssss.sssxssssx.ssssssssssss.sssssssssss.sssxssss [ 46%]
ssxssssssssssssssssssssssssssssssss.sssssssssssssss.sssssssxssssssssssssssssss [ 46%]
sssssssssssssssssssssssssss.ssssssssssssss.sssssssssssssssssssssssssssssssss [ 46%]
sssssssssssssssssss.ssssssssssssssssssssssssss.ssssssssssssssssssssssxss [ 46%]
s.sssssssssssss.ssss.sssss.ssssssssssssssssssss.sssss..ssss.ssssssssssss [ 46%]
ssssssss.sss.ssss.sssssssssssssssssxsssssssssssxssssssssssss.sssssssssssss [ 46%]
ssssssxssssssssssssssssssssssssxsssssxsss.s.ssssssssssssssssssssssxsssss [ 47%]
sssssssssssssssssssssssxxsssssssssssssssssssssssssssssssssssssssssssssss [ 47%]
sxxsssssssssssssssssxssssssssssssssssssssssssssssss.ssssssssssssssssssss [ 47%]
ssssssssssssxssssssxssssxx.x....x..xx...x..xxssx.x.xxxx.xsssxssx.xxxxxxs [ 47%]
sxssssssxxx.ssssssssssssxssssssss.sssss.xx.sx..xxsssssssss.sssssss...sss [ 47%]
ssssssssssssssssxsssss.ssssssssssssssssssxssssssssssssssssssssssssss..xs [ 47%]
ssssssssssssssssxxxsssssssssssssssssssssssssssss.xssssssssssssssxsssssss [ 47%]
ssxssssssssssssssssssssssssssssssssss.ssxsssssssssssssxssssxsssssxxsssss [ 48%]
sssssssssssssssssssssssssssssssss.ssssssssssssssssssssssssssssssss.sssss [ 48%]
ssssssssssssssssssxxsssssssssssxxssssssssssssssssssxssssssssssssssssssss [ 48%]
sssssssssssxsssssssssssssssssssssssssss.xxsssssssssssssssssssssssxxssxss [ 48%]
ss.ssssssxssssssssssssxsssssx...xxssssxxssssssssssssssssssss.ssssssssssxxs.xs [ 48%]
.ssssssssxssssssssssssxsssxssssssssssssssssssssssssssssssx.sssxxssssssss [ 48%]
ssxsss.ssxssssxsssssssssssssssssssxssssssssssssxssssssss.sxssssssssssxss [ 48%]
ssxsssssssssssssssss.sssssssssssssxs.sssssssssssxsssssssssssssssssssssss [ 49%]
sxsssxsssssssssssssssssxsxsssssssssssssssssxsxs.ssssssxsssssssssssssssssss [ 49%]
sssssxssssssssssxssssssssssssssxssssssssssssssssssssssssssssssssxssssxss [ 49%]
ssssssssssssssssssssssssssssssssss.sssssssssssssxsssssxssssssssssssssssss [ 49%]
sssssssxssxssssssssssssssssxssssssssssssssssssssss.ssssssssssssssssssssxs [ 49%]
ssssssssssss.sssssssssssssss.sss..sssss.ssss..ssssssssxssxssx.sssssxx.ss [ 49%]
ssssssssss.sssssssssssssssssssssssssssssssssssssssss.ssssxsssssssssssssss [ 49%]
ssssssxxsssssssssxssxssssssssssssssssssssssxssssssssssssssssssssssx..sss [ 50%]
sssxxsssssssssxssssssssssssssssssssssx.ssssssssss.ssxssssssssxsssssss.sss [ 50%]
ssssssssss.ssssssssss.s.ssssssxxx.ssssssssxsssssxssssssxssssxssssssss.ss [ 50%]
sssss.sxsssssssssssssssssssssssssssssssssssssssssssxssssssssssssssssssss [ 50%]
ssssssssssssxsssssssssssssssxxssssssssssssssssssssssxxssssssssssssssss.s [ 50%]
ssssssssssxxssssxsssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 50%]
sx.ssssssssssssssssssssssssssssssssssssssssssssssssssssss.ssssssssssssss [ 51%]
sssssssssssssssssssssssssssssssssssssssssssxxsssssssssssssssssssssssssss [ 51%]
sssssssssssssssssssssssssssssssssssssxsssssssssssxssssssssss.xxsssssssss [ 51%]
sxxxssssssssssssxssssssssssssssssssssssssssssxsssssssssssxssssssssssssss [ 51%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssxssssss [ 51%]
ssssssssssssssssxssssxssssssssxxxxssssxxsssssssssssssssssss.ssssssssssss [ 51%]
ssssssssssx.ssssssssss.ssssssssssssssssssssssssssssssssssssx.sssssssssss [ 51%]
ssxsssssssssssssssssssssss.x.xxssxssssssssssssssssssssssssssss..xssssxss [ 52%]
sssssssssssssssssssssssssssssssssxssssxxsssssssssssssssssssssssss.s.xxss [ 52%]
ssxsssssssssss.sxssss.xsssss.ssssssxx.sssssssssssssx.xxx.ssssssxxsxxxxxx [ 52%]
xx.xxxx..xsssxx.sxxsxxxssxssssssssssssssssssssx.xx.xssssssssssssssssssss [ 52%]
sssss.sssssxxssssxsssssssssssssssssssssssssxsssssssssxsssssssss.sxxssssss [ 52%]
ssssssxsssssssssssssssssssssssssssssss.ssssssssssssssssssssssxsss.ssssss [ 52%]
ssssssssssssxxxssssxsssssssssxssssxssssssssxxxsssssssxssssssssssssssssss [ 52%]
ssss.xxsssssssssssssssssssssssxsssssssssssssssssssssssssxsssssssxxssssss [ 53%]
ssssss.xssssssssss.xxsssxxsxsssssxsssssssssssxssssssssssssssssssssssssss [ 53%]
ssssssssssssssxxsssssssssssssssssssss.ss.ssssssssxssssss.ssssssssssssssx [ 53%]
sssssssssssssssssssssssxssssssssssssssssxxsxssssssssssssssssssssss.sssss [ 53%]
ssssssssssssssssssxsssssssssxsssxsssssssxssssxxxxssssxs.ssssssssssssssss [ 53%]
sssssssssssssssssssssssxssssxxssssssssssssssssssssssss.xxxxxssssssxxssss [ 53%]
xssssssss.sssssssssssssssssssssss.sssssssssssss.ssssssssssssssssssssssss [ 53%]
sssx..xxxsxsxs.ssssssx.ssssssssssssxssssssxsssxsssssxssssssss.ssssssxxss [ 54%]
xs.ssssssssssssssxsssssssssxsssssxsssssssssssssssxssssssssssssss.ssxssssx [ 54%]
sssssssssssssxssssssss.sxs.sssxsssss.sssssssxssssxsssxssxxsssssxssxsssss [ 54%]
ssssssssssssssssssssssssxsxsssssssssssss.sssss.sxsxsssssssssssssssssssssss [ 54%]
ssssssssxssssssssssssxssssssssssssssssxsssssssssssssxsssssssssssssxxsx..s [ 54%]
sssxssss.sxsssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssxs [ 54%]
.sssxssxsssss.sss.ssssssssssssssssssxs.ssssxssssssssxssssssssssssssssxss [ 55%]
ssss.xsssssssssssssxssxxssxssssssssxsssssssxssssxssssxsssss.sssssssssssx [ 55%]
ssssxxssssssssssssssssssssssssssssssssssssssssssxxxssssxssssssxsssssssssx [ 55%]
sssssssssssssssssxsxssssxssssssssssssssssssssssssxxsssssssssss.sssssssss [ 55%]
sssssss.sssssssssssssssssss.s.sssssss.ssssssssssssssssssssxssssssxxxssss [ 55%]
ssssssssxssssssss..sssssssssssx..sssssssssssssssssssssssssssxssss.ssssss [ 55%]
ssxxxxssssssssssssssssssssssxx.sssxsxssssxsssxxsssxxsssssssxssssssssssss [ 55%]
ssssssssssxsssssxsssssxxsssssssssssssssssssssssssxssssssxxssssssssssssss [ 56%]
ssssssssssssssssssssssssssssssssxsssssssssxssssssssssssssssxxxssss.sssss [ 56%]
sssssssssssssssxssssx.sssssssssssssssssssssxsss..ssssssssssssssss.xsssss [ 56%]
sssssssssxssss..sssssxss..x.xsssssssssxssxx.xxxxxxxssssxsxxxssssxssssxx. [ 56%]
xsxsssxxssss.xs..sssxsssssxsxxxxsssssss.xsx.sx.x.xxxx..xxx.xx.xxxxxxx... [ 56%]
xssssssss.xxxssssssssxxxxxxxxxxx.xxxxxxsxxxsxssss..ssssssssxxxxssss..sss [ 56%]
sssssssssxssssssss..xxxsssssssssssssssss.xxssssxssss..xxxsssssssssssssss [ 56%]
xxxsssssssssxxxsssssssssssssssssss.sssx.x.xxxsssssxxxsssssssssssssssssss [ 57%]
sssssssssssssssssssssssssssss.xxssssssxssssssss...ssssxssssssssssssxxsss [ 57%]
ssssssx.xssssssssssssssssssssssssssssssxsssssssssssxxxsssssxsssssssssx.xx. [ 57%]
.ssxxsxsssssxsssssssxsxsssssssssxsssxssssss.ssssssssx.sssxssssx.ssssxsss [ 57%]
xssssxsxssssssssss.ssssssxssssxssssssssssssssssssssss..xssss...sssssssss [ 57%]
sssssssssssssssss.xssssssssx.xxxssssssssssssssssssssssssssssssxxxsssssxx [ 57%]
xssssssssssssssssssssssssssssssssssssxxxssss.xsss.ssssxxsssxsxssssssssss [ 57%]
sssssssssxxsssssssssssssssssssssxsssssx.s.xxxxssxssxssssssssxssssssxss.s [ 58%]
ssxsssssssssxssssssssssssssxssxs.s.sssssssssssssssssxsssss.xsxsssssssssxs [ 58%]
sxs..xsssxssssssssssssssss.xxsss.sssssxssss.sssxxsssssxxxsssxxxssssxssxs [ 58%]
ssssxsssssxxsssssssssxssssssssssssssssssssssss.sssxssssssssxsssss.xsssss [ 58%]
sssssssssssssssssssxssssssssssssssssssssssxsssssssssssssssssxsssx.ssssss [ 58%]
sssssssssssssxxxxsssssssssssssxxssssxsssssssssssssssssssssssss...xssssss [ 58%]
sssssssssssssssxxssssxssssxsssss..ssssssssssxxxssssssssssssssssssssssxss [ 58%]
ssssssssssssssssx.sssssssssssssssssss.sssssxsssssssssssssssssssssx..ssss [ 59%]
ssssss.sssssssssssxssssssssssssssssssssssssssxxssssxxssssssssxxxssssxxxx [ 59%]
sss.x.ssssssssssssssssxsssssssssxssssssssssssssssssxxxsxxsxsxssss.ssxx.s [ 59%]
sssss.ssssxsxsxsxssssssssxssssssxssssssss.ssxssssssssxssssss.s.ssxsssssx [ 59%]
xsss.ssssssssxsssssxssssssssx.sssssxs.sssxsssssssxssxs.ssxssssssxs.sssxxss [ 59%]
ssxxsssxssxsssssssssss.ssxsxssxsssss.xsssssxssssssssssssxsssx.ssss.xsssxx [ 59%]
sssxsss.xx.ssssssxsssssssxssxssssssxssssssssssssxsss.ssxsss.ssssssssssss [ 60%]
sxsxssxsssssxssss.sssxssssssssxxxs.sssxsssssssssxxsxsssssssxssxsssssssss [ 60%]
sssssssxxsx.xxsssxssssssssxxxxxssss..ssssssssssssssssssssssxssssxxssssss [ 60%]
ssssssssssssssssssssxssssssssssssssss.sxsssssssssx.ssssssssssssssssssss. [ 60%]
.sssssssxsxssssxx..xssssssssssss.xsssssssssxxsssssssssxxssss..xsssssssss [ 60%]
sxsssssssssssssssxxxsssssxss..xxsssss.xxxxxxssssssssssssssssssssssxxxx.ss [ 60%]
ssxss.ssxsxs.ssssss.sssssx.sssssxssx.xssssssss...ssssssxssxssss.sssxssssss [ 60%]
ssxssx.xssx.ssxssss.ssssxssxssx.ss.xssssssxsssxxsxxxxxssssxxxx.xxssss.xx [ 61%]
xxxxs.xsssxxxxxxssssxss..xssssxxxssss..xxsss.xs.x..x.xxx.xxx.x.x...xxx.x [ 61%]
xxxxssss..x.x.xsx.x.xxxxsssxssssxxssss.xxs.xssxxssxsssssssssssssssssssss [ 61%]
sxxxxxssssssssssxxxxxssssxxxx.x.xx.sssssssssssssssssssss.xssssxxxxxxxs.x [ 61%]
sssxsxss.ssssxsss.xsxxxssxssssxs.xssxsss.ssssssxssssssssssssssssssssss.. [ 61%]
xsssss..ssssssxsssss.sx.xxsssssssssssxx.xsssx.sxssssx.ssssssxxxxssssssss [ 61%]
ssssssxssx.ssssxsssss.ssssssssssxssssssxssssxss.xsssssss.sssxxsssssssssss [ 61%]
sss.xxsssssssxsssssxssssxsssssssssxsssssssxsssssxssss.ssssxss.ssxxsssssss [ 62%]
ss.xssssssssssssss.sssssxssssxsxxssssxssssssxsssssssxsssssssssssxsssssssss [ 62%]
xxsss.xsssxxs.sssssssssssssssxsxsxssssxsssssssssssssxxsssssxxssxsssxssxs [ 62%]
ssssssxx.xsxsss.xssssxxx.xxxsssxxssxxssxssss.sssssssssssssssssssssss.sss [ 62%]
ssssssss.sssssssssssssxssssssssssssxsssssssssssssss.sssss.ssssssssssssss [ 62%]
sssssssssssssssssssssssssssssssssssssssssssssssssssxxssssssssssssxxsssss [ 62%]
sssssssss.x.xssssssssssxssssssssssssssssssssssssssssssssssssssssssssssss [ 62%]
ssssssssssssxxsssssssss.sssssssssssssssxsssxxx.sssssssssssssssssssssssss [ 63%]
sssssssxxssssssssssssssssssssssssssssxssssssxssssssssxxssssxss.sssssssss [ 63%]
ssssssssssssssssssxxssssxx.xxxssssx..x..x.sssssssssssssxssssssssxxssssss [ 63%]
ssssssssx.xsssssssssssssx...ssssssssssssssssssssssss.sssssssssssssssssss [ 63%]
sssssssssssssssssssssssssssssssssssssssxx.ssssssssxxxxxssssssssssssssxxs [ 63%]
sssssssss.sssssssssssssssssxxssssssss.xssssssssssssssssssssssxsssssssssx [ 63%]
xsssssssssssssssxsssssssssxxxxxssssssssssxxssssssssssssssssssss.s.xxxxxs [ 64%]
ssssssssxxxx.xxxsssssssssssxsxsssssssssssssssxssssssssss.xxssssxssssssss [ 64%]
sssssssssssx.ssssssssssssssssssxsssssssxsssssxssssssssssssxssssssxssssss [ 64%]
sxxxssssssssssssssssssssssss.ssssssssssssssssssssssssssssssssxxsssssssss [ 64%]
xsssssssssssssss...ssssssssssssssss.xxssssxssssssssxxsssssssssxssxsssss. [ 64%]
xsssssssxxssssssssx..sssssssssssssssssssss.xxxssxxxxsssssss.ssxxssssssss [ 64%]
sssxxx.xxx.xssssssxxxx.xxssssxssssxssssxsssssssxx.xxxxssss.ssssssx.sssss [ 64%]
xsssxssssssssx.xxxxxsxssssssssssssssssssssssss.ssssssssssssssssssssssxxs [ 65%]
ssssssxxssss.xssssx.sssssssssssssss.xssss.xsssssssssxxxssssssssssssxxs....s.x [ 65%]
sss.sss.xsss..sssss.ssssssss.ssxsxsssssssssxssx.sss.ssxxsxssss.xss.xxsxs [ 65%]
.xsssxsxxxxssxsxxssxssssssxsssxxssxssxxsxsssxssxxssssssssxxxxssx.xxssssx [ 65%]
xxxxxxssssxxxssssssxssssxssss..xxsss.sxx...xsx..xxxxxsxsssssssxxxss..s.s [ 65%]
xxxxssxxss..xx.xsxsx..xxxxxssssxsssxsssx.xx.xxssssss.sssxssxssxsssssssss [ 65%]
.xssssxxssxxssssxsssssxssssssssxs.ssxxssssxsssxs.sssxsssssssssxxssssxsss [ 65%]
sssssxxsssssssx.ssssssxs.sssssssssxsssssssssx.sxssssxsssssssssssssssssss [ 66%]
ssxxxssssssssssssssssssssssssssssssssssxxxx.xsxxss.xsxssssssssssss.s.sss [ 66%]
ssxsssssxssssss.ss.ssxs.ssssxsss..ssssssssssssxx.xx...xxx.xx..ssssxxxxxxx.xxx.xxxsx [ 66%]
sssssxxxxxxxxxxxxxxxssssssxssxsxxxxxxxxssssssssxxxxxxssss.sssssxxxsss..sxxxsx [ 66%]
ssxxssssssssxssssssx.ssssssssssssssssx...sssxssssssx.sxssssss.ssssssss.s [ 66%]
sssssssssxssssssssssssssssssssssssssss.xsss..ssxsssssssxssssxsxxsssssxs. [ 66%]
sssxsssssssxsxsssssssssxssssssssxsx.xsxxsxssxsss.ssssssssssxssssssssssss [ 67%]
ssxsssssssssssssssss.xssss.ssssssssxs.sssssssssssxxssssx.xs.xsss.ssssxxs [ 67%]
sssxssxsssss.sxssss.ssxssssssss.sssssssx.ssssx.sssssssssssssssxsssssssss [ 67%]
sxssssss.sssssssssssssssssssssssxx.sssssssssss.xssssssssssssssssssssssss [ 67%]
sssxxssssssssssssxxssssssss.xsssssssssssxsss.sssssss.sssssssxxsssxssssss [ 67%]
sssssssssxxx.xsssssssxxxxxxxxssssssssssssxxxxxxxsssssssssssssssssssssxxx [ 67%]
sssssxsssssssxsssxxxxxxxxsssssssxssssx.xssssxssssssssxssssssssxsxssssssx [ 67%]
sssssssssssssssss.x.ssssssssssssssssssssssssssssss.sssssssssxssssssss.xs [ 68%]
sssxssssxsssssssssssssssssxssssss.xsssssssssssssx.ssssssssssssssssssssss [ 68%]
sssssssssssxsssssssssssssssssssssssxsssssssssssss.xxsssss.xxxsssssssssss [ 68%]
sssssssssssxxssssssssssx.xssssssssssxxxxxsssxx.x.x..xxsxxxx.sssxxx.xxsxx [ 68%]
xsssxssssssssssssxxxssssxxssssssssssss.xxssssssssssssssssssssssssxsssss. [ 68%]
.xxsssssssssssssxxs.sssssssssxssssxsssxxsssssssssssxss.ssss.xsxssssxssss [ 68%]
.ssssssssssssssssssssxssss..xxsssssss.xssssssssssssssssssssss.xs..ssssss [ 68%]
ssssssssss.sssssssssssssxss.x...xssssssssssxxxxsssssssx.ssssssssssssssss [ 69%]
ssssssxssssxxxssssssssxx.xxsssssssssx.ssxsxx.xssssssssssssssx.xxxxxxxxxx [ 69%]
xss.xxssx.xssssxxxxxssssssssxsssxssssxssxxsssssxxss.ssssssssssssssssssssx [ 69%]
xxxxxsssssxssssssssssssssssssssssssssssssxssxsssssssssssssssssss.xssssss [ 69%]
sssssssxssssssssssssssssssssxsssxsssssssssxsssssssxxssssssssssssssssssss [ 69%]
ssxxss..sssxssssssssssssssssssssssssssssss.ssssxssssssssssssssssssssssss [ 69%]
sssssssssxsssssssssssssxsssssxxssxsssssssssss.ssssxxsssssssssssssxssssxs [ 69%]
sssxx.sssssssssssssssssssssssssssssssssssssxsssssssssssssss.xx.xssssssxs [ 70%]
ssssssssssss.ss.sss.s.ssxssssx.sssssssx.sssssss.xssssxsssssssxssxx.x..xx [ 70%]
xsss..ssxx.x.xsx...xx.ssss.xssssssssssssssssxssssssssssxssssssssssssssss [ 70%]
ssssss.sssssssssxssssssssssssssssssssssx.ssssssss..sssssssssssssssssx..s [ 70%]
ssx.ssssssssssxxsssx.ssssssxssssssssssssssxssssssssssssssssssssxssssssxs [ 70%]
ssxxsssssssssssssssssssssxxx.xxxxsssssssssssssssssssssssssssssssssssxxss [ 70%]
ssssssssssssssxxxx.ssxxssssxssssssssssssssxxxxxxsssssssxx.ssssssssssssss [ 70%]
ssssxxxxsssssssssssssxxxs.xxx.xxssssxssssssssssssxsssssssxsssss.sss.sxssss [ 71%]
sssssxssxsssxxsssssssssssssssssxssssss.xssssssssssxxssssssxxxxssssssssss [ 71%]
sssssssssssssssssssssssssssssssssssssssssssssss.xssxsssssss.ssxsssssssss [ 71%]
sssssssssssssssssssssssxsssssxssssssssssxxssxssssssxssssssssss.x.xssssxs [ 71%]
ssssssssssssssssxxxsssssssssssssssssss..sssxssx.ssssssssssssssssssssssss [ 71%]
s.sssssssssx.xx.xxssssx.xssssxsssxs.ssssssssssssxssssssssssssxxxssxsssss [ 71%]
ssssssxssssxssssssssssssssssxxssssss.xssssssssssssssssssssss.sssssssssss [ 72%]
ssssssss.xsssssssssssssssssssssssxssssssssxssssssssssssssssssssssssssxss [ 72%]
.ssssss..ssxssxssssssssxxxssssss....ssssssssssssssssssxsssssx.ssssssssss [ 72%]
sssssssssssssssxsssssxxssssss.ssssssxxxxxsssssss..xxxx..sssssssxxxssssss [ 72%]
ssxssss.ssssssxx.ssssssssxssxxssssxsssssssssssssssssssssssssssssssss.sss [ 72%]
sssssssss.ssssssss.xx.x.xx.ssssssssssssx.x....xsssssxxxxsssss.ssssssxssx [ 72%]
sssxsssssxxsssssxsxsssxssssssssssssssssssssssxssssxssssxssssss.sssssssss [ 72%]
ssssssssssssxssssssssssxsssssssxssssssssssssss.xssssssssss.sssssssssssss [ 73%]
sxsssssssssssssssssssssssssssssssxsssssssssss.sssssssssssssssxsss.sssxss [ 73%]
ssssss.xsssxx.xsxsssxxssssssssssxxssssssssssssssxssssssssssssssxssssxsss [ 73%]
sssssssssssssssssssssssssssssssssssssssssssss.xxsssssss.xsssssssssssssss [ 73%]
sssssssss.ssss.sssssss.x.xssssssssssssssssssssssssssssssssxsssssssssssss [ 73%]
sssssssxsssssssssssssssssssssxsssssssssssssssxssssssssssssssssssssss.sss [ 73%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssxsssxx.ssxx...ss [ 73%]
sssssssssssssssxssxssssssssssssssssssssssssssssssssxssssssssssssssssssss [ 74%]
ssssssssssssssssssssssssxsssssssssssssxssssssssssssssssssssssssssssssssx [ 74%]
xssssssssssssssssssssssssssssssxxssssssssssssxssssssssssssssssssssssssss [ 74%]
sssssssssssssssssssssssssssssssssssssssxssssssssssssssssssssssssssssssss [ 74%]
ssxssssssss.sssssssssssxxsssssxxsssssssssssss.sssssss.s.ssssssssxsssssss [ 74%]
ssssssssxssxsssssssssssssssssssssss.ssssssssxsssss.ss.ssssssssssssssssss [ 74%]
ss.sssxssssssssss.sxsssssssssssxssxsssssssssssssssssssssssssssssssssssss [ 74%]
sssssssssxsssss.sssxsssxssxs.ssssssxsssssssss.ssss.sssxsssssssss.sxsssssss [ 75%]
sssssssssssssssssssss..ssssssssssssssxsssssssssssssssssxx.xs.xsxssssssss [ 75%]
sssxsssssssxxsssssssss.sssssssxssssssssssssssssssssssssssssssssss.xsssss [ 75%]
ssssssxs..ssssxxsssssxss.xx.xssxsxs.sxsssxssss.sssxssss.ssssssxsxsssssss [ 75%]
ssssssssxsssxsssssssxsssssssssssxsssxsssssssssssssssssssssssssssssssssssx [ 75%]
.ssssssssssssssssssssssssssssssssxxxsssssssssx.xxxsssssssxssxsxssssssssss [ 75%]
ssxsxssssssssssssssssssxssssssssssssssssss.sssssxxssssssssxsssssssssssss [ 76%]
ssssssssssss....sssssssssxssssxssssxsssss..x.xxsssssssxxssssssssssssssss [ 76%]
ssssssxssssssxxxssssssssssssssx.sx.ssssssssssssss.sssxssssssssssss.sssss [ 76%]
ssssxsssxxsssxssssssssxsssxsssssssssssssssssssssssssssssxssssss.ssssssss [ 76%]
ssssssss.ssxssxxsssxssssssssssss.sxsssss..ssssssxssssssssssssssssssssssss [ 76%]
sssssxssssxssssssssssxxxsssxss.sssssssssssssssxssss.sss.xssxssssssssssssx.xx.xx.ss [ 76%]
ssxsxsssx.sssssssssssssssssssssssssxssssssssssssssssssssssssssss.sssssss [ 76%]
sssxssssssxs.ssssx.sxssssxsxsssssssssssssssssssxssssssssssssssssssxsss.ss [ 77%]
ssssssssssssssssssssssxssssssssssssssssxssssss.ssssssxssssssssxssssssssssss [ 77%]
ssxssxss.sssssssssssssss.ssssssssssssssssssxsssssssssssxssssssssssssssxxsxs [ 77%]
sssssssssssssssssssssxsssssssssssssssssssssssxxx.ssssssssssxxsssssssssss [ 77%]
s.xxsssssssssssssssssssssssssssssssssssssssssssssssssssxsssssssssxssss... [ 77%]
.xxxxssssssxxxsssssssssssssssssxxxxxxssssssssssssssssssssssxxxxssssssxxx [ 77%]
xxsssssssssssssssssssssssssxsssssssxssssssxsssssssssssssssssssssssssssss [ 77%]
ssssssssssssssssssssssssxxxsssssxsssssssssssssssssssssssssssxsssssssssss [ 78%]
sssssssssssssssssssssssssssxsssssssssssssssssssssssssssssssxssssssssssss [ 78%]
ssssssssssssssssssssssssssssssssssssssssssssssssxxssssssssssssssssssssss [ 78%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 78%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 78%]
xssssxsssssssssssssssss.ssssssssxssssssssssssssssssssssssssssssssssssssss [ 78%]
sssssssssssssssssssssxssssssssssssssssssssssssssssssssssssssssssssssssss [ 79%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssxsssss [ 79%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 79%]
ssssssssssssssssss.sssssssssssssssssssssssssssssssssssssssssssssssssssss [ 79%]
sssssssssssssssssssssssxsssssssssssssssssssssssssssssx..sssssssss..xxxss [ 79%]
ssssssssssssssxxsssssxxssss.x.s.sssx.ssx.x..xxxxx..ssss.xssssssssssxxxss [ 79%]
s.ssss.xsssssssssssssss.xsssssssssssssxxxxsssssssssssssx.xsxxxxx.xxsssxx [ 79%]
sssxxxxxssssss.ssxxxxsxssxsss.ssssxxsxssss..sxssxssss..ssxsss.ssxxss.sxxss [ 80%]
s.sssssxssssssxssss.xssxsssxs.sssssxsxsxssx.xssssxs.ssssssxxxsxxssssxxss [ 80%]
s.ssxsssssxsssxxssxsssssxsssssxsssssxssxs.sssssxsssssssssxxssssssxxsssss [ 80%]
sssssxssssssssxsssx.sxssssssssssssssxxsssssss.xxsssssssssxsssxsssssssssss [ 80%]
sssssssssss.sssxssssxsssssssss.sss.ssssssssssssssss..sssxsss.ssssssssxsss [ 80%]
.s.sssssxsssssssssssssx.xsssxssxssx.x.xxsssssssssssssssssssxx.ssssxsssss [ 80%]
sssssssssssssxxssssssssssssxxssssssssssssssssssssssssssssssssssssssxxsss [ 80%]
ssssssssssssssssxxxssxxxssssssssssssssssssssssssssssssssssxsssssssssssss [ 81%]
sssssssssssssssxxxssssssssxsssssssxxxssssxxx.ssssssssxxsssssssss..ssssss [ 81%]
ssssssssssssssssssss.xssssssss..ssssssssssssssxsssssssss.xsssssssxssssss [ 81%]
sssssssssssssssssssxssssxxssssssssssssssxxxxx.xsssssssxxsssssssssssxxsxss [ 81%]
.ssssssxssssssssxsxsss.sxssssssssssssssssssssssssxssxssssssssssssssssssss [ 81%]
sssssssxsxsssssssssssssssssxs.sssssssssssssssx.sssssssssssssssssssssssss [ 81%]
sss.ssssxssssssssssssxssssssssssssssssssssssssssssssssssssssssssss.x.sss [ 81%]
ssssssxssssssssssssssssssssssxssssssxxxsssssssssssss...ssssssssxsx.xxxs. [ 82%]
ssssssssssssxss.xssssxsssssss.sssxsssssssssssssssssxxsxssssssssssssssssxx.ss [ 82%]
s.sssxsssssss.ss.sxsssxssssssxssssssssssssssssxssss.ssssssxssssssssssssss [ 82%]
sssx.ssxssssxs.ssssxxssssssxsssxsssssxssssxsssxsssssxssssssssssssssxssxss [ 82%]
sssssxxsssssxsssssssssssssxssssssxssssssssssssssssssxxsssxxsssssssssssss [ 82%]
xssssxsssssssxxssssxsssssssssssssssssssssssssssssssssssxss.sssssssssssss [ 82%]
sssss.ssssssss.ssssssssssssssssssssssssssssxssssssssssssssssssssxsssssss [ 83%]
ssssssssxssssssss.xssssssssss.s.ssssxssssssxsxsssssssssssssxxsssssssssss [ 83%]
sssssssssxssssssssssxssssssssssssssxssssssssssssssxssssss.ssssssssssssss [ 83%]
ssssssxxssssssssssss.xsssssssssxsssssssssssssssssssssssxssssxssssssssss. [ 83%]
xxsssssssxsss.xsssss.sssssssssssssssssxx.ssssssss.ssssssssssxxssssssssss [ 83%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.x.xsssssssss [ 83%]
ssssssssssxsxssssssssssss.x.xssss.xxssxssxssssssssxsssss.sssssxxxxsssssx [ 83%]
ssssssxs.sssxss.sssxs.xsssssxssss.ssxssxs.ssxs.xss.sx.sss.sssssssssx.x.x [ 84%]
sssssxxss.xsxxx.xssssss.sxxssxsss.xsxssssxssxssssxxxss..ssxx.xsssssssxsx [ 84%]
xs.ssxssssxsxxss.ssssssssxxsssxxsssxxxsssxxssx.sxxxsss.ssssxssxssssss.xs [ 84%]
ssssssssssxssssx.xssxxssssssssx.xssssxsss.sssssss.sss.xssssxsssxssss.xss [ 84%]
ssx.xssssssxsssssssssssssssssssssssssssssssssssssssxsxx.xsssxsssssssssss [ 84%]
sssssssssssssssxssssssss..sx.xssssxx..xxsssssssx.sssssssssssssssssssssxx [ 84%]
ssssxxssss..xxssssxs.xsxsssssssssssssxssssssssxssssxxxssssssssssxxssxsss [ 84%]
s.ssssssxsssssssssssssssssssssssssssssssssssssssxx.xxssssssxxxxxsssssx.s [ 85%]
sxssx.ssssssxxxxssssssssssssssssssxsssssssss.sssssssssssssssssssssssssss [ 85%]
ssxsssssssssssssssss..sssssssssssssxsssssssssssssssxssssssssssssssss.xss [ 85%]
ssxxssssxxxssxxxssx.sssssxxssssxsssssss.xsssssssssxssssssssssxsxssssssss [ 85%]
ss..ssxxsssssx.sssxsssssssssxs.ssss.ssssssssssssssssssssssssssssssssssss [ 85%]
sxxssssssssssssssssssssssss..xxsssssssssxxsx.xxsssssssxxxssss.sssx.xxsss [ 85%]
sss.xssssssxxssssssssxxssssxsssssssssssssxxssssssssssssssssssssssss.xxss [ 85%]
sssssssssssss.x.sxsxxxxsssssssxsssssssssssss.xsssssssssssssssssssxssssss [ 86%]
sssxssssssssssssxxssssssssss.x.ssssssssssssxsssssssssssssxssssssssssssss [ 86%]
ssssssssssssssssssssssssssxxxsssssssssxxxssssxsssxx..xsssssss.xx..xsssss [ 86%]
sssssssssssssx.xxxssssssssxssss.xxxsssx..sssssssxx..ssssssssssssssx.ssxs [ 86%]
ssxxxsssssssssssssssssssssssssxx.xssssssssssssssxsssssxx.xsssssssss.x.ss [ 86%]
sssxssssxssssssssssssssssssssssxsssxssss.xxssx..ssxssxxx.sssssxsssssssss [ 86%]
ssssxxxsssssssssssssssssssxxxxxsssss..ssssssssssxssssssssss.ssssssssssss [ 86%]
sssxxxxssss.x.ss...sssssssssssssssssssssssssssssssssx..xssss.xssssssssss [ 87%]
sssxxssssssssssssssssssxxsssssssssssssssssssssssssssssssssssssssssssxsss [ 87%]
ssssssssssssssssssssxssssss...xxsssss.sssssssssssssxxssssssssssssssssssx [ 87%]
xxxx.sssssssssssssssssssssss.sssssssssssssssssssss.ssssssssssssssssssssx [ 87%]
sssssssssssssssssssss.sssssssssssssx.ssxxxxxssssssssssssssxssssxsxsssssxss [ 87%]
ssssxsssssssxssssss.ssssssssssssssssssssssssxxssssssssssssssssssssssssxs [ 87%]
sssssxssssssssssssssxsssss.ssssssssssss.sssssssssssssssssssssssssxssssss [ 88%]
xsssssssssssssssssxssssx.ssssssss.ssxsssxssssxssssxssssxsssxssss.x.sssss [ 88%]
sssxsssssssxs.xssssssssssssxssssssss.sssssssssssxssxssssssss.sxssxs.ss.s [ 88%]
sx.ssssxxss.xsssssssxxssssssssssssssss..xxss.ssssxxssssss.xsssssssss.xss [ 88%]
ssssxsssssssssssxsssssssxssssssssssx.ss.ssssssssssss...xsssssssssssxssss [ 88%]
ssssssssssssssssxx.sxsssssssxssss.ss.xxsss.sxssssxxxx.x.xss.x.sssxssssss [ 88%]
ssssssssss..xs..ssssssssxxxxxxx.xsssx.ssxxxssssssssssssss..xsssssxssssss [ 88%]
sssxxx..xxxxxsxssxsxxxxsxssxsxxxxxssxsxsxssxxs.sxx.xsxsssssxss.xsssxsxss [ 89%]
sssssssssssssssssssssxssssxxsssssssssssxxxxssssxsssssxxx.xssssxssxxss..s [ 89%]
sssxxsssssss.ssss..sssxsssssssssssssssssssssssssssss.x.x.ssssxssssssssss [ 89%]
xxssssxsssss..sssssxx..xsxssss..ssssssssssssssssxx..xxssxsxsxsssxsssxxsxx [ 89%]
ssssx.ss.s.ssxss.s.xssssxsssxxsxx.xxxxxx.xxxxxxxss.xssssxsssxxxxxxxxxxxxsxx [ 89%]
ss.sssssxssssxssxxxxxxxxxxssssxssxsssssxssssssxsssxxxxxxsxssxs.xsss..s.sx [ 89%]
sssssxxssxxsxxxxss.ssx......xssssssxssxxxxssxsxsss.x.x.sx.ssssssssxxssssx [ 89%]
xssxsxxxxsx.ssssssssssxsxs.sxsxxxsx.xxssss.ssssss.xsxxxx..xxsssssss.sssxs [ 90%]
x.sxxx.x.xxsssssxssssx.x.xss.ssssxxxxxss.ssssssxxxxsxssssssxxsxssxssssxsxxs [ 90%]
xsssxsxxxxssssssssssxssssssssssssssssssxssssssssssssssssssssssssssssssss [ 90%]
sssssss.sssssssssssssssssssssssssssssssssssssssxssssssssssx.ssssssssssss [ 90%]
sssssssssssssssssssxsssssssssssssssss.xssssssssss.ssssssssssssssssssssss [ 90%]
ssxsssssssssssssssxxxs.ssssssssssssssssssssssssssssssssssxssssssssssssss [ 90%]
ssssssx.xsssssssssssssssssssssssssssssssssssssssssssxsssssssssssssssssss [ 90%]
sssssssssssssssssssxxsssssssssssssssssssssssssssssssssssssssssssssssssss [ 91%]
sssssssssssssssxssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 91%]
ssssssssssssssssssssssssssssssssssssxsssssssssssssssxsssssssssssssssssss [ 91%]
ssssssssssssssssssssssssssssssssssssssssssxsssssssssssssssssssssssssssxs [ 91%]
sssssssssssssssxssssssssssssssssssssssssssssssssssssssssssxsssssssssssss [ 91%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 91%]
sssssssssssssssssssssssssssssssssssssssssssssssssxsssssssssssssxssssssss [ 92%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssxss.s [ 92%]
ssssssssssssssssssssssxsssssssssssssssssssssssssssssssssxsssssssssssssss [ 92%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 92%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 92%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 92%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 92%]
sssssssssssssssxssxssssssssssxssss.s..ssssssssssssxsssxxsssssssssssxxs.ss [ 93%]
.sxx..sssssssssssssssssssssssssssssss.xxxx..xsssx.sxssssssssssssxxssssss [ 93%]
sssssssx...xxxx..xxxssssssxxxssx..xs..xxssssssssxxx.xxsxx.xxxssxxxxxxxxx [ 93%]
xssssssxxxxxxxssssxs.xsxxsssssssxxx.xx..xs.xss.xxxssssx.sssxxxxsssxsssxs [ 93%]
ssss.xsxxsssxs.s.sssssssssssssss.x.xsssssssssssx...sssssssssxsssssxx.xxx [ 93%]
x..ssssssx.xs.x.xssssxsxssxxxx.sss.xssssxssxxxssss.xss.ssxssssx..xxxxxxx [ 93%]
s..xs.xsssssssssxxxxssss.sssssssxxssssssssssxx..xsssssx..xsssssssssss.xx [ 93%]
xxxx.sxxxsssssxx..xssssxssssx.sssxx.sssssx.xx..x.xxxxxxsssxxsssxss.xssss [ 94%]
ssssssssssss.xxsssssssss.xxsssssssssssssxsxxsxx.ssssxxx.xx.sxxxxxxx.xx.x [ 94%]
x.xxxx.xxssssssssssxsssssssssssssssxxsxsssxsxssssx.sssxxssssxxxxssssssss [ 94%]
sssxx.xxxssssssssssssss.sssssssssxxxssssxxsssssssssxssss.xxxs.ssssssssss [ 94%]
xsssxxsssxxxxxssssxxssssssssssssxxsssssxssssssssssssxsssssssssssssss.x.s [ 94%]
ssxxsssssssxx.x.sxsssxxsxsxxssssss..sxxssssssssssssssssssx.xsxsssssssss. [ 94%]
ssssssss.ssssssxssssssssssssssssssxsssxssxsssssssssxssxsxssssssssssssxssss [ 94%]
sxssssxxssssxsssssssssxsssssssssssssssssssssssssssssssssssssssssxssssssss [ 95%]
ssssssssxssssssxsssssssxssssssssssssssss.ssssssxsxsssssssssssxssssssssss [ 95%]
ssssssssssssxssssssssssssssssssssssxssssssssssssssssssssss..xsssssssssss [ 95%]
ssssssssssxsssssssssssxsssssssssssssxssssssssssssssssssss.ssssssssssssss [ 95%]
xsxsssssssssssssxsssssssssssssssssssssssssssssssssssssssssssssssxsssssss [ 95%]
sssssssssssxsssssssss.ssssss.sxsssssssssss..sssssssss.sss.s...ssxssss.s. [ 95%]
s.....s..s.sss..ssssssss....xs................x.s.......s....s.........s [ 96%]
..s.s.............s..s......s.....s...........sss.......s...s......s.s.. [ 96%]
.......................................s.......s........................ [ 96%]
..................s..................................................... [ 96%]
......................................................................... [ 96%]
.............sss........................................................ [ 96%]
............................x..........x.........x...................x.. [ 96%]
.....................x........................xssss....................... [ 97%]
........................................................................ [ 97%]
.............................x.......................................... [ 97%]
..............................x.x..............x........................ [ 97%]
....x..........xx.....x.....x..............................x............ [ 97%]
....................x.....................................x............. [ 97%]
........................................................................ [ 97%]
.....................xssss.............................................. [ 98%]
........................................................................ [ 98%]
....................x................................................... [ 98%]
.................x.....................................................x [ 98%]
.......................................................x................ [ 98%]
....................................x................................... [ 98%]
.....................................xsss...s........................... [ 98%]
..x.................xx..x......xx.x...x..xx..x.x..xxxxxssss............. [ 99%]
.x..xx.x..xxxxssss...................xx.x....x....x........sxssss....... [ 99%]
.s....x....x..s..s.....x........x.............sss.ss.sssssssssss.ssss.ss [ 99%]
.s.ssss.ss.sssss..x.....s.xs..s.....s...ss.....s..xssss...s.........s... [ 99%]
......................x.......x......sx..s...sxs......s..x.sxx...s.sxsss [ 99%]
s...........x....x.s..x...x...sx..x..ssss................x...s..x....x... [ 99%]
........x.................s...........................................   [100%]error: object directory /localdisk/tc_agent/system/git/git-CE4319E5.git/objects does not exist; check .git/objects/info/alternates
fatal: bad object HEAD


---------- coverage: platform linux, python 3.8.10-final-0 -----------
Coverage XML written to file coverage.xml

= 8342 passed, 37237 skipped, 3276 xfailed, 20928 warnings in 192.71s (0:03:12) =
PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.
Tests are successful


<b>Remaining output truncated<b>

[codecov]

Codecov Report

Merging #3298 (009e5e7) into master (0bc409d) will decrease coverage by 23.82%.
The diff coverage is 58.25%.

❗ Current head 009e5e7 differs from pull request most recent head caed341. Consider uploading reports for the commit caed341 to get more accurate results

@@             Coverage Diff             @@
##           master    #3298       +/-   ##
===========================================
- Coverage   85.32%   61.50%   -23.83%     
===========================================
  Files         150      145        -5     
  Lines       15826    15681      -145     
===========================================
- Hits        13504     9645     -3859     
- Misses       2322     6036     +3714     

Impacted Files	Coverage Δ
modin/backends/base/doc_utils.py	100.00% <ø> (ø)
modin/distributed/dataframe/pandas/partitions.py	0.00% <0.00%> (-53.66%)	⬇️
modin/engines/base/io/text/json_dispatcher.py	97.72% <ø> (-0.06%)	⬇️
...in/experimental/backends/omnisci/query_compiler.py	0.00% <ø> (-92.54%)	⬇️
.../experimental/engines/omnisci_on_ray/frame/data.py	0.00% <0.00%> (-91.49%)	⬇️
modin/experimental/engines/pandas_on_ray/io_exp.py	0.00% <0.00%> (-100.00%)	⬇️
modin/experimental/pandas/__init__.py	0.00% <0.00%> (-100.00%)	⬇️
modin/experimental/pandas/io_exp.py	0.00% <0.00%> (-83.79%)	⬇️
modin/experimental/xgboost/test/test_xgboost.py	0.00% <0.00%> (-100.00%)	⬇️
modin/experimental/xgboost/xgboost.py	0.00% <0.00%> (-97.23%)	⬇️
... and 102 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0bc409d...caed341. Read the comment docs.

[modin-bot]

TeamCity Ray test results bot

Tests PASSed

Tests Logs

ced_errors":[{"created":"@1630540459.390618410","description":"Address family not supported by protocol","errno":97,"file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":446,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:46021"},{"created":"@1630540459.390650703","description":"Unable to configure socket","fd":19,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1630540459.390646222","description":"Address already in use","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":190,"os_error":"Address already in use","syscall":"bind"}]}]}]}
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>
�[2m�[33m(raylet)�[0m     agent = DashboardAgent(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__
�[2m�[33m(raylet)�[0m     self.grpc_port = self.server.add_insecure_port(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port
�[2m�[33m(raylet)�[0m     return _common.validate_port_binding_result(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result
�[2m�[33m(raylet)�[0m     raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)
�[2m�[33m(raylet)�[0m RuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.
...x--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>\n    agent = DashboardAgent(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__\n    self.grpc_port = self.server.add_insecure_port(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port\n    return _common.validate_port_binding_result(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result\n    raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)\nRuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.\n'
Arguments: ()
�[2m�[33m(raylet)�[0m E0901 23:54:21.761229787    9626 server_chttp2.cc:49]        {"created":"@1630540461.761145479","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":872,"referenced_errors":[{"created":"@1630540461.761140303","description":"Failed to add any wildcard listeners","file":"src/core/lib/iomgr/tcp_server_posix.cc","file_line":340,"referenced_errors":[{"created":"@1630540461.761108654","description":"Address family not supported by protocol","errno":97,"file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":446,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:46021"},{"created":"@1630540461.761139035","description":"Unable to configure socket","fd":19,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1630540461.761134668","description":"Address already in use","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":190,"os_error":"Address already in use","syscall":"bind"}]}]}]}
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>
�[2m�[33m(raylet)�[0m     agent = DashboardAgent(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__
�[2m�[33m(raylet)�[0m     self.grpc_port = self.server.add_insecure_port(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port
�[2m�[33m(raylet)�[0m     return _common.validate_port_binding_result(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result
�[2m�[33m(raylet)�[0m     raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)
�[2m�[33m(raylet)�[0m RuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>\n    loop.run_until_complete(agent.run())\n  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete\n    return future.result()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run\n    modules = self._load_modules()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules\n    c = cls(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__\n    self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__\n    prometheus_exporter.new_stats_exporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter\n    exporter = PrometheusStatsExporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__\n    self.serve_http()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http\n    start_http_server(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server\n    httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server\n    server = server_class((host, port), handler_class)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__\n    self.server_bind()\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind\n    HTTPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind\n    socketserver.TCPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind\n    self.socket.bind(self.server_address)\nOSError: [Errno 98] Address already in use\n'
Arguments: ()
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>
�[2m�[33m(raylet)�[0m     loop.run_until_complete(agent.run())
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
�[2m�[33m(raylet)�[0m     return future.result()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run
�[2m�[33m(raylet)�[0m     modules = self._load_modules()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules
�[2m�[33m(raylet)�[0m     c = cls(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__
�[2m�[33m(raylet)�[0m     self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__
�[2m�[33m(raylet)�[0m     prometheus_exporter.new_stats_exporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter
�[2m�[33m(raylet)�[0m     exporter = PrometheusStatsExporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__
�[2m�[33m(raylet)�[0m     self.serve_http()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http
�[2m�[33m(raylet)�[0m     start_http_server(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server
�[2m�[33m(raylet)�[0m     httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server
�[2m�[33m(raylet)�[0m     server = server_class((host, port), handler_class)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__
�[2m�[33m(raylet)�[0m     self.server_bind()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind
�[2m�[33m(raylet)�[0m     HTTPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind
�[2m�[33m(raylet)�[0m     socketserver.TCPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind
�[2m�[33m(raylet)�[0m     self.socket.bind(self.server_address)
�[2m�[33m(raylet)�[0m OSError: [Errno 98] Address already in use
..--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>\n    loop.run_until_complete(agent.run())\n  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete\n    return future.result()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run\n    modules = self._load_modules()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules\n    c = cls(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__\n    self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__\n    prometheus_exporter.new_stats_exporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter\n    exporter = PrometheusStatsExporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__\n    self.serve_http()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http\n    start_http_server(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server\n    httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server\n    server = server_class((host, port), handler_class)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__\n    self.server_bind()\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind\n    HTTPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind\n    socketserver.TCPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind\n    self.socket.bind(self.server_address)\nOSError: [Errno 98] Address already in use\n'
Arguments: ()
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>
�[2m�[33m(raylet)�[0m     loop.run_until_complete(agent.run())
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
�[2m�[33m(raylet)�[0m     return future.result()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run
�[2m�[33m(raylet)�[0m     modules = self._load_modules()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules
�[2m�[33m(raylet)�[0m     c = cls(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__
�[2m�[33m(raylet)�[0m     self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__
�[2m�[33m(raylet)�[0m     prometheus_exporter.new_stats_exporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter
�[2m�[33m(raylet)�[0m     exporter = PrometheusStatsExporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__
�[2m�[33m(raylet)�[0m     self.serve_http()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http
�[2m�[33m(raylet)�[0m     start_http_server(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server
�[2m�[33m(raylet)�[0m     httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server
�[2m�[33m(raylet)�[0m     server = server_class((host, port), handler_class)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__
�[2m�[33m(raylet)�[0m     self.server_bind()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind
�[2m�[33m(raylet)�[0m     HTTPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind
�[2m�[33m(raylet)�[0m     socketserver.TCPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind
�[2m�[33m(raylet)�[0m     self.socket.bind(self.server_address)
�[2m�[33m(raylet)�[0m OSError: [Errno 98] Address already in use
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>\n    agent = DashboardAgent(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__\n    self.grpc_port = self.server.add_insecure_port(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port\n    return _common.validate_port_binding_result(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result\n    raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)\nRuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.\n'
Arguments: ()
�[2m�[33m(raylet)�[0m E0901 23:54:24.014653259    9644 server_chttp2.cc:49]        {"created":"@1630540464.014569482","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":872,"referenced_errors":[{"created":"@1630540464.014564406","description":"Failed to add any wildcard listeners","file":"src/core/lib/iomgr/tcp_server_posix.cc","file_line":340,"referenced_errors":[{"created":"@1630540464.014532656","description":"Address family not supported by protocol","errno":97,"file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":446,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:46021"},{"created":"@1630540464.014563080","description":"Unable to configure socket","fd":19,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1630540464.014558632","description":"Address already in use","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":190,"os_error":"Address already in use","syscall":"bind"}]}]}]}
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>
�[2m�[33m(raylet)�[0m     agent = DashboardAgent(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__
�[2m�[33m(raylet)�[0m     self.grpc_port = self.server.add_insecure_port(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port
�[2m�[33m(raylet)�[0m     return _common.validate_port_binding_result(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result
�[2m�[33m(raylet)�[0m     raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)
�[2m�[33m(raylet)�[0m RuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>\n    loop.run_until_complete(agent.run())\n  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete\n    return future.result()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run\n    modules = self._load_modules()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules\n    c = cls(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__\n    self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__\n    prometheus_exporter.new_stats_exporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter\n    exporter = PrometheusStatsExporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__\n    self.serve_http()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http\n    start_http_server(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server\n    httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server\n    server = server_class((host, port), handler_class)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__\n    self.server_bind()\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind\n    HTTPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind\n    socketserver.TCPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind\n    self.socket.bind(self.server_address)\nOSError: [Errno 98] Address already in use\n'
Arguments: ()
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>
�[2m�[33m(raylet)�[0m     loop.run_until_complete(agent.run())
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
�[2m�[33m(raylet)�[0m     return future.result()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run
�[2m�[33m(raylet)�[0m     modules = self._load_modules()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules
�[2m�[33m(raylet)�[0m     c = cls(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__
�[2m�[33m(raylet)�[0m     self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__
�[2m�[33m(raylet)�[0m     prometheus_exporter.new_stats_exporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter
�[2m�[33m(raylet)�[0m     exporter = PrometheusStatsExporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__
�[2m�[33m(raylet)�[0m     self.serve_http()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http
�[2m�[33m(raylet)�[0m     start_http_server(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server
�[2m�[33m(raylet)�[0m     httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server
�[2m�[33m(raylet)�[0m     server = server_class((host, port), handler_class)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__
�[2m�[33m(raylet)�[0m     self.server_bind()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind
�[2m�[33m(raylet)�[0m     HTTPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind
�[2m�[33m(raylet)�[0m     socketserver.TCPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind
�[2m�[33m(raylet)�[0m     self.socket.bind(self.server_address)
�[2m�[33m(raylet)�[0m OSError: [Errno 98] Address already in use
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>\n    agent = DashboardAgent(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__\n    self.grpc_port = self.server.add_insecure_port(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port\n    return _common.validate_port_binding_result(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result\n    raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)\nRuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.\n'
Arguments: ()
�[2m�[33m(raylet)�[0m E0901 23:54:26.297383349    9665 server_chttp2.cc:49]        {"created":"@1630540466.297290025","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":872,"referenced_errors":[{"created":"@1630540466.297283507","description":"Failed to add any wildcard listeners","file":"src/core/lib/iomgr/tcp_server_posix.cc","file_line":340,"referenced_errors":[{"created":"@1630540466.297235488","description":"Address family not supported by protocol","errno":97,"file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":446,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:46021"},{"created":"@1630540466.297282093","description":"Unable to configure socket","fd":19,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1630540466.297274301","description":"Address already in use","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":190,"os_error":"Address already in use","syscall":"bind"}]}]}]}
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>
�[2m�[33m(raylet)�[0m     agent = DashboardAgent(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__
�[2m�[33m(raylet)�[0m     self.grpc_port = self.server.add_insecure_port(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port
�[2m�[33m(raylet)�[0m     return _common.validate_port_binding_result(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result
�[2m�[33m(raylet)�[0m     raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)
�[2m�[33m(raylet)�[0m RuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>\n    loop.run_until_complete(agent.run())\n  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete\n    return future.result()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run\n    modules = self._load_modules()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules\n    c = cls(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__\n    self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__\n    prometheus_exporter.new_stats_exporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter\n    exporter = PrometheusStatsExporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__\n    self.serve_http()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http\n    start_http_server(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server\n    httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server\n    server = server_class((host, port), handler_class)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__\n    self.server_bind()\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind\n    HTTPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind\n    socketserver.TCPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind\n    self.socket.bind(self.server_address)\nOSError: [Errno 98] Address already in use\n'
Arguments: ()
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>
�[2m�[33m(raylet)�[0m     loop.run_until_complete(agent.run())
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
�[2m�[33m(raylet)�[0m     return future.result()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run
�[2m�[33m(raylet)�[0m     modules = self._load_modules()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules
�[2m�[33m(raylet)�[0m     c = cls(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__
�[2m�[33m(raylet)�[0m     self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__
�[2m�[33m(raylet)�[0m     prometheus_exporter.new_stats_exporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter
�[2m�[33m(raylet)�[0m     exporter = PrometheusStatsExporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__
�[2m�[33m(raylet)�[0m     self.serve_http()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http
�[2m�[33m(raylet)�[0m     start_http_server(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server
�[2m�[33m(raylet)�[0m     httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server
�[2m�[33m(raylet)�[0m     server = server_class((host, port), handler_class)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__
�[2m�[33m(raylet)�[0m     self.server_bind()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind
�[2m�[33m(raylet)�[0m     HTTPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind
�[2m�[33m(raylet)�[0m     socketserver.TCPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind
�[2m�[33m(raylet)�[0m     self.socket.bind(self.server_address)
�[2m�[33m(raylet)�[0m OSError: [Errno 98] Address already in use
.                                              [100%]--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>\n    agent = DashboardAgent(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__\n    self.grpc_port = self.server.add_insecure_port(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port\n    return _common.validate_port_binding_result(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result\n    raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)\nRuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.\n'
Arguments: ()
�[2m�[33m(raylet)�[0m E0901 23:54:28.565553381    9680 server_chttp2.cc:49]        {"created":"@1630540468.565480135","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":872,"referenced_errors":[{"created":"@1630540468.565474673","description":"Failed to add any wildcard listeners","file":"src/core/lib/iomgr/tcp_server_posix.cc","file_line":340,"referenced_errors":[{"created":"@1630540468.565446019","description":"Address family not supported by protocol","errno":97,"file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":446,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:46021"},{"created":"@1630540468.565473530","description":"Unable to configure socket","fd":19,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1630540468.565470059","description":"Address already in use","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":190,"os_error":"Address already in use","syscall":"bind"}]}]}]}
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>
�[2m�[33m(raylet)�[0m     agent = DashboardAgent(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__
�[2m�[33m(raylet)�[0m     self.grpc_port = self.server.add_insecure_port(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port
�[2m�[33m(raylet)�[0m     return _common.validate_port_binding_result(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result
�[2m�[33m(raylet)�[0m     raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)
�[2m�[33m(raylet)�[0m RuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>\n    loop.run_until_complete(agent.run())\n  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete\n    return future.result()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run\n    modules = self._load_modules()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules\n    c = cls(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__\n    self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__\n    prometheus_exporter.new_stats_exporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter\n    exporter = PrometheusStatsExporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__\n    self.serve_http()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http\n    start_http_server(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server\n    httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server\n    server = server_class((host, port), handler_class)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__\n    self.server_bind()\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind\n    HTTPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind\n    socketserver.TCPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind\n    self.socket.bind(self.server_address)\nOSError: [Errno 98] Address already in use\n'
Arguments: ()
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>
�[2m�[33m(raylet)�[0m     loop.run_until_complete(agent.run())
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
�[2m�[33m(raylet)�[0m     return future.result()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run
�[2m�[33m(raylet)�[0m     modules = self._load_modules()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules
�[2m�[33m(raylet)�[0m     c = cls(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__
�[2m�[33m(raylet)�[0m     self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__
�[2m�[33m(raylet)�[0m     prometheus_exporter.new_stats_exporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter
�[2m�[33m(raylet)�[0m     exporter = PrometheusStatsExporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__
�[2m�[33m(raylet)�[0m     self.serve_http()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http
�[2m�[33m(raylet)�[0m     start_http_server(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server
�[2m�[33m(raylet)�[0m     httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server
�[2m�[33m(raylet)�[0m     server = server_class((host, port), handler_class)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__
�[2m�[33m(raylet)�[0m     self.server_bind()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind
�[2m�[33m(raylet)�[0m     HTTPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind
�[2m�[33m(raylet)�[0m     socketserver.TCPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind
�[2m�[33m(raylet)�[0m     self.socket.bind(self.server_address)
�[2m�[33m(raylet)�[0m OSError: [Errno 98] Address already in use
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>\n    loop.run_until_complete(agent.run())\n  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete\n    return future.result()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run\n    modules = self._load_modules()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules\n    c = cls(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__\n    self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__\n    prometheus_exporter.new_stats_exporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter\n    exporter = PrometheusStatsExporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__\n    self.serve_http()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http\n    start_http_server(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server\n    httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server\n    server = server_class((host, port), handler_class)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__\n    self.server_bind()\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind\n    HTTPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind\n    socketserver.TCPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind\n    self.socket.bind(self.server_address)\nOSError: [Errno 98] Address already in use\n'
Arguments: ()
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>
�[2m�[33m(raylet)�[0m     loop.run_until_complete(agent.run())
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
�[2m�[33m(raylet)�[0m     return future.result()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run
�[2m�[33m(raylet)�[0m     modules = self._load_modules()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules
�[2m�[33m(raylet)�[0m     c = cls(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__
�[2m�[33m(raylet)�[0m     self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__
�[2m�[33m(raylet)�[0m     prometheus_exporter.new_stats_exporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter
�[2m�[33m(raylet)�[0m     exporter = PrometheusStatsExporter(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__
�[2m�[33m(raylet)�[0m     self.serve_http()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http
�[2m�[33m(raylet)�[0m     start_http_server(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server
�[2m�[33m(raylet)�[0m     httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server
�[2m�[33m(raylet)�[0m     server = server_class((host, port), handler_class)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__
�[2m�[33m(raylet)�[0m     self.server_bind()
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind
�[2m�[33m(raylet)�[0m     HTTPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind
�[2m�[33m(raylet)�[0m     socketserver.TCPServer.server_bind(self)
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind
�[2m�[33m(raylet)�[0m     self.socket.bind(self.server_address)
�[2m�[33m(raylet)�[0m OSError: [Errno 98] Address already in use
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>\n    agent = DashboardAgent(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__\n    self.grpc_port = self.server.add_insecure_port(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port\n    return _common.validate_port_binding_result(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result\n    raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)\nRuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.\n'
Arguments: ()
�[2m�[33m(raylet)�[0m E0901 23:54:30.898443232    9701 server_chttp2.cc:49]        {"created":"@1630540470.898352990","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":872,"referenced_errors":[{"created":"@1630540470.898347555","description":"Failed to add any wildcard listeners","file":"src/core/lib/iomgr/tcp_server_posix.cc","file_line":340,"referenced_errors":[{"created":"@1630540470.898316146","description":"Address family not supported by protocol","errno":97,"file":"src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":446,"os_error":"Address family not supported by protocol","syscall":"socket","target_address":"[::]:46021"},{"created":"@1630540470.898346153","description":"Unable to configure socket","fd":19,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":216,"referenced_errors":[{"created":"@1630540470.898342155","description":"Address already in use","errno":98,"file":"src/core/lib/iomgr/tcp_server_utils_posix_common.cc","file_line":190,"os_error":"Address already in use","syscall":"bind"}]}]}]}
�[2m�[33m(raylet)�[0m Traceback (most recent call last):
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 382, in <module>
�[2m�[33m(raylet)�[0m     raise e
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 352, in <module>
�[2m�[33m(raylet)�[0m     agent = DashboardAgent(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 87, in __init__
�[2m�[33m(raylet)�[0m     self.grpc_port = self.server.add_insecure_port(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/aio/_server.py", line 83, in add_insecure_port
�[2m�[33m(raylet)�[0m     return _common.validate_port_binding_result(
�[2m�[33m(raylet)�[0m   File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/grpc/_common.py", line 166, in validate_port_binding_result
�[2m�[33m(raylet)�[0m     raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)
�[2m�[33m(raylet)�[0m RuntimeError: Failed to bind to address [::]:46021; set GRPC_VERBOSITY=debug environment variable to see detailed error message.
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/worker.py", line 1197, in listen_error_messages_raylet
    threads_stopped.wait(timeout=0.01)
Message: 'The agent on node 3855add55d23 failed with the following error:\nTraceback (most recent call last):\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 370, in <module>\n    loop.run_until_complete(agent.run())\n  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete\n    return future.result()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 151, in run\n    modules = self._load_modules()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/agent.py", line 105, in _load_modules\n    c = cls(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/new_dashboard/modules/reporter/reporter_agent.py", line 153, in __init__\n    self._metrics_agent = MetricsAgent(dashboard_agent.metrics_export_port)\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/metrics_agent.py", line 75, in __init__\n    prometheus_exporter.new_stats_exporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 333, in new_stats_exporter\n    exporter = PrometheusStatsExporter(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 266, in __init__\n    self.serve_http()\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/ray/_private/prometheus_exporter.py", line 320, in serve_http\n    start_http_server(\n  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/prometheus_client/exposition.py", line 149, in start_wsgi_server\n    httpd = make_server(addr, port, app, ThreadingWSGIServer, handler_class=_SilentHandler)\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 154, in make_server\n    server = server_class((host, port), handler_class)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 452, in __init__\n    self.server_bind()\n  File "/root/anaconda3/envs/modin/lib/python3.8/wsgiref/simple_server.py", line 50, in server_bind\n    HTTPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/http/server.py", line 138, in server_bind\n    socketserver.TCPServer.server_bind(self)\n  File "/root/anaconda3/envs/modin/lib/python3.8/socketserver.py", line 466, in server_bind\n    self.socket.bind(self.server_address)\nOSError: [Errno 98] Address already in use\n'
Arguments: ()
error: object directory /localdisk/tc_agent/system/git/git-CE4319E5.git/objects does not exist; check .git/objects/info/alternates
fatal: bad object HEAD


---------- coverage: platform linux, python 3.8.10-final-0 -----------
Coverage XML written to file coverage.xml

= 8342 passed, 37237 skipped, 3276 xfailed, 20696 warnings in 315.46s (0:05:15) =
PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.
Tests are successful


<b>Remaining output truncated<b>

[modin-bot]

TeamCity Dask test results bot

Tests PASSed

Tests Logs

un_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
--- Logging error ---
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7fb154275940>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:40419')>),)
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:45567'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7fecf8ec2d30>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:45567')>),)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:46155'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:46155'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7f36e43379d0>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:46155')>),)
--- Logging error ---
Traceback (most recent call last):
--- Logging error ---
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:45681'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:40909'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
--- Logging error ---
KeyError: 'tcp://127.0.0.1:45681'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:40909'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7f4250142040>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:45681')>),)
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7f1214565310>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:40909')>),)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:35329'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
--- Logging error ---
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:45517'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:35329'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7efc5adce8b0>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:35329')>),)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:45517'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7f2730100700>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:45517')>),)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:32795'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
--- Logging error ---
--- Logging error ---
Traceback (most recent call last):
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
--- Logging error ---
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:32795'

During handling of the above exception, another exception occurred:

KeyError: 'tcp://127.0.0.1:35425'
Traceback (most recent call last):

During handling of the above exception, another exception occurred:

  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
Traceback (most recent call last):
ValueError: I/O operation on closed file.
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
Call stack:
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7fecd899c5e0>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:32795')>),)
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:44077'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:35425'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7fe2045a25e0>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:35425')>),)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:44077'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7f0ea906bd30>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:44077')>),)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:33433'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:33433'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7f676c2b8d30>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:33433')>),)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:43799'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
Message: 'Exception while handling op %s'
Arguments: ('register-client',)
--- Logging error ---
Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/tcpserver.py", line 331, in <lambda>
    gen.convert_yielded(future), lambda f: f.result()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py", line 510, in _handle_stream
    await self.comm_handler(comm)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/core.py", line 502, in handle_comm
    result = await result
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5243, in add_client
    await self.handle_stream(comm=comm, extra={"client": client})
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5272, in remove_client
    self.client_releases_keys(
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5010, in client_releases_keys
    self.transitions(recommendations)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 7054, in transitions
    self.send_all(client_msgs, worker_msgs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/scheduler.py", line 5536, in send_all
    w = stream_comms[worker]
KeyError: 'tcp://127.0.0.1:43799'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/envs/modin/lib/python3.8/logging/__init__.py", line 1088, in emit
    stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/modin/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/utils.py", line 403, in run_loop
    loop.start()
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
    handle._run()
  File "/root/anaconda3/envs/modin/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 688, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/root/anaconda3/envs/modin/lib/python3.8/site-packages/tornado/ioloop.py", line 761, in _run_callback
    app_log.error("Exception in callback %r", callback, exc_info=True)
Message: 'Exception in callback %r'
Arguments: (functools.partial(<function TCPServer._handle_connection.<locals>.<lambda> at 0x7fbf9c862af0>, <Task finished name='Task-27' coro=<BaseTCPListener._handle_stream() done, defined at /root/anaconda3/envs/modin/lib/python3.8/site-packages/distributed/comm/tcp.py:493> exception=KeyError('tcp://127.0.0.1:43799')>),)
error: object directory /localdisk/tc_agent/system/git/git-CE4319E5.git/objects does not exist; check .git/objects/info/alternates
fatal: bad object HEAD


---------- coverage: platform linux, python 3.8.10-final-0 -----------
Coverage XML written to file coverage.xml

= 8342 passed, 37237 skipped, 3276 xfailed, 20695 warnings in 562.21s (0:09:22) =
PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.
Tests are successful


<b>Remaining output truncated<b>

[YarShev]

@dchigarev , fix pydocstyle job, please.

[dchigarev]

This is a bug in a BaseQueryCompiler.view regarding to the incorrect default values (implicitly causes some problems in a Groupby.getitem for a base backend). Ideally, this should be a separate PR fixing this, but maybe we can keep it here since the fix is just about fixing the typo?

[YarShev]

Let's leave this here.

[devin-petersohn]

I would prefer to move this to a new PR to track the change at a finer granularity.

[dchigarev]

this is a full code duplication from _wrap_aggregation because these functions are actually equal. In #3197 _apply_agg_function will be removed as well as this duplication

[dchigarev]

this attribute was missing

[dchigarev]

this condition has the same meaning as here, the difference is that the base backend already handles all of the cases naturally except this one with the selection.

Ideally, we want to move all of the exceptions checks to the front-end, this would help us to get rid of the code duplication for different backend in terms of raising proper errors and would help to get errors as soon as possible. But this is a task for a different PR

[dchigarev]

we don't want to drop conflicting columns if they're presented in the selection

[YarShev]

Let's leave this here.

[YarShev]

@dchigarev , what is the issue with partitions communication?

[dchigarev]

apply_indices is a parameter of broadcast_apply_full_axis which indicates the indices to apply a function on, so the function could be applied only to the required partitions

[dchigarev]

I found inconsistent behavior between Pandas and your branch on this test. On Modin column "a" is not present in the result dataframe, only in its index.

@gshimansky it's directly caused by recently reported #3376: aggregated columns has an intersection with the "by", because of that conflict, "a" columns is not presented in the result. The reasons for that are described in the actual issue.

btw, on the current master the reproducer you provided also does not include "a" to the result

[gshimansky]

I found inconsistent behavior between Pandas and your branch on this test. On Modin column "a" is not present in the result dataframe, only in its index.

@gshimansky it's directly caused by recently reported #3376: aggregated columns has an intersection with the "by", because of that conflict, "a" columns is not presented in the result. The reasons for that are described in the actual issue.

btw, on the current master the reproducer you provided also does not include "a" to the result

Ok I see. I'll add a comment with this code into that bug too, to be included in tests.

[dchigarev]

Why is this skipped for BaseOnPython?

Because base query compiler (as any other non-pandas QCs) can't handle SeriesGroupBy specific behaviour. I've prepared an issue about that, its content is the following:

I'll create it as soon as we merge this PR

Content to copy into a future issue

BaseOnPython backend can't execute SeriesGroupBy methods natively

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Any
• Modin version (modin.__version__): commit hash
• Python version: 3.7.10
• Code we can use to reproduce:

from modin.conftest import set_base_backend

set_base_backend() # Setting BaseOnPython backend

from modin.pandas.test.utils import create_test_dfs

data = {"a": [1, 1, 2, 2], "b": ["a", "b", "c", "d"], "c": [3, 4, 5, 6]}
md_df, pd_df = create_test_dfs(data)

md_res = md_df.groupby("a")["b"].sum()
pd_res = pd_df.groupby("a")["b"].sum()

print(f"pd_res:\n{pd_res}\n")
print(f"md_res:\n{md_res}\n")

Output:

pd_res:
a
1    ab
2    cd
Name: b, dtype: object

md_res:
Empty DataFrame
Columns: []
Index: [1, 2]

Describe the problem

Suddenly, pandas DataFrameGroupBy and SeriesGroupBy sometimes have different behaviors for certain aggregations, referencing to the example above, the SeriesGroupBy.sum() produces concatenation of string columns, when for DataFrameGroupBy.sum() it drops string columns as non-suitable for numeric aggregation:

>>> df = pandas.DataFrame({"col1": ["a", "b", "c"]})
>>> df.groupby([1, 1, 2]).sum()
Empty DataFrame
Columns: []
Index: [1, 2]
>>> df.squeeze(axis=1).groupby([1, 1, 2]).sum()
1    ab
2     c
Name: col1, dtype: object

The reason for the different results of pandas and modin is the same: doing __getitem__("b") produces SeriesGroupBy object, however, for a base backend we're always applying DataFrameGroupBy methods, and so are getting incorrect behavior:

modin/modin/backends/base/query_compiler.py

Line 2305 in dd91a78

return GroupByDefault.register(pandas.core.groupby.DataFrameGroupBy.sum)(

[dchigarev]

@YarShev @gshimansky @modin-project/modin-core all comments were addressed, PR is ready for review now

[YarShev]

@dchigarev , LGTM, thanks! Left one comment.

[YarShev]

@modin-project/modin-core , @modin-project/modin-omnisci reminder to review.

[devin-petersohn]

This is a really big diff, and I'm working my way through it. I left a couple of questions for my understanding.

[devin-petersohn]

This is like doing a mask before the groupby, right? Trying to understand.

[dchigarev]

Not exactly, it's more like doing a mask on the data, that's required to be aggregated, so it's like mask of the raw frame's data after grouping:

Code

>>> df
   a  b  c
0  1  5  3
1  1  4  4
2  2  3  5
>>> df[["a", "b"]].groupby("a").sum()
   b
a
1  9
2  3
>>> df.groupby("a")[["a", "b"]].sum()
   a  b
a
1  2  9
2  2  3

We can't achieve an effect of aggregating 'by' columns only doing a mask before groupby, unless we'll detach 'by' data from the frame:

Code

>>> df[["a", "b"]].groupby(df["a"]).sum()
   a  b
a
1  2  9
2  2  3

For now, there's no convention on what backend's groupby obtains as 'by', in some cases, it's detached columns (as query compiler of the 'by' columns), in other cases, it's a list of string column names.

It's not obvious which way of passing 'by' was added by mistake and so which one should be avoided, most of our implementation implies that 'by' is a query compiler, supposing that 'by' as query compiler is the true way, however #2744 notation (as I thought this PR should show an idealistic API that we want to follow) it forces us to always use columns names as 'by's.

To be able to work properly with both of 'by' types, we must pass extra information about actual columns that have to be aggregated (selection parameter).

BTW, our current groupby implementation for pandas backend always translates "detached" 'by' into the "attached" ones ([1] [2]) so we will always need to generate some kind of meta-info about actual selection (if we remove selection backend parameter, we still need this info, so it would be computed anyway inside backend's groupby).

[devin-petersohn]

Ok, this seems like it would be a great developer blog post for how to support this because of how inconsistent and weird it is. @aregm @Garra1980 agree?

It is tempting to add the new API and pass the state through to the backend (and eventually the compute kernels), but I would like to avoid this if possible. We can get the desired behavior by adding a .copy() to the grouping column if the __getitem__ comes after the groupby:

In [1]: import modin.pandas as pd
In [2]: df = pd.DataFrame([[1,5,3], [1,4,4], [2,3,5]], columns=list("abc"))
In [3]: df
Out[3]:
   a  b  c
0  1  5  3
1  1  4  4
2  2  3  5

In [4]: df.groupby("a")[["a", "b"]].sum()  # incorrect behavior
Out[4]:
   b
a
1  9
2  3

In [5]: df.groupby(df["a"])[["a", "b"]].sum()  # also incorrect, semantically equivalent to In[4] in pandas and modin
Out[5]:
   b
a
1  9
2  3

In [6]: df.groupby(df["a"].copy())[["a", "b"]].sum()  # adding copy at the API layer fixes it
Out[6]:
   a  b
a
1  2  9
2  2  3

In [7]: df[["a", "b"]].groupby("a").sum()  # correct behavior for __getitem__ before groupby
Out[7]:
   b
a
1  9
2  3

Does this make sense? We can avoid adding additional internal APIs this way.

[dchigarev]

In the case of doing a copy, we lose information about where this column came from (whether it originally was from the 'self' frame or it was an external Series). In some cases, GroupBy results depend on the origin of the 'by' data. Anyway, I can try this and see how it goes.

As for now, I decided to detach and move a small piece of this PR that's fixing the actual #3252 issue into a separate PR (#3463), so we can merge the fix until the 0.11 release.

As for this PR (#3298), it's more about supporting groupby.__getitem__ in a proper way, it's much more general than the bug fix, so I convert it to draft for now and will try to get rid of adding 'selection' as a new backend's API.

[devin-petersohn]

I would prefer to move this to a new PR to track the change at a finer granularity.

[devin-petersohn]

Why move the mask logic here?

[dchigarev]

it's not masking, it's actual selection. grouped_df here is a GroupBy object

[YarShev]

@dchigarev , what is the status here?

[dchigarev]

@YarShev
The current status is that we want to redo the solution from the beginning without adding new QC parameters. I've written the new proposal for the implementation in the corresponding issue (#3490). There's no work done yet, only a proposal.

Regarding this PR, we probably need to close this or convert it to a draft until the new implementation will come along.

[YarShev]

@dchigarev , I am okay with the proposal. We should try to put the fix in 0.12 release. With respect to this PR, I suggest closing it and opening a new one.

[Garra1980]

#3490 is planned for 0.13

[YarShev]

#3490 is planned for 0.13

Ok, don't mind.

[vnlitvinov]

@dchigarev please convert this to draft if it's not ready to be reviewed.

[dchigarev]

please convert this to draft if it's not ready to be reviewed.

I'm not author of this PR, @YarShev can you do it?

[YarShev]

please convert this to draft if it's not ready to be reviewed.

I'm not author of this PR, @YarShev can you do it?

done