Skip to content

Commit 5a0764d

Browse files
committed
Merge branch 'main' into flox-smart
* main: Link to second cubed blog post (#702) Bump webpack from 5.92.1 to 5.93.0 (#700) Bump @fontsource-variable/inter from 5.0.18 to 5.0.20 (#698) Bump typescript from 5.5.3 to 5.5.4 (#699) Bump the dependencies group with 7 updates (#697)
2 parents dc559a7 + 231c2f3 commit 5a0764d

File tree

3 files changed

+211
-223
lines changed

3 files changed

+211
-223
lines changed

package.json

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -19,22 +19,22 @@
1919
"@chakra-ui/icons": "^2.1.1",
2020
"@chakra-ui/react": "^2.8.2",
2121
"@chakra-ui/system": "^2.6.2",
22-
"@emotion/react": "^11.11.4",
23-
"@emotion/styled": "^11.11.5",
24-
"@fontsource-variable/inter": "^5.0.18",
22+
"@emotion/react": "^11.13.0",
23+
"@emotion/styled": "^11.13.0",
24+
"@fontsource-variable/inter": "^5.0.20",
2525
"@giscus/react": "^3.0.0",
2626
"@mdx-js/loader": "^3.0.1",
2727
"@mdx-js/react": "^3.0.1",
28-
"@next/mdx": "^14.2.4",
29-
"apexcharts": "3.49.2",
28+
"@next/mdx": "^14.2.5",
29+
"apexcharts": "3.51.0",
3030
"d3": "^7.9.0",
3131
"date-fns": "^3.0.0",
3232
"date-fns-tz": "^3.1.3",
3333
"feed": "^4.2.2",
34-
"framer-motion": "^11.2.12",
34+
"framer-motion": "^11.3.21",
3535
"gray-matter": "^4.0.3",
36-
"isomorphic-dompurify": "^2.12.0",
37-
"next": "^14.2.4",
36+
"isomorphic-dompurify": "^2.14.0",
37+
"next": "^14.2.5",
3838
"next-mdx-remote": "^5.0.0",
3939
"react": "^18.3.1",
4040
"react-apexcharts": "^1.4.1",
@@ -49,7 +49,7 @@
4949
"eslint": "^9.8.0",
5050
"eslint-config-next": "14.2.5",
5151
"playwright": "^1.45.3",
52-
"typescript": ">=5.5.3",
53-
"webpack": "^5.92.1"
52+
"typescript": ">=5.5.4",
53+
"webpack": "^5.93.0"
5454
}
5555
}

src/posts/cubed-xarray/index.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,7 @@ FIGURE: Cubed's actual memory usage vs projected memory usage, for the largest w
150150

151151
You can see again that the [projected memory](https://tom-e-white.com/cubed/user-guide/memory.html) usage is below the allowed memory usage (else Cubed would have raised an exception before the job even started running), and the actual peak memory used was lower still. We've also plotted the reserved memory, which is a parameter intended to account for the memory usage of the executor itself (i.e. Lithops here), and was estimated by measuring beforehand using [`cubed.measure_reserved_memory()`](https://tom-e-white.com/cubed/generated/cubed.measure_reserved_memory.html#cubed.measure_reserved_memory).
152152

153-
One obvious tradeoff for this memory stability is that Cubed took **a lot longer** to complete - roughly 4x longer then dask for the 1.5TB workload (45m 22s vs 11m 26s). We will come back to discuss this shortly.
153+
One obvious tradeoff for this memory stability is that Cubed took **a lot longer** to complete - roughly 4x longer then dask for the 1.5TB workload (45m 22s vs 11m 26s). We will come back to discuss this shortly. (EDIT: Since then a lot of work has been put into optimizing Cubed's performance - see the [follow-up blog post](https://medium.com/pangeo/optimizing-cubed-7a0b8f65f5b7).)
154154

155155
Finally it's interesting to look at Cubed's equivalent of the task graph. To calculate one array (the product $UV$ from the quadratic means problem), Cubed's "Plan" for processing 1.5TB of data looks like this:
156156

@@ -167,7 +167,7 @@ Cubed uses a completely different paradigm to Dask (and other frameworks), and s
167167
### Disadvantages
168168

169169
1. **Writing to persistent storage is slow**
170-
In general writing and reading to persistent storage (disk or object store) is slow, and doing it repeatedly even more so. Whilst there is scope for considerable [optimization within Cubed](https://github.com/tomwhite/cubed/issues?q=is%3Aissue+is%3Aopen+label%3Aoptimization), the model of communicating between processes by writing to disk will likely always be slower for many problems than communicating using RAM like dask does. One idea for mitigating this might be to use a very fast storage technology like Redis to store intermediate results.
170+
In general writing and reading to persistent storage (disk or object store) is slow, and doing it repeatedly even more so. Whilst there is scope for considerable [optimization within Cubed](https://github.com/tomwhite/cubed/issues?q=is%3Aissue+is%3Aopen+label%3Aoptimization) (EDIT: see the [follow-up blog post](https://medium.com/pangeo/optimizing-cubed-7a0b8f65f5b7) for subsequent performance improvements), the model of communicating between processes by writing to disk will likely always be slower for many problems than communicating using RAM like dask does. One idea for mitigating this might be to use a very fast storage technology like Redis to store intermediate results.
171171
2. **Spinning up cloud services can be slow**
172172
There is also a time cost to spinning up the containers in which each task is performed, which can vary considerably between cloud services.
173173
3. **Higher monetary cost**

0 commit comments

Comments
 (0)