Performance/Memory wise, it makes sense to just append the results of generator-experiments to file. Right now it just overwrites the results, so you need to build a growing data structure in your experiments and yield the whole thing every time.
We should still leave in the old option though as not all generator-experiment will involve linearly growing data structures.
To do this properly, we should have a way of accessing the latest element without reading the whole file, so we need to make or find something like an IndexedPickleFile object that maintains both a data and an index file.
Performance/Memory wise, it makes sense to just append the results of generator-experiments to file. Right now it just overwrites the results, so you need to build a growing data structure in your experiments and yield the whole thing every time.
We should still leave in the old option though as not all generator-experiment will involve linearly growing data structures.
To do this properly, we should have a way of accessing the latest element without reading the whole file, so we need to make or find something like an IndexedPickleFile object that maintains both a data and an index file.