prover/compute_proof: track duration of proving time

[pinkiebell]

No description provided.

[ed255]

LGTM!

A part from reporting the total proving time, maybe it's also useful to report the proving time for each individual proof?

[pinkiebell]

LGTM!

A part from reporting the total proving time, maybe it's also useful to report the proving time for each individual proof?

CC @AronisAt79 - What do you think?

[AronisAt79]

LGTM!
A part from reporting the total proving time, maybe it's also useful to report the proving time for each individual proof?

CC @AronisAt79 - What do you think?

hi @pinkiebell , i somewhow missed previous comment from Edu; For sure, that would help in terms of more accurately benchmarking proof generation times. Is ti easy to add duration fields per proof, in the results struct? And a question: proofs get calculated in parallel or in sequence?

[pinkiebell]

The proofs are processed in sequence. We can of course add more timing information here. Furthermore, the prover will support multiple instances of itself so that we can compute proofs in parallel - for each block.

@ed255 What happens when the supercircuit lands? Does it include all sub-circuits or do we still have dedicated components?

[AronisAt79]

The proofs are processed in sequence. We can of course add more timing information here. Furthermore, the prover will support multiple instances of itself so that we can compute proofs in parallel - for each block.

@pinkiebell, are there any dependencies regarding different parts of a single block proof? For example, evm must be computed first and so on?

p.s. please let us know if you proceed with additional duration fields so we can adapt the test code :)

[ed255]

@ed255 What happens when the supercircuit lands? Does it include all sub-circuits or do we still have dedicated components?

The SuperCircuit will include all sub-circuits, which will be proved all at once. In the PR that I'm preparing now the included circuits will be: EVM Circuit, Tx Circuit and Bytecode Circuit. In a following iteration I'll be adding more circuits (starting with the State Circuit).

[pinkiebell]

The proofs are processed in sequence. We can of course add more timing information here. Furthermore, the prover will support multiple instances of itself so that we can compute proofs in parallel - for each block.

@pinkiebell, are there any dependencies regarding different parts of a single block proof? For example, evm must be computed first and so on?

p.s. please let us know if you proceed with additional duration fields so we can adapt the test code :)

There are no dependencies (yet). Given the fact that the SuperCircuit will include all sub circuits, I'm wondering if it makes sense to include more timing information at the moment that will be obsolete in the future. On another note, I think it makes more sense to focus on scaling the block production (for all proofs) per block in the proverd thus, scaling the number of machines. Any sub-circuit parallel improvements can be focused on the other crates - especially because the circuit_benchmarks keeps track of individual performance for each circuit.

In practice, we have to limit the production to a 'sane' value. For example, assuming a block proof completes in 1h and we like to limit the number of not finalized blocks to 256 (on l1). We could produce a new block each ~14 seconds and have up to 256 prover instances active to process backlogs. This is already a bit heavy but needs to taken into account :)

[ed255]

I'm wondering if it makes sense to include more timing information at the moment that will be obsolete in the future.

You're completely right. In the near future we'll have the super circuit for which we will compute a proof, and then the aggregation circuit for which we will compute a proof (this one will depend on the super circuit proof). In a more distant future we may go back to having individual circuits for which we can compute their proofs in parallel, and then one or more aggregation circuits that verify all the individual circuits proofs.

The reason why I was asking for the sub-circuit timing information was because I think it's very useful to figure out which circuits take more time, so that we know where to focus on optimizations; but I agree that this would soon be obsolete in the tesnet! We can still get those numbers by running benchmarks outside of the testnet.

On another note, I think it makes more sense to focus on scaling the block production (for all proofs) per block in the proverd thus, scaling the number of machines. Any sub-circuit parallel improvements can be focused on the other crates - especially because the circuit_benchmarks keeps track of individual performance for each circuit.

Sounds good! Computing several block proofs in parallel doesn't require any support from halo2 nor circuits side. Distributing the computation of a single proof requires some work. So it makes sense to focus on the first option.

In practice, we have to limit the production to a 'sane' value. For example, assuming a block proof completes in 1h and we like to limit the number of not finalized blocks to 256 (on l1). We could produce a new block each ~14 seconds and have up to 256 prover instances active to process backlogs. This is already a bit heavy but needs to taken into account :)

This sounds great! And I think it's the simplest way to increase the throughput we have :)

About this PR, having said that I agree on not adding sub-circuit timings: from my side this can be merged any time now :)

[AronisAt79]

I'm wondering if it makes sense to include more timing information at the moment that will be obsolete in the future.

You're completely right. In the near future we'll have the super circuit for which we will compute a proof, and then the aggregation circuit for which we will compute a proof (this one will depend on the super circuit proof). In a more distant future we may go back to having individual circuits for which we can compute their proofs in parallel, and then one or more aggregation circuits that verify all the individual circuits proofs.

The reason why I was asking for the sub-circuit timing information was because I think it's very useful to figure out which circuits take more time, so that we know where to focus on optimizations; but I agree that this would soon be obsolete in the tesnet! We can still get those numbers by running benchmarks outside of the testnet.

On another note, I think it makes more sense to focus on scaling the block production (for all proofs) per block in the proverd thus, scaling the number of machines. Any sub-circuit parallel improvements can be focused on the other crates - especially because the circuit_benchmarks keeps track of individual performance for each circuit.

Sounds good! Computing several block proofs in parallel doesn't require any support from halo2 nor circuits side. Distributing the computation of a single proof requires some work. So it makes sense to focus on the first option.

In practice, we have to limit the production to a 'sane' value. For example, assuming a block proof completes in 1h and we like to limit the number of not finalized blocks to 256 (on l1). We could produce a new block each ~14 seconds and have up to 256 prover instances active to process backlogs. This is already a bit heavy but needs to taken into account :)

This sounds great! And I think it's the simplest way to increase the throughput we have :)

About this PR, having said that I agree on not adding sub-circuit timings: from my side this can be merged any time now :)

to the extent that i understand the concept of supercircuit/aggregation circuit and so on.... i also agree. On the testing front, we could start parallelizing proof computations for different blocks among individual prover instances, if this is already supported/implemented

[pinkiebell]

we could start parallelizing proof computations for different blocks among individual prover instances, if this is already supported/implemented

I will implement this soon 👍