[R-299] Add cost estimation to testset generator

[ahgraber]

Describe the Feature
RAGAS supports cost estimation during evaluation; it would also be useful to know how much it costs to construct the synthetic dataset.

I would like to have the ability to use the CostCallbackHandler in both KnowledgeGraph creation and in TestsetGenerator.generate() calls. Specifically, it would be useful to track them separately and jointly:

• if I create a KnowledgeGraph with apply_transforms(), I would like to know the token count / cost
• if I call TestsetGenerator.generate() with that previously created KnowledgeGraph, I would like to know just the token count / cost of scenario generation and question synthesis
• if I call TestsetGenerator.generate_with_langchain_docs(), I would like to know the token count / cost of both creating the KnowledgeGraph and scenario + question generation

Why is the feature important for you?
Cost management is critical especially if source material is dynamic and requires frequent dataset synthesis.

_R-299

[Kefan-pauline]

+1

[jjmachan]

that is a very good suggestion @ahgraber - will get this sorted for you, hopefully by next week 🙂

[Kefan-pauline]

Thanks for adding this feature, however, it does not yet taken into account the cost of generating the KG used for testset generation, which is probably the most costly part.

[jjmachan]

hey @Kefan-pauline that is true - just created #1671 to track this. Will get to this soon 🙂