Draft Breaking FEAT: Refactoring Single turn objective

[rlundeen2]

Separating objective from SeedPrompts and bringing single turn orchestrators closer to a unified orchestrator. It also allows for better attack separation

• Making Prepending Conversations a part of the run_attack_asnyc instead of a part of the orchestrator class. This allows unique prepended conversations per objective
• Separated SeedPrompt from objective
• Separated objective_scorer from auxiliary scorers
• Now returning OrchestratorResult (in common with multi-turn)
• Allowing converterconfigurations to be passed. Converters can now be applied to prepended conversations, and responses
• Allowing retry on objective not achieved

Tests and Documentation

I've currently updated all single turn orchestrators and ran their notebooks. I still need to clean things up and run tests, but would like feedback on the approach.

NOTE: I haven't run the cookbook yet

[bashirpartovi]

I would think on how you can encapsulate configurations. Right now, anything you would need in an orchestrator gets passed in as a single argument, and changes to init is usually breaking changes.

For example, you could encapsulate converter configuration inside ConverterConfiguration object. E.g.

@dataclass
class ConverterConfiguration:
       request_converter_configurations: Optional[....]
       ...
       
def __init__(self, *, ...., converter_config: Optional[ConverterConfiguration] = None, ... ):

This way, if for any reason, you need to change the converter config (add more fields), you won't need to touch the init, same with Scorers

[rlundeen2]

I love this, but also thinking about punting for your refactor.

[bashirpartovi]

You are technically doing broadcasting here and the fact that you are adjusting seed_prompts to match the size tells me, you want to implement batch_task_async to actually broadcast the calls. I'll give you an example hopefully tonight when I have some time to test things out

[bashirpartovi]

I have been playing around with this, and I think I have a great implementation that also handles broadcasting. For now we can go with what you have, I'll push that under a separate PR.

[bashirpartovi]

Added some comments on the syntax and the structure, will look at the design later tonight but looks promising :)

[romanlutz]

This will break the scanner and need an adjustment

[rlundeen2]

I'll keep this open, I have a million notebooks and tests to fix once we're happy with how it looks :)

[bashirpartovi]

<< this is mostly preference, nothing wrong with the code >>
I think this could be simplified a bit:

                # add `or []` it makes your life much easier :)
                auxiliary_scores = self._memory.get_scores_by_prompt_ids(prompt_request_response_ids=[str(piece.id)]) or []
                for auxiliary_score in auxiliary_scores:
                     if not self.score or auxiliary_score.id != self.score.id:
                         print(f"{Style.RESET_ALL}auxiliary score: {auxiliary_score} : {auxiliary_score.score_rationale}")

if you use or [], you initialize your auxiliary_scores to an empty list if self._memory.get_scores_by_prompt_ids(prompt_request_response_ids=[str(piece.id)]) is None.

That way, you can eliminate this check

if auxiliary_scores and len(auxiliary_scores) > 0:

If auxiliary_scores is an empty list, then for auxiliary_score in auxiliary_scores: won't even run.

[rlundeen2]

agreed! This is a lot more readable!