TTCs as Reward

[sandorstormen]

One way you can represent cost for RL algorithms is to adjust the reward function. We should implement an option to represent TTCs in terms of reward. The reward function (or more aptly named utility function) should give the negative TTC when performing an attack step that always succeeds.

There are two formulations for this:

$r(a) = -\texttt{ttc value}$, where $\texttt{ttc value}\sim\text{TTCDist}(a)$

$r(a) = -\texttt{ttc value}$, where $\texttt{ttc value} = \mathbb{E}[\text{TTCDist}(a)]$

such that $r(\cdot)$ is the reward/utility function and $a$ is an attack step.

Let me note here that this has already been done by Sandor and Manuel with a simplified coreLang.

[sandorstormen]

This may be implemented already. But I need pointers to how it is implemented and if all cases are covered.

[mrkickling]

This is not yet implemented, but something that we should have.
It could perhaps be another RewardMode in the malsim settings, I will try to find a solution that makes it understandable.

[mrkickling]

In this version would the agent have to attempt a step n times to succeed or only once and get the ttc as a penalty?

[mrkickling]

One question that comes up is if then the reward of an attack step is different for attackers and defenders in this mode. An attacker gets a ttc penalty/reward, the defender gets a penalty aswell, but using the ttc penalty for defenders probably makes no sense? I assume you are mainly thinking of the attacker perspective, but let me know if you had already thought about it.

[sandorstormen]

In this version would the agent have to attempt a step n times to succeed or only once and get the ttc as a penalty?

This is a good question. Let's say TTCs are turned on, then it would take x amount of attack step selections until you succeed. Each time you select an attack step you could give a reward of $-1$. It would then be equivalent to having the TTCs turned off and receiving $-x$ in reward when you perform the attack step. So there's a relation for the reward mode to whether the TTCs are turned on or if they are turned off.

One question that comes up is if then the reward of an attack step is different for attackers and defenders in this mode. An attacker gets a ttc penalty/reward, the defender gets a penalty aswell, but using the ttc penalty for defenders probably makes no sense? I assume you are mainly thinking of the attacker perspective, but let me know if you had already thought about it.

I don't think the defender reward works the same here since the defender penalty is based on the evaluation of the asset and the effect caused on the asset. But maybe @kasanari can give his thoughts here.

[mrkickling]

I first understood it as if this option (TTCs as reward) would have all steps succeed on the first attempt in the simulator but with a penalty representing the time it took.

In your new comment it sounds like the attacker could also be penalized for each attempt it does. It would be the same reward in the end, but it would happen instantaneously in the first approach and step by step in the second.

Which would make more sense?

I also thought about another approach that could fit both of these:

• Letting agents give a max budget when performing step (budget = ttc attempts, so time or resources). The default would be to run a step with just budget 1 (like it is now), but it would be possible to give a higher budget to be able to perform more things. The budget used could also represent the penalty/reward for that step.

don't think the defender reward works the same here

No, I also believe the defender rewards are different, they shouldn't have anything to do with TTCs.

[sandorstormen]

Which would make more sense?

Both of them make sense from my perspective. It depends on what you want to do.

Letting agents give a max budget when performing step (budget = ttc attempts, so time or resources). The default would be to run a step with just budget 1 (like it is now), but it would be possible to give a higher budget to be able to perform more things. The budget used could also represent the penalty/reward for that step.

This also makes sense to me.

Maybe it's best that you can define your own reward function? It seems like this would be a lot for you to do.

[mrkickling]

If we want the 'budget' approach it would not be enough to just define a reward function, but otherwise, yes it might be a better solution. You could inherit the simulator and overwrite _attacker_step_reward(), or would you prefer giving a lambda to the init or something?