Add SynthID (watermerking by Google DeepMind) #34350
Conversation
# Add PT version of the bayesian detector
Further changes
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| return all_masks, all_g_values | ||
|
|
||
|
|
||
| def tpr_at_fpr(detector, detector_inputs, w_true, minibatch_size, target_fpr=0.01) -> torch.Tensor: |
There was a problem hiding this comment.
I have no idea what tpr and fpr means, let's either be explicit, or have a small docstring
| self.beta = torch.nn.Parameter(-2.5 + 0.001 * torch.randn(1, 1, watermarking_depth)) | ||
| self.delta = torch.nn.Parameter(0.001 * torch.randn(1, 1, self.watermarking_depth, watermarking_depth)) |
There was a problem hiding this comment.
these are usually initialized in the _init_weights function rather than here!
There was a problem hiding this comment.
Ihere we init with zeros or empty
There was a problem hiding this comment.
It's not common, but we do have this pattern in other places (e.g.)
(I also have no idea how to set this specific initialization in _init_weights 😅 )
|
|
||
| # [batch_size, seq_len, watermarking_depth] | ||
| # Long tensor doesn't work with einsum, so we need to switch to the same dtype as self.delta (FP32) | ||
| logits = torch.einsum("ijkl,ijkl->ijk", self.delta, x.type(self.delta.dtype)) + self.beta |
There was a problem hiding this comment.
would be a lot better if we can avoid einsums! 🤗
(i, j, k, l) x (i, j, k, l) -> (i, j, k)
would be:
(i, j, k, 1, l) x (i, j, k, l, 1) -> (i, j, k,1)
so:
self.delta[.., None,:] @ x.transpose(-2,-1)[..., None])
There was a problem hiding this comment.
Good idea!
(the correct form is then (self.delta[.., None,:] @ x[..., None]).squeeze())
What does this PR do?
Adds SynthID, a watermarking by DeepMind.
https://deepmind.google/technologies/synthid/
Applying watermarking and using a detector is added to
transfomers. Training a detector is added as a research project.