Question about copy attention and batching by tokens

[weiyengs]

Hi all,
In a traditional batches of sentences setting, at each decoder time-step, it would attend to the encoder states of words in a sentence with regards to the probability distribution learnt by the copy mechanism. I'm confused that if we batch by tokens, and a sentence is actually split into 2 batches(due to chance), wouldn't it affect the decoder from attending to the encoded states from the first batch?

Example sentence: The light show at Jewel Changi has just ended.
Batch 1:
The light show at Jewel
Batch 2:
Changi has just ended.

How could the decoder attend to the encoded states from 2 separate batches?

Thanks all!

[vince62s]

"token batch" is just a wording to say "the maximum number of tokens in a batch is X" but the batch remains a list of sentence. the number of sentences varies depending on how the batch has been consituted.
hope this helps

[weiyengs]

"token batch" is just a wording to say "the maximum number of tokens in a batch is X" but the batch remains a list of sentence. the number of sentences varies depending on how the batch has been consituted.
hope this helps

Hey @vince62s, thanks for your quick answer. Let me get this straight with an example.

Batch size: 10 tokens
Dataset:
Example sentence #1: The light show at Jewel Changi . (7 tokens)
Example sentence #2: One question about copy attention and batching . (8 tokens)

Batch 1:
The light show at Jewel Changi . [pad] [pad] [pad]

Batch 2:
One question about copy attention and batching . [pad] [pad]

Is that right? thanks for your help!

[vince62s]

no. Padding is not related to the "token batch" size. Padding is only done so that all sentences in a batch have the same length.

[weiyengs]

I'm still confused at the boundary of the sentence when it happens between the batches.

Okay..let me try it again:
Batch size: 10 tokens
Dataset:
Example sentence #1: The light show at Jewel Changi . (7 tokens)
Example sentence #2: One question about copy attention and batching . (8 tokens)

Batch 1:
The light show at Jewel Changi . One question about
Batch 2:
copy attention and batching .

Is that right?

[vince62s]

no.
Your examples are not correct.

Sent 1: The light show at Jewel Changi . (7 tokens)
Sent 2: The dull light show at Jewel Changi . (8 tokens)
Sent 3: The very dull light show at Jewel Changi . (9 tokens)
if Batch size = 10 then it will be 3 batches
if batch size = 16 then
Batch 1 = sent 1 and Sent 2 (because 7+8 <= 16)
Batch 2 = sent 3
clear ?

[weiyengs]

Thanks for the examples! It's much clearer.