What would it take to 100x the context window?

[awfulcooking]

Thinking about what could be done when large language models can operate on phenomenally large context, and wondering what it might actually take to get there.

And realised this repo has a ton of really bright people in orbit, who actually understand brass tacks what might be involved.

Assuming it's really desirable, what hacks could be done to get there?

• What options are there?
• How much space or time would be involved?
• Would the models need full retraining?
• What about for 10x?

🙏

[redthing1]

The true context window of transformers is effectively baked in to the model. It's an inherent property, and it's tied to how the model learns positional embeddings and all of its layer states. If you want larger context you must completely retrain the model. Context is typically limited because of the computational complexity of the type of attention used by these models. However there are some variants like longformer which support 16k context.

[SlyEcho]

There used to be recurrent networks (RNN, LSTM) that are infinite but they are too hard to train using backpropagation and so after "Attention is all you need" the everybody switched to transformer models.

EDIT: Apparently RNN's are not dead yet: #846

[awfulcooking]

So the abstract from the Longformer paper answers some questions about computational complexity:

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length

They then say that mere linear scaling "makes it easy" !

Has a Longformer inspired mechanism made it into any of the huge models?

It still ultimately depends on a discrete set of positions for text to be in, right, modelled like pigeonholes / slots.

Maybe there's a way to project it continuously.

[tyzoid]

In theory, you could approximate it by using the model to compress the existing context window. For example, let's say you have a maximum window of 2048. Once you get 2048 tokens of context, take the first (perhaps 1024?) tokens of the current window, and ask the model to summarize it. Then you could replace the first tokens with the new "compressed" summarized context.

[jon-chuang]

You can use a layer on top of llama.cpp that either:

1. stores old context in a searcheable knowledge base
2. compresses old context via:
   • summarization. This just needs an instruction following model that can recursively summarise a context prompt prefix. I think langchain offers this ability.
   • latent compression (RNN-style). This requires a specific model architecture.

The memory required for larger context windows scales quadratically with the window size. 100x lossless context window is impractical from what I understand.

[funnbot]

A PR that implements what chatgpt does would be neat, https://www.pinecone.io/learn/langchain-conversational-memory/
but that isn't much of an increase. It also has the issue that summarizing usually loses important information, its at the whims of what the language model finds important.

I am curious if it would be possible to bypass the low compression of written language by adding custom embedding tokens that represent phrases/concepts instead of letters, would need to get tuned/trained in, and I guess set to zero probability when sampling.

edit: i guess that idea would be a multi-modal model

[xloem]

finetuning / lora with a longer max sequence length?

[SlyEcho]

The max context length is not something that can be extended because there are no weights for it. I think it may be possible to mess with the KV cache to compress it down or something.

[xloem]

The context length isn’t associated with weights is it? The position encodings are summed into each token embedding, the existing weights would simply need to be finetuned for the new position encodings, no?

[MariasStory]

https://paperswithcode.com/paper/scaling-transformer-to-1m-tokens-and-beyond

[awfulcooking]

I woke up thinking about interpolating input vectors, and/or the early representations that they make - whatever is created when the attention matrices are applied to the input embedding.

For example, could you blend "Explain quantum physics in easy to understand terms", "Gandalf", "Cyberpunk" and "Lord of the Rings", in an associative / multiplicative way that uses many more bits, and bits not associated directly with positional encoding, to move around in the latent / representational space in a new interesting way.

Like how visual AIs can blend between arbitrary points in their representation space.

In some way, maybe the compression of particularly large contexts that we do in our own brains, is more akin to that than sentence stringification.

For whatever it's worth.

[SlyEcho]

I was thinking that maybe the K and V buffers can be manipuleted using some image processing techniques like maxpool or convolutions.

[xloem]

If I were innovating something new here, my present idea is to organize an architecture similar to the concepts used in langchain, autogpt, and llama-index, but in-training: that is, I might wire the layers so they can navigate and update knowledge graphs of logits and weights, with concepts of nearness and relevance, and finetune them to learn to do this.

[SlyEcho]

The context length isn’t associated with weights is it? The position encodings are summed into each token embedding, the existing weights would simply need to be finetuned for the new position encodings, no?

I was wrong about this, I thought there were tensors that had size 2048 in them, but there aren't. The KQV weights are applied to the layer input size (so 4096). The self attention happens on all the past context, the huge matrixes are multiplied but no matter the size of the context, the output is still the embedding size that is passed to the next layer or is the result.

So it should be possible to increase the limits without any issues? Yes. You need to change the hard-coded limit in llama.cpp and also increase the memory to be allocated. Everything works, there are no crashes.

Except it doesn't work, once you go past some token number, everything breaks down and the output becomes more and more garbage. I don't know what the reason is, it could also be technical like NaNs or Infs appearing or maybe it is never going to work because at some point we are "taking the average" of too many tokens and there is nothing meaningful to extract from it.