Initial commit #1
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
carmocca
commented
May 4, 2023
| ## License | ||
|
|
||
| Lit-LLaMA is released under the [Apache 2.0](https://github.com/Lightning-AI/lightning-llama/blob/main/LICENSE) license. | ||
| # FIXME |
Contributor
Author
There was a problem hiding this comment.
We probably want to refresh this
| > **Note** | ||
| > All scripts support argument [customization](customize_paths.md) | ||
| ### FIXME: update this |
Contributor
Author
There was a problem hiding this comment.
Need to try this on a A100
lantiga
approved these changes
May 5, 2023
|
|
||
| if hasattr(self, "bias"): | ||
| # causal self-attention; Self-attend: (B, nh, T, hs) x (B, nh, hs, T) -> (B, nh, T, T) | ||
| # NOTE: cannot use flash attention because it takes q.size(-1) as the norm factor which is different to the |
Contributor
There was a problem hiding this comment.
why is this conditioned on bias being there?
|
|
||
| class Tokenizer: | ||
| def __init__(self, vocabulary_path: Path, config_path: Path) -> None: | ||
| # https://github.com/Stability-AI/StableLM/blob/e60081/configs/stablelm-base-alpha-3b.yaml#L108 |
Contributor
There was a problem hiding this comment.
I think we should just vendor the yaml file in the repo directly
Contributor
Author
There was a problem hiding this comment.
Are you suggesting this as a showcase of the configs used?
Because this is a gpt-neox config, meaning we don't need to use it
Or do you want to add support for running the scripts by passing it?
Contributor
|
There's a test failing in windows and the readme to complete. I can work on the readme. |
Co-authored-by: Luca Antiga <luca@lightning.ai>
gkroiz
pushed a commit
to gkroiz/lit-gpt
that referenced
this pull request
Aug 31, 2023
* Make trainer configurable and add docker file * Fix bugs * Add dockerignore * Fix config * Fix bug * Fix big * Fix bug * Fix bug * Try dlprof * fix bug * Add pytorch logger * FIx import * Add pytorch profiler * Fix bug * Reorder docker file * Fix bug * Make pytorch profiler optional * Try to fix profiler * Pytorch profiler working. Shunt torch comms again * tune profiler params * Make pt profiler configurable and run for global batches * Fix bug * Fix batch offset * Fix bug * Debug print issues * More print stuff * Add nvtx ranges * Adjust model sizes * tune validation iters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Generation works.
I removed all files that are not updated for simplicity. We can port them from upstream on demand.