Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Adding xLSTM #3464

Open
HallerPatrick opened this issue Jun 4, 2024 · 0 comments
Open

[Feature]: Adding xLSTM #3464

HallerPatrick opened this issue Jun 4, 2024 · 0 comments
Labels
feature A new feature

Comments

@HallerPatrick
Copy link
Collaborator

Problem statement

As part of this issue following improvements will be added:

  • Expand LanguageModel and LanguageModelTrainer that it can use xLSTM (sLSTM and mLSTM)
  • Allow subtoken tokenization fo LM training

Solution

The new officially xLSTM library just dropped. So its probably wise to use it instead of implementing it from scratch (though we have to check of dependency issues)

Additional Context

No response

@HallerPatrick HallerPatrick added the feature A new feature label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature A new feature
Projects
None yet
Development

No branches or pull requests

1 participant