Integrate with Ray for serving/training?

[richardliaw]

Hi there! am really excited about the progress you have made towards replicating ChatGPT in open source! We are strong believers in open models here and appreciate your contributions.

We noticed that you are using docker-compose as a cluster orchestration layer to train and serve your models. We wanted to suggest Ray as an alternative -- it is an open-source python-first cluster framework that has emerged as the training platform of choice for major players in this space, including OpenAI (link).

Have you tried using Ray to train and serve your models? I believe that it would yield substantial productivity improvements for your team. We would also be happy to help you get set up with Ray -- we'd love to see more open source large models trained on Ray.

cc @tchordia

[CloseChoice]

Thanks for reaching out @richardliaw. Currently we are using deepspeed for training our models. Could elaborate a bit on the differences and to deepspeed and the advantages of ray or point to some literature regarding that?

[richardliaw]

Hey @CloseChoice sorry for the slow response!

Ray sort of sits in a different part of the stack than deepspeed - here's a blog post we wrote recently about using the stack together.

One of the things about Ray for training is that it allows users to easily spin up machines and scale on the cloud, whereas deepspeed is mostly focused on optimizing the training process itself. Together, Ray and Deepspeed really simplify the experience for scaling training for large models.

Does that help/make sense?