Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

希望给数据集也添加深度重启功能 #881

Open
monet-joe opened this issue Jun 15, 2024 · 2 comments
Open

希望给数据集也添加深度重启功能 #881

monet-joe opened this issue Jun 15, 2024 · 2 comments
Assignees

Comments

@monet-joe
Copy link

Describe the feature
Features description
在modelscope以往的MsDataset框架中
ds = MsDataset.load(dataset, subset_name=subset, split="test")
想获得 ds 的 features 属性需要 ds._hf_ds.features
而新版框架中不需要 ._hf_ds ,也就是说MsDataset框架经过这么久的迭代已经有很大变化
那么在以往框架下开发的数据集,无法适配新框架的API调用代码的
即使将 modelscope pip upgrade到最新版本,在旧版框架下创建的数据集,其调用仍然需要 ._hf_ds
只有删除数据集再重建才能适配新框架

还包括以往的框架下创建数据集需要先指定是否是脚本模式,如果创建的是非脚本模式的数据集,在新框架下已经不区分了,此时即使上传同名脚本也不会激活viewer,也就是说旧数据集仍然保持在旧版框架的逻辑限制之内,无法跳脱出来

如果想既要保留star,还要让其刷新适配新框架,可能只有深度重启这个方法了,然而不支持
Motivation
A clear and concise description of the motivation of the feature. Ex1. It is inconvenient when [....]. Ex2. There is a recent paper [....], which is very helpful for [....].

Related resources
If there is an official code release or third-party implementations, please also provide the information here, which would be very helpful.

Additional context
Add any other context or screenshots about the feature request here. If you would like to implement the feature and create a PR, please leave a comment here and that would be much appreciated.

@monet-joe
Copy link
Author

@wenmengzhou @yingdachen @wangxingjun778 @tastelikefeet

@wangxingjun778
Copy link
Collaborator

Thanks for your suggestion, we are considering to support this feature in the next version. )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants