ValueError: not enough values to unpack (expected 2, got 1)

[lazir0lufias]

Hi, sorry im new in this field.

!python tune_gpt.py --khan-dataroot /content/amps/khan/ --save-dir /content/drive/MyDrive/model/

when i using the above code on google colap, i got error

Traceback (most recent call last):
File "tune_gpt.py", line 333, in
main()
File "tune_gpt.py", line 318, in main
train_data = get_dataset(args)
File "tune_gpt.py", line 239, in get_dataset
len_multiplier, dirname = args.khan_dataroot.split("@")
ValueError: not enough values to unpack (expected 2, got 1)

How to fix this?

[hendrycks]

On colab there must not have been a directory name, as directories on drive might be structured unusually. Best, Dan Hendrycks

…

On Tue, Jun 21, 2022 at 7:07 PM lazir0lufias ***@***.***> wrote: Hi, sorry im new in this field. !python tune_gpt.py --khan-dataroot /content/amps/khan/ --save-dir /content/drive/MyDrive/model/ when i using the above code on google colap, i got error Traceback (most recent call last): File "tune_gpt.py", line 333, in main() File "tune_gpt.py", line 318, in main train_data = get_dataset(args) File "tune_gpt.py", line 239, in get_dataset len_multiplier, dirname = args.khan_dataroot.split("@") ValueError: not enough values to unpack (expected 2, got 1) How to fix this? — Reply to this email directly, view it on GitHub <#11>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACZBITT7QYZZD52LS2UTWITVQJYOJANCNFSM5ZOQHKGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[lazir0lufias]

free drive just 15gb but amps 23gb, how can i edit the scripts so that can be trained on colap ?

[gcalabria]

I have the same problem here. I am running it on my machine, not on colab. Any ideas?

Update:
I figured out what was causing this problem. You were probably passing the argument as a string. For example:

python t5_tune.py \
  --mathematica-dataroot "/home/gui/dev/t5math/data/amps/mathematica/*/*/*.txt"

However, it should be passed as a path (i.e., without the quotes):

python t5_tune.py \
  --mathematica-dataroot /home/gui/dev/t5math/data/amps/mathematica/*/*/*.txt

The problem now is that I am getting an error zsh: argument list too long: python, which I believe is caused because there is simply too many files in the corpus.

[ayaka14732]

I believe that dataroot is the path to the directories, i.e. /home/gui/dev/t5math/data/amps/mathematica, not a list of files.

[ayaka14732]

I understand the problem now. It should be

python t5_tune.py \
  --mathematica-dataroot="/home/gui/dev/t5math/data/amps/mathematica/*/*/*.txt"

in your case.