Update numbers in retrieval statistics

[orionw]

Update the retrieval statistics to match the new metadata processing.

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[orionw]

@KennethEnevoldsen @isaac-chung I replaced one of the keys with the full information.

How do we want to break this up to fit in the existing schema?

[KennethEnevoldsen]

I think this looks very good. We can def. keep this as is.

[orionw]

I think I got every retrieval dataset, except for MSMarcoV2. My machine unfortunately died every time I tried to calculate it, apparently it requires a lot of RAM.

Edit: the tests failing appears to be a pydantic error. I am fairly new to pydantic and their error messages confuse me a bit -- @isaac-chung do you have any ideas on what is wrong?

[isaac-chung]

@orionw the pydantic error seems to be related to the presence of "task_name" in some dicts, e.g. mteb/tasks/Retrieval/dan/TV2Nordretrieval.py. The validation checks for a dict of {str:float} (for other tasks) or {str:{str: dict}} (for this change). The extra key will make it upset.

[orionw]

Ah thank you! I tried to delete all of them but clearly missed some. I'll update it. EDIT: you're fast, you already did - thanks!

[isaac-chung]

I'll try to run calculate_metadata_metrics for MSMARCOv2 on my machine.
Otherwise this is good to merge :D

[orionw]

Thanks @isaac-chung!! Good luck, it takes quite a while!

TBH if neither of us can load it, perhaps we need a more efficient dataset loader or simply to remove MSMarcoV2 from our list. My machine had quite a lot of RAM so it's pretty inaccessible -- looking at the specs on IR datasets it has 138 million passages!!

[isaac-chung]

Wow!
It seems to be running still. I'll give it till the morning (GMT+3) and report back.

[isaac-chung]

It went OOM killed already. Let's merge this and consider our options with MSMARCOv2.