Update argilla integration to use argilla_sdk v2

[alvarobartt]

Description

This PR renames and updates Argilla to ArgillaBase, since now the client in argilla_sdk (later to be renamed to argilla only as per a recent discussion with @frascuchon) is named Argilla too. Besides that, the code has been updated to use the latest Python client instead not only for ArgillaBase but also for the subclasses TextGenerationToArgilla and PreferenceToArgilla.

Warning

This change here implies that the argilla server version should be 1.27.0 or higher, otherwise the argilla_sdk won't work.

Closes argilla-io/argilla#4880

[burtenshaw]

@frascuchon What do you think the timeline on renaming argilla_sdk >> argilla is? Is it worth waiting for the changes in argilla so that it's in line?

[frascuchon]

These changes are minimal. We can leave it as is until we havemore stable versions of argilla v2.

[burtenshaw]

Nice start. Just a few high level comments.

[alvarobartt]

Edit: the issue was with the Argilla Server version as I was using 1.26.0 while 1.27.0 or higher was required 👍🏻

As an update @burtenshaw @frascuchon I've installed argilla_sdk from the latest version of argilla-python in main and running the code below, leads to the following exception, complaining about not finding the /records/bulk endpoint, could you try to reproduce on your end?

---

Install as pip install "distilabel[hf-inference-endpoints] @ git+https://github.com/argilla-io/[email protected]" and then run the code below with your personal HF_TOKEN and your Argilla credentials (feel free to use dev):

from uuid import uuid4

from distilabel.llms import InferenceEndpointsLLM
from distilabel.pipeline.local import Pipeline
from distilabel.steps import (
    LoadDataFromDicts,
    TextGenerationToArgilla,
)
from distilabel.steps.tasks import TextGeneration

if __name__ == "__main__":
    with Pipeline(name="my-pipeline") as pipeline:
        load_dataset = LoadDataFromDicts(
            name="load_dataset",
            data=[
                {
                    "instruction": "Write a short story about a dragon that saves a princess from a tower.",
                },
            ],
        )

        text_generation = TextGeneration(
            name="text_generation",
            llm=InferenceEndpointsLLM(
                model_id="meta-llama/Meta-Llama-3-8B-Instruct",
                tokenizer_id="meta-llama/Meta-Llama-3-8B-Instruct",
                api_key="...",  # type: ignore
            ),
            num_generations=4,
            group_generations=True,
        )

        text_generation_to_argilla = TextGenerationToArgilla(
            name="text_generation_to_argilla",
            api_url="...",
            api_key="...",  # type: ignore
            dataset_name=f"text-generation-{uuid4()}",
            dataset_workspace="admin",
        )

        (  # type: ignore
            load_dataset
            >> text_generation
            >> text_generation_to_argilla
        )

    pipeline.run(
        parameters={
            text_generation.name: {  # type: ignore
                "llm": {
                    "generation_kwargs": {
                        "max_new_tokens": 512,
                        "temperature": 0.7,
                    },
                },
            },
        }
    )

The logs then look like:

[frascuchon]

dataset.exists() should return already a bool value. So maybe extra checks are not needed.

Suggested change

	return (
	True
	if self._client.datasets( # type: ignore
	name=self.dataset_name, # type: ignore
	workspace=self._client.workspaces(name=self.dataset_workspace) # type: ignore
	if self.dataset_workspace is not None
	else None,
	).exists()
	is not None
	else False
	)
	workspace = (
	self._client.workspaces(name=self.dataset_workspace) # type: ignore
	if self.dataset_workspace is not None
	else None
	)
	return (
	self._client.datasets( # type: ignore
	name=self.dataset_name, # type: ignore
	workspace=workspace,
	).exists()
	)

[alvarobartt]

I didn't do it that way since it was returning None not sure if that is still a thing within the version in the argilla repository

[frascuchon]

@burtenshaw I think this is a side effect of calling API on Dataset.__init__. Maybe we should review to force client-dependant actions only for methods calling the API and not init ones (for example workspace is only needed when creating the dataset)