Huggingface Transformers/Tokenizers do not have required attributes for outlines.models.Transformers

[ahgraber]

I am trying to use the HuggingFace version of Phi3-mini, but I get the error AttributeError: 'TransformerTokenizer' object has no attribute 'all_special_tokens' when trying to instantiate an outlines.models.Transformers object:

hf_tkz = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
tokenizer = TransformerTokenizer(hf_tkz)

llm = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct",
    output_attentions=True,
    trust_remote_code=True,
)
model = Transformers(llm, tokenizer)

Traceback:

File ~/.../site-packages/outlines/models/transformers.py:74, in TransformerTokenizer.__init__(self, tokenizer, **kwargs)
     71     self.pad_token_id = self.tokenizer.pad_token_id
     72     self.pad_token = self.tokenizer.pad_token
---> 74     self.special_tokens = set(self.tokenizer.all_special_tokens)
     76     self.vocabulary = self.tokenizer.get_vocab()
     77     self.is_llama = isinstance(self.tokenizer, get_llama_tokenizer_types())

Package versions:

python                                   3.11.8           hdf0ec26_0_cpython    conda-forge
outlines                                 0.0.43           pyhd8ed1ab_0          conda-forge
transformers                             4.41.2           pyhd8ed1ab_0          conda-forge

[ahgraber]

I think this is caused by the messy namespace collisions possible with the overlapping modules (openai.Openai vs outlines.models.openai.OpenAI, transformers vs outlines.models.transformers vs outlines.models.transformers.transformers)

[lapp0]

You shouldn't be passing a TransformerTokenizer, it is created during initialization. Could you please try passing the AutoTokenizer instead?

class Transformers:
    """Represents a `transformers` model."""

    def __init__(
        self,
        model: "PreTrainedModel",
        tokenizer: "PreTrainedTokenizer",
    ):
        self.model = model
        self.tokenizer = TransformerTokenizer(tokenizer)

Also I'm noticing people using the internals rather than creating via model = outlines.models.transformers(<model_uri>). Could you help me understand why you're using the classes directly rather than the helper function / what your use case is?

[ahgraber]

That worked! I was using the models.transformers constructor previously but caused myself some problems with poor import sanitation -- when following the docs:

from outlines import models
model = models.transformers("mistralai/Mistral-7B-v0.1", device="cuda")

pylance reports "transformers" is not exported from module "outlines.models". So instead I did from outlines.models import transformers and it didn't occur to me to check the collisions vs the transformers package...

As far as why I'm trying to use the internals -- I am trying to follow along with this approach to optimizing the FSM, which calls for using the tokenizer directly. I suppose the recommended approach would be to use model.tokenizer?