Multilingual text classification with BERT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "# Multilingual text classification with BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In this notebook we'll build a multilingual text classification model with [BERT](https://arxiv.org/abs/1810.04805). To this goal, we can use a multilingual BERT model:\n",
    "\n",
    "- BERT-multilingual was pretrained on monolingual text in 104 languages. A wordpiece tokenizer is used to map these texts to a shared vocabulary.\n",
    "- Because of this multilingual pre-training phase, BERT-multilingual can be finetuned in any of these languages to perform a task in the same language.\n",
    "- However, BERT-multilingual also has cross-lingual capabilities. This means it can generalize to a language it was not finetuned on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To investigate BERT's cross-lingual capabilities, we'll test it out on a multilingual sentiment analysis task: \n",
    "\n",
    "- Our data consists of product reviews in six languages: English, Dutch, German, Italian, French and Spanish.\n",
    "- These product reviews have been grouped into three classes: negative (reviews with one or two stars), mixed (three stars) and positive (four or five stars).\n",
    "\n",
    "Unfortunately, we cannot share this data. However, this notebook should work fine if you use your own data in ndjson format, where each document has a title, body and rating field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "languages = [\"en\", \"nl\", \"es\", \"fr\", \"de\", \"it\"]\n",
    "data_paths = {lang: f\"/home/yves/data/sentiment_all/{lang}/test_3.ndjson\" for lang in languages}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As usual, we start by splitting up the data in a training, development and testing portion. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== en ===\n",
      "train: 2430 documents\n",
      "dev: 270 documents\n",
      "test: 300 documents\n",
      "=== nl ===\n",
      "train: 2430 documents\n",
      "dev: 270 documents\n",
      "test: 300 documents\n",
      "=== es ===\n",
      "train: 2430 documents\n",
      "dev: 270 documents\n",
      "test: 300 documents\n",
      "=== fr ===\n",
      "train: 2430 documents\n",
      "dev: 270 documents\n",
      "test: 300 documents\n",
      "=== de ===\n",
      "train: 2430 documents\n",
      "dev: 270 documents\n",
      "test: 300 documents\n",
      "=== it ===\n",
      "train: 2430 documents\n",
      "dev: 270 documents\n",
      "test: 300 documents\n"
     ]
    }
   ],
   "source": [
    "import ndjson\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data = {}\n",
    "for lang in data_paths:\n",
    "    data[lang] = {\"train\": {}, \"dev\": {}, \"test\": {}}\n",
    "    \n",
    "    with open(data_paths[lang]) as i:\n",
    "        langdata = ndjson.load(i)\n",
    "        \n",
    "    texts = [\" \".join([doc[\"title\"], doc[\"body\"]]).strip() for doc in langdata]\n",
    "    labels = [doc[\"rating\"] for doc in langdata]\n",
    "    \n",
    "    rest_texts, test_texts, rest_labels, test_labels = train_test_split(texts, labels, test_size=0.1, random_state=1)\n",
    "    train_texts, dev_texts, train_labels, dev_labels = train_test_split(rest_texts, rest_labels, test_size=0.1, random_state=1)\n",
    "\n",
    "    data[lang][\"train\"][\"texts\"] = train_texts\n",
    "    data[lang][\"dev\"][\"texts\"] = dev_texts\n",
    "    data[lang][\"test\"][\"texts\"] = test_texts\n",
    "    \n",
    "    data[lang][\"train\"][\"labels\"] = train_labels\n",
    "    data[lang][\"dev\"][\"labels\"] = dev_labels\n",
    "    data[lang][\"test\"][\"labels\"] = test_labels\n",
    "    \n",
    "for lang in data: \n",
    "    print(f\"=== {lang} ===\")\n",
    "    for c in [\"train\", \"dev\", \"test\"]:\n",
    "          print(f\"{c}: {len(data[lang][c]['texts'])} documents\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Next we map each of the labels in our data to an index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'pos': 0, 'neg': 1, 'mixed': 2}\n"
     ]
    }
   ],
   "source": [
    "target_names = list(set(data[\"en\"][\"train\"][\"labels\"]))\n",
    "label2idx = {label: idx for idx, label in enumerate(target_names)}\n",
    "print(label2idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Initializing a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Most of the code we'll use to finetune and train BERT is the same as in our previous notebook in text classification with BERT. However, instead of using one of the English models, this time we're going to use one of BERT's multilingual models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/opt/anaconda3/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "I1031 17:56:15.545178 140624693438272 file_utils.py:39] PyTorch version 1.1.0 available.\n",
      "I1031 17:56:15.582974 140624693438272 modeling_xlnet.py:194] Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex .\n",
      "I1031 17:56:16.167633 140624693438272 tokenization_utils.py:374] loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-vocab.txt from cache at /home/yves/.cache/torch/transformers/bb773818882b0524dc53a1b31a2cc95bc489f000e7e19773ba07846011a6c711.535306b226c42cebebbc0dabc83b92ab11260e9919e21e2ab0beb301f267b4c7\n"
     ]
    }
   ],
   "source": [
    "from transformers.tokenization_bert import BertTokenizer\n",
    "\n",
    "BERT_MODEL = \"bert-base-multilingual-uncased\"\n",
    "tokenizer = BertTokenizer.from_pretrained(BERT_MODEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "As sentiment analysis is a type of sequence classification, we'll again load a model of type BertForSequenceClassification. This consists of BERT's pretrained core with a simple one-layer classifier on top that maps the output for the `[CLS]` token to the required number of classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1031 17:56:16.779504 140624693438272 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json from cache at /home/yves/.cache/torch/transformers/33b56ce0f312e47e4d77a57791a4fc6233ae4a560dd2bdd186107058294e58ab.3d3939ed0a2ebc86dd3c4703e5a591dcfb3341e9bbf87d411dea195948a37c74\n",
      "I1031 17:56:16.781740 140624693438272 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 105879\n",
      "}\n",
      "\n",
      "I1031 17:56:17.386830 140624693438272 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/cc4042a0d6f70eae595ea0e6d49521b17bd6251f973b3e37d303ce7945b90eed.54b4dad9cc3db9aa8448458b782d11ab07c80dedb951906fd2f684a00ecdb1ee\n",
      "I1031 17:56:20.645111 140624693438272 modeling_utils.py:405] Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\n",
      "I1031 17:56:20.645735 140624693438272 modeling_utils.py:408] Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "BertForSequenceClassification(\n",
       "  (bert): BertModel(\n",
       "    (embeddings): BertEmbeddings(\n",
       "      (word_embeddings): Embedding(105879, 768, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 768)\n",
       "      (token_type_embeddings): Embedding(2, 768)\n",
       "      (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1)\n",
       "    )\n",
       "    (encoder): BertEncoder(\n",
       "      (layer): ModuleList(\n",
       "        (0): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (1): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (2): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (3): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (4): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (5): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (6): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (7): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (8): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (9): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (10): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "        (11): BertLayer(\n",
       "          (attention): BertAttention(\n",
       "            (self): BertSelfAttention(\n",
       "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "            (output): BertSelfOutput(\n",
       "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "              (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "              (dropout): Dropout(p=0.1)\n",
       "            )\n",
       "          )\n",
       "          (intermediate): BertIntermediate(\n",
       "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
       "          )\n",
       "          (output): BertOutput(\n",
       "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (LayerNorm): LayerNorm(torch.Size([768]), eps=1e-12, elementwise_affine=True)\n",
       "            (dropout): Dropout(p=0.1)\n",
       "          )\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (pooler): BertPooler(\n",
       "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
       "      (activation): Tanh()\n",
       "    )\n",
       "  )\n",
       "  (dropout): Dropout(p=0.1)\n",
       "  (classifier): Linear(in_features=768, out_features=3, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers.modeling_bert import BertForSequenceClassification\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "model = BertForSequenceClassification.from_pretrained(BERT_MODEL, num_labels = len(label2idx))\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Preparing the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Next we need to prepare our data for BERT. We'll present every document as a BertInputItem, which contains all the information BERT needs: the input ids (the ids of the word pieces in the text), the input mask (which masks out padding tokens), the segment ids (which tell BERT how many segments are present in the input, and where these segments are) and the label id (the id of the label for the item).\n",
    "\n",
    "To speed up the training process, we set the maximum sequence length to 100. This means we'll cut off our texts after 100 wordpiece tokens.\n",
    "\n",
    "Finally, we'll also create a DataLoader for each of our data sets. These dataloaders will feed our data to the model during training, development and testing. They take care of presenting batches of the right size and shuffling the data when necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:22.987053 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (527 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.001937 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1144 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.035306 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (925 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.048370 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1132 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.080740 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (554 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.107719 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (812 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.154371 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (731 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.166877 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (559 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.191500 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (732 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.198900 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (808 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.242585 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (562 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.265081 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (818 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.278940 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1502 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.296676 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (571 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.314459 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (874 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.322872 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (566 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.354040 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (548 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.365204 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (567 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.371930 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (786 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.399598 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (966 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.410046 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (702 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.439943 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1195 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.450742 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (541 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.483396 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (551 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.511932 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (969 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.565696 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1130 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.584767 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (581 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.600724 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1219 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.633939 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2721 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.644096 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (727 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.659443 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (622 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.682989 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (953 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.705125 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (780 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.754305 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (596 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:23.765532 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1169 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.778315 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (617 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.809127 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (717 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.838254 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (623 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.889850 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (949 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.904124 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (538 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.924475 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1423 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.966064 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (557 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:23.996551 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (565 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.008564 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (951 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.019214 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (650 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.038560 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (959 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.087192 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (611 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.117128 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (978 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.144579 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (541 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.152284 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (725 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.161344 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (642 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.169282 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (941 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.194446 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (749 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.202700 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (710 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.217591 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1041 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.249990 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (558 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.313178 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1170 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.353433 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (624 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.359170 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (522 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.363988 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (521 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.388058 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (685 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.403534 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (605 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.467105 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (769 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.505591 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (519 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.528090 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (534 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.536189 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (588 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.590267 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (963 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.602980 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (584 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:24.635043 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (651 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.643235 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (683 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.661886 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1395 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.674306 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (550 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.691431 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (825 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.700796 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (673 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.715419 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (580 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.726869 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (635 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.733598 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (816 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.744076 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (741 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.750457 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (531 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.819085 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (546 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.824224 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (541 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.838743 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1203 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.863009 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1813 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.872420 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (677 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.881350 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (518 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.892876 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (565 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.920783 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2916 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.950252 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (876 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:24.957826 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (550 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.010146 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (666 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.032368 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2468 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.059163 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (726 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.094535 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2547 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.115612 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (567 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.134974 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1173 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.141195 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (623 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.148622 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (593 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.163454 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (552 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.178473 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (534 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.187258 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (762 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.223522 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1339 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.241462 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (730 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:25.324532 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (561 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.341262 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (720 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.376079 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1912 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.395441 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (892 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.453721 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1734 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.484867 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (679 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.536891 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (581 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.559548 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (611 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.567792 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (932 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.576152 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (514 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.590639 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (646 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.599809 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (824 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.604753 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (540 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.663163 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (703 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.674541 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (682 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.742476 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (983 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.749901 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (623 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.766328 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (816 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.827808 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (914 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.835095 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (590 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.878580 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (805 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.894935 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1030 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.922468 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (878 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.971182 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1485 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:25.986743 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (816 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.005184 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (668 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.019572 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (647 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.058099 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1178 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.084328 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2957 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.139877 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (885 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.148016 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (515 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.177417 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1617 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.207114 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1004 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.237292 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1039 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:26.254839 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (895 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.331607 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (641 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.359093 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (529 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.365629 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (593 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.375991 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (558 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.413379 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1691 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.425266 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (519 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.477528 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (818 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.511317 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (755 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.517134 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (575 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.549327 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1499 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.649325 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1093 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.655605 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (527 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.664692 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (973 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.682091 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1473 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.710813 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2215 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.727765 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (895 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.741061 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (853 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.759648 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (948 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.777469 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (769 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.791342 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (657 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.812294 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1341 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.836191 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (699 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.859824 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (722 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.879023 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (639 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.897283 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1294 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.910621 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (515 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.925137 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (717 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:26.977841 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (516 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.061858 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (566 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.106858 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (517 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.125979 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2482 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.209062 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (945 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.253312 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (891 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:27.288460 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (3810 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.307774 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1028 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.314847 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (657 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.325520 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1331 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.340651 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1006 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.346168 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (609 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.365716 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.398562 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (599 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.404917 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (624 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.470411 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (819 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.645401 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (550 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.685674 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (555 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.816443 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (858 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.901955 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (525 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:27.958243 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (518 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.196202 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (729 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.355239 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (516 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.364209 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (936 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.462100 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (993 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.500612 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (641 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.593674 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (752 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.607345 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (713 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.616411 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (878 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.645403 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (550 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.695599 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (625 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.713856 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (690 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.792044 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (566 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.853903 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (639 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:28.878833 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (936 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:29.001945 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (583 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:29.015173 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (673 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:29.022889 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (519 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:29.040555 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (936 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:29.091344 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1001 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:29.100605 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (823 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:29.445366 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (808 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:29.479968 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (602 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:29.560234 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (552 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:29.608459 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (870 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:30.259968 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1324 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:30.361489 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (720 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:30.660652 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (585 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:30.747972 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (977 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:30.975799 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (910 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.011152 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (520 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.071499 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (549 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.103560 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (541 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.154678 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (535 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.191703 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1444 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.206797 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (840 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.222527 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (675 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.278548 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (658 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.331640 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (559 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.339669 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (578 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.361763 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (666 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.437574 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (740 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.473361 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1008 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.521265 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (559 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.770988 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (625 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.816788 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1162 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.848764 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (636 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.893418 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2617 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:31.900999 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (654 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.039664 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (981 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.120140 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1019 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.148393 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1158 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.156111 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (595 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.178198 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (802 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:32.253149 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (628 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.339672 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (934 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.352094 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (516 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.401155 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (520 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.412586 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (551 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.441330 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (518 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.480341 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (831 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.557142 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (918 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.563667 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (557 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.706001 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (556 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.718721 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (760 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.728999 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (653 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.784002 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (740 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.910974 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1156 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.953785 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:32.987860 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (827 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.019957 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (640 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.033687 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1089 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.087524 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1007 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.147629 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (677 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.164791 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (623 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.197264 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1406 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.211608 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (613 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.342046 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (684 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.349043 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.361370 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (740 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.383387 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (578 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.392512 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (602 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.402824 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (560 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.442173 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1028 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.481569 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (624 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.501643 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (517 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.533345 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (523 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.542773 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1036 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:33.586807 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (532 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.596908 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (799 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.661094 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2647 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.779202 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (683 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.795449 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (700 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.835079 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (578 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.872224 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (899 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.893576 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1303 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.937419 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (846 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.962177 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (642 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:33.998711 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1427 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.052951 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (947 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.081424 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (693 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.095835 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1010 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.106696 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (545 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.149083 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (586 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.184228 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (548 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.239320 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1673 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.251733 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (981 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.278244 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1179 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.310668 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1051 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.382274 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (594 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.390280 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (781 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.409930 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (681 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.452709 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (902 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.503567 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1585 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.556236 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (646 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.584089 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (630 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.593418 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (709 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.608713 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1164 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.642897 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (720 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.663765 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (586 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.742139 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1599 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.767658 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1639 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:34.800009 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (528 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.819178 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1507 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.858129 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1756 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.885199 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (749 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.904210 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (704 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.916414 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (860 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:34.956565 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (575 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.004776 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (886 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.026178 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (524 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.091856 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (637 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.141181 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (673 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.161740 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (631 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.178461 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1493 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.202327 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (686 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.342939 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1483 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.353313 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (900 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.361658 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (663 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.383859 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1175 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.488249 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1096 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.501002 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (563 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.528375 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (641 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.646022 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (7984 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.673784 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (720 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.719832 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (568 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.734413 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (579 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.760051 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (548 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.782291 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (679 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.892167 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (540 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.955439 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (598 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.979612 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1005 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:35.997056 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (607 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.014487 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (980 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.055629 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (857 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.243073 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (796 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1031 17:56:36.301810 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (579 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.314114 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (806 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.378123 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (607 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.386794 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (584 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.523332 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (941 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.653620 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1095 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.713399 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (664 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.738361 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2580 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.805798 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1224 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.821997 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1981 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.889528 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (634 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:36.933118 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (768 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.059367 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (681 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.091317 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1036 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.116612 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (587 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.176826 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (730 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.189602 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (953 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.200180 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (530 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.317448 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (944 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.370078 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (582 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.383965 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (1059 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.552509 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2383 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.564127 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (584 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.580842 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (531 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.759118 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (2158 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.892441 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (585 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:37.901967 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (535 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:38.005033 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (632 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:38.039770 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (529 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:38.103783 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (742 > 512). Running this sequence through the model will result in indexing errors\n",
      "W1031 17:56:38.146802 140624693438272 tokenization_utils.py:677] Token indices sequence length is longer than the specified maximum sequence length for this model (937 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import numpy as np\n",
    " \n",
    "from torch.utils.data import TensorDataset, DataLoader, SequentialSampler\n",
    "\n",
    "logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',\n",
    "                    datefmt = '%m/%d/%Y %H:%M:%S',\n",
    "                    level = logging.INFO)\n",
    "logger = logging.getLogger(__name__)\n",
    "\n",
    "MAX_SEQ_LENGTH=100\n",
    "\n",
    "class BertInputItem(object):\n",
    "    \"\"\"An item with all the necessary attributes for finetuning BERT.\"\"\"\n",
    "\n",
    "    def __init__(self, text, input_ids, input_mask, segment_ids, label_id):\n",
    "        self.text = text\n",
    "        self.input_ids = input_ids\n",
    "        self.input_mask = input_mask\n",
    "        self.segment_ids = segment_ids\n",
    "        self.label_id = label_id\n",
    "        \n",
    "\n",
    "def convert_examples_to_inputs(example_texts, example_labels, label2idx, max_seq_length, tokenizer, verbose=0):\n",
    "    \"\"\"Loads a data file into a list of `InputBatch`s.\"\"\"\n",
    "    \n",
    "    input_items = []\n",
    "    examples = zip(example_texts, example_labels)\n",
    "    for (ex_index, (text, label)) in enumerate(examples):\n",
    "\n",
    "        # Create a list of token ids\n",
    "        input_ids = tokenizer.encode(f\"[CLS] {text} [SEP]\")\n",
    "        if len(input_ids) > max_seq_length:\n",
    "            input_ids = input_ids[:max_seq_length]\n",
    "\n",
    "        # All our tokens are in the first input segment (id 0).\n",
    "        segment_ids = [0] * len(input_ids)\n",
    "\n",
    "        # The mask has 1 for real tokens and 0 for padding tokens. Only real\n",
    "        # tokens are attended to.\n",
    "        input_mask = [1] * len(input_ids)\n",
    "\n",
    "        # Zero-pad up to the sequence length.\n",
    "        padding = [0] * (max_seq_length - len(input_ids))\n",
    "        input_ids += padding\n",
    "        input_mask += padding\n",
    "        segment_ids += padding\n",
    "\n",
    "        assert len(input_ids) == max_seq_length\n",
    "        assert len(input_mask) == max_seq_length\n",
    "        assert len(segment_ids) == max_seq_length\n",
    "\n",
    "        label_id = label2idx[label]\n",
    "\n",
    "        input_items.append(\n",
    "            BertInputItem(text=text,\n",
    "                          input_ids=input_ids,\n",
    "                          input_mask=input_mask,\n",
    "                          segment_ids=segment_ids,\n",
    "                          label_id=label_id))\n",
    "\n",
    "        \n",
    "    return input_items\n",
    "\n",
    "def get_data_loader(features, max_seq_length, batch_size=8, shuffle=True): \n",
    "\n",
    "    all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)\n",
    "    all_input_mask = torch.tensor([f.input_mask for f in features], dtype=torch.long)\n",
    "    all_segment_ids = torch.tensor([f.segment_ids for f in features], dtype=torch.long)\n",
    "    all_label_ids = torch.tensor([f.label_id for f in features], dtype=torch.long)\n",
    "    data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids)\n",
    "\n",
    "    dataloader = DataLoader(data, shuffle=shuffle, batch_size=batch_size)\n",
    "    return dataloader\n",
    "\n",
    "\n",
    "all_train_features, all_dev_features = [], []\n",
    "for lang in data:\n",
    "    \n",
    "    train_features = convert_examples_to_inputs(data[lang][\"train\"][\"texts\"], \n",
    "                                                data[lang][\"train\"][\"labels\"], \n",
    "                                                label2idx, MAX_SEQ_LENGTH, tokenizer, verbose=0)\n",
    "\n",
    "    dev_features = convert_examples_to_inputs(data[lang][\"dev\"][\"texts\"], \n",
    "                                              data[lang][\"dev\"][\"labels\"], \n",
    "                                              label2idx, MAX_SEQ_LENGTH, tokenizer)\n",
    "\n",
    "    test_features = convert_examples_to_inputs(data[lang][\"test\"][\"texts\"], \n",
    "                                               data[lang][\"test\"][\"labels\"], \n",
    "                                               label2idx, MAX_SEQ_LENGTH, tokenizer)\n",
    "\n",
    "    all_train_features.extend(train_features)\n",
    "    all_dev_features.extend(dev_features)\n",
    "    \n",
    "    data[lang][\"train\"][\"dataloader\"] = get_data_loader(train_features, MAX_SEQ_LENGTH, shuffle=True)\n",
    "    data[lang][\"dev\"][\"dataloader\"] = get_data_loader(dev_features, MAX_SEQ_LENGTH, shuffle=False)\n",
    "    data[lang][\"test\"][\"dataloader\"] = get_data_loader(test_features, MAX_SEQ_LENGTH, shuffle=False)\n",
    "\n",
    "dataloader_train_all = get_data_loader(all_train_features, MAX_SEQ_LENGTH, shuffle=True)\n",
    "dataloader_dev_all = get_data_loader(all_dev_features, MAX_SEQ_LENGTH, shuffle=False)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Evaluation method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Our evaluation method, too, is the same as before. It takes as input a model and dataloader and returns the loss, the correct labels for the data and the labels that were predicted for the model. This evaluation method will be used several times in our experiment: \n",
    "\n",
    "- During training we will use it to evaluate the loss on the development data. If this loss stops improving, we will stop training. \n",
    "- When we're done training we will use it to evaluate the model's performance on the held-out test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from tqdm import tqdm_notebook as tqdm\n",
    "\n",
    "def evaluate(model, dataloader, device=\"cpu\"):\n",
    "    model.eval()\n",
    "    \n",
    "    eval_loss = 0\n",
    "    nb_eval_steps = 0\n",
    "    predicted_labels, correct_labels = [], []\n",
    "\n",
    "    model.to(device)\n",
    "    for step, batch in enumerate(tqdm(dataloader, desc=\"Evaluation iteration\")):\n",
    "        batch = tuple(t.to(device) for t in batch)\n",
    "        input_ids, input_mask, segment_ids, label_ids = batch\n",
    "\n",
    "        with torch.no_grad():\n",
    "            tmp_eval_loss, logits = model(input_ids, attention_mask=input_mask,\n",
    "                              token_type_ids=segment_ids, labels=label_ids)\n",
    "\n",
    "        outputs = np.argmax(logits.to('cpu'), axis=1)\n",
    "        label_ids = label_ids.to('cpu').numpy()\n",
    "        \n",
    "        predicted_labels += list(outputs)\n",
    "        correct_labels += list(label_ids)\n",
    "        \n",
    "        eval_loss += tmp_eval_loss.mean().item()\n",
    "        nb_eval_steps += 1\n",
    "\n",
    "    eval_loss = eval_loss / nb_eval_steps\n",
    "    \n",
    "    correct_labels = np.array(correct_labels)\n",
    "    predicted_labels = np.array(predicted_labels)\n",
    "        \n",
    "    return eval_loss, correct_labels, predicted_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We also write a helper function that evaluates a given model on the test set for every language and returns the accuracies in a Pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d789fd0b4e824bc4b95ff3550fa62ef1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a766dbab605942549b4e3ef97c023aae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32584a501bbc42cdb3d58f0a849e93db",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b87f441c2313480dbc5bd84c3f901a16",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2efe56516eef49bb9f544a6b91c0e7ec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c46525f3a2ef4de2806303474bd652c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    accuracy\n",
      "en  0.316667\n",
      "nl  0.323333\n",
      "es  0.303333\n",
      "fr  0.280000\n",
      "de  0.283333\n",
      "it  0.333333\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def evaluate_all_languages(model, source_language=None, device=\"cpu\"):\n",
    "    accuracies = []\n",
    "    for lang in languages:\n",
    "        _, correct_labels, predicted_labels = evaluate(model, data[lang][\"test\"][\"dataloader\"], device=device)\n",
    "        accuracy = np.mean(predicted_labels == correct_labels)\n",
    "        accuracies.append(accuracy)\n",
    "        \n",
    "    df = pd.DataFrame({source_language: accuracies}, index=languages)\n",
    "    return df\n",
    "        \n",
    "print(evaluate_all_languages(model, \"accuracy\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "We're finally ready to train our model. At each epoch, we're going to train it on our training data and evaluate it on the development data. We keep a history of the loss, and stop training when the loss on the development set doesn't improve for a certain number of steps. Whenever the development loss of our model improves, we save it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import os\n",
    "from tqdm import trange\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "from sklearn.metrics import classification_report, precision_recall_fscore_support\n",
    "from transformers.optimization import AdamW, WarmupLinearSchedule\n",
    "\n",
    "def train(model, train_dataloader, dev_dataloader, output_model_file=\"/tmp/bert.bin\",\n",
    "          num_train_epochs=20, patience=2, gradient_accumulation_steps=1, max_grad_norm=5,\n",
    "          warmup_proportion=0.1, batch_size=8, learning_rate=5e-5): \n",
    "    \n",
    "    num_train_steps = int(len(data[\"en\"][\"train\"][\"texts\"]) / batch_size / gradient_accumulation_steps * num_train_epochs)\n",
    "    num_warmup_steps = int(warmup_proportion * num_train_steps)\n",
    "\n",
    "    param_optimizer = list(model.named_parameters())\n",
    "    no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']\n",
    "    optimizer_grouped_parameters = [\n",
    "        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},\n",
    "        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n",
    "        ]\n",
    "\n",
    "    optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate, correct_bias=False)\n",
    "    scheduler = WarmupLinearSchedule(optimizer, warmup_steps=num_warmup_steps, t_total=num_train_steps)\n",
    "    \n",
    "    loss_history = []\n",
    "    no_improvement = 0\n",
    "    for _ in trange(int(num_train_epochs), desc=\"Epoch\"):\n",
    "        model.train()\n",
    "        tr_loss = 0\n",
    "        nb_tr_examples, nb_tr_steps = 0, 0\n",
    "        for step, batch in enumerate(tqdm(train_dataloader, desc=\"Training iteration\")):\n",
    "            batch = tuple(t.to(device) for t in batch)\n",
    "            input_ids, input_mask, segment_ids, label_ids = batch\n",
    "\n",
    "            outputs = model(input_ids, attention_mask=input_mask, token_type_ids=segment_ids, labels=label_ids)\n",
    "            loss = outputs[0]\n",
    "\n",
    "            if gradient_accumulation_steps > 1:\n",
    "                loss = loss / gradient_accumulation_steps\n",
    "\n",
    "            loss.backward()\n",
    "\n",
    "            tr_loss += loss.item()\n",
    "\n",
    "            if (step + 1) % gradient_accumulation_steps == 0:\n",
    "                torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm) \n",
    "                optimizer.step()\n",
    "                optimizer.zero_grad() \n",
    "                scheduler.step()\n",
    "\n",
    "        dev_loss, _, _ = evaluate(model, dev_dataloader, device=\"cuda\")\n",
    "\n",
    "        print(\"Loss history:\", loss_history)\n",
    "        print(\"Dev loss:\", dev_loss)\n",
    "\n",
    "        if len(loss_history) == 0 or dev_loss < min(loss_history):\n",
    "            no_improvement = 0\n",
    "            model_to_save = model.module if hasattr(model, 'module') else model\n",
    "            torch.save(model_to_save.state_dict(), output_model_file)\n",
    "        else:\n",
    "            no_improvement += 1\n",
    "        \n",
    "        if no_improvement >= patience:\n",
    "            print(\"No improvement on development set. Finish training.\")\n",
    "            break\n",
    "\n",
    "        loss_history.append(dev_loss)\n",
    "        \n",
    "    return output_model_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Experiment 1: zero-shot learning\n",
    "\n",
    "In our first experiment, we want to test BERT's ability to do zero-shot cross-lingual transfer. In other words, we're going to train BERT on one language and evaluate its performance on another language. We'll repeat this process for all six languages in our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== Training en ===\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1031 17:57:25.967924 140624693438272 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json from cache at /home/yves/.cache/torch/transformers/33b56ce0f312e47e4d77a57791a4fc6233ae4a560dd2bdd186107058294e58ab.3d3939ed0a2ebc86dd3c4703e5a591dcfb3341e9bbf87d411dea195948a37c74\n",
      "I1031 17:57:25.970035 140624693438272 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 105879\n",
      "}\n",
      "\n",
      "I1031 17:57:26.490439 140624693438272 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/cc4042a0d6f70eae595ea0e6d49521b17bd6251f973b3e37d303ce7945b90eed.54b4dad9cc3db9aa8448458b782d11ab07c80dedb951906fd2f684a00ecdb1ee\n",
      "I1031 17:57:29.785684 140624693438272 modeling_utils.py:405] Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\n",
      "I1031 17:57:29.786309 140624693438272 modeling_utils.py:408] Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "Epoch:   0%|          | 0/20 [00:00<?, ?it/s]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d4f3aad729f54a439ea729ed69cde306",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0869743496e2467597d35e4600038567",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: []\n",
      "Dev loss: 0.8892643521813786\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:   5%|▌         | 1/20 [00:35<11:18, 35.68s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a524a4ec09fb4bc192f19a63e6fe197a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "38ea313f7c3a4a03b3de9598a8ecbd24",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  10%|█         | 2/20 [01:10<10:39, 35.51s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8892643521813786]\n",
      "Dev loss: 1.0420478897936203\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f65841c93c2b452096b87edd6c82ec66",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c107954140234b5cafec3003dfa838ee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8892643521813786, 1.0420478897936203]\n",
      "Dev loss: 0.7991069559665287\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  15%|█▌        | 3/20 [01:46<10:07, 35.72s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "487c9faa495b48f18e613917adaf0db4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c30792d3482416fad62f746b82e7047",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch:  20%|██        | 4/20 [02:22<09:30, 35.65s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8892643521813786, 1.0420478897936203, 0.7991069559665287]\n",
      "Dev loss: 0.9805790615432403\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81e9b3f1b0f54a7088a266e97c76f1cd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb897c6a0d5242628cd705caef776eea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8892643521813786, 1.0420478897936203, 0.7991069559665287, 0.9805790615432403]\n",
      "Dev loss: 0.9208172397578464\n",
      "No improvement on development set. Finish training.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4acc7f75aea4cb99ceba2f01f274dec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d4649d4231a46b19f4f9f1e401b756e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7952f47c455f4d799214f6b6f94aca04",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bca05468cda14b48a46e12fd275959ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2cbe532136b14786a741bd36f95f71d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b5b77d4d3ac84018a6de6583d4d8a2aa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "          en\n",
      "en  0.660000\n",
      "nl  0.460000\n",
      "es  0.633333\n",
      "fr  0.560000\n",
      "de  0.570000\n",
      "it  0.576667\n",
      "=== Training nl ===\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1031 18:01:16.805742 140624693438272 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json from cache at /home/yves/.cache/torch/transformers/33b56ce0f312e47e4d77a57791a4fc6233ae4a560dd2bdd186107058294e58ab.3d3939ed0a2ebc86dd3c4703e5a591dcfb3341e9bbf87d411dea195948a37c74\n",
      "I1031 18:01:16.807965 140624693438272 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 105879\n",
      "}\n",
      "\n",
      "I1031 18:01:17.343017 140624693438272 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/cc4042a0d6f70eae595ea0e6d49521b17bd6251f973b3e37d303ce7945b90eed.54b4dad9cc3db9aa8448458b782d11ab07c80dedb951906fd2f684a00ecdb1ee\n",
      "I1031 18:01:20.623367 140624693438272 modeling_utils.py:405] Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\n",
      "I1031 18:01:20.624025 140624693438272 modeling_utils.py:408] Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "\n",
      "Epoch:   0%|          | 0/20 [00:00<?, ?it/s]\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b995fc5731d4ef1a87035da10efc027",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "780d08da5a2745d391f33675615fc4a9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: []\n",
      "Dev loss: 1.067294890389723\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch:   5%|▌         | 1/20 [00:35<11:15, 35.56s/it]\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "28bbe8bfdc0a4be999d48d00c6f3547e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "353d5ec1572245538df9fb80b6a166ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [1.067294890389723]\n",
      "Dev loss: 0.9490295297959271\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch:  10%|█         | 2/20 [01:11<10:41, 35.65s/it]\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a21e144a1d8b4d84874758fc11de2859",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d7071b711174da6ac43866da39e5486",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [1.067294890389723, 0.9490295297959271]\n",
      "Dev loss: 0.9130261803374571\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch:  15%|█▌        | 3/20 [01:47<10:07, 35.76s/it]\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c310cf946544c8aabf6840e4674188a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b71bc629f6f4e8eb313d0fcc104e28e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch:  20%|██        | 4/20 [02:22<09:30, 35.63s/it]\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [1.067294890389723, 0.9490295297959271, 0.9130261803374571]\n",
      "Dev loss: 0.9443715819541145\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b75e6a53eec84de48a33ffd4321e658d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "397eadc8236b4f55ac4e64359fdf3188",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [1.067294890389723, 0.9490295297959271, 0.9130261803374571, 0.9443715819541145]\n",
      "Dev loss: 0.825725401587346\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch:  25%|██▌       | 5/20 [02:59<08:57, 35.82s/it]\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f66da9912a034ab1af7779a339373a48",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d8a5a69f3fe47a5bf6f0eaa4b43218f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch:  30%|███       | 6/20 [03:34<08:20, 35.74s/it]\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [1.067294890389723, 0.9490295297959271, 0.9130261803374571, 0.9443715819541145, 0.825725401587346]\n",
      "Dev loss: 0.9627383167252821\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "70c3362ec42e4f9eab6880fdd7165875",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "764c2cd6ffa843e8a8995f93c030b0b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [1.067294890389723, 0.9490295297959271, 0.9130261803374571, 0.9443715819541145, 0.825725401587346, 0.9627383167252821]\n",
      "Dev loss: 1.179090435452321\n",
      "No improvement on development set. Finish training.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f1c1dc7211b947fb8d5bf80e1ceb101a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c160ef9b8faf4aac8db491d472099314",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b484f017175347d39cb535aa7459bd0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fe1650f88fb54325b33b217d01cf0631",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c0fddefbcd1542eba7bddd13313e5076",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1e26b15bcfa346f29d7e6a06daad5f75",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "          nl\n",
      "en  0.460000\n",
      "nl  0.596667\n",
      "es  0.520000\n",
      "fr  0.470000\n",
      "de  0.503333\n",
      "it  0.503333\n",
      "=== Training es ===\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1031 18:06:19.408812 140624693438272 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json from cache at /home/yves/.cache/torch/transformers/33b56ce0f312e47e4d77a57791a4fc6233ae4a560dd2bdd186107058294e58ab.3d3939ed0a2ebc86dd3c4703e5a591dcfb3341e9bbf87d411dea195948a37c74\n",
      "I1031 18:06:19.411678 140624693438272 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 105879\n",
      "}\n",
      "\n",
      "I1031 18:06:19.927417 140624693438272 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/cc4042a0d6f70eae595ea0e6d49521b17bd6251f973b3e37d303ce7945b90eed.54b4dad9cc3db9aa8448458b782d11ab07c80dedb951906fd2f684a00ecdb1ee\n",
      "I1031 18:06:23.291018 140624693438272 modeling_utils.py:405] Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\n",
      "I1031 18:06:23.291646 140624693438272 modeling_utils.py:408] Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "\n",
      "\n",
      "Epoch:   0%|          | 0/20 [00:00<?, ?it/s]\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "25b7dd5f2db64412aca55e36df34e607",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eeef25202a2f4ef9b4583fb39a7f26d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: []\n",
      "Dev loss: 0.8810234096120385\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch:   5%|▌         | 1/20 [00:35<11:15, 35.53s/it]\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "200e2b0b75ef4249b971adac002689d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d0115d024284792960fca9957a6f683",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8810234096120385]\n",
      "Dev loss: 0.7474475787842975\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch:  10%|█         | 2/20 [01:11<10:41, 35.66s/it]\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d0794d98a0345a6872d0f05f1c14caa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97f21186e03046baa403abec66cbaaf1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Epoch:  15%|█▌        | 3/20 [01:46<10:04, 35.56s/it]\u001b[A\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8810234096120385, 0.7474475787842975]\n",
      "Dev loss: 1.0352991258396822\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f26a1d9f98045fe98ec281843eb5b90",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da7b7912ef0c4d019412ec2e3a268c3f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8810234096120385, 0.7474475787842975, 1.0352991258396822]\n",
      "Dev loss: 0.8174160219290677\n",
      "No improvement on development set. Finish training.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5397ce222c38447ca46875dba4b3f817",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a5f6824a66c040d1a94bdf3e58d7887f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b2045a10fb40405b9299d4eda91c82b0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "422a1acfe5a1437686d012a2bc46c600",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2e2e1684b696489e9c33bcdfd7cdf657",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c39c88d8da2545a0ba7110b3ea223825",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "          es\n",
      "en  0.533333\n",
      "nl  0.473333\n",
      "es  0.676667\n",
      "fr  0.570000\n",
      "de  0.523333\n",
      "it  0.546667\n",
      "=== Training fr ===\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1031 18:09:34.049234 140624693438272 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json from cache at /home/yves/.cache/torch/transformers/33b56ce0f312e47e4d77a57791a4fc6233ae4a560dd2bdd186107058294e58ab.3d3939ed0a2ebc86dd3c4703e5a591dcfb3341e9bbf87d411dea195948a37c74\n",
      "I1031 18:09:34.051923 140624693438272 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 105879\n",
      "}\n",
      "\n",
      "I1031 18:09:34.573785 140624693438272 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/cc4042a0d6f70eae595ea0e6d49521b17bd6251f973b3e37d303ce7945b90eed.54b4dad9cc3db9aa8448458b782d11ab07c80dedb951906fd2f684a00ecdb1ee\n",
      "I1031 18:09:37.807969 140624693438272 modeling_utils.py:405] Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\n",
      "I1031 18:09:37.808603 140624693438272 modeling_utils.py:408] Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "\n",
      "\n",
      "\n",
      "Epoch:   0%|          | 0/20 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b772e6ee0d5447c0af47a74d5a1237b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a28f9658fd434a9daca83cdadc069938",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: []\n",
      "Dev loss: 0.7781718399594811\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Epoch:   5%|▌         | 1/20 [00:35<11:13, 35.47s/it]\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0015fc7c353949f18fa0ebccdb3b3322",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9ce229674941429fb19f06b9ba2fbf65",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.7781718399594811]\n",
      "Dev loss: 0.7284165655865389\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Epoch:  10%|█         | 2/20 [01:11<10:41, 35.65s/it]\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3cbe5f88d83047f2966523b3c8fd2139",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "13268025d5ee4d1888724932b9ade311",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Epoch:  15%|█▌        | 3/20 [01:47<10:05, 35.63s/it]\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.7781718399594811, 0.7284165655865389]\n",
      "Dev loss: 0.7338471167227801\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b28fb599c454cb590769b0224a7924f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbd17b9f896e419bb2ba7d819e62483e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.7781718399594811, 0.7284165655865389, 0.7338471167227801]\n",
      "Dev loss: 0.886058055302676\n",
      "No improvement on development set. Finish training.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eccaa84404bb43d1a25842f039244bac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "59c6fd3b46c24c8fb2566b993e38a1c4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b6d26cfdce640638300923108f8619f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aefb3ad986d44047bf03b4552e5720f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b5da2658b5a14af490e057040679f47e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d24bc9a107fb48019c8961e32919864b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "          fr\n",
      "en  0.580000\n",
      "nl  0.486667\n",
      "es  0.590000\n",
      "fr  0.693333\n",
      "de  0.553333\n",
      "it  0.606667\n",
      "=== Training de ===\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1031 18:12:48.783101 140624693438272 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json from cache at /home/yves/.cache/torch/transformers/33b56ce0f312e47e4d77a57791a4fc6233ae4a560dd2bdd186107058294e58ab.3d3939ed0a2ebc86dd3c4703e5a591dcfb3341e9bbf87d411dea195948a37c74\n",
      "I1031 18:12:48.785574 140624693438272 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 105879\n",
      "}\n",
      "\n",
      "I1031 18:12:49.295864 140624693438272 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/cc4042a0d6f70eae595ea0e6d49521b17bd6251f973b3e37d303ce7945b90eed.54b4dad9cc3db9aa8448458b782d11ab07c80dedb951906fd2f684a00ecdb1ee\n",
      "I1031 18:12:52.508860 140624693438272 modeling_utils.py:405] Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\n",
      "I1031 18:12:52.509431 140624693438272 modeling_utils.py:408] Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:   0%|          | 0/20 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a0359c3cb844690ad91ef81b770f0f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "100d22d8ab65496290e452efdc8bab68",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: []\n",
      "Dev loss: 0.9197206234230715\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:   5%|▌         | 1/20 [00:35<11:16, 35.61s/it]\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4878f45a9e244252acd5e7b46d961537",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a36921027284b818797dfcc09aaf782",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  10%|█         | 2/20 [01:10<10:39, 35.52s/it]\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.9197206234230715]\n",
      "Dev loss: 0.9257739273940816\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f6bdfc9e2ac4a23abc96533d751e170",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3c0df2751aef4d55930639d8146eda66",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.9197206234230715, 0.9257739273940816]\n",
      "Dev loss: 0.7630253039738711\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  15%|█▌        | 3/20 [01:47<10:07, 35.72s/it]\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3de2c1cce46f4492b066bc092a328a68",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "008f34f3778a4808af41e1cc9d32e372",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.9197206234230715, 0.9257739273940816, 0.7630253039738711]\n",
      "Dev loss: 0.708569827982608\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  20%|██        | 4/20 [02:23<09:33, 35.87s/it]\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3df308122ece4b58a3d7c43f87849f0a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73e79fd4ed014f8e98ccf825831e72f8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  25%|██▌       | 5/20 [02:58<08:56, 35.78s/it]\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.9197206234230715, 0.9257739273940816, 0.7630253039738711, 0.708569827982608]\n",
      "Dev loss: 0.9534998378332924\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "22e8d37fa61f4b16b146159593273ea8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d99ba524d94f4849857cde701d64eaf5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.9197206234230715, 0.9257739273940816, 0.7630253039738711, 0.708569827982608, 0.9534998378332924]\n",
      "Dev loss: 0.8452849550282254\n",
      "No improvement on development set. Finish training.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7efa6ca0e41146fa9ed1a9135c87f313",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f1b6d6225684268b9ef212bf9b5ecdf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7b4f5aab42647a5a372fe003bb32bcc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "087b043956324eb3bccc26113fa38e36",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "193fa0ee953d42778f00c577226ba4c3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0e38e577b1b34acd98b31e18ad523f79",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "          de\n",
      "en  0.583333\n",
      "nl  0.480000\n",
      "es  0.536667\n",
      "fr  0.533333\n",
      "de  0.693333\n",
      "it  0.470000\n",
      "=== Training it ===\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1031 18:17:15.677266 140624693438272 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json from cache at /home/yves/.cache/torch/transformers/33b56ce0f312e47e4d77a57791a4fc6233ae4a560dd2bdd186107058294e58ab.3d3939ed0a2ebc86dd3c4703e5a591dcfb3341e9bbf87d411dea195948a37c74\n",
      "I1031 18:17:15.679254 140624693438272 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 105879\n",
      "}\n",
      "\n",
      "I1031 18:17:16.190802 140624693438272 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/cc4042a0d6f70eae595ea0e6d49521b17bd6251f973b3e37d303ce7945b90eed.54b4dad9cc3db9aa8448458b782d11ab07c80dedb951906fd2f684a00ecdb1ee\n",
      "I1031 18:17:19.316194 140624693438272 modeling_utils.py:405] Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\n",
      "I1031 18:17:19.316824 140624693438272 modeling_utils.py:408] Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:   0%|          | 0/20 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac1043855ea648cea23152e8bc2eb966",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "69def8d06f3f45d0b2246c5c612348ce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: []\n",
      "Dev loss: 0.8076019760440377\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:   5%|▌         | 1/20 [00:35<11:17, 35.65s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0888fe93f3f14cf98acc57f7fc72ce5f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "325f5813a71a481fb8cd79dd8905be40",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  10%|█         | 2/20 [01:10<10:37, 35.41s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8076019760440377]\n",
      "Dev loss: 0.9971894152900752\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc6621ecb7d648759cb76f286c27d412",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c50298b371034ebc99e14e5067281bdb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8076019760440377, 0.9971894152900752]\n",
      "Dev loss: 0.7530732295092415\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  15%|█▌        | 3/20 [01:46<10:02, 35.46s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1ddc8041c89c4123a680bee788dc32f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "84d9a31c57294ef39880129ca2ae9eba",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  20%|██        | 4/20 [02:21<09:24, 35.30s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8076019760440377, 0.9971894152900752, 0.7530732295092415]\n",
      "Dev loss: 0.8962154769722153\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4c1af69f2fa54dfc9d1cd1bc21a53412",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=304, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6ff1ad5ab12841a18fb8e671be2a4748",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=34, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.8076019760440377, 0.9971894152900752, 0.7530732295092415, 0.8962154769722153]\n",
      "Dev loss: 1.1358241881517803\n",
      "No improvement on development set. Finish training.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb8e088fcdca4dc6aededd46e0fff884",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "142f568010954e5da4a0d30de2db9bcc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "87bf98d73d914431b774404067a9ff19",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eec00be40c43404f9cf13aa83e8b7339",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5b2d85eea703413eb53fad41c21f5611",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a130d87c5cad42da97c8e16d05d65ab0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "          it\n",
      "en  0.546667\n",
      "nl  0.433333\n",
      "es  0.523333\n",
      "fr  0.550000\n",
      "de  0.496667\n",
      "it  0.660000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>en</th>\n",
       "      <th>nl</th>\n",
       "      <th>es</th>\n",
       "      <th>fr</th>\n",
       "      <th>de</th>\n",
       "      <th>it</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>en</th>\n",
       "      <td>0.660000</td>\n",
       "      <td>0.460000</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.580000</td>\n",
       "      <td>0.583333</td>\n",
       "      <td>0.546667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nl</th>\n",
       "      <td>0.460000</td>\n",
       "      <td>0.596667</td>\n",
       "      <td>0.473333</td>\n",
       "      <td>0.486667</td>\n",
       "      <td>0.480000</td>\n",
       "      <td>0.433333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>es</th>\n",
       "      <td>0.633333</td>\n",
       "      <td>0.520000</td>\n",
       "      <td>0.676667</td>\n",
       "      <td>0.590000</td>\n",
       "      <td>0.536667</td>\n",
       "      <td>0.523333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fr</th>\n",
       "      <td>0.560000</td>\n",
       "      <td>0.470000</td>\n",
       "      <td>0.570000</td>\n",
       "      <td>0.693333</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>de</th>\n",
       "      <td>0.570000</td>\n",
       "      <td>0.503333</td>\n",
       "      <td>0.523333</td>\n",
       "      <td>0.553333</td>\n",
       "      <td>0.693333</td>\n",
       "      <td>0.496667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>it</th>\n",
       "      <td>0.576667</td>\n",
       "      <td>0.503333</td>\n",
       "      <td>0.546667</td>\n",
       "      <td>0.606667</td>\n",
       "      <td>0.470000</td>\n",
       "      <td>0.660000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          en        nl        es        fr        de        it\n",
       "en  0.660000  0.460000  0.533333  0.580000  0.583333  0.546667\n",
       "nl  0.460000  0.596667  0.473333  0.486667  0.480000  0.433333\n",
       "es  0.633333  0.520000  0.676667  0.590000  0.536667  0.523333\n",
       "fr  0.560000  0.470000  0.570000  0.693333  0.533333  0.550000\n",
       "de  0.570000  0.503333  0.523333  0.553333  0.693333  0.496667\n",
       "it  0.576667  0.503333  0.546667  0.606667  0.470000  0.660000"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = []\n",
    "for language in languages:\n",
    "    print(f\"=== Training {language} ===\")\n",
    "    torch.cuda.empty_cache()\n",
    "    model = BertForSequenceClassification.from_pretrained(BERT_MODEL, num_labels = len(label2idx))\n",
    "    model.to(device)\n",
    "    model_file_name = train(model, data[language][\"train\"][\"dataloader\"], data[language][\"dev\"][\"dataloader\"], \n",
    "                           gradient_accumulation_steps=4)\n",
    "    language_df = evaluate_all_languages(model, source_language=language, device=\"cpu\")\n",
    "    print(language_df)\n",
    "    results.append(language_df)\n",
    "\n",
    "\n",
    "zero_shot_df = pd.concat(results, axis=1)\n",
    "zero_shot_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The results of our first experiment confirm that BERT-multilingual can do zero-shot transfer indeed. Because our three data labels are uniformly distributed, a BERT model that has not been finetuned should score around 33% on the test set. However, the figure below shows that our finetuned BERT scores significantly better than that for all languages, even when it has not been finetuned for that language. \n",
    "\n",
    "The horizontal axis of the figure shows the language on which BERT was finetuned; the vertical axis shows the language on which it was tested. Obviously, the highest scores are on the diagonal: for every language, we get the best performance with a model finetuned on that language. The accuracies on the diagonal lie somewhere between 60% and 70%. However, even if the model was finetuned on a different language, BERT does significantly better than random guessing. \n",
    "\n",
    "These zero-shot capabilities are likely the result of several factors (see e.g. Pires et al. 2019, [How multilingual is multilingual BERT?](https://arxiv.org/abs/1906.01502):\n",
    "\n",
    "- Many languages (and certainly those in our data) have similar words with a similar meaning. Words such as English _fantastic_, Dutch _fantastisch_ and French _fantastique_ share word pieces, and therefore help zero-shot learning.\n",
    "\n",
    "- However, Pires et al. show that zero-shot learning also occurs between languages with a different script, which do not share any word pieces. This means that during its pre-training phase, BERT builds multilingual representations.\n",
    "\n",
    "- Finally, Pires et al. also show that zero-shot transfer does not work as well between languages with different word orders (for example, SVO vs VSO languages, or adjective-noun vs noun-adjective languages)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAD8CAYAAADUv3dIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAE4RJREFUeJzt3X/wZXV93/Hn6/tdQKILSlCa8kMWi0aqiQGEJOhEq1gwDTiVGhDToI7bpJLSEDKFkVpLa5u0EzNDstF8nVASxpaYtiQ7uAm2Bgd/BN0FBGWRzA6MssSpjQJiiC4/3v3je9de1v3ee+5+7z333MPz4Zzh3nPP/dz3ft15fT/7vud8TqoKSVI7luZdgCQ9kxi6ktQiQ1eSWmToSlKLDF1JapGhK0ktMnQlaQ1Jrkny9SRfWuP1JLk6ya4kdyU5edyYhq4kre1a4KwRr58NnDjYNgMfHDegoStJa6iqW4BvjjjkXOAPatWtwHOT/NCoMTdMs8D9eeG//18Ldcnb6WccMu8SJvbVby7e785Hv/XUvEuYyCJeuPns5yze34vtb3lV1jvGocdd0Pj/re88cP0/Y3WGutdKVa1M8HFHAw8MPd892Pe1td4w89CVpK4aBOwkIbtuhq6kXklaneE/CBw79PyYwb41Ld6/PyRphKVsaLxNwVbgnw7OYvhx4JGqWrO1AM50JfXMNGe6Sf4b8BrgyCS7gX8DHARQVR8CtgFvBHYBjwFvHzemoSupV5J1fxf3PVV1wZjXC3j3JGMaupJ6pttdU0NXUq+0/EXaxAxdSb1i6EpSi6Z0VsLMdLs6SZqQM11JapGhK0ktCtM7ZWwWDF1JveJMV5JatLTU7VjrdnWSNDFnupLUGtsLktSiXoRukucD7wKOH35PVb1jjeM3M1iN/YhzLuE5r/zpdRcqSU2kJ+2FPwE+Bfxv4MlxBw+vxr5ot+uRtNh6MdMFfqCq/tVMK5GkKVhaWp53CSM1/ZVwY5I3zrQSSZqCsNR4m4emM91LgCuS7AEeB8Lq+r2HzawySToAfWkvHA5cCGyqqquSHAeMvLe7JM1D10O3aXVbgB8H9t664lHgt2dSkSStQ1/aC6dX1clJ7gCoqoeSHDzDuiTpgKQnlwE/nmQZKPjeebtPzawqSTpA07wx5Sw0Dd2rgRuAFyR5P3AecOXMqpKkA9SLiyOq6iNJbgNex+qZC2+qqntmWpkkHYCuf5HWuPlRVV8GvjzDWiRp/XrSXpCkxdDtia6hK6lnlrqduoaupH7pduYaupL6pezpSlKLup25hq6knlnqdurOPHRPP+OQWX/EVH1h17wrmNxLNy3eOvHP29jxxts+NmTxfsZHHfrEvEuYD9sLktSiZUNXktrjTFeSWtTtzDV0JfVMx79IW6xvMyRpnEywjRsqOSvJvUl2Jbl8P68fl+TmJHckuavJvSSd6UrqlVqezlxysIb4FuBMYDewPcnWqto5dNiVwEer6oNJTgK2AcePGteZrqR+md5M9zRgV1XdV1V7gOuBc/c5poC9N+g9HPircYMaupL6JWm8JdmcZMfQtnlopKOBB4ae7x7sG/Y+4G1JdrM6y/2lceXZXpDULxN8kVZVK8DKOj7tAuDaqvqNJD8BXJfkZVW15u3MnOlK6pfptRceBI4den7MYN+wdwIfBaiqvwCeBRw5alBDV1K/TNBeGGM7cGKSTYO7n58PbN3nmK+yehszkryU1dD9v6MGtb0gqV+mdBlwVT2R5GLgJmAZuKaq7k5yFbCjqrYCvwJ8OMkvs/ql2kVVNXKhDkNXUr9M8TLgqtrG6hdkw/veO/R4J3DGJGMaupL6pdsXpBm6kvqlOn4ZsKErqV9cZUySWtTtzB0dukn+8ajXq+p/TrccSVqnKa29MCvjZro/s8/zvadCZPB4v6E7uJRuM8Apl/4KJ/yjc9ZToyQ1t8gz3ap6O0CSZwFvZnX1nL3vWfNctOFL695y8y2Ld3MpSYurJ1+k/THwMHA78J3BPsNUUvf0JHSPqaqzZlqJJE1BdTtzG4fuZ5O8vKq+ONNqJGm9FvyLtL1eBVyU5H7guwy+SKuqH5lZZZJ0IHrSXjh7plVI0rR0e6LbLHSr6iuzLkSSpsIr0iSpRT1pL0jSQihnupLUog2GriS1x5muJLXInq4ktajbmWvoSuoX7xwhSW0ydCWpRVO6BfuszDx0b73jyVl/xFQd96KD5l3CxO785KPzLmFihxz/7HmXMJHnPrfj15bux0OPPUPnVJ69IEktsr0gSS0ydCWpPV4GLElteqZ/kSZJrbK9IEktMnQlqUXdzlxDV1K/eBmwJLXJsxckqUWevSBJ7Vnq+BXbHS9PkiaTNN/Gj5WzktybZFeSy9c45i1Jdia5O8l/HTemM11JvTKtlm6SZWALcCawG9ieZGtV7Rw65kTgCuCMqnooyQvGjetMV1KvJGm8jXEasKuq7quqPcD1wLn7HPMuYEtVPQRQVV8fN6ihK6lXlpaab0k2J9kxtG0eGupo4IGh57sH+4a9GHhxks8kuTXJWePqs70gqVcywVSyqlaAlXV83AbgROA1wDHALUleXlUPr/UGZ7qSemWKX6Q9CBw79PyYwb5hu4GtVfV4Vd0P/CWrIbymRqGb5JIkh2XV7yW5PckbmrxXktq0lObbGNuBE5NsSnIwcD6wdZ9j/pjVWS5JjmS13XDfyPoa/jneUVXfAt4APA/4OeDX1jp4uE/y7b+4seFHSNL6TWumW1VPABcDNwH3AB+tqruTXJXknMFhNwHfSLITuBn41ar6xqhxm/Z095b308B1gw9es+ThPslxH7i5Gn6GJK3bNK8CrqptwLZ99r136HEBlw62RpqG7m1JbgJOAC5PshF4qumHSFJbljp+GXDT9sI7gc8AN1bVY6y2GP7lzKqSpAM0zSvSZqFp6G4BjgL2noP2KPCBmVQkSevQ9dBt2l44vapOTnIHwOByt4NnWJckHZCOr+zYOHQfH1yHXABJno89XUkd1PE1zBu3F64GbgBekOT9wKeB/zCzqiTpAPWivVBVH0lyG/A6Vk8fe1NV3TPTyiTpAHT97IXGay9U1ZeBL8+wFklat770dCVpIRi6ktQiQ1eSWtT1sxcMXUm9srQ87wpGM3Ql9YrtBUlqUYN7n82VoSupVzqeuYaupH55xofuEUctVq4//9mLt+b6t17ynHmXMLFHv/jIvEuYSE5/7rxLmNh3/nbx/i5PwzM+dCWpTRs6frtdQ1dSryyl2zN8Q1dSr3hxhCS1qOPdBUNXUr/YXpCkFtlekKQWbTB0Jak9sb0gSe2xvSBJLfLsBUlqkWcvSFKL/CJNklpkT1eSWmR7QZJa5ExXklrU9bMXRtaXZDnJR9oqRpLWaynVeJtLfaNerKongRcmOXiSQZNsTrIjyY5v/PnWdRUoSZPYsNR8m4cmH3sf8Jkk/zrJpXu3UW+oqpWqOrWqTv3Bf3DOdCqVpAaWJtjGSXJWknuT7Epy+Yjj3pykkpzapL61Brlu8PAc4MbBsRuHNknqnGm1F5IsA1uAs4GTgAuSnLSf4zYClwCfa1LfqC/STknyd4GvAr/VZDBJmrcpnr1wGrCrqu4DSHI9cC6wc5/j/h3w68CvNhl0VOh+CPgEsAnYMbQ/QAEnNCpbklo0Sas2yWZg89CulapaGTw+Gnhg6LXdwOn7vP9k4Niq+liS9YVuVV0NXJ3kg1X1i00Gk6R5m2SmOwjYlbEH7keSJeADwEWTvG/seboGrqRFsrw0tVPBHgSOHXp+zGDfXhuBlwGfTALwd4CtSc6pquHuwNN4cYSkXpnimWDbgROTbGI1bM8H3rr3xap6BDhy7/MknwQuGxW4YOhK6plpXfRQVU8kuRi4CVgGrqmqu5NcBeyoqgO6CMHQldQr01x7oaq2Adv22ffeNY59TZMxDV1JveKCN5LUooNc2lGS2uNMV5JaZOhKUouWDV1Jao8zXUlqkfdIk6QWHfRMn+k+9dSsP2G6Ht7T9Tssfb/DDuv2b/b9OejHDp93CRP5P597eN4lTOyE1z5v3iXMhe0FSWqR7QVJapFnL0hSi2wvSFKL5nWX36YMXUm9smxPV5La0/GJrqErqV/s6UpSiwxdSWqRPV1JapFnL0hSi2wvSFKLvCJNklrk2guS1KKOt3QNXUn9Yk9Xklp00JLtBUlqTddnuo3aH0lenOQTSb40eP4jSa4ccfzmJDuS7PjmzVunVaskjbWU5ttc6mt43IeBK4DHAarqLuD8tQ6uqpWqOrWqTj3iteesv0pJamhpgm0emrYXfqCqPp887VfDEzOoR5LWJR1vLzQN3b9O8iKgAJKcB3xtZlVJ0gHqek+3aei+G1gBfjjJg8D9wIUzq0qSDtBCn6eb5NKhp9uAm1n9M/0N8GbgA7MrTZImlwW/Im3j4L8vAV4J/AkQ4OeAz8+wLkk6IB3vLowO3ar6twBJbgFOrqpHB8/fB3xs5tVJ0oS6/kVa0/bHUcCeoed7BvskqVMywTZ2rOSsJPcm2ZXk8v28fmmSnUnuGlzL8MJxYzb9Iu0PgM8nuWHw/E3AtQ3fK0mtmdbSjkmWgS3AmcBuYHuSrVW1c+iwO4BTq+qxJL8I/CfgZ0eN22imW1XvB94OPDTY3l5V/3HyP4YkzVbSfBvjNGBXVd1XVXuA64Fzhw+oqpur6rHB01uBY8YN2njthaq6Hbi96fGSNA+TTHSTbAY2D+1aqaqVweOjgQeGXtsNnD5iuHcCfzruM13wRlKvTBK6g4BdGXvguM9M3gacCvzUuGMNXUm9MsUr0h4Ejh16fsxg39MkeT3wHuCnquq7Y+ubWnmS1AFTPHthO3Bikk1JDmZ1ka+nLZuY5MeA3wXOqaqvN6nPma6kXpnWPdKq6okkFwM3AcvANVV1d5KrgB1VtRX4z8BzgD8aLAj21aoaubSioSupV6Z5cURVbWN1CYThfe8devz6Scc0dCX1Std7pjMP3aeenPUn6LtjW/fd8/DD3V6UZF+veMPh8y5hYrff+tj4g7rm7PUP0fXLgJ3pSuqVjmeuoSupX/qyiLkkLQRDV5Ja1PHMNXQl9cui3zlCkhaKM11JapGnjElSi5bnXcAYhq6kXnGmK0mt6nbqGrqSeiWGriS1J+n2kjeGrqSecaYrSa1Jxxd3NHQl9YrtBUlqVbfbC41+JSS5pMk+SZq3TPC/eWg6D//5/ey7aIp1SNJUdD10R7YXklwAvBXYlGT41sMbgW+OeN9mYDPAD/38ZRzxmpE3x5SkqUm6fSHwuJ7uZ4GvAUcCvzG0/1HgrrXeVFUrwArAy679VLfXWZPUM93u6Y4M3ar6CvAV4CfaKUeS1mehr0hL8umqelWSR4HhGWuAqqrDZlqdJE1sgU8Zq6pXDf67sZ1yJGl9FnqmK0mLJh1f29HQldQr6fgy5oaupJ5xpitJrbG9IEmtMnQlqTUu7ShJrXKmK0mtWXI9XUlqk6ErSa3p+hVp3f6VIEkTywTbmJGSs5Lcm2RXksv38/ohSf5w8Prnkhw/bkxDV1KvJGm8jRlnGdgCnA2cBFyQ5KR9Dnsn8FBV/T3gN4FfH1efoSupV8Jy422M04BdVXVfVe0BrgfO3eeYc4HfHzz+78DrMibNZ97T/dJFr55ZgyXJ5sGC6Qth0eqFxat50eqFGdb8+qmP+D3d/jm/uHHmDN/lZmBl6M91NPDA0Gu7gdP3GeJ7x1TVE0keAX4Q+Ou1PnPRZ7qbxx/SKYtWLyxezYtWL1jz3FTVSlWdOrTN/BfJooeuJM3Kg8CxQ8+PGezb7zFJNgCHA98YNaihK0n7tx04McmmJAcD5wNb9zlmK///bunnAX9eVSPvC7no5+l2tKe0pkWrFxav5kWrF6y5kwY92ouBm4Bl4JqqujvJVcCOqtoK/B5wXZJdrN4h/fxx42ZMKEuSpsj2giS1yNCVpBYZunOQ5Nok5827jj5J8i+S3JPkI/Oupakk70ty2bzraCrJZwf/PT7JW+ddz6IydNUX/xw4s6ou3LtjcAqPpqSqfnLw8HjA0D1ACxO6Sd6W5PNJvpDkd5MsJ/l2kvcnuTPJrUmOmnedwwYzgnuSfDjJ3Uk+nuTQedc1yho/52uTfCnJF5P88rxr3FeSDwEnAH+a5JEk1yX5DHDdnEv7Pknek+Qvk3waeMlg34uS/FmS25J8KskPz7nM/Ury7cHDXwNePfg70rm/D123EKGb5KXAzwJnVNUrgCeBC4FnA7dW1Y8CtwDvml+VazoR2FJVfx94GHjznOtZ0xo/5yuBo6vqZVX1cuC/zLPG/amqXwD+Cngtq4uOnAS8vqoumGth+0hyCqunFL0CeCPwysFLK8AvVdUpwGXA78ynwsYuBz5VVa+oqt+cdzGLZlH++fU64BRg+2AtiUOBrwN7gBsHx9wGnDmX6ka7v6q+MHh8G6v/NOuq/f2c/ww4IclvAR8DPj6/8hrbWlV/O+8i9uPVwA1V9RhAkq3As4CfBP5oaJ2UQ+ZTntqwKKEb4Per6oqn7UwuG7r640m6+ef57tDjJ1kNsq5a6+f8HuAfAr8AvAV4xxxqm8TfzLuACSwBDw/+ZaFngIVoLwCfAM5L8gKAJEckeeGca+qjtX7OS1X1P1htNZw8zwIX3C3Am5IcmmQj8DPAY8D9Sf4JQFb96DyLbOBRYOO8i1hUXZwZfp+q2pnkSuDjSZaAx4F3z7ms3lnj53wpcMPgOcAVaw6gkarq9iR/CNzJants++ClC4EPDn72B7G6buud86mykbuAJ5PcCVxrX3cyXgYsSS1alPaCJPWCoStJLTJ0JalFhq4ktcjQlaQWGbqS1CJDV5Ja9P8AS56Q1uGn9nMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import seaborn as sns\n",
    "\n",
    "plot = sns.heatmap(zero_shot_df, linewidths=.0, cmap=\"YlGnBu\", vmin=0, vmax=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Experiment 2: multilingual learning\n",
    "\n",
    "If languages can reinforce each other, this may mean that we can further improve the performance of our model by finetuning it on all languages in our data set together. Let's test this out by using a new training dataloader that shuffles the training data for all six languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1031 18:21:03.952027 140624693438272 configuration_utils.py:151] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-config.json from cache at /home/yves/.cache/torch/transformers/33b56ce0f312e47e4d77a57791a4fc6233ae4a560dd2bdd186107058294e58ab.3d3939ed0a2ebc86dd3c4703e5a591dcfb3341e9bbf87d411dea195948a37c74\n",
      "I1031 18:21:03.953774 140624693438272 configuration_utils.py:168] Model config {\n",
      "  \"attention_probs_dropout_prob\": 0.1,\n",
      "  \"directionality\": \"bidi\",\n",
      "  \"finetuning_task\": null,\n",
      "  \"hidden_act\": \"gelu\",\n",
      "  \"hidden_dropout_prob\": 0.1,\n",
      "  \"hidden_size\": 768,\n",
      "  \"initializer_range\": 0.02,\n",
      "  \"intermediate_size\": 3072,\n",
      "  \"layer_norm_eps\": 1e-12,\n",
      "  \"max_position_embeddings\": 512,\n",
      "  \"num_attention_heads\": 12,\n",
      "  \"num_hidden_layers\": 12,\n",
      "  \"num_labels\": 3,\n",
      "  \"output_attentions\": false,\n",
      "  \"output_hidden_states\": false,\n",
      "  \"output_past\": true,\n",
      "  \"pooler_fc_size\": 768,\n",
      "  \"pooler_num_attention_heads\": 12,\n",
      "  \"pooler_num_fc_layers\": 3,\n",
      "  \"pooler_size_per_head\": 128,\n",
      "  \"pooler_type\": \"first_token_transform\",\n",
      "  \"pruned_heads\": {},\n",
      "  \"torchscript\": false,\n",
      "  \"type_vocab_size\": 2,\n",
      "  \"use_bfloat16\": false,\n",
      "  \"vocab_size\": 105879\n",
      "}\n",
      "\n",
      "I1031 18:21:04.488624 140624693438272 modeling_utils.py:337] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-pytorch_model.bin from cache at /home/yves/.cache/torch/transformers/cc4042a0d6f70eae595ea0e6d49521b17bd6251f973b3e37d303ce7945b90eed.54b4dad9cc3db9aa8448458b782d11ab07c80dedb951906fd2f684a00ecdb1ee\n",
      "I1031 18:21:07.738768 140624693438272 modeling_utils.py:405] Weights of BertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']\n",
      "I1031 18:21:07.739465 140624693438272 modeling_utils.py:408] Weights from pretrained model not used in BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:   0%|          | 0/20 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "02796a502b30454e90aaf10665a9e803",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=1823, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "15fd7291949d4f0283dbc22419a0b850",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=203, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: []\n",
      "Dev loss: 0.9228241321782173\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:   5%|▌         | 1/20 [03:32<1:07:09, 212.10s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cf68680c23dd48eaacf47fae7b261d51",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=1823, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f87c8dc514b24b6f9083b496cac2afed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=203, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.9228241321782173]\n",
      "Dev loss: 0.837513931682838\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  10%|█         | 2/20 [07:05<1:03:43, 212.42s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6bcffa4774ff414ca56702b35176bac3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=1823, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "42d8edcf5445462684f855e6294d608a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=203, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.9228241321782173, 0.837513931682838]\n",
      "Dev loss: 0.7727996630621661\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  15%|█▌        | 3/20 [10:38<1:00:16, 212.74s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f41821fd71984c6b84a5f8cdc5aa3e28",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=1823, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e97557365a084d6ebfe36ad2f4a57531",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=203, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Epoch:  20%|██        | 4/20 [14:11<56:44, 212.77s/it]  \u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.9228241321782173, 0.837513931682838, 0.7727996630621661]\n",
      "Dev loss: 0.8220019129476523\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e5ef1e6688a841559a5efb9b248a86c7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Training iteration', max=1823, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "285f71634e704a50ab9ad097a8071865",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=203, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Loss history: [0.9228241321782173, 0.837513931682838, 0.7727996630621661, 0.8220019129476523]\n",
      "Dev loss: 0.8220019129476523\n",
      "No improvement on development set. Finish training.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4b18c64e4ebb4a9d9aee7d673976364e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fed005f21289491aa828a8af488033c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "15570cded77c4b6dac066da8c1a4d3ed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "98e6adc3e5d84e48806505c6cc2b2452",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "013d4a76084b4506b8950ed84f9c48cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7475a72a55b47c29f4974ca2e15b718",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Evaluation iteration', max=38, style=ProgressStyle(descriptio…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "model = BertForSequenceClassification.from_pretrained(BERT_MODEL, num_labels = len(label2idx))\n",
    "model.to(device)\n",
    "model_file_name = train(model, dataloader_train_all, dataloader_dev_all, gradient_accumulation_steps=4)\n",
    "combined_df = evaluate_all_languages(model, source_language=\"all\", device=\"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The results show that our truly multilingual BERT indeed outperforms the single-language ones on every single one of our test sets. Its accuracy exceeds 70% for most of the test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>single</th>\n",
       "      <th>all</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>en</th>\n",
       "      <td>0.660000</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nl</th>\n",
       "      <td>0.596667</td>\n",
       "      <td>0.646667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>es</th>\n",
       "      <td>0.676667</td>\n",
       "      <td>0.703333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fr</th>\n",
       "      <td>0.693333</td>\n",
       "      <td>0.723333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>de</th>\n",
       "      <td>0.693333</td>\n",
       "      <td>0.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>it</th>\n",
       "      <td>0.660000</td>\n",
       "      <td>0.716667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      single       all\n",
       "en  0.660000  0.666667\n",
       "nl  0.596667  0.646667\n",
       "es  0.676667  0.703333\n",
       "fr  0.693333  0.723333\n",
       "de  0.693333  0.700000\n",
       "it  0.660000  0.716667"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined_df.insert(0, \"single\", np.diag(zero_shot_df))\n",
    "combined_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(-0.4,0.29,'baseline')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa8AAAD+CAYAAACNzornAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAGuVJREFUeJzt3Xu8VXWd//HXh5sHEsjglCYQmGiRMUgk3qZh0kY0QktyvBZjDk0OpdPlMVhmZkYXtZwMHXF0HG18INkNlcKfllOmKUcjUxFFRDmmBSfywv3A5/fH2ei5wdlwNmezOq/n48GDvdb3u9b67HVgvc93rbXXjsxEkqQi6VHtAiRJ2lGGlySpcAwvSVLhGF6SpMIxvCRJhWN4SZIKx/CSJBWO4SVJKhzDS5JUOL2qteHBgwfn8OHDq7V5SdJu6MEHH1yVmbUd9ataeA0fPpy6urpqbV6StBuKiGfK6edpQ0lS4RhekqTCMbwkSYVjeEmSCsfwkiQVjuElSSocw0uSVDiGlySpcKr2IWVJu5ELB3Zy+RcrU4dUJkdekqTCceQlSWpSoBG4Iy9JUuGUFV4RMTEilkTE0oiY0U77tyNiUenPExHxl8qXKklSkw5PG0ZET2AW8D6gHlgYEfMy87GtfTLz35r1/yRw8C6oVZIkoLxrXocASzNzGUBEzAGOBx7bRv9TgC9VpjypCxToPL+kJuWE177AimbT9cD49jpGxFuAEcDPt9E+DZgGMGzYsB0qVJJ2KX+JKZRK37BxMnBLZm5urzEzZ2fmuMwcV1vb4RdlSpLUrnJGXs8BQ5tNDynNa8/JwL92tihJO2b4jNs7tfzymgoVUkXug+6lnJHXQmBkRIyIiD40BdS81p0i4m3AXsB9lS1RkqSWOgyvzGwEpgMLgMXA3Mx8NCIuiojJzbqeDMzJzNw1pUqS1KSsJ2xk5nxgfqt5F7SavrByZUmStG0+YUOSVDg+21CF54V6qftx5CVJKpzuPfLyQ4mSVEiOvCRJhVPYkVdnr3OA1zok/XXpTtd/HXlJkgrH8JIkFY7hJUkqnMJe81KFeMelpAJy5CVJKhzDS5JUOIaXJKlwDC9JUuEYXpKkwjG8JEmFY3hJkgrH8JIkFY7hJUkqnLLCKyImRsSSiFgaETO20eekiHgsIh6NiJsqW6YkSa/p8PFQEdETmAW8D6gHFkbEvMx8rFmfkcB5wBGZuToi3rirCpYkqZyR1yHA0sxclpkbgTnA8a36/DMwKzNXA2TmnypbpiRJryknvPYFVjSbri/Na+4A4ICI+HVE/CYiJlaqQEmSWqvUU+V7ASOBCcAQ4JcR8c7M/EvzThExDZgGMGzYsAptuvvy26QldVfljLyeA4Y2mx5SmtdcPTAvMzdl5tPAEzSFWQuZOTszx2XmuNra2p2tWZLUzZUTXguBkRExIiL6ACcD81r1+TFNoy4iYjBNpxGXVbBOSZJe1WF4ZWYjMB1YACwG5mbmoxFxUURMLnVbADRExGPAL4DPZWbDripaktS9lXXNKzPnA/Nbzbug2esEPl36I0nSLuUTNiRJhWN4SZIKx/CSJBWO4SVJKhzDS5JUOIaXJKlwDC9JUuEYXpKkwjG8JEmFY3hJkgrH8JIkFY7hJUkqHMNLklQ4hpckqXAML0lS4RhekqTCMbwkSYVjeEmSCsfwkiQVjuElSSqcssIrIiZGxJKIWBoRM9ppnxoRKyNiUenPWZUvVZKkJr066hARPYFZwPuAemBhRMzLzMdadb05M6fvgholSWqhnJHXIcDSzFyWmRuBOcDxu7YsSZK2rZzw2hdY0Wy6vjSvtRMj4uGIuCUihra3ooiYFhF1EVG3cuXKnShXkqTK3bBxKzA8M0cD/w/4n/Y6ZebszByXmeNqa2srtGlJUndTTng9BzQfSQ0pzXtVZjZk5obS5H8B76pMeZIktVVOeC0ERkbEiIjoA5wMzGveISL2aTY5GVhcuRIlSWqpw7sNM7MxIqYDC4CewHWZ+WhEXATUZeY84FMRMRloBP4MTN2FNUuSurkOwwsgM+cD81vNu6DZ6/OA8ypbmiRJ7fMJG5KkwjG8JEmFY3hJkgrH8JIkFY7hJUkqHMNLklQ4hpckqXAML0lS4RhekqTCMbwkSYVjeEmSCsfwkiQVjuElSSocw0uSVDiGlySpcAwvSVLhGF6SpMIxvCRJhWN4SZIKx/CSJBVOWeEVERMjYklELI2IGdvpd2JEZESMq1yJkiS11GF4RURPYBZwLDAKOCUiRrXTrz9wDnB/pYuUJKm5ckZehwBLM3NZZm4E5gDHt9PvK8A3gPUVrE+SpDbKCa99gRXNputL814VEWOBoZl5+/ZWFBHTIqIuIupWrly5w8VKkgQVuGEjInoA3wI+01HfzJydmeMyc1xtbW1nNy1J6qbKCa/ngKHNpoeU5m3VHzgIuDsilgOHAvO8aUOStKuUE14LgZERMSIi+gAnA/O2Nmbmi5k5ODOHZ+Zw4DfA5Mys2yUVS5K6vQ7DKzMbgenAAmAxMDczH42IiyJi8q4uUJKk1nqV0ykz5wPzW827YBt9J3S+LEmSts0nbEiSCsfwkiQVjuElSSocw0uSVDiGlySpcAwvSVLhGF6SpMIxvCRJhWN4SZIKx/CSJBWO4SVJKhzDS5JUOIaXJKlwynqq/K6wZMkSJkyY0GLe+eefz9FHH82iRYs499xz2ywzc+ZMDj/8cO69915euGlGm/Y3HDWNPm/aj3XLF/HivXPatA86Zjq9Bw1h7dL7eemBHzGhx5oW7Td+sC9DB/bg5kc2cVXdxjbL33JSXwb368H1izZy/aJNcHfL+ufPn0+/fv248sormTt3bpvl7777bgAuvfRSbrvtthZtffv25ac//SkAX/nKV7jrrrta1j5oED/4wQ8AOO+887jvvvt4YVnDq+29+g9m8Ac+C8Cf75zNxj8ta7F87zfsy6CJnwSg4WdXsOnPTd8nunUfjNm7J5dPrAHg9B+uo/6lLS2WP2xIT752dFP7iXPX0rA2S2+qaR8cddRRfPGLXwTg2GOPZd26dS2WnzRpEp/9bFN9rX/uACeddBJnn302a9eu5bjjjmvTPnXqVKZOncqqVauYMmVKi7YXljXQ/+DjeN3b30PjSytZddtlbZYfcMgH6bf/eDY11NOw4Lst2ib0WMP579mDo/frxaIXNnPuz9a3WX7mUXtw+NBe3Luikc/ftaFl490TuPzyyxkzZgx33nknF198cZvlr776ag488EBuvfVWLrusbX033ngjQ4cO5eabb+aqq65q037LLbcwePBgrr/+eq6//vo27Vve9Ul69K7h5YduZ83jv2rTvvepXwfgxft/yLqnHmjRFr32gI80vf7K/23grqcbW7QP6hf84KR+AJx353ruq9/con3IgB5878Km1+eeey6LFi1q0X7AAQcwe/ZsAKZNm8YTTzzRon3MmDFcfvnlAJx++unU19e3aD/ssMP42te+BsCJJ55IQ0NDi/bm//b+OPdLZGPLn0/ftx7CwPEfAmj3uPG6t/0t/ce+n7WbkuP+d22b9qljejN1TB9Wrd3ClLnr2rR/Ylwf/vGg3qxYsYIzzjijTftnPvMZPvCBD7BkyRI+/vGPt2nfkePe5z//+Tbtzf/ttff+Wh/3Whs86TP0GlDLmsW/ZMLv1rRpb3Pca2X+af3o1zu4cuFG5rbzf7szx73tceQlSSqcyMyqbHjcuHFZV7fzX7Y8fMbtna5hec2pnVvBhS92uobOcB806ex+cB+4D8B9ALvHPoiIBzNzXEf9HHlJkgrH8JIkFc5uGV7Lly/noIMO2iXrXv/sw/zpli8DMG/JJr5+z4YOlpAk7W7KCq+ImBgRSyJiaUS0uZ0lIv4lIn4fEYsi4p6IGFX5Uitv8oG9mXHkHtUuQ5K0gzq8VT4iegKzgPcB9cDCiJiXmY8163ZTZv5nqf9k4FvAxM4U1tjYyGmnncZDDz3EO97xDm644QYuvfRSbr31VtatW0dDryG84ZjpRAQv1c3jlUU/hR496T1oKLXH/ztbNq7nz3f+J5tWPUNu3szrjzyVfiMPbbGN6xdtpO4Pm/nucX2Z+uN1DNgjqPvDZl54ZQvffF8NU0b1BuCSX29g7mOb2NAIH3xbL7789zWdeWuSpE4qZ+R1CLA0M5dl5kZgDnB88w6Z+VKzydcBnb6FccmSJZx99tksXryYAQMGcOWVVzJ9+nQWLlzII488QjZufPXzKi/dfwv7TP0Obz7zuww65l8BePG+m6l5y9+wz0e+zd6nzGT1L65jy8a2n99p7vlXtnDPmf247dR+zLizqe8dTzXy5J+38MBZr2PRv7yOB5/fwi+fadzueiRJu1Y5H1LeF1jRbLoeGN+6U0T8K/BpoA/w3vZWFBHTgGkAw4YN2+5Ghw4dyhFHHAE0fXDxO9/5DiNGjOCb3/wma9euZf0zf6D34GGw/3h61w5n1a2X0PeAw14dXa1f/lvWLb2flx74IQDZuInNL63c7jZPOLA3PSIYVduTP65pyt87nmrkjqcaOfjqpg/vvbIxebJhC+95y3ZXJUnahSr2hI3MnAXMiohTgfOBj7bTZzYwG5o+57W99UVEm+mzzz6buro6hg4dyuuPPJVsbHoKxhunfIkNKx5l7dL7efHem3nzx2ZBJrUnfJ7eg4a0WM/mZ1dvc5t7NNsbWz/+lgnnHbkHHx/XZ3vlSpK6UDmnDZ8DhjabHlKaty1zgBM6UxTAs88+y3333QfATTfdxJFHHgnA4MGDeeWVV1i75NcAZG5h88urqHnLaPaa8E/kxrXkxnXUjBjLyw/dytYPYW/841M7Vccx+/fiukUbeWVj03qee2kLf1qzpYOlJEm7Ujkjr4XAyIgYQVNonQy0+Bh2RIzMzCdLk+8HnqSTDjzwQGbNmsWZZ57JqFGj+MQnPsHq1as56KCD2Hvvvemz9wFNHbdsYdWtl7Flwxog6f+uD9CjZk8GHn4yq++6huevmw6Z9Hr9m3jjlC/tcB3/8NZeLF7Zm8OubTptuGef4Hsf7MsbX9fZdyhJ2lkdhldmNkbEdGAB0BO4LjMfjYiLgLrMnAdMj4ijgU3Aato5Zbgjhg8fzuOPP95m/sUXX/zqQ0+bPwZl79O/2aZvj957MGji9Dbza4aNpmbYaACmjunD1DFN868/oW+Lfq98fsCrr885dA/OOdRb6iVpd1HWNa/MnA/MbzXvgmavz6lwXZIkbdNu+YQNSZK2x/CSJBWO4SVJKhzDS5JUOIaXJKlwDC9JUuEYXpKkwjG8JEmFY3hJkgrH8JIkFY7hJUkqHMNLklQ4hpckqXAML0lS4RhekqTCMbwkSYVjeEmSCsfwkiQVjuElSSocw0uSVDhlhVdETIyIJRGxNCJmtNP+6Yh4LCIejoi7IuItlS9VkqQmHYZXRPQEZgHHAqOAUyJiVKtuvwXGZeZo4Bbgm5UuVJKkrcoZeR0CLM3MZZm5EZgDHN+8Q2b+IjPXliZ/AwypbJmSJL2mnPDaF1jRbLq+NG9bPgb8tL2GiJgWEXURUbdy5cryq5QkqZmK3rAREacD44BL2mvPzNmZOS4zx9XW1lZy05KkbqRXGX2eA4Y2mx5SmtdCRBwNfAH4u8zcUJnyJElqq5yR10JgZESMiIg+wMnAvOYdIuJg4Gpgcmb+qfJlSpL0mg7DKzMbgenAAmAxMDczH42IiyJicqnbJcCewPcjYlFEzNvG6iRJ6rRyThuSmfOB+a3mXdDs9dEVrkuSpG3yCRuSpMIxvCRJhWN4SZIKx/CSJBWO4SVJKhzDS5JUOIaXJKlwDC9JUuEYXpKkwjG8JEmFY3hJkgrH8JIkFY7hJUkqnLKeKi9JqpxNmzZRX1/P+vXrK7reaybv06nlF8fczhWweHHZXWtqahgyZAi9e/feqU0ZXpLUxerr6+nfvz/Dhw8nIiq23k31f+nU8m/v0cla3vz2srplJg0NDdTX1zNixIid2pSnDSWpi61fv55BgwZVNLiKJCIYNGhQp0aehpckVUF3Da6tOvv+DS9JUuF4zUuSqmz4jNsrur5504/Y4WUu/NynuPjjxzPqgP12eNnlK/7ApI+ewyOPL93hZXeW4SVJ4sJLvsOoHk9Xu4yylXXaMCImRsSSiFgaETPaaX9PRDwUEY0RMaXyZUqSKmXt2jVM/+hJfPgfjuRDRx3Gz+b9kI99eBJ1v3sMgD1HHsEXvv5d/ubof+TQSR/hjysbAHhq+QoOnfQR3nnUSZz/jVnsObLtCG/z5s187nOf493vfjejR4/m6quv3iXvocPwioiewCzgWGAUcEpEjGrV7VlgKnBTpQuUJFXWvXffRe2b9uH7d9zDD++6jyMmHNWifc3adRw69p387s6bec+hY7nmf38EwDkXXMI5Z53K7++ay5B93tTuuq+99loGDhzIwoULWbhwIddccw1PP135EV05I69DgKWZuSwzNwJzgOObd8jM5Zn5MLCl4hVKkipq/7eN4je/+gXfnvklHrr/XvoPGNiivU+f3kx633sAeNc7387y+j8AcN+Dv+fDk44G4NQPTmx33XfccQc33HADY8aMYfz48TQ0NPDkk09W/D2Uc81rX2BFs+l6YPzObCwipgHTAIYNG7Yzq5AkddLw/fZnzvz/41e/uIPvXvJVDjny71q09+7V69Vb2Xv27Elj4+ay152ZXHHFFRxzzDEVrbm1Lr1VPjNnZ+a4zBxXW1vblZuWJJX86YXnqenbl0kf+kc++i+f5PHf/66s5Q4d+05+cPtdAMz5yYJ2+xxzzDFcddVVbNq0CYAnnniCNWvWVKbwZsoZeT0HDG02PaQ0T5JUAcu//v6KrOfhMh8P9eTjj/Htr15Ajx496NWrN1+YeRnfuviLHS53+Zc/y+mfOp+vXnEtEycczsABe7bpc9ZZZ7F8+XLGjh1LZlJbW8uPf/zjHX4vHSknvBYCIyNiBE2hdTJwasUrkSR1iSMmHNXmJo1rv38bo0u3yr/y5K9fnT9l0tFMKV3n2nefWn5z6/8QEcz5yQKWPPUMAMOHvplHfv59AHr06MHMmTOZOXPmLn0PHYZXZjZGxHRgAdATuC4zH42Ii4C6zJwXEe8GfgTsBXwgIr6cme/YpZVLkrrUgw8vZvoXvkGSvH5Af6677EtVq6WsDyln5nxgfqt5FzR7vZCm04mSpL9Sfzt+LL+78+ZqlwH4bENJUgEZXpKkwjG8JEmFY3hJkgrHp8pLUrVdOLDjPmUYXfr74bOe2anlh49/P3U//R6D37AXe448osUt87sbR16SpMJx5CVJ3dC5HzuNF55/jg0bNnDamR9nymlTq13SDjG8JKkb+vKl32XgXnuxft06Tp30Xo4+bnK1S9ohhpckdUM3/ffV/PxntwHwx+ef49mnn6pyRTvG8JKkbmbhfffwm3vu5oaf3EHfvv342IcnsWHDhmqXtUO8YUOSuplXXnqJAQNfT9++/Xh66RM8/Nu6ape0wxx5SVK1XfhiRVZT7leiHDHhKL7/ves44e/HM3y//Rl98LiKbL8rGV6S1M302WMPrrzxljbzl99/+6uvd+fPeIGnDSVJBWR4SZIKx/CSpCrIzGqXUFWdff+GlyR1sZqaGhoaGrptgGUmDQ0N1NTU7PQ6vGFDkrrYkCFDqK+vZ+XKlRVd7x9Xr+vU8oujk/W8uLjsrjU1NQwZMmSnN2V4SVIX6927NyNGjKj4eo+dcXvHnbZjec2pnSugQrf8l6Os04YRMTEilkTE0oiY0U77HhFxc6n9/ogYXulCJUnaqsPwioiewCzgWGAUcEpEjGrV7WPA6szcH/g28I1KFypJ0lbljLwOAZZm5rLM3AjMAY5v1ed44H9Kr28BjoqIqFyZkiS9Jjq62yUipgATM/Os0vQZwPjMnN6szyOlPvWl6adKfVa1Wtc0YFpp8kBgSaXeyE4aDKzqsNdfN/eB+wDcB+A+gN1jH7wlM2s76tSlN2xk5mxgdlduc3sioi4zi/dQrwpyH7gPwH0A7gMo1j4o57Thc8DQZtNDSvPa7RMRvYCBQEMlCpQkqbVywmshMDIiRkREH+BkYF6rPvOAj5ZeTwF+nt3103eSpF2uw9OGmdkYEdOBBUBP4LrMfDQiLgLqMnMecC1wY0QsBf5MU8AVwW5zCrOK3AfuA3AfgPsACrQPOrxhQ5Kk3Y3PNpQkFY7hJUkqHMNLklQ4hpckqXB8qnw3EhEf2l57Zv6wq2pR9ZSeV3pDZp5W7VpUfRFxTmb+R0fzdjfd6m7DiKgF/hkYTrPgzswzq1VTV4qI/241a+sPP4DsRvvhHOC/gZeB/wIOBmZk5h1VLawLRcQ9wHtLzyvttiLiAOAq4E2ZeVBEjAYmZ+bFVS6ty0TEQ5k5ttW832bmwdWqqRzdbeT1E+BXwJ3A5irX0uUy858AIqIGOJGWId59fouBMzPzPyLiGGAv4AzgRqDbhBewDPh1RMwD1mydmZnfql5JVXEN8DngaoDMfDgibgL+6sMrIk4BTgVGlP4dbNWfps/r7ta6W3j1y8x/r3YRu4EfA38BHgLWl+Z1p/Da+o0H7wduLH3ovlt8C0JE3JiZZwCTafr6oh40Hay6q36Z+UCrH39jtYrpYvcCz9P0MN7Lms1/GXi4KhXtgO4WXrdFxHGZOb/ahVTZkMycWO0iqujBiFgA7AfMiIj+wJYq19RV3hURbwaeBa6odjG7gVUR8VZKv7yVvkXj+eqW1DUy8xngGeCwateyM7rbNa+XgX7ARmATr13rGVDVwrpYRMwGrsjM31e7lmqIiB7A+cBemflvETGMpq9h+FWVS9vlIuJTwCeAEcAfmjfR9H9hv6oUViURsR9Nj0Q6HFgNPA2cVjqw/1WLiHsy88jScbF5EBTiuNjdwqsHcBowIjMvKh209snM+6tcWpeKiMeA/Wn6j7qB1/6xjq5qYV0kIq6iaaT13sx8e0TsBdyRme+ucmldJiKuysxPVLuOaomIT7ea1ZemU6hroFte+yuc7nbacBalgxZwEU3ndn8AdJuDVsmx1S6gysZn5tiI+C1AZq4ufWNCt9Gdg6tk63W+A2n6//8Tmn6JOwN4oFpFqXzdLby6/UELXj3X3Z1tKn3Waet1jlq6zzUvAZn5ZYCI+CUwNjNfLk1fCNxexdJUpu72hA0PWgL4DvAj4I0R8VXgHmBmdUtSlbyJpmvgW20szdNurruNvFoftKbQdOFe3Uhm/m9EPAgcRdOpohMyc3GVy1J13AA8EBE/Kk2fAFxfvXJUrm51wwZARLyN1w5ad3nQkrq3iBgL/G1p8peZ+dtq1qPydLvwkiQVX3e75iVJ+itgeEmSCsfwkiQVjuElSSqc/w/mVwQ9e0TbjQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 504x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (7,4)\n",
    "ax = combined_df.plot(kind=\"bar\")\n",
    "ax.legend(loc=4)\n",
    "ax.hlines(0.33, -1, 6, linestyles='dashed')\n",
    "ax.annotate('baseline',(-0.4,0.29))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Conclusions\n",
    "\n",
    "In short, our experiments have shown that: \n",
    "\n",
    "- BERT-multilingual can do zero-shot transfer between typologically related languages.\n",
    "- Its performance further improves when you add data from the target language.\n",
    "- A model that was finetuned on several languages can outperform language-specific models."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}