[Bug]: Validation F1 score is consistently 0 across training epochs but test F1 is ~0.85

[choomegan]

Describe the bug

When training a SequenceTagger for NER with the last layer of RoBERTa embeddings, the micro average F1 score on the validation set is consistently 0, but the training loss is decreasing (as expected). However, the test set F1 score is 0.8490. There is an issue with the logging of validation F1 scores.

My dataset only has 3 possible tags: B-SHI, I-SHI and O.

To Reproduce

from flair.data import Corpus
from flair.datasets import ColumnCorpus
from typing import List
from flair.trainers import ModelTrainer
from flair.models import SequenceTagger
from flair.embeddings import (
    TransformerWordEmbeddings,
)

columns = {0: "text", 1: "ner"}
data_folder = "/datasets"

corpus: Corpus = ColumnCorpus(
    data_folder=data_folder,
    column_format=columns,
    train_file="train.txt",
    dev_file="dev.txt",
    test_file="test.txt",
)

# make label dictionary from the corpus
label_dictionary = corpus.make_label_dictionary(label_type="ner")

embeddings = TransformerWordEmbeddings(
    "roberta-base",
    layers="-1",
    layer_mean=False,
    subtoken_pooling="first",
    fine_tune=True,
)

tagger: SequenceTagger = SequenceTagger(
    hidden_size=256,
    embeddings=embeddings,
    tag_dictionary=label_dictionary,
    tag_type="ner",
    use_crf=False,  # Conditional Random Field (CRF) layer - capture dependencies between tags
    use_rnn=False,
    reproject_embeddings=False,  # Embeddings are not 'transformed' before being fed into hidden layer, raw embeddings are used
)

trainer: ModelTrainer = ModelTrainer(tagger, corpus)

trainer.fine_tune(
    base_path="/flair-output",
    learning_rate=5e-5,
    mini_batch_size=64,
    eval_batch_size=64,
    max_epochs=150,
    embeddings_storage_mode="gpu",
    use_final_model_for_eval=False,
)

Expected behavior

Non-zero F1 validation scores as the training loss is decreasing. Validation F1 score near the end of 150 epochs should be comparable to the test set F1.

Logs and Stack traces

1710395179672 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:46:14,945 epoch 140 - iter 4/45 - loss 0.00025502 - time (sec): 1.30 - samples/sec: 1889.46 - lr: 0.000005 - momentum: 0.000000
2024-03-14 05:46:16,494 epoch 140 - iter 8/45 - loss 0.00012755 - time (sec): 2.85 - samples/sec: 1728.27 - lr: 0.000005 - momentum: 0.000000
2024-03-14 05:46:17,621 epoch 140 - iter 12/45 - loss 0.00008608 - time (sec): 3.97 - samples/sec: 1843.39 - lr: 0.000005 - momentum: 0.000000
2024-03-14 05:46:18,725 epoch 140 - iter 16/45 - loss 0.00015033 - time (sec): 5.08 - samples/sec: 1971.81 - lr: 0.000005 - momentum: 0.000000

1710395184748 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:46:19,721 epoch 140 - iter 20/45 - loss 0.00017150 - time (sec): 6.07 - samples/sec: 2044.11 - lr: 0.000005 - momentum: 0.000000
2024-03-14 05:46:20,877 epoch 140 - iter 24/45 - loss 0.00033717 - time (sec): 7.23 - samples/sec: 2078.60 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:21,833 epoch 140 - iter 28/45 - loss 0.00038059 - time (sec): 8.18 - samples/sec: 2147.68 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:23,076 epoch 140 - iter 32/45 - loss 0.00033360 - time (sec): 9.43 - samples/sec: 2128.28 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:24,187 epoch 140 - iter 36/45 - loss 0.00033955 - time (sec): 10.54 - samples/sec: 2151.70 - lr: 0.000004 - momentum: 0.000000

1710395189808 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:46:25,264 epoch 140 - iter 40/45 - loss 0.00030646 - time (sec): 11.62 - samples/sec: 2163.60 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:26,416 epoch 140 - iter 44/45 - loss 0.00027796 - time (sec): 12.77 - samples/sec: 2171.25 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:26,556 ----------------------------------------------------------------------------------------------------
2024-03-14 05:46:26,557 EPOCH 140 done: loss 0.0003 - lr: 0.000004
100%|██████████| 6/6 [00:00<00:00, 37.68it/s]
2024-03-14 05:46:26,743 DEV : loss 0.8289894461631775 - f1-score (micro avg)  0.0
2024-03-14 05:46:26,755 ----------------------------------------------------------------------------------------------------
2024-03-14 05:46:28,041 epoch 141 - iter 4/45 - loss 0.00123978 - time (sec): 1.28 - samples/sec: 1922.74 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:29,143 epoch 141 - iter 8/45 - loss 0.00061903 - time (sec): 2.39 - samples/sec: 2074.17 - lr: 0.000004 - momentum: 0.000000

1710395194888 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:46:30,636 epoch 141 - iter 12/45 - loss 0.00046702 - time (sec): 3.88 - samples/sec: 1974.32 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:31,757 epoch 141 - iter 16/45 - loss 0.00034711 - time (sec): 5.00 - samples/sec: 2062.41 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:32,826 epoch 141 - iter 20/45 - loss 0.00027852 - time (sec): 6.07 - samples/sec: 2119.61 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:33,897 epoch 141 - iter 24/45 - loss 0.00023577 - time (sec): 7.14 - samples/sec: 2129.48 - lr: 0.000004 - momentum: 0.000000

1710395199950 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:46:34,920 epoch 141 - iter 28/45 - loss 0.00020447 - time (sec): 8.16 - samples/sec: 2149.70 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:36,031 epoch 141 - iter 32/45 - loss 0.00017895 - time (sec): 9.27 - samples/sec: 2164.59 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:37,181 epoch 141 - iter 36/45 - loss 0.00020030 - time (sec): 10.42 - samples/sec: 2164.62 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:38,290 epoch 141 - iter 40/45 - loss 0.00025158 - time (sec): 11.53 - samples/sec: 2178.68 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:39,275 epoch 141 - iter 44/45 - loss 0.00022910 - time (sec): 12.52 - samples/sec: 2205.34 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:39,445 ----------------------------------------------------------------------------------------------------
2024-03-14 05:46:39,445 EPOCH 141 done: loss 0.0002 - lr: 0.000004
100%|██████████| 6/6 [00:00<00:00, 37.13it/s]
2024-03-14 05:46:39,635 DEV : loss 0.8288939595222473 - f1-score (micro avg)  0.0
2024-03-14 05:46:39,647 ----------------------------------------------------------------------------------------------------

1710395204998 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:46:41,040 epoch 142 - iter 4/45 - loss 0.00000121 - time (sec): 1.39 - samples/sec: 1815.02 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:42,008 epoch 142 - iter 8/45 - loss 0.00042547 - time (sec): 2.36 - samples/sec: 2000.71 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:42,945 epoch 142 - iter 12/45 - loss 0.00028173 - time (sec): 3.30 - samples/sec: 2165.59 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:44,324 epoch 142 - iter 16/45 - loss 0.00020488 - time (sec): 4.67 - samples/sec: 2101.54 - lr: 0.000004 - momentum: 0.000000

1710395210056 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:46:45,505 epoch 142 - iter 20/45 - loss 0.00016289 - time (sec): 5.86 - samples/sec: 2113.29 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:46,698 epoch 142 - iter 24/45 - loss 0.00013305 - time (sec): 7.05 - samples/sec: 2152.31 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:47,737 epoch 142 - iter 28/45 - loss 0.00011378 - time (sec): 8.09 - samples/sec: 2196.90 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:48,788 epoch 142 - iter 32/45 - loss 0.00024703 - time (sec): 9.14 - samples/sec: 2204.08 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:49,780 epoch 142 - iter 36/45 - loss 0.00022194 - time (sec): 10.13 - samples/sec: 2229.44 - lr: 0.000004 - momentum: 0.000000

1710395215152 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:46:50,860 epoch 142 - iter 40/45 - loss 0.00025818 - time (sec): 11.21 - samples/sec: 2243.51 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:51,930 epoch 142 - iter 44/45 - loss 0.00030082 - time (sec): 12.28 - samples/sec: 2255.56 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:52,096 ----------------------------------------------------------------------------------------------------
2024-03-14 05:46:52,096 EPOCH 142 done: loss 0.0003 - lr: 0.000004
100%|██████████| 6/6 [00:00<00:00, 37.97it/s]
2024-03-14 05:46:52,281 DEV : loss 0.8288004994392395 - f1-score (micro avg)  0.0
2024-03-14 05:46:52,293 ----------------------------------------------------------------------------------------------------
2024-03-14 05:46:53,421 epoch 143 - iter 4/45 - loss 0.00070282 - time (sec): 1.13 - samples/sec: 2113.99 - lr: 0.000004 - momentum: 0.000000
2024-03-14 05:46:54,807 epoch 143 - iter 8/45 - loss 0.00047642 - time (sec): 2.51 - samples/sec: 1908.56 - lr: 0.000004 - momentum: 0.000000

1710395220216 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:46:55,925 epoch 143 - iter 12/45 - loss 0.00031238 - time (sec): 3.63 - samples/sec: 2017.59 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:46:57,294 epoch 143 - iter 16/45 - loss 0.00034553 - time (sec): 5.00 - samples/sec: 1968.77 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:46:58,494 epoch 143 - iter 20/45 - loss 0.00027237 - time (sec): 6.20 - samples/sec: 2016.32 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:46:59,572 epoch 143 - iter 24/45 - loss 0.00022680 - time (sec): 7.28 - samples/sec: 2063.95 - lr: 0.000003 - momentum: 0.000000

1710395225273 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:00,759 epoch 143 - iter 28/45 - loss 0.00019481 - time (sec): 8.46 - samples/sec: 2067.46 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:01,750 epoch 143 - iter 32/45 - loss 0.00025657 - time (sec): 9.45 - samples/sec: 2112.43 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:02,845 epoch 143 - iter 36/45 - loss 0.00022801 - time (sec): 10.55 - samples/sec: 2131.41 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:04,067 epoch 143 - iter 40/45 - loss 0.00023507 - time (sec): 11.77 - samples/sec: 2136.02 - lr: 0.000003 - momentum: 0.000000

1710395230417 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:05,279 epoch 143 - iter 44/45 - loss 0.00021349 - time (sec): 12.98 - samples/sec: 2133.60 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:05,427 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:05,427 EPOCH 143 done: loss 0.0002 - lr: 0.000003
100%|██████████| 6/6 [00:00<00:00, 37.55it/s]
2024-03-14 05:47:05,610 DEV : loss 0.8286968469619751 - f1-score (micro avg)  0.0
2024-03-14 05:47:05,622 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:06,677 epoch 144 - iter 4/45 - loss 0.00079839 - time (sec): 0.95 - samples/sec: 2487.58 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:08,072 epoch 144 - iter 8/45 - loss 0.00038394 - time (sec): 2.45 - samples/sec: 2014.87 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:09,234 epoch 144 - iter 12/45 - loss 0.00032369 - time (sec): 3.61 - samples/sec: 2099.26 - lr: 0.000003 - momentum: 0.000000

1710395235468 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:10,414 epoch 144 - iter 16/45 - loss 0.00055661 - time (sec): 4.79 - samples/sec: 2139.06 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:11,562 epoch 144 - iter 20/45 - loss 0.00044751 - time (sec): 5.94 - samples/sec: 2146.83 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:12,533 epoch 144 - iter 24/45 - loss 0.00037702 - time (sec): 6.91 - samples/sec: 2191.44 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:13,766 epoch 144 - iter 28/45 - loss 0.00032281 - time (sec): 8.14 - samples/sec: 2172.84 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:14,676 epoch 144 - iter 32/45 - loss 0.00028423 - time (sec): 9.05 - samples/sec: 2221.15 - lr: 0.000003 - momentum: 0.000000

1710395240525 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:15,562 epoch 144 - iter 36/45 - loss 0.00028622 - time (sec): 9.94 - samples/sec: 2264.12 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:16,801 epoch 144 - iter 40/45 - loss 0.00025668 - time (sec): 11.18 - samples/sec: 2246.06 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:18,051 epoch 144 - iter 44/45 - loss 0.00039253 - time (sec): 12.33 - samples/sec: 2248.91 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:18,222 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:18,222 EPOCH 144 done: loss 0.0004 - lr: 0.000003
100%|██████████| 6/6 [00:00<00:00, 37.53it/s]
2024-03-14 05:47:18,409 DEV : loss 0.8287426233291626 - f1-score (micro avg)  0.0
2024-03-14 05:47:18,421 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:19,641 epoch 145 - iter 4/45 - loss 0.00000091 - time (sec): 1.22 - samples/sec: 2126.37 - lr: 0.000003 - momentum: 0.000000

1710395245577 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:20,673 epoch 145 - iter 8/45 - loss 0.00000097 - time (sec): 2.25 - samples/sec: 2259.85 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:21,777 epoch 145 - iter 12/45 - loss 0.00033357 - time (sec): 3.35 - samples/sec: 2257.54 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:22,948 epoch 145 - iter 16/45 - loss 0.00025010 - time (sec): 4.53 - samples/sec: 2234.28 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:24,231 epoch 145 - iter 20/45 - loss 0.00020042 - time (sec): 5.81 - samples/sec: 2174.79 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:25,409 epoch 145 - iter 24/45 - loss 0.00032537 - time (sec): 6.99 - samples/sec: 2191.26 - lr: 0.000003 - momentum: 0.000000

1710395250676 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:26,377 epoch 145 - iter 28/45 - loss 0.00036992 - time (sec): 7.95 - samples/sec: 2244.85 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:27,381 epoch 145 - iter 32/45 - loss 0.00033398 - time (sec): 8.96 - samples/sec: 2263.60 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:28,390 epoch 145 - iter 36/45 - loss 0.00090788 - time (sec): 9.97 - samples/sec: 2275.17 - lr: 0.000003 - momentum: 0.000000
2024-03-14 05:47:29,568 epoch 145 - iter 40/45 - loss 0.00081781 - time (sec): 11.14 - samples/sec: 2259.15 - lr: 0.000003 - momentum: 0.000000

1710395255736 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:30,814 epoch 145 - iter 44/45 - loss 0.00076414 - time (sec): 12.39 - samples/sec: 2233.55 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:30,960 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:30,960 EPOCH 145 done: loss 0.0008 - lr: 0.000002
100%|██████████| 6/6 [00:00<00:00, 37.96it/s]
2024-03-14 05:47:31,146 DEV : loss 0.8288417458534241 - f1-score (micro avg)  0.0
2024-03-14 05:47:31,158 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:32,515 epoch 146 - iter 4/45 - loss 0.00000091 - time (sec): 1.36 - samples/sec: 1810.91 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:33,616 epoch 146 - iter 8/45 - loss 0.00051328 - time (sec): 2.46 - samples/sec: 2064.83 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:34,625 epoch 146 - iter 12/45 - loss 0.00034056 - time (sec): 3.46 - samples/sec: 2208.46 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:35,704 epoch 146 - iter 16/45 - loss 0.00025569 - time (sec): 4.54 - samples/sec: 2245.03 - lr: 0.000002 - momentum: 0.000000

1710395260820 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:36,872 epoch 146 - iter 20/45 - loss 0.00028284 - time (sec): 5.71 - samples/sec: 2223.58 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:38,074 epoch 146 - iter 24/45 - loss 0.00023631 - time (sec): 6.91 - samples/sec: 2200.06 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:39,168 epoch 146 - iter 28/45 - loss 0.00020549 - time (sec): 8.01 - samples/sec: 2185.98 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:40,246 epoch 146 - iter 32/45 - loss 0.00017904 - time (sec): 9.09 - samples/sec: 2212.91 - lr: 0.000002 - momentum: 0.000000

1710395265903 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:41,326 epoch 146 - iter 36/45 - loss 0.00019267 - time (sec): 10.17 - samples/sec: 2243.45 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:42,473 epoch 146 - iter 40/45 - loss 0.00022236 - time (sec): 11.31 - samples/sec: 2233.79 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:43,621 epoch 146 - iter 44/45 - loss 0.00020302 - time (sec): 12.46 - samples/sec: 2222.02 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:43,768 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:43,768 EPOCH 146 done: loss 0.0002 - lr: 0.000002
100%|██████████| 6/6 [00:00<00:00, 38.03it/s]
2024-03-14 05:47:43,952 DEV : loss 0.8287931680679321 - f1-score (micro avg)  0.0
2024-03-14 05:47:43,964 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:44,859 epoch 147 - iter 4/45 - loss 0.00000129 - time (sec): 0.89 - samples/sec: 2477.30 - lr: 0.000002 - momentum: 0.000000

1710395270958 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:46,256 epoch 147 - iter 8/45 - loss 0.00000106 - time (sec): 2.29 - samples/sec: 2117.90 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:47,348 epoch 147 - iter 12/45 - loss 0.00000107 - time (sec): 3.38 - samples/sec: 2174.00 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:48,478 epoch 147 - iter 16/45 - loss 0.00000110 - time (sec): 4.41 - samples/sec: 2252.03 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:49,786 epoch 147 - iter 20/45 - loss 0.00028940 - time (sec): 5.82 - samples/sec: 2163.00 - lr: 0.000002 - momentum: 0.000000

1710395276030 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:51,067 epoch 147 - iter 24/45 - loss 0.00030542 - time (sec): 7.10 - samples/sec: 2135.87 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:52,126 epoch 147 - iter 28/45 - loss 0.00026485 - time (sec): 8.16 - samples/sec: 2145.40 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:53,020 epoch 147 - iter 32/45 - loss 0.00023234 - time (sec): 9.05 - samples/sec: 2205.17 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:54,099 epoch 147 - iter 36/45 - loss 0.00025540 - time (sec): 10.13 - samples/sec: 2213.89 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:55,417 epoch 147 - iter 40/45 - loss 0.00022812 - time (sec): 11.45 - samples/sec: 2194.30 - lr: 0.000002 - momentum: 0.000000

1710395281200 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:47:56,495 epoch 147 - iter 44/45 - loss 0.00036235 - time (sec): 12.53 - samples/sec: 2212.08 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:56,636 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:56,636 EPOCH 147 done: loss 0.0004 - lr: 0.000002
100%|██████████| 6/6 [00:00<00:00, 41.98it/s]
2024-03-14 05:47:56,802 DEV : loss 0.828815221786499 - f1-score (micro avg)  0.0
2024-03-14 05:47:56,814 ----------------------------------------------------------------------------------------------------
2024-03-14 05:47:57,950 epoch 148 - iter 4/45 - loss 0.00040544 - time (sec): 1.13 - samples/sec: 2358.91 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:47:59,087 epoch 148 - iter 8/45 - loss 0.00021431 - time (sec): 2.27 - samples/sec: 2232.62 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:48:00,157 epoch 148 - iter 12/45 - loss 0.00014433 - time (sec): 3.34 - samples/sec: 2259.77 - lr: 0.000002 - momentum: 0.000000

1710395286261 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:48:01,252 epoch 148 - iter 16/45 - loss 0.00021919 - time (sec): 4.44 - samples/sec: 2271.84 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:48:02,401 epoch 148 - iter 20/45 - loss 0.00017655 - time (sec): 5.58 - samples/sec: 2243.06 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:48:03,561 epoch 148 - iter 24/45 - loss 0.00014843 - time (sec): 6.75 - samples/sec: 2211.97 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:48:04,596 epoch 148 - iter 28/45 - loss 0.00012678 - time (sec): 7.78 - samples/sec: 2247.75 - lr: 0.000002 - momentum: 0.000000
2024-03-14 05:48:05,650 epoch 148 - iter 32/45 - loss 0.00083240 - time (sec): 8.83 - samples/sec: 2274.20 - lr: 0.000001 - momentum: 0.000000

1710395291329 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:48:06,604 epoch 148 - iter 36/45 - loss 0.00079076 - time (sec): 9.79 - samples/sec: 2303.32 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:07,937 epoch 148 - iter 40/45 - loss 0.00073942 - time (sec): 11.12 - samples/sec: 2252.44 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:09,231 epoch 148 - iter 44/45 - loss 0.00066961 - time (sec): 12.41 - samples/sec: 2228.35 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:09,395 ----------------------------------------------------------------------------------------------------
2024-03-14 05:48:09,395 EPOCH 148 done: loss 0.0007 - lr: 0.000001
100%|██████████| 6/6 [00:00<00:00, 39.66it/s]
2024-03-14 05:48:09,574 DEV : loss 0.8287803530693054 - f1-score (micro avg)  0.0
2024-03-14 05:48:09,686 ----------------------------------------------------------------------------------------------------
2024-03-14 05:48:10,845 epoch 149 - iter 4/45 - loss 0.00000130 - time (sec): 1.16 - samples/sec: 2199.36 - lr: 0.000001 - momentum: 0.000000

1710395296378 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:48:11,853 epoch 149 - iter 8/45 - loss 0.00000128 - time (sec): 2.16 - samples/sec: 2290.53 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:12,920 epoch 149 - iter 12/45 - loss 0.00025843 - time (sec): 3.23 - samples/sec: 2325.94 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:13,862 epoch 149 - iter 16/45 - loss 0.00029740 - time (sec): 4.17 - samples/sec: 2403.18 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:14,931 epoch 149 - iter 20/45 - loss 0.00024071 - time (sec): 5.24 - samples/sec: 2365.89 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:15,992 epoch 149 - iter 24/45 - loss 0.00020109 - time (sec): 6.30 - samples/sec: 2358.45 - lr: 0.000001 - momentum: 0.000000

1710395301469 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:48:16,943 epoch 149 - iter 28/45 - loss 0.00022697 - time (sec): 7.25 - samples/sec: 2388.62 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:18,194 epoch 149 - iter 32/45 - loss 0.00021418 - time (sec): 8.51 - samples/sec: 2323.12 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:19,661 epoch 149 - iter 36/45 - loss 0.00018777 - time (sec): 9.97 - samples/sec: 2261.62 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:20,845 epoch 149 - iter 40/45 - loss 0.00020345 - time (sec): 11.16 - samples/sec: 2245.29 - lr: 0.000001 - momentum: 0.000000

1710395306528 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:48:22,053 epoch 149 - iter 44/45 - loss 0.00018429 - time (sec): 12.36 - samples/sec: 2237.68 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:22,216 ----------------------------------------------------------------------------------------------------
2024-03-14 05:48:22,216 EPOCH 149 done: loss 0.0002 - lr: 0.000001
100%|██████████| 6/6 [00:00<00:00, 38.16it/s]
2024-03-14 05:48:22,400 DEV : loss 0.8287989497184753 - f1-score (micro avg)  0.0
2024-03-14 05:48:22,411 ----------------------------------------------------------------------------------------------------
2024-03-14 05:48:23,561 epoch 150 - iter 4/45 - loss 0.00039111 - time (sec): 1.15 - samples/sec: 2080.72 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:24,715 epoch 150 - iter 8/45 - loss 0.00018762 - time (sec): 2.30 - samples/sec: 2197.38 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:25,772 epoch 150 - iter 12/45 - loss 0.00018687 - time (sec): 3.36 - samples/sec: 2275.98 - lr: 0.000001 - momentum: 0.000000

1710395311595 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:48:26,889 epoch 150 - iter 16/45 - loss 0.00014197 - time (sec): 4.47 - samples/sec: 2251.76 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:27,830 epoch 150 - iter 20/45 - loss 0.00014523 - time (sec): 5.42 - samples/sec: 2300.47 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:28,772 epoch 150 - iter 24/45 - loss 0.00019992 - time (sec): 6.36 - samples/sec: 2331.30 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:29,740 epoch 150 - iter 28/45 - loss 0.00017246 - time (sec): 7.33 - samples/sec: 2346.96 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:30,983 epoch 150 - iter 32/45 - loss 0.00015023 - time (sec): 8.57 - samples/sec: 2305.34 - lr: 0.000001 - momentum: 0.000000

1710395316691 clearml-id-b5f6f528cc0a49569e781aef9aed0d9e:gpuGPU-ed7a4a88-2f57-3dc0-10c2-b7a8ed494e37 DEBUG 2024-03-14 05:48:32,230 epoch 150 - iter 36/45 - loss 0.00013200 - time (sec): 9.82 - samples/sec: 2292.88 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:33,460 epoch 150 - iter 40/45 - loss 0.00011843 - time (sec): 11.05 - samples/sec: 2272.57 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:34,887 epoch 150 - iter 44/45 - loss 0.00079182 - time (sec): 12.47 - samples/sec: 2223.46 - lr: 0.000001 - momentum: 0.000000
2024-03-14 05:48:35,034 ----------------------------------------------------------------------------------------------------
2024-03-14 05:48:35,034 EPOCH 150 done: loss 0.0008 - lr: 0.000001
100%|██████████| 6/6 [00:00<00:00, 35.10it/s]
2024-03-14 05:48:35,232 DEV : loss 0.8287793397903442 - f1-score (micro avg)  0.0

Results:
- F-score (micro) 0.849
- F-score (macro) 0.849
- Accuracy 0.7376

By class:
              precision    recall  f1-score   support

         SHI     0.8421    0.8560    0.8490       243

   micro avg     0.8421    0.8560    0.8490       243
   macro avg     0.8421    0.8560    0.8490       243
weighted avg     0.8421    0.8560    0.8490       243

### Screenshots

_No response_

### Additional Context

_No response_

### Environment

### Versions:
### Flair
0.13.1

### Pytorch
2.0.1

### Transformers
4.35.2

### GPU
True

[stefan-it]

Hi @choomegan ,

could you check if you see that the best model was saved in previous epochs? I hypothesize that the best model with non Zero F1-Score was found in previous epochs, which would explain why you actually achieve a non-zero F1-Score on the test set. So maybe you can post the full log output here :)

[choomegan]

Hi @stefan-it I have attached the full logs here: flair_finetune.log

I ran inference with final-model.pt model and the results (0.8490) match the test results seen at the end of the training log file.

Seems like the best model is not saved as only ['training.log', 'final-model.pt', 'test.tsv', 'dev.tsv', 'loss.tsv'] is found in /flair-output which is the base_path I specified in the trainer.finetune method.

[Aakame]

Hello @choomegan. Have you found the solution to the issue yet? I have the same problem with validation F1 being ~0 in the loss.tsv, while the test F1 takes good values ( 0.76, 0.8,..)

[choomegan]

Hello @choomegan. Have you found the solution to the issue yet? I have the same problem with validation F1 being ~0 in the loss.tsv, while the test F1 takes good values ( 0.76, 0.8,..)

Hi @Aakame, I have not found a solution to the issue yet :( @stefan-it would you be able to assist? Thanks!

[Aakame]

Hello @choomegan. Have you found the solution to the issue yet? I have the same problem with validation F1 being ~0 in the loss.tsv, while the test F1 takes good values ( 0.76, 0.8,..)

Hi @Aakame, I have not found a solution to the issue yet :( @stefan-it would you be able to assist? Thanks!

I've downgraded Flair to version 12.2, and it appears that the tsv.loss now produces the correct values for DEV_F1. It seems that the bug may be present only in the most recent version.

[petermartens1992]

The only difference between the faulty DEV evaluations that happen after each epoch and the correct final TEST evaluation is the storage of the embeddings, which doesnt happen in the latter case:

store_embeddings(evaluation_split_data, embeddings_storage_mode)

I found out that when I set the embeddings_storage_mode to "none" the DEV evaluation happens correctly again and the score becomes higher than zero.

@stefan-it I guess the gold labels get wiped out as part as the data_point.clear_embeddings()?