NNCF v2.5.0 accuracy and performance degradation of EfficientDet

[lisosia]

I tried NNCF 2.5.0 for my effdet model and observed accuracy and performance degradation.
Even if I use pot-backend (use_pot=True), the accuracy and performance is degraded compared to nncf 2.4.0.

The issue is reproduced with the efficientdet-d0 which was exported by the official procedure.
The throughput was measured by benchmark-app of openvino-dev=2023.0.0.

model	coco_precision	Throughput
v2.4.0	30.68%	67.14 FPS
commit ca54404 (pot backend)	30.50%	65.40 FPS
commit ca54404	24.53%	60.92 FPS

I attaches the reproduction code and environments.
I hope the issue will be fixed so that we can migrate to the newer NNCF.

repro_effdet_issue.zip

[MaximProshin]

@l-bat , please help with this issue
cc @alexsu52

[l-bat]

@lisosia have you measured accuracy of initial (FP32/FP16) OpenVINO IR? Have you tried to obtain OpenVINO IR with the latest instruction?
Could you please share efficientdet-d0_frozen.xml and efficientdet-d0_frozen.bin files?

[lisosia]

@l-bat @MaximProshin

have you measured accuracy of initial (FP32/FP16) OpenVINO IR?

model	coco_precision	Throughput
FP32	31.93%	33.79 FPS

Could you please share efficientdet-d0_frozen.xml and efficientdet-d0_frozen.bin files?

I attaches the zip file
repro_effdet_issue.0627.zip

Have you tried to obtain OpenVINO IR with the latest instruction?

No.
I refered to doc of 2022.3 instead of the doc of 2023.0.
See 01_PREPARE_MODEL.sh for the detail.

[l-bat]

@lisosia could you please try to use latest NNCF version?
I tried to reproduce problem on 240cc24

python ~/nncf/tests/openvino/tools/calibrate.py -c quantization_config.json --impl native
accuracy_check -c accuracy_check.yaml -m efficientdet-d0_quantization/efficientdet-d0_frozen.xml 

coco_precision: 31.33%

python ~/nncf/tests/openvino/tools/calibrate.py -c quantization_config.json --impl pot
accuracy_check -c accuracy_check.yaml -m efficientdet-d0_quantization/efficientdet-d0_frozen.xml 

coco_precision: 30.87%

quantization_config.json

{
    "compression": {
        "algorithms": [
            {
                "name": "DefaultQuantization",
                "params": {
                    "preset": "performance",
                    "stat_subset_size": 300
                }
            }
        ],
        "dump_intermediate_model": true
    },
    "engine": {
        "config": "<path to accuracy_check.yaml>"
    },
    "model": {
        "model": "efficientdet-d0_frozen.xml",
        "model_name": "efficientdet-d0",
        "weights": "efficientdet-d0_frozen.bin"
    }
}

I couldn't reproduce accuracy drop with provided script too.

[lisosia]

@l-bat
Performance degradation is also observed in 240cc24.
Throughput degradation is minor when I retried.

Model	coco_precision	Throughput
v2.4.0	30.68%	61.80 FPS (retry)
240cc24 (pot backend)	30.55%	61.98 FPS
240cc24	24.53%	61.10 FPS

I found benchmark-app -api sync yields more reliable (less variance) results.
The throughput gap between pot/nncf(pot-backend) and latest(native-backend) is small but still visible.

Model	benchmark-app -api sync
v2.4.0	54.24 FPS
ca54404 (pot)	54.22 FPS
ca54404	53.90 FPS
240cc24 (pot)	54.28 FPS
240cc24	53.87 FPS

---

coco_precision: 31.33%

It's weird that FP32 accuracy is different from my result 31.93%

[lisosia]

Note that I hope nncf (native-backend) will yield the same accuracy/throughput with pot-backend
because use_pot=True will be deleted in the future (#1923 (comment))

[l-bat]

It's weird that FP32 accuracy is different from my result 31.93%
The accuracy of the FP32 is the same as your result.

INT8 native:
coco_precision: 31.33%

INT8 use_pot:
coco_precision: 30.87%

[lisosia]

Oh, I misread that. 31.33% was the int8(native) accuracy.
How was the accuracy of FP32 model?

BTW, I'll try nncf/tests/openvino/tools/calibrate.py tomorrow.

[l-bat]

FP32 accuracy is 31.93%.

Why do you use has_background: False parameter in accuracy_check.yaml?
I could reproduce the degradation in accuracy on colab, but it still works fine locally.
On colab with PERFORMANCE preset I obtained default_coco_precision: 30.98% for use_pot=False.

[l-bat]

@lisosia Could you please try to use

advanced_parameters=AdvancedQuantizationParameters(overflow_fix="enable")

?
I think the issue is only reproducible on machines without VNNI support.

[lisosia]

@l-bat

Why do you use has_background: False parameter in accuracy_check.yaml?

Regarding dataset settings, I copied the following sample

• https://github.com/openvinotoolkit/open_model_zoo/blob/c7a13a842d41333397ea6c3f9bc5a7053da00eec/models/public/efficientdet-d0-tf/accuracy-check.yml#L7C15-L7C60
• https://github.com/openvinotoolkit/open_model_zoo/blob/c7a13a842d41333397ea6c3f9bc5a7053da00eec/data/dataset_definitions.yml#L119

Could you please try to use advanced_parameters=AdvancedQuantizationParameters(overflow_fix="enable")?
I think the issue is only reproducible on machines without VNNI support.

I use Intel(R) Core(TM) i7-8700 CPU, which does not support VNNI I think.

Model	coco_precision	benchmark-app -api sync
FP32	31.93%	36.65 FPS
v2.4.0	30.68%	54.24 FPS
240cc24 (pot)	30.55%	54.28 FPS
240cc24 (native)	24.53%	53.87 FPS
240cc24 (pot, overflow-fix=enable)	30.55%	-
240cc24 (native, overflow-fix=enable)	24.27%	53.92 FPS
240cc24 (native, PERFORMANCE)	30.98%	53.53 FPS

overflow-fix="enable" didn't make significant effect.
native+PERFORMANCE yields 30.98%, which is the same with you. (the FPS is a little more slower, that is strange).

[l-bat]

But you provided config with dataset: name: ms_coco_detection_91_classes https://github.com/openvinotoolkit/open_model_zoo/blob/c7a13a842d41333397ea6c3f9bc5a7053da00eec/data/dataset_definitions.yml#L72-L76

[lisosia]

@l-bat
Sorry for the confusion. It should have been named ms_coco_detection_90_class_without_background.
While looking for a working config, the name seems to have been inappropriate.
I didn't use the --definitions option of accuracy-checker, so the name wouldn't affect the behavior I believe.

[lisosia]

@l-bat
From the above, the following problems were identified at the latest NNCF

• native+preset=MIXED: the accuracy and throughput is worse than pot/nncf240
• native+preset=PERFORMANCE: accuracy is good, but throughput is even worse than native+preset=MIXED (this is not expected)

We hope these problems will be corrected.

[l-bat]

@lisosia We currently working on ChannelAlignment algorithm which should improve accuracy on your machine.
Could you please verify changes at #1928?
Please enable channel_alignment in your code:

advanced_parameters=AdvancedQuantizationParameters(disable_channel_alignment=False)

[lisosia]

@l-bat
sorry for the late reply. I tested commit 76d6b3a.

Model	coco_precision	benchmark-app -api sync
FP32	31.93%	36.65 FPS
v2.4.0	30.68%	54.24 FPS
240cc24 (pot)	30.55%	54.28 FPS
240cc24 (native)	24.53%	53.87 FPS
240cc24 (native, PERFORMANCE)	30.98%	53.53 FPS
76d6b3a (pot, disable_channel_alignment=False)	30.56%	54.17 FPS
76d6b3a (native, disable_channel_alignment=False)	30.73%	54.15 FPS
76d6b3a (native, PERFORMANCE, disable_channel_alignment=False)	30.95%	53.33 FPS

76d6b3a (native, disable_channel_alignment=False) yields almost same accuracy and throughput compared to v2.4.0.
Thank you!

76d6b3a (native, PERFORMANCE, disable_channel_alignment=False) result is still strange (better accuracy but slower than preset=MIXED).