The ir model does not deduce a target greater than 0.98

[tonycc521]

OpenVINO Version

openvino2022.3

Operating System

Windows System

Device used for inference

CPU

Framework

ONNX

Model used

yolov8

Issue description

I have a picture that the inference score of onnx model is 0.98, but the recognition score after converting to IR model is only 0.025
XML.zip

Step-by-step reproduction

No response

Relevant log output

No response

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[andrei-kochin]

Hello @tonycc521, Thank you for reaching the OpenVINO!

From the IR you've shared I've seen that you've used the NNCF for quantization and your IR is quantized. Have you followed some instruction or made it on your own? We have a pretty good tutorial regarding yolov8 optimization https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks/yolov8-optimization. Have you checked that?

To compare results could you please share how was the original ONNX model got? Have you got it through the Ultralytics export?

Thank you!

[tonycc521]

Hello! We used YOLOV8's int8 quantization method to deduce this image with onnx and xml respectively, the former score is 0.98, the latter is 0.028. The version we are using is openvino 2024.1.0 nncf 2.8.0

[tonycc521]

We deduced another picture using onnx and xml, and the results were 0.83 and 0.86. We guess if there is a case where the result is more than 1, then the data after the decimal point will be displayed

[andrei-kochin]

YOLOV8's int8 quantization

Which exact method have you used? could you please share the exact steps made to generate ONNX INT8 and IR INT8 models?
Best way would be to share the python code reproducer if possible. As currently it is hard to follow what is exactly wrong and what to compare with.

Thank you!

[tonycc521]

model = YOLO(r'C:\Users\nuc\Desktop\222\best.pt')
model.export(format='openvino', int8=True)

@try_export

# def export_openvino(self, prefix=colorstr("OpenVINO:")):
#     """YOLOv8 OpenVINO export."""
#     # check_requirements("openvino>=2024.0.0")  # requires openvino: https://pypi.org/project/openvino/
#     import openvino as ov

#     LOGGER.info(f"\n{prefix} starting export with openvino {ov.__version__}...")
#     assert TORCH_1_13, f"OpenVINO export requires torch>=1.13.0 but torch=={torch.__version__} is installed"
#     ov_model = ov.convert_model(
#         self.model.cpu(),
#         input=None if self.args.dynamic else [self.im.shape],
#         example_input=self.im,
#     )

#     def serialize(ov_model, file):
#         """Set RT info, serialize and save metadata YAML."""
#         ov_model.set_rt_info("YOLOv8", ["model_info", "model_type"])
#         ov_model.set_rt_info(True, ["model_info", "reverse_input_channels"])
#         ov_model.set_rt_info(114, ["model_info", "pad_value"])
#         ov_model.set_rt_info([255.0], ["model_info", "scale_values"])
#         ov_model.set_rt_info(self.args.iou, ["model_info", "iou_threshold"])
#         ov_model.set_rt_info([v.replace(" ", "_") for v in self.model.names.values()], ["model_info", "labels"])
#         if self.model.task != "classify":
#             ov_model.set_rt_info("fit_to_window_letterbox", ["model_info", "resize_type"])

#         ov.runtime.save_model(ov_model, file, compress_to_fp16=self.args.half)
#         yaml_save(Path(file).parent / "metadata.yaml", self.metadata)  # add metadata.yaml

#     if self.args.int8:
#         fq = str(self.file).replace(self.file.suffix, f"_int8_openvino_model{os.sep}")
#         fq_ov = str(Path(fq) / self.file.with_suffix(".xml").name)
#         if not self.args.data:
#             self.args.data = DEFAULT_CFG.data or "coco128.yaml"
#             LOGGER.warning(
#                 f"{prefix} WARNING ⚠️ INT8 export requires a missing 'data' arg for calibration. "
#                 f"Using default 'data={self.args.data}'."
#             )
#         # check_requirements("nncf>=2.8.0")
#         import nncf

#         def transform_fn(data_item):
#             """Quantization transform function."""
#             assert (
#                 data_item["img"].dtype == torch.uint8
#             ), "Input image must be uint8 for the quantization preprocessing"
#             im = data_item["img"].numpy().astype(np.float32) / 255.0  # uint8 to fp16/32 and 0 - 255 to 0.0 - 1.0
#             return np.expand_dims(im, 0) if im.ndim == 3 else im

#         # Generate calibration data for integer quantization
#         LOGGER.info(f"{prefix} collecting INT8 calibration images from 'data={self.args.data}'")
#         data = check_det_dataset(self.args.data)
#         dataset = YOLODataset(data["val"], data=data, task=self.model.task, imgsz=self.imgsz[0], augment=False)
#         n = len(dataset)
#         if n < 300:
#             LOGGER.warning(f"{prefix} WARNING ⚠️ >300 images recommended for INT8 calibration, found {n} images.")
#         quantization_dataset = nncf.Dataset(dataset, transform_fn)

#         ignored_scope = None
#         if isinstance(self.model.model[-1], Detect):
#             # Includes all Detect subclasses like Segment, Pose, OBB, WorldDetect
#             head_module_name = ".".join(list(self.model.named_modules())[-1][0].split(".")[:2])

#             ignored_scope = nncf.IgnoredScope(  # ignore operations
#                 patterns=[
#                     f".*{head_module_name}/.*/Add",
#                     f".*{head_module_name}/.*/Sub*",
#                     f".*{head_module_name}/.*/Mul*",
#                     f".*{head_module_name}/.*/Div*",
#                     f".*{head_module_name}\\.dfl.*",
#                 ],
#                 types=["Sigmoid"],
#             )

#         quantized_ov_model = nncf.quantize(
#             ov_model, quantization_dataset, preset=nncf.QuantizationPreset.MIXED, ignored_scope=ignored_scope
#         )
#         serialize(quantized_ov_model, fq_ov)
#         return fq, None

#     f = str(self.file).replace(self.file.suffix, f"_openvino_model{os.sep}")
#     f_ov = str(Path(f) / self.file.with_suffix(".xml").name)

#     serialize(ov_model, f_ov)
#     return f, None

[andrei-kochin]

@tonycc521 thank you! looks like you are using Ultralytics export to get the IR model. Could you please either share the original model if possible so we can compare with the original framework?

Do you see any issues with the original yolov8 from the Ultralytics?

[eaidova]

Hello @tonycc521, I have a couple of questions, could you please answer?

• is model provided in https://github.com/openvinotoolkit/openvino/files/15425236/XML.zip is actual model what you use? If yes, could you please try to update ultralytics and openvino export model one more time (what I see in IR that it was exported using openvino 2023.2 that is slightly outdated and not aligned with your message that you use openvino 2024.1)?
• If not, could you please check, do you see the same issue if export model with int8=False?
• which pytorch and ultralytics versions do you use?
• which code do you use for comparing onnx and openvino models inference? Can be issue with difference in conf be connected with difference pre or postprocessing steps (e.g. providing image in BGR instead of RGB, different resize, different thresholds or conf and iou)

[tonycc521]

pytorch : 2.0.1+cu117
Ultralytics: 8.1.47

from openvino.runtime import Core
import cv2
import numpy as np

ie = Core()
model = ie.read_model(r'C:\Users\nuc\Desktop\222\best_int8_openvino_model\best.xml')
compile_model = ie.compile_model(model=model, device_name='CPU')
input_blob = compile_model.input()

img = cv2.imread(r'C:\Users\nuc\Desktop\222\123.jpg')
h, w, ch = img.shape
scale = min(640 / h, 640 / w)
img = cv2.resize(img, (int(round(wscale)), int(round(hscale))), cv2.INTER_CUBIC)
top, button = int(round((640 - h * scale) / 2 - 0.1)), int(round((640 - hscale) / 2 +0.1))
lt, rt = int((640 - w * scale) / 2 -0.1), int((640 - wscale) / 2 +0.1)
img = cv2.copyMakeBorder(img, top, button, lt, rt, cv2.BORDER_CONSTANT, (144,144,144))
img = img.astype(np.float32)
img = img / 255.
img = img.transpose(2,0,1)
img = np.expand_dims(img, axis=0)

res = compile_model(img)

[tonycc521]

url

[eaidova]

@tonycc521 thank you for sharing new openvino model and inference code that you use. Do you use the same preprocessing when running onnx model that you describe in this message:

We deduced another picture using onnx and xml, and the results were 0.83 and 0.86.

From provided inference code snippet, I see that you did not transpose channels from BGR to RGB in your code, while as I know ultralytics preprocessing code do that https://github.com/ultralytics/ultralytics/blob/main/ultralytics/engine/predictor.py#L125. (opencv read images in BGR channels order, it is important to preserve channels order the same like used during model training, because wrong channels order may lead to incorrect data representation and may affect accuracy) and meta info from provided XML also contains info that this step is required:

<reverse_input_channels value="YES" />

[eaidova]

@tonycc521, if you export mode, without flag int8=True like:

model = YOLO(r'C:\Users\nuc\Desktop\222\best.pt')
model.export(format='openvino')

do you see the same wrong probabilities result? I have suspicious that problem may be in quantization code and usage wrong dataset for calibration instead of specific for your task data.

[tonycc521]

Hello, thanks for the advice! I got the same result with ultralytics as I did with openvino. I also tried to put BRG2RGB and the result was the same. In addition, the results of reasoning onnx model with openvino and XMl model with FP32 are the same. However, if converted to INT8, the score will change, and the score after quantization will be higher.

[eaidova]

Hello @tonycc521, did I understand you correctly, that FP32 model results are the same like for original model and low probabilities issue only present if you use int8?

[tonycc521]

yes

[eaidova]

@tonycc521 could you please try to separate export and quantization steps like demonstrated in this notebook
https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/yolov8-optimization/yolov8-object-detection.ipynb
but using subset from your training/validation dataset?

As I already said before issue with int8 model can be connected with non-representative data used for model quantization (as I understand, your model was trained for detecting some defects on manufactoring , while by default yolov8 trained on coco dataset for object detection). if default ultralytics exporter selects different dataset, it may lead to out of distribution data used and as the result it may lead to unrepresentative quantization statistics and become a reason of inaccurate detection results

[tonycc521]

model = TRAIN(r'C:\Users\nuc\Desktop\222\best.mt')
with open(r'D:\project\M-Trainer_src\M-Train_V2.00\NEW\M-Train\MT\train\cfg\default.yaml', 'r', encoding='utf-8') as f:
  data = yaml.safe_load(f)
model.export(**data)

default.yaml:
model: # (str, optional) path to model file, i.e. yolov8n.pt, yolov8n.yaml
data: D:\MT_WorkSpace\projects\detect_zheng0509\train\data\data.yaml # (str, optional) path to data file, i.e. coco128.yaml
epochs: 1000 # (int) number of epochs to train for
time: # (float, optional) number of hours to train for, overrides epochs if supplied
patience: 100 # (int) epochs to wait for no observable improvement for early stopping of training
batch: 1 # (int) number of images per batch (-1 for AutoBatch)
imgsz: 640 # (int | list) input images size as int for train and val modes, or list[w,h] for predict and export modes
save: True # (bool) save train checkpoints and predict results
save_period: -1 # (int) Save checkpoint every x epochs (disabled if < 1)
cache: False # (bool) True/ram, disk or False. Use cache for data loading
device: cpu # (int | str | list, optional) device to run on, i.e. cuda device=0 or device=0,1,2,3 or device=cpu
workers: 4 # (int) number of worker threads for data loading (per RANK if DDP)
project: C:\Users\nuc\Desktop\222 # (str, optional) project name

Export settings ------------------------------------------------------------------------------------------------------

format: openvino # (str) format to export to, choices at https://docs.ultralytics.com/modes/export/#export-formats
keras: False # (bool) use Kera=s
optimize: False # (bool) TorchScript: optimize for mobile
int8: True # (bool) CoreML/TF INT8 quantization
dynamic: False # (bool) ONNX/TF/TensorRT: dynamic axes
simplify: False # (bool) ONNX: simplify model
opset: # (int, optional) ONNX: opset version
workspace: 4 # (int) TensorRT: workspace size (GB)
nms: False # (bool) CoreML: add NMS

[tonycc521]

Hello! The data set that int8 quantifies is the trained data set, but the result is the same

[andrei-kochin]

@MaximProshin seems like we require some help with quantization of the custom Yolov8 model so assigning to you

Please contact @eaidova for the further details if required

[MaximProshin]

internal ref: 143651
@andrey-churkin , please take a look

[andrey-churkin]

@tonycc521 Hello, thanks for your question. After the quantization process, the INT8 model may have some insignificant accuracy degradation. So it may happen that there are some inputs on which the FP32 model has good results, but the INT8 model
does not. For this reason, it is better to compare the accuracy of FP32 and INT8 models using a representative dataset.

Could you please provide the following information?

1. Metrics of your initial PyTorch model on a representative dataset.
2. Metrics of your model converted to float-precision IR on a representative dataset.
3. Metrics of your model converted to INT8 IR on a representative dataset.

Please note that during export to INT8, a quantization dataset is used. It should also be representative.

[andrey-churkin]

@tonycc521 Do you have any updates regarding my last comment?