Fine-tune a YOLOv8 model to detect Pikachu without worrying about the training data.
3D Pikachu modeling, synthetic data generation with Unity Perception.
YOLOv8 fine-tuning exclusively on synthetic data.
The main problem when training a detection model, or generally when dealing with machine learning models, is the data. The goal of this project is to train an object detection model (YOLOv8) without manually creating every training image and annotation. Instead, we create a 3D model of the object and we exploit the Unity Perception package to automatically generate several images and annotations.
The overall pipeline looks as follows.
- Use MagiScan 3D to make the 3D model of Pikachu. This model is in a raw format and needs some cleaning.
- The raw 3D Pikachu model is post-processed in Blender, where small noises are deleted.
- Export the 3D model and its texture, and move it to Unity, where a Perception project needs to be set up.
- Generate the detection data. Afterwards, modify the generated coordinates to match the YOLOv8 format.
- Finally, train the YOLOv8 model solely with synthetic data.
