Instance Segmentation Mask/Bbox Relation

[FrsECM]

Describe the bug

I work on a usecase of instance segmentation with torchvision. In this case, i have :

• image
• bboxes
• masks
• labels
  I'have created an augmentation set with albumentation, something like that :

def augmentations(instance_item:dict):
    transform = A.Compose([
        A.LongestMaxSize(max_size=MAX_SIZE),
        A.PadIfNeeded(
                min_height=MIN_IMG_HEIGHT,
                min_width=MIN_IMG_WIDTH,
                border_mode=cv2.BORDER_CONSTANT,
                value=0,
                always_apply=True),
        A.HorizontalFlip(),
        A.RandomCrop(MIN_IMG_HEIGHT,MIN_IMG_WIDTH),
        A.ToFloat(max_value=255),
        ToTensorV2()
    ],
        bbox_params=A.BboxParams(format='pascal_voc',label_fields=['labels'],min_visibility=BBOX_MIN_VISIBILITY),
        is_check_shapes=False
    )
    output = transform(
        image=instance_item['image'],
        masks=instance_item['masks'],
        bboxes=instance_item['boxes'],
        labels=instance_item['labels'])
    return output

I would expect that the parameter related to the visibility is applyable at the instance level.
I mean if the bbox is not enough visible, it should remove the corresponding mask.

I tried adding "masks" in the bbox_params :

A.BboxParams(format='pascal_voc',label_fields=['labels','masks'],min_visibility=BBOX_MIN_VISIBILITY),

but in that case it does not apply augmentation on the masks.

To Reproduce

In order to reproduce, you can use the code bellow :

import albumentations as A
import numpy as np

img = np.zeros((2048, 2048, 3), dtype=np.uint8)

# Format des bbox : [x_min, y_min, x_max, y_max]
bboxes = [
    [800, 800, 1200, 1200],  # bbox à l'intérieur du recadrage
    [1500, 1500, 1700, 1700] # bbox à l'extérieur du recadrage
]
labels = [0,1]

masks = [
    np.zeros((2048, 2048), dtype=np.uint8),
    np.zeros((2048, 2048), dtype=np.uint8)
]
masks[0][800:1200, 800:1200] = 1
masks[1][1500:1700, 1500:1700] = 1

aug = A.Compose(
    [
        A.CenterCrop(1024, 1024)
    ],
    bbox_params=A.BboxParams(format='pascal_voc', label_fields=['labels'],min_visibility=0.3),
)

# Apply Transformation
augmented = aug(image=img, bboxes=bboxes, masks=masks, labels=[0, 1])

# Get back results
print(len(augmented['bboxes']))
print(len(augmented['labels']))
print(len(augmented['masks']))
# Returns
1
1
2 => Should be 1

Expected behavior

I would expect that the parameter related to the visibility is applyable at the instance level.
I mean if the bbox is not enough visible, it should remove the corresponding mask.

Actual behavior

If i do not add masks to label_fields, i have inconsistency.
If i do add masks to label_fields, augmentations are not applyed to it.

[FrsECM]

A workarround i've find :

def iris2_training(segmentation_item:dict):
    transform = A.Compose([
        A.LongestMaxSize(max_size=MAX_SIZE),
        A.PadIfNeeded(
                min_height=MIN_IMG_HEIGHT,
                min_width=MIN_IMG_WIDTH,
                border_mode=cv2.BORDER_CONSTANT,
                value=0,
                always_apply=True),
        A.HorizontalFlip(),
        A.RandomCrop(MIN_IMG_HEIGHT,MIN_IMG_WIDTH),
        A.ToFloat(max_value=255),
        ToTensorV2()
    ],
        bbox_params=A.BboxParams(format='pascal_voc',label_fields=['labels','ids'],min_visibility=BBOX_MIN_VISIBILITY),
        is_check_shapes=False
    )
    output = transform(
        image=segmentation_item['image'],
        masks=segmentation_item['masks'],
        bboxes=segmentation_item['boxes'],
        labels=segmentation_item['labels'],
        ids=range(len(segmentation_item['labels']))
    )

    return dict(
        image=output['image'],
        boxes=output['bboxes'],
        labels=output['labels'],
        masks=[output['masks'][i] for i in output['ids']],
        name=segmentation_item['name']
        )

But it should be working without it.

[ternaus]

Thanks for the proposed solution!

Yep, we do have this issue that masks, boxes and keypoints and not binded on the instance level.

#1716

Your approach is the best that I have seen so far for this problem.