Add video support

[iejMac]

Current plan is to use a ViViT architecture (factorised encoder variant) with the image and text encoder initialized with CLIP weights. We can also opt to train this CoCa style, then we would initialize the image encoder and entire text tower (including decoder) with CoCa weights so we'd likely want to train an L/14 size model.

This PR will likely require the addition/adaptation of:

• vivit.py - the implementation of the ViViT model which would depend on 2 layers of transformers coming from our transformer.py file. It would implement the whole CLIP-like model that would be used in train.py
• data.py - need to add a video dataloader

Will use this PR to track progress. Lmk if there's a better way of approaching this
@rwightman @rom1504 @mitchellnw

[iejMac]

Hmmm I don't see how we can reliably train this though. Let's say for L/14 max local batch size on a 40GB GPU is like a few hundred images so if a video has 100 frames at 1 FPS that gives us a local batch size <10 for rather short videos which probably still won't work very well unless we have more GPU's than we have

[rom1504]

Gradient accumulation maybe ?

…

On Wed, Feb 15, 2023, 12:48 Maciej Kilian ***@***.***> wrote: Hmmm I don't see how we can reliably train this though. Let's say for L/14 max local batch size on a 40GB GPU is like a few hundred images so if a video has 100 frames at 1 FPS that gives us a local batch size <10 for rather short videos which probably still won't work very well unless we have more GPU's than we have — Reply to this email directly, view it on GitHub <#430 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437QTXKWHS75KDPHW6OTWXS7CFANCNFSM6AAAAAAU4WTXDE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[lucidrains]

@iejMac nice! i can contribute to this

i believe for video, we can do much more aggressive patch dropout in the beginning. well, if the video does not resemble this lol

[lucidrains]

[iejMac]

@lucidrains cool! I'll start filling out the code a bit today. And yeah good idea with aggressive patch dropout. So currently we have:

• aggressive patch dropout
• grad accumulation

as some tricks to make this a bit more tractable. Anything to maximize batch size here will be really important.

[iejMac]

ah also another thing I want to add here: simultaneous (image, video) - text training. I.e. the final model should be able to both handle temporal sequences and static images. The thing I'm unsure about is if we apply the image loss after the temporal decoder essentially saying "the time transformer should understand singular images" or before, just at the spatial transformer side

[iejMac]

@lucidrains, does anything look wrong to you in the modeling code? Specifically this ViViT class. I'm getting very strange loss curves and was wondering if you might have an idea.

[iejMac]

could also be the data loader code but I figured I'd ask you about the model since I'm comparing with your vivit

[lucidrains]

@iejMac nice! i'll do a code review later this week when i find some downtime

[iejMac]

Next task - initialize spatial and text transformer from CLIP model

[iejMac]

Next things to do:

• Benchmarks benchmarks benchmarks
• https://arxiv.org/pdf/2303.15343v1.pdf
• aggressive patch dropout