Hello everyone!
This tutorial is dedicated to a beautiful technique called "Invariant Information Clustering" (IIC) introduced here. With its help one can can encounter many tasks, such as:
- Image (and other data types) clustering
- Unsupervised image segmentation
- Weakly supervised classification
- Encoder training
- and many others ...
According to paperswithcode.com, this method showed several SOTA results, therefore I believe it's worth considering.
This tutorial covers a very simple setting this technique is applicable to -- the clustering and weakly-supervised classification of MNIST. However, after solving this problem it's easier handle more complicated tasks and focus on the details, but not on the method itself.
This tutorial consits of three parts. The first part is a method overview -- here we state the problems, refresh some general DL concepts and see how IIC approach is organized. If you are willing to cover this topic in depth or see the applications to other problems, i.e. unsupervised segmentation, it's highly recommended to read the original paper or to visit official repo.
The second part describes a notion of mutual information and shows, how can it be estimated in IIC method. To enable TeX formulas it's written in Jupyter notebook. You can open it both through github or through Colab, but the latter one is better in rendering TeX .
The third part shows the implementation and focuses on applied nuances. It is also made in Jupiter, so you can run it in colab or locally and try IIC yourself.
- Part 1: Method overview
- Part 2: IIC loss function
- Part 3: Practice
If you are running part 3 locally, it's better to install the libraries using the requirements.txt file. It can be done with the following line:
pip install -r requirements.txtAfter this restart the jupyter runtime and begin exploring.