Patches Are All You Need? 🤷

This repository contains an implementation of ConvMixer for the ICLR 2022 submission “Patches Are All You Need?” by Asher Trockman and Zico Kolter.

The tweetable version of ConvMixer, which requires from torch.nn import *:

def ConvMixer(h,d,k,p,n):
 S,C,A=Sequential,Conv2d,lambda x:S(x,GELU(),BatchNorm2d(h))
 R=type('',(S,),{'forward':lambda s,x:s[0](x)+x})
 return S(A(C(3,h,p,p)),*[S(R(A(C(h,h,k,groups=h,padding=k//2))),A(C(h,h,1))) for i in range(d)],AdaptiveAvgPool2d(1),Flatten(),Linear(h,n))

Cite

@article{trockman2022patches,
  title={Patches Are All You Need?},
  author={Trockman, Asher and Kolter, J Zico},
  journal={arXiv preprint arXiv:2201.09792},
  year={2022}
}