Featured Stuff

Training with quantization noise for extreme model compression

Audio noise reduction on Nvidia RTX

https://www.nvidia.com/en-us/geforce/guides/nvidia-rtx-voice-setup-guide/
You can unzip the exe file, there is a model, but looks like it is encrypted (open-ssl is also there)
Typical dick move by Nvidia - release something useful as an exe file, encrypt it, claim it to work only with new RTX GPUs (which is false btw)

NLP

Key insights
- Given a pair of regions that are similar to each other, most corresponding subsets of those regions are also similar
- Lightweight (75k parameter) convolutional architecture, using IR brightness and neighbor information to adjust incorrect matches
- System is self-calibrating
- End-to-end deep learning architecture that enhances the raw uDepth data, inferring a complete, dense 3D depth map

Gradient Centralization: A New Optimization Technique for Deep Neural Networks - http://arxiv.org/abs/2004.01461
Dynamic Relu - http://arxiv.org/abs/2003.10027

Designing Network Design Spaces - http://arxiv.org/abs/2003.13678

Very similar to how I do it, albeit with 1000x more compute
Essentially this is differential evolution of network design in a low-compute,low-epoch training regime with only a single block type
Interesting design principles: (i) shared bottleneck ratio (ii) shared group width (iii) good network have increasing widths
The depth of best models is stable across regimes, with an optimal depth of ∼20 blocks (60 layers)
The best models use a bottleneck ratio b of 1.0 which effectively removes the bottleneck
The width multiplier wm of good models is ∼2.5 similar but not identical to the popular recipe of doubling widths across stages
Inverted bottleneck degrades the quality
SE modules add a good gain
A blog of an open-source search engine - https://0x65.dev/ ?

Some ez tweaks to the transformer architecture for NER - http://arxiv.org/abs/1911.04474

Looks like a very cool set of CUDA based ML libraries - https://github.com/rapidsai

Advancing Self-Supervised and Semi-Supervised Learning with SimCLR

https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html
Contrastive learning
A combination of simple augmentations (random cropping, random color distortion, and Gaussian blur)
ResNet
Two transformations stand out: random cropping and random color distortion

Interesting, new “internal” NN visualizations: