PyTorch

PyTorch 1.1 release

• Tensorboard loggers (beta);
• DistributedDataParallel new functionality and tutorials;
• Multi-headed attention block;
• EmbeddingBag enhancements: from pretrained and trainable weights;
• Other cool, but more niche features:
  • nn.SyncBatchNorm
  • optim.lr_scheduler.CyclicLR

STT

Apply image augmentations to spectrogram;

NLP

A set of wrappers to load all of the latest trendy huge models;
Word2Vec visualized. An article from the amazing “Illustrated X” article series;

An idea from fast.ai:

• Crappify an image. Train a UNET to fix it;
• It converges quickly;
• Replace with some generative loss / GAN;
• Other ideas - self-attention, pre-trained UNET;
• NoGAN training:
  • Pretrain the Generator;
  • Save Generated Images From Pretrained Generator;
  • Pretrain the Critic as a Binary Classifier;
  • Train as GAN;
• This also stabilizes videos;
• Nice notes on ML bias;
• Sparse Transformers from OpenAI;
• New epoch in CPU design?
• Transformers can also learn music;
• Google intends … to kill SMB calls market?;
• Controversial topics about medical AI:

No medical advance is going to be achieved by a team who has designed a fancy new model for the task.

• Some current remarks on unsupervised modelling;
• Training transformers on 128 GPUs;
• New MobileNet;

Datasets

• Google landmarks 2019 - 5m images, 700k landmarks;
• Open images v5 - now it boasts semseg data as well;
• Russian Open Speech To Text (STT/ASR) Dataset;

Papers

Cool paper about automated augmentations. Key idea - sample unlabeled images, corrput them, use KLD loss to enforce that the model produces the same label.