2019 DS/ML digest 19

2019 DS/ML digest 19

Posted by snakers41 on December 23, 2019

Internet

  • Ben Evans about 5G
  • A good piece on toxicity and hypocrisy in IT
  • You will be able to export all of your photos from FaceBook to google photos
  • How to check Russian passports

Code / python

  • Viewing diffs for notebooks
  • Most copied SO answer is a bit wrong and even Oracle does not respect SO license
  • Useful pandas functions

ML

  • Double descent phenomenon occurs in CNNs, ResNets, and transformers
  • AI misinformation
  • Suprise-surprise, FaceBook uses statistical analysis to mine its server logs
  • How FaceBook mines the app problem logs
  • Develop python packages in nbdev in notebooks?
  • Cool idea - use cryptography as a part of your neural network with homomorphic algorithms
  • Reproducibility crisis in ML and science

Datasets

Papers

  • What’s Hidden in a Randomly Weighted Neural Network?

    • Link
    • TLDR - show practically that the lottery ticket hypothesis works on ImageNet
    • Randomly weighted neural networks contain subnetworks achieve impressive performance without ever training the weight values
    • There is a subnetwork of Wide ResNet-50 (with random weights) that is smaller than but matches the performance ofa ResNet-34 trained on ImageNet
    • Edge-popup algorithm
    • Despite the hype that they do not actually “learn” weights, they essentially learn scores to include some weights with something very similar to SGD
    • Also as usual there is no comparsion of the compute required vs. actual training
  • Fast Sparse ConvNets

    • Link
    • Before - use dense “regularized” blocks (separable convs, inverted bottlenecks) => higher efficiency / higher performance
    • A family of efficient sparse kernels for ARM and WebAssembly
    • Efficient implementations of sparse primitives => sparse versions of MobileNet v1, MobileNet v2 and EfficientNet
    • MobileNets => FLOPs and parameter counts in these architectures are dominated by the 1×1 convolutions
    • Fast kernels for sparse matrix-dense matrix multiplication (SpMM) specifically targeted at the acceleration of sparse neural networks
    • Target the sparsity range of 70-95%
    • Reduces inference times by 1.3 − 2.4×, parameter counts by over 2× and number of floating-point operations (FLOPs) by up to 3× relative to the previous generations
    • They use To make the networks sparse we use the gradual magnitude pruning technique