Spark in me - annual retrospective 2018

Spark in me - annual retrospective 2018

Posted by snakers41 on December 29, 2018

Buy me a coffeeBuy me a coffee

Become a PatronBecome a Patron


Spark in me - annual retrospective 2018


Today’s buzzword - disenchantment and reality.
Turn on this track and this track for the mood.
I decided to make this one a bit more emotional / realistic.

A bit of harsh words and in the face attitude.
Do not get your opinion in a twist.

Yeah, I probably should have switched to Russian, but only me and my cat (which I do not have because of allergy) will read this, so who cares.

И да, не хочу никого обидеть, но если вы где-то тут узнали себя (в плохом смысле), то конечно же срочно напишите мне об этом!

Также я заранее понимаю, что по-английски “I am very dissapointed” на русский переводится гораздо веселее.


TLDR



Mood swings this year. Ranging from this …




To this


Following my last post and drooling over the popularity of vas3k’s annual posts (here and here - it is his 10 year anniversary and this is only my second post) I made just a few bullets to save your time:

  • Got thoroughly acquainted with a new domain - NLP (see details below);
  • Last year’s retrospective was a bit random. This one probably will just as random, but it will be more structured xD;
  • As I did more and more public stuff I grew more and more disappointed in people in general. It does not mean it’s not worth it. It’s worth because you meet amazing people, but they are in minority. It just means having no illusions;
  • Decent people also usually are silent, because of the previous bullet;
  • I genuinely have no idea how we continue to develop as a species. I guess the oscillations cancel out and the net trend is upward;
  • Having no intention of making my content more mainstream and “popular” I turned down 2 or 3 “ad deals” this year. Because the deals itself were trash, their content was maybe ok in one case, and because integrity matters. We even trolled a bit, which was fun becase the menagers selling their “content” had no idea what they were doing. The funniest case was when some unnamed educational company selling their courses for US$3k was offering … pro bono ads in their online channels with real readership … of around zero people and were very bullish about it;
  • I / we (honestly, I am not a fan of Anglo-Saxon / American tradition of bragging like hell about the most insignificant bs, doing something is always a team effort - regardless this team being a literal team or your closest people) made a lot of progress in the field of ML this year:
    • Worked on 3 full-time jobs, finally landed on a job that arguably is the BEST on this market (CIS) in this field. Value cohesion and integrity matter;
    • Successfully jump-started a DS / ML team from scratch (I feel that I did this several time during competitions / personal projects, but who cares) in a corporate setting;
    • Built 2 or 3 (depends on how to count) dev boxes this year;
    • Participated in 10+ ML competitions this year, achieved some decent results in 5 (?) of them (1st place - CFT 2018, 2nd place MCS 2018, 7th SpaceNet3, 3rd Jungle video competition, top 1% solo in Airbus 2018 ship detection challenge, some high percentile in DS Bowl given a total lack of stacking) - and almost did not employ cringe tactics (2-3+ level stacking, leak exploitation, etc);
    • SpaceNet 4 was a misstep - I believed my team to be mature enough (so I refrained from actively writing code / competing) to get in top5 with only minor guidance, but due to some issues, they got only in top 9. I understand this is a very good result, but alas only victors write the history. For me as a manager it is a lesson that probably you should apply not only a carrot, but a stick sometimes;
    • Learnt to code properly a bit in Python (pythonic code, whatever), write proper commits and such stuff;
    • Did not yet dip into machine orchestration / active learning / auto ML / queues and stuff yet. I guess in 1Q 2019 there will be a project with active learning xD;
  • More or less built 3 real-world “in the wild” production ready / as-is deployable models this year (not counting countless times when EDA / exploration itself was a product in itself, but useful only as a chain of pipeline);
  • Even though did not build any production ready models on one of the jobs, I found a critical bug, the elimination of which added 5-10% to their core accuracy as-is, which was hilarious for me - I found this just out of curiosity. Always be curious! Ask why and wtf! Ask stupid questions!. You will be amazed at answers;
  • Almost lived in ~3 countries;

I Telegram channel itself

The channel is growing. But I continue to post only what inspires me.

The curve looks like a generic log-curve. I.e. in the long run it achieves infinity, albeit so much slower than y = x. Meh.

Maybe I will have to change the platform soon or seriously expand to a new platform. I do not know. I played with Twitter a bit. It will work, when / if my blog gets tractions in the US, which is 1000 harder in terms of competition. And I do not want to produce cringe content like “how to earn US$1bn in 3 seconds with these 10 awesome ML models!”.

TLDR:

  • There are probably ~100k-300k ML people / data scientists in the world;
  • The vast majority of them do no use Telegram;
  • The majority of (tech) people in general (tech has been compared to religion sometimes, lol) naively buy into one or another form of bs (i.e. TensorFlow is cool mantra) or just follow herd mentality;
  • 100k-300k * 0.1 * 0.5 = around 10-30k people, which is the ceiling readership of such a channel;
  • Alas, the popularity of channels written by people WHO DO NOT KNOW WHAT THEY ARE DOING is staggering. Best examples:
    • All “X is sexy / cool!!! channels”;
    • All “from zero to hero in 5 seconds content”;
    • Just any sort of visual / consumer porn;
    • These people do not answer 3 main questions - как, зачем и почему (in English all of them translate to why, but they roughly mean by which means, for which purpose and because of which reasons)

Just do not repost and spread rubbish. Seriously. Follow some information etiquette. You do not litter outside, right?

The thing with my channel is:

  • I post only articles / news / papers - that I (i) either have read (ii) are endorsed at least by 2-3 people I respect and trust (iii) are of quality and / or their values correlate with mine (iv) something I did or checked that it works / is tractable / usable;
  • I sift the content by the simple rule - if I find it useful enough for me to use somehow in my work - I post it;


Channel subscribers


Average post coverage / related to subscribers

Yeah, if you saw the chart with monthly posts - you would see a different story - I post less and less, in bursts. That is because I like sharing useful / novel / great things - and there is saturation / decreasing returns whatever you do.

The channel has always been a source of inspiration and a means of meeting new cool people with real opinions. And it serves this purpose well, for which I am grateful =).

In terms of blog posts - the number is more or less stable.

YearTelegram postsBlog - my postsBlog - digestsBlog - other people
20162277
2017 (*)14365111
201855023136

(*) In 2017 we had a lot of post series with 3-4 parts. The deflated number would be around 35 posts. Also note that for the majority of 2017 I did not really “work”.

I also more or less failed to do 2 things:

  • Bring SEO traffic to my blog. It is too niche;
  • Bring a lot (10+) of authors;

Telegram disenchantment and the path of all media


Reality is a harsh mistress

Surprise surprise. All media and technologies follow the same cycle.

For media you easily spot the parts of this cycle by the types of ads you can see on the platform. I believe that Telegram now is somewhere in the Trough of Disillusionment zone. The freedom of speech and thought in interesting communities is still staggering in Telegram, though.

Even ODS (varies by chat wildly, but whatever) has some strong internal self-censorship rules. I would argue, that among all the major chats, only #nlp + small thematic channels lately have had the spirit of free cultural and technical exchange.

II Freedom of speech in 2018


Make your time count

If you live in the CIS, probably this is not news for you.
If you live in Europe or on the Americas, you probably have heard of the Arab spring and that freedom of speech took a great hit in surveillance states lately. All this stuff with Google building censored search for China, Internet freedom, militarization of AI etc.

They always try to protect us from some enemies. But the reality is more harsh - in modern times there are hardly external enemies - normal people just do not give a crap about politics. They only enemies are internal - consumer society and laziness.

But this hardly matters. What matters is how you take it and what you do with your time.

Helping people have their voice

All people deserve the right to HAVE a voice. It is a different thing whether and how they should exercise this right. It is a different thing whether some people should be heard at all. Each person can learn complex advanced calculus. The only difference is how much time he/she will require.

Today the general crowd becomes the judge of merit in the online society which leads to all sorts of cringe (just look at recent Google’s YouTube “adpocalypse”, or just watch some popular music video on YouTube). But it is a different thing to entirely be deprived of access to free information in the Internet era. It is also very funny, because in Moscow ~1 Gb/s uplink/downlink costs US$10 per month (compare to American prices, or prices in the UK).

But in reality the usual outcome of such government’s anti-human measures, done incompetently, is just an increase in technical savviness of the general folk.

Anyway this year, we played with sock5 proxies (here and here) and even played with Open VPN a bit.

The idea behind these posts was to encourage all people to actively choose a way to make sure that (i) they free have power to choose which information to use (ii) the technical services they use are adequate (i.e. not some random free proxies, not some unknown people with ads all over the Internet reselling OpenVPN for US$100 with “a unique deal for 3 years”).

Each person has a right to choose how to handle his online security and should not be forced to pay some vultures in the process.

In doing so, I earned some Digital Ocean credits. 411 people used the link, and over 120 used DO afterwards. I wonder how many people just did not use DO. Probably 10x.

There were some downsides, though:

  • My post was banned from geektimes.ru without any apparent reason (even though it gained positive consensus in the end) and then was stolen by internet marketers, who refused to delete it and then - говоря по русски обложили меня;
  • I got acquainted with a nice guy behind this channel, he was rude at first (as all of the IT people with the attitude of over-privileged assholes), but the he even publicly apologized (geektimes article is down);
  • Overall, many people just wrote me via email / telegram and thanked for a plain down-to-earth explanation;

All in all - managing your servers with Linux is easy, you just need some patience to read some manuals and how-to articles.

III Raising visibility, doing publications properly, my path this year

Well, ideally ideas should be popularized as follows:

  • There is some challenge. First you read all you can swallow;
  • You solve the challenge;
  • You understand which part of what you did is YOUR contribution;
  • You publish your code, write accompanying articles, tell people of your solution, contribution, path and why it is unique (or not);
  • You publish it to all media channels you can find / own;

Do not want to copy-paste everything I included here, but this year my publication story was as follows:

  • I switched to PyTorch end of last year;
  • At the beginning of year I learned how to build custom models (not just your Imagenet classifier stacking) for video / satellite imaging and even got into top-10 in a prestigious competition;
  • I learned a bit about VAEs and GANs and came to conclusion that in real work they are not yet useful for complicated domains (or are just very difficult to train);
  • Reality is a harsh mistress, and probably this is the last post in Russian in my “looking for a job series”;
  • We played with time series (and ofc used all the available models, perhaps except for TCNs and Trellis networks, of which I learned later), and even got high in the leaderboard - but our consensus was that either the competition was leaky or we just did not get something or had a random error somewhere (2 people with different frameworks and code);
  • It was an interesting experience to get a medal wo stacking in a DS Bowl 2018 competition - it was funny - I just forgot to properly unfreeze my encoder;
  • I even tapped a bit into RL (I did not gather strength to participate in some competition) and adversarial attacks;
  • I also played with understanding of proper sampling and using domain information in semi-supervised way;
  • I failed to replicate one of Google’s papers I needed at that time, but a friend managed to get 73%+ (maybe he will share later);
  • The end of year was dedicated to NLP, where me and my team managed to do a lot of things within a short time span, both in supervised and unsupervised fashion:
    • This is the end result of our work during 2-3 months - input agnostic classifier robust to broken input + some hands on research + some ideas on building unsupervised clusters;
    • Shared some pre-trained vectors in ODS;
    • Learned how to work with Wikipedia and Common Crawl;
    • Winning a seq2seq competition also did not hurt;
    • Shared a lot of NLP insights on the channel;
  • At the end of the year I even was accepted to be an author on Towardsdatascience - the most popular English DS blog. To be honest, the majority of its content is decent, albeit there are a lot of superficial articles;
  • In 2018 Q4 habr.com finally became multi-lingual. As I expected - as a platform for publications this website is not interesting anymore (just read the comments), for several main reasons (despite me having a top? account there):
    • After a merge with geektimes (reverse merge, it was separated some time ago) all feeds are now dominated by “everything is awesome” posts;
    • Some professional communities are highly toxic and averse to new things (i.e. admins);
    • In new spheres - there is no expertise there but toxicity is already present, i.e. NLP;
    • New IT content is mostly published in watered down paid corporate blogs - I guess the revenues of TM are hitting all time highs - you are right - if you publish on a personal account - you are subjected to all kinds of two-faced crap (i.e. be our free author) - but if you pay them you can post any bs wo moderation;
    • I have seen only 2-3 quality articles there during last 3 months, despite that fact that I read daily their digests / google technical issues with habr keyword frequently - meh;

IV What about work? And why I love my current job and my team


Just stole this from here - make your time count

Just because people just set goals, and work together to achieve them.
No politics, no bs, no corporate lies, no false promises (at least that I know of).

If you read my blog, you know that I am quite an anal person (if you read the blog). I worked hard to spot anything murky (not in legal terms - but in terms of promised / delivered) - and could not.

V “Conquering” new domains

A short list of new domains to “conquer” (or just get familiar with, or maybe to build a new deployable model?):

  • Machine text understanding, i.e. datasets like SQUAD, but in more chat-like setting in Russian;
  • Siamese / triplet networks with all sorts of clever cosine losses for computer vision / NLP - moreover this is highly deployable and usable in practice;
  • Build a working active learning pipeline, build some orchestration / queue or scheduling / technical solution for managing a system of several ML components (i.e. deployable model, automatic re-training, automatic re-deploy);

VI Great inspirations this year

  • NLP in general and unsupervised NMT by FAIR;
  • FAIR’s products and papers I used / examined a bit (FastText, FAISS, FAIRseq, PyTorch). Did not get PyText though;
  • PyTorch 1.0, its distributed wrappers;
  • Finally Bitcoin bubble was burst - cheap second hand GPUs!;
  • Embedding bags for RNNs, advanced training regimes for neural networks;
  • Learning about chainer and its influence on PyTorch;
  • UMAP and HDBSCAN;

VII Greatest disappointments of this year

  • SpaceNet 4 (mentioned above);
  • Computer vision in general getting industrialized and militarized;
  • ML community in general;
  • New Nvidia GPUs - (at least in terms of value for money) - only an incremental improvement (I guess nv-link);
  • General mood in the CIS;
  • Tightening Internet censorship;
  • I understood this year (maybe too late), that too many gate keepers’ processes are well-dressed vanity fairs;

VIII Random cool stories

50% of amazing things, that happened to me this year, happened at random.
There was little agency in them, but a lot of agency to get in the situation where they could happen.

IX Long path of self-improvement / not losing hope in future, after all*

TLDR:

If you value rationality, then you will doubt yourself. Then you will doubt everything. Unless you pave your way with optimal and rational decisions, you may follow a self-destructive “what-if” spiral. If you make an effort always to make the best / most rational / most optimal decision - the you will have less doubts. You will have to revisit your theories sometimes.

After all there are only following ways to know something:

  • Deduce it logically;
  • Test it empirically;
  • Test and doubt everything all the time;
  • Employ some large scale graph based trust-based self-tuning mechanism (i.e. the standard broken window parable);

And in the society always exist structures who will exploit this doubt. Just be yourself, do your best and continue struggling. The world is big and full of terrors as well as wonders.