Cambrian explosion in AI: Sunday night musings

I don't know about you, but my LinkedIn and Twitter feeds are overflowing with excitement, opinions, prognostications, and at times, a more serious technical content attempting to discern the implications of OpenAI's work and that of others since the launch of ChatGPT.

I didn't plan to offer another opinion, but the amount of content has begun to affect me as well, so I thought I would offer some musings on the subject - if only to organize my own thoughts. I'm not sure this will even age well.

To be fully honest, I am probably as conflicted as you are. On the one hand, seeing so much excitement, whether informed or not, inflated with an outsized amount of marketing or not, is great as it will drive the resources, market education, and adoption that we need. In fact, this could be our chance to change the trajectory of the economy for the better in the long run.

On the other hand, the amount of ongoing FOMO and seemingly unlimited quantity of born-again AI experts reminds me of the .com bubble, with all of its requisite dangers (such as drowning good ideas and teams in a sea of bad ones).

We must try to keep both of these opposing thoughts in our heads at the same time.

Let's try to do that by re-examining the ground truth, if you will. What actually occurred? (Note: I am going to use some gross simplifications in the interests of brevity.)

In 2017, Google introduced the Transformer architecture for NLP problems, such as sequence prediction (i.e., when you ask ChatGPT to answer a question). What did it do that various RNN architectures with attention couldn’t do before? Nothing accuracy-wise, really, except one thing: the attention mechanism itself turned out to be sufficient, and one could dispense with the whole idea of sequential processing, making it possible to train large models effectively.

Before this current Cambrian explosion, we had sort of the same thing occur with CNNs with respect to visual processing, enabling autonomous driving and other fields relying on efficient image processing. It's amazing how simplification often drives change.

So, what do we have so far? Companies like OpenAI, Microsoft, and others with requisite resources (yes, it is very expensive) are now able to build large, useful NLP systems relying on vast amounts of public data and private data they either own or can access through their customers. It's pretty straightforward business applications and product enhancements from there. Needless to say, the one who has the most differentiated data in size will, at least as a starting point, have the most advantage and from there it’s up to network effect gods.

A twist I see here, however, is that the outcome of the model will depend on the quality of the data. To illustrate this point better: Github CoPilot could do a halfway decent job because it was trained on presumably mostly "correct" code that at least compiles and hopefully works. The same may not be true in other fields - because there are no "correct" answers in, for example, economics. So, we are back to tuning the model either by prompt engineering (human labeling in effect), model architecture, or biasing the data (i.e., weighting that paper or author as more credible than the other one).

That leaves us with an already useful workflow tool that is augmented such that the human becomes a what would have been called in the old days a labeler. The same goes for new tools (i.e., a legal contract review system that redlines contracts automatically but needs an attorney to check and choose the best option. I'd say why not multiple attorneys junior associates, and then the model decides who's performance is better/cheaper in the end).

That’s how I see it at the moment - those two major buckets basically with a few more minor ones. There are indeed many hidden opportunities there.

I’ll leave you a snippet from encyclopedia Britannica:

“Cambrian explosion, the unparalleled emergence of organisms between 541 million and approximately 530 million years ago at the beginning of the Cambrian Period. The event was characterized by the appearance of many of the major phyla (between 20 and 35) that make up modern animal life. Many other phyla also evolved during this time, the great majority of which became extinct during the following 50 to 100 million years.”