The Overhang Edition

Noah here. Over the weekend, the FT’s Big Read was devoted to transformers, the machine learning architecture sweeping the world (the “T” in GPT). The paper introducing the concept initially came out in 2017 from a group of researchers at Google and represented a fairly significant departure from the approaches folks were taking at the time, which—to overly simplify things—tended to look at each word sequentially and use the previous ones to predict the next. The new transformer model was able to parallelize the process, looking at all words simultaneously while also taking account of their position in the sequence and using all that info to make predictions.

WITI Classifieds:

We are experimenting with running some weekly classifieds in WITI. If you’re interested in running an ad, you can purchase one through this form. If you have any questions, don’t hesitate to drop a line.

Another generative AI report? Take it from our insights & strategy friends at R/GA, Landor & Fitch, & EY—we promise this one’s a good read. this one’s a good read

In addition, transformers utilized an approach called attention, which essentially allowed them to look at all the words in a set at one time and use the model’s understanding of those words to focus on the most important ones, thus better zeroing in on the key concepts (see the image below from the FT piece). Combined with increases in compute and more powerful graphical processing units (GPUs), we’ve seen an explosion in the power of these systems to perform heretofore unimaginable tasks.

Why is this interesting?

Beyond being worth understanding, a particular word in the article caught my eye (this being metaphor week, it made sense). See if you can spot it:

This has led to a period that Silicon Valley insiders call a technology overhang — time that industries will spend integrating the latest AI developments into products, even if research does not progress at all.

Overhang. 

I’ve never heard that particular term used before, but the concept is one I’m pretty familiar with, particularly as it relates to the ideas of Venezuelan economist Carlota Perez. Perez argues that the best way to understand macroeconomic history is through long waves of technological innovation. Each wave looks like an S-curve, with two distinct phases: the installation period, when a new technology like the microchip or internet is introduced to the world, and the foundation is laid for its use in society, and the deployment phase, when that foundation is utilized to spread the new technology far and wide. The two periods are broken up by a bust, when the new technology’s hype overwhelms its ability to actually deliver. The dot com bubble, for instance, represented excitement about the possibility the internet offered, but at the time, the total number of people connected numbered in the low hundreds of millions.

While overhang doesn’t quite match up with Perez’s ideas as far as cycle time is concerned, the core idea that it takes time to fully diffuse new technology into society and culture is well-founded, and overhang is an excellent metaphor to go along with it. (NRB)

—

Thanks for reading,

Noah (NRB) & Colin (CJN)

—

Why is this interesting? is a daily email from Noah Brier & Colin Nagy (and friends!) about interesting things. If you’ve enjoyed this edition, please consider forwarding it to a friend. If you’re reading it for the first time, consider subscribing.