Some things left lying around
A few interesting AI things worth your attention including my new book!
This is an intermittent newsletter whereby I write when I have something to say that might be a little different from others. But as it is the end of the year, I want to do a bit of ‘deck clearing’ of things I thought might warrant a longer post, but haven’t found the time to give them the treatment they deserve. Nonetheless, I suspect they will point readers to avenues of interest here.
Compelling AI in eCommerce
I really enjoyed this discussion with a troupe of AI and economic researchers with Tobi Lutke, the CEO of Shopify.
There are three things of much interest. First, Shopify has launched a fantastic set of AI tools. What struck me when I heard about them is that they use current AI models but are designed squarely with Shopify merchants in mind. Not only are the tools impressive, but I believe that how Shopify are developing them is a message for all of us interested in how AI might actually become useful in the wild.
Second, there was a broader philosophical discussion about data generation that might drive future AI productivity. Ajay Agrawal posited that Shopify’s data was extraordinary precisely because it is a far more experimentation-focused ecosystem than other places (including Amazon’s and Google’s of the world, which tend to push people into a “one size fits all” behavioural context).
Finally, right at the very end, in the last minute of the podcast, Tobi drops a bomb of an insight on goals.
Benchmarks currently and may always suck
The AI industry is benchmark-focused. Basically, they are leapfrogging themselves on benchmarks of various kinds, but when you actually try a new model, it can be a significant change, or it might even feel worse. Why? Because you aren’t automating everything, you are conversing and iterating. This is something we noted as a discord in our paper “The Economics of Bicycles for the Mind” (paper here, and some easy-to-digest slides here).
A new paper found that this is really impacting how we can use LLMs and their offshoots to accelerate science. Basically, as the following chart shows, high benchmark scores are uncorrelated with scientific performance.
I am beginning to wonder if this is a solvable problem at all. This is because we are all so different in how we use tools. I mean, imagine if Microsoft Word were designed based on benchmarks rather than using usual customer feedback to inform product features and performance.
I have a new book!
It is called The Microeconomics of Artificial Intelligence, and MIT Press just released it. It was a two-year labour of love for me, collecting all the economics I thought would be relevant to those researching the economics of AI. It is more of a graduate-level textbook than a comfortable read. I also designed the cover (with the help of ChatGPT).
You’d think this is something I would have devoted a whole post to, but it is a bit pricey and will also have chapters available as Open Access somewhat soon. I was waiting for that to happen (but it hasn’t yet). Nonetheless, it is undoubtedly a good stocking stuffer (although it might not fit into one) for the economist in your life.