My Recent Podcast
My most recent podcast was with McKinsey senior partner Shelley Stewart. Shelley is Director of the McKinsey Institute for Black Economic Mobility, whose goal is to advance racial equity and inclusive growth in the world. Shelley shares some surprises in the data that run counter to our stereotypes and make us ask new questions.
So, check it out.
Machines Analyzing Human Language
Last week, I heard from one of my former students, Wilson Kung. Wilson is now Head of Data Science at CapitalOne Commercial. “Vasant! Look here,” his message began, “you were seven years ahead of the times.” He pointed me to an AI seminar at Cornell titled “Leveraging Text Mining to Extract Insights from Earnings Call Transcripts.”
Indeed, Wilson and I did just that several years ago as part of his Master’s thesis at NYU.
Once every quarter, publicly traded companies have an “earnings call” with analysts, where company leaders report on the previous quarter, provide future guidance and engage in an active Q&A at the end of the call with analysts. The transcripts of these calls are publicly available. I had spent a few years collecting and reading these reports, so when Wilson approached me for an idea for his thesis, I shared the dataset and my thinking about how machines might extract sentiment from such data. I had collected almost 15 years of call reports for companies in the S&P500 index. That was sixty reports per company, for a total of roughly thirty thousand reports.
This was three years before we had GPT, the current state of the art language model. Our methods were simple. We used an existing lexicon that assigned positive and negative sentiment scores to words and phrases. We aggregated these into cumulative positive (P) and negative (N) sentiment scores for each story. We then constructed a sentiment score, calculated as 100*(P-N)/(P+N). In this way scores varied between -100 and +100, for extremely negative and positive stories respectively. A neutral story, one with equal positive and negative sentiment, would have a sentiment score of zero. Think back to Alan Greenspan, whose Fed announcements always went something like “On the one hand we are optimistic that inflation is under control; on the other hand, inventories are low and may stoke inflation.” With Greenspan, P and N were very close. He must have been a good poker player.
What did we find? Several things, but I will report two of our interesting findings.
First, the average sentiment score was not zero, but +42. Maybe Douglas Adams was onto something after all in his Hitchhikers Guide to the Galaxy! Earnings call data are clearly biased towards optimism.
Here’s what is even more interesting. Sentiment scores above 42 were predictive of an “abnormal” positive return the next day, whereas scores below 42 were predictive of an abnormal negative return the next day! By abnormal, I mean the return of the stock relative to the market.
How cool is that? The machine reads 20 pages of text and Q&A, and predicts how the stock will perform tomorrow!
It would be interesting to test whether this would still work today. It would be equally interesting to see whether the state-of-the-art methods such as GPT do any better. My hypothesis is “probably not,” based on the fact that finance is a highly adversarial domain in general where such patterns get discovered and therefore disappear. Whenever an interesting pattern becomes known, an adversary will invariably act to nullify it. In this case, if CEOs are aware that their transcripts are being analyzed by machines, they are likely to pump their speech through the algorithm first, and craft their sentences accordingly.
Wilson and I analyzed data from a period where humans were not aware about the possibility of such analysis. Now, I’d be surprised if there’s still any juice left here for AI.
Many problems in life are adversarial. If you play poker or chess, for example, past data about opponents won’t be useful if they modify their strategy sensing that you’re onto them. Analyzing historical data in adversarial domains is of limited value for prediction.
The Indian Chief
At a university leadership meeting many years ago, we were broken up into groups of three and asked to analyze a situation. I was in a group with two other compatriots. Our deans were in another group. After a few minutes of deliberation, we took turns presenting our analyses.
The three deans reported their thoughts. Then it was our turn. I opened by saying that the Indians had a very different view of the situation than the chiefs. The room cracked up, but I wonder whether we can still say things like that now.
Talking about Indians, I would never have imagined that a brown dude would become the British chief in my lifetime. When I left India in 1978, everyone in my dad’s generation, except my dad, said “son, I don’t know why you want to go America. You realize that you can never make it to the top?”
My response was “right, but they just landed a man on moon and put on a show like Woodstock, how cool is that?” Some discrimination seemed like a small price to pay. Many years later, as my former Stern colleague and economic historian Richard Sylla put it, “people tend to treat newcomers to their country poorly. It’s just the way things are.” Indeed, the thought of “one of us” becoming the prime minister of the UK was laughable at the time, or even as recently as a decade ago.
The fact that it has happened is indicative of how much the UK and the US have changed during my lifetime. Unlike the colonial experience of my elders, I feel that on balance, my generation of Americans has been much more generous than discriminatory towards us. And it was relatively easy to fit into a culture whose language we already shared, complete with the slang. Many of us grew up much liking or even preferring rock n roll to our vernacular pop. Despite the colonial humiliation our parents had felt, weirdly enough, my generation felt no animosity towards the nation of our former colonizers. Indeed, we liked and adopted many of their ways. As immigrants, we expected to be treated badly, as Sylla noted. We largely kept our heads down and laughed off the occasional condescension.
With that attitude, how can you lose? Everything is just gravy. I’m sure Rishi Sunak has faced his share of crap, has focused on what is important. Through a Shakespearean kind of twist that played out in his political party, he has been rewarded with the top job without regard to his color. Now, he his work cut out for him. He must deliver to a country that has put the highest level of trust in him.