Erez Katz, CEO and Co-founder of Lucena Research

How to use Machine Learning to Measure the Predictive Nature of Analyst Ratings 

 

Change in sentiment by reputable analysts can often lead to stock movement. Some investors are influenced by “expert” opinions, but many wonder if by the time an analyst publishes a new report on a publicly traded company, if the sentiment is already baked into the price.

In addition, investors are often suspect of analysts’ true motivations and question whether their reports are bias-free and not aimed at furthering their firm’s (or their own) investment goals.

Insights from Analyst Sentiment

In recent years, we’ve witnessed the emergence of several data vendors who specialize in deriving actionable insights from analysts’ sentiment through “smart” scoring. Rather than averaging all analysts’ scores for a given stock, they apply various weights in a way to increase/decrease the influence of some analysts over others when calculating the overall sentiment score of a particular stock.

It turns out that how much weight should be placed on each factor in the overall calculation is a perfect problem for a deep neural network regressor. In general, we want to train our model to assign an overall analysts’ sentiment score based on how consistent that score was to predict an impending price action in the past.

Factors to consider for sentiment score:

• – Adjust an analyst’s ratings as a portion of the total score based on how accurate he/she were historically.
•  
• – Apply additional weight as a portion of the total score for scores that were against common belief, but which turned out to be accurate.
•  
• – Rank analysts by their professional credentials, such as years of experience and education level.
•  
• – Evaluate how the market reacted historically when a specific analyst upgraded or downgraded a particular stock.

How to Measure if Analyst Scores are Predictive

Here I’ll discuss specifically how to measure if analyst sentiment scores are predictive of future asset prices for profitable investment. In particular, the longer term impact (up to one year) of how stocks perform following an upgrade or a downgrade of an overall analysts consensus sentiment. 

In this example, we will measure market relative performance. In other words, how specific selected stocks have outperformed the Russell 1K index ($RUI).

How Are Analysts’ Ratings Scored?

To keep things simple, I won’t dive into the weight of individual factors and how analysts’ scores are measured. Assume for now, that some smarts where factored into how scoring was done during the training period, the years prior to our evaluation periods (completely out of sample).

In our example, we will break the analysts’ consensus score into 5 sequential ranks from 1 (weak outlook) to 5 (strong bullish outlook), which is the inverse of how they are measured in Yahoo Finance (see below). 

 

An example of analysts recommendation as measured on Yahoo Finance.

  

Using QuantDesk Event Analyzer

Using the QuantDesk Event Analyzer, we can construct a stock screener (a scan) to determine which stocks in the Russell 1K meet or exceed overall analysts’ score threshold (for example, above 4 for bullish signals).

Next we’ll measure how the stocks that met the criteria of the scan moved afterwards relative to the Russell 1K.

As you can see from the bold blue line in the graph below which represents the average price move of the stocks that matched the scan, the results are quite favorable. Given our validation time frame (1/1/2010 to 12/31/2010), stocks with sentiment score ratings above 4, on average, handsomely outperformed the Russell 1K by more than 5%.  

 

Strong buy ratings on average outperformed the Russell 1K by 5.12% one year later.

Conversely, scores below 2 underperformed and lagged the benchmark significantly (by more than 28%), although for a fair comparison, returns should have been compared to shorting the Russell 1K.

 

Strong sell ratings on average underperformed the Russell 1K by 28% one year later.

Since the assessment above was made on a specific time frame (the year 2010), we wanted to determine if the signal carries forward the same potency into the future.

To that end, we conducted a backtest out of sample, from 2011 through 2018. Finally, we wanted to continue to measure the signals effectiveness into the future. QuantDesk is fully capable of converting any backtest into a model portfolio that will carry the buy/sell signals perpetually on a roll forward basis.

 

 

Out-of-sample backtest simulating trade execution based on analysts consensus score. Backtest considers transaction costs and slippage. Portfolio performance in orange vs. benchmark S&P 500 in blue. Past performance is not indicative of future returns.

 

 

Perpetual simulation of a paper traded model portfolio which carries the backtest into the future. Please note the start date as of 1/24 vs. 1/1/2019 was chosen for no particular reason other than an arbitrary date we decided to start tracking the portfolio on. Model portfolio performance in orange vs. benchmark S&P 500 in blue. Past performance is not indicative of future returns.

Based on QuantDesk’s event study, backtesting and model portfolio it is evident that our analysts consensus signals contain actionable information over multiple periods of time and perpetually into the future.

It’s important to note that the empirical evidence above is not meant to dictate how stocks should be traded but rather to showcase that the data is predictive at which point it will be up to the user to integrate that knowledge into their own investment approach as they deem fit.

 

Using Machine Learning to Gather Unbiased Signals

There is a growing demand for intelligent signals (often backed by machine learning) that distinguish good and reliable data from the rest. Read more about how we use machine learning to validate data for stock forecasting.

The challenge remains on the buy-side, as there is a great deal of misinformation and no industry standards geared to ascertain if a data source is worthy of further research or of inclusion in a quantitative investment. At Lucena, we aim to offer an unbiased view of data validation and investment research capabilities.

 

Looking to quickly evaluate alternative data for your desired investment approach? Check out our Data Analytics Suite. 

Test your own signals and research in QuantDesk with a free trial.