In this stock market forecast evaluation report, we will examine the performance of the forecasts generated by the I Know First AI Algorithm for Brazilian Bovespa component stocks which were daily sent to our customers. Our analysis covers the time period from 19 June, 2019, to December 26, 2019. The report also demonstrates how our Global Stock Picking method compares to our Daily Forecast model and how those generate predictions resulting in returns that outperform the Bovespa Index by a significant margin.

Best Brazilian Stocks Evaluation Highlights:

• Stock market forecasts that were generated by both the Daily and Global models achieved positive returns that outperformed the Ibovespa index.
• Except in a few cases, signal filtering proved again to have a strong effect on returns. Top 5 filtering achieved returns of up to 12.01% and outperformed the Ibovespa index by a significantly high margin.
• The Global model effect was observed only for long time horizons, where it achieved impressive returns and exceptional accuracy: for the 30 and 90 days, average returns were 10.56% and 20.25% respectively, while hit ratios were 71% and 90%.

The above results were obtained based on the stock market forecast evaluation over the specific time period using a consecutive filtering approach – by predictability and then by signal, to give a general overview of the forecasting capabilities of the algorithm for specific stock universe. In addition, we conducted alternative filtering based on our Global Model which is based on special signals filtering and will be explained further in a respective section. We will start with an introduction to our stock picking and benchmarking methods and then apply it to the Brazilian stocks universe covered by us in the Brazilian Package. After introducing the evaluation methodology in detail, we develop the analysis and present the evaluation results with relevant conclusions. This stock market forecast evaluation is part of our continuous studies of live I Know First’s AI predictive algorithm performance.

Evaluating Brazilian Stocks Forecasts: The Stock Picking Method

In this evaluation analysis, we use two stock-picking methodologies – our Daily Forecast model and our new Global Model forecast method:

The Daily Forecast Mode

We take the top 30 most predictable assets, and then we apply a set of signal-based filters: top 20, 10 and 5 based on predictability.

By doing so we focus on the most predictable assets on the one hand, while capturing the ones with the highest signal on the other.

We use absolute signals since these strategies are long and short ones. If the signal is positive, then we buy and, if negative, we short.

For example, a top 30 predictability filter with a top 10 signal filter means that on each day we take only the 30 most predictable assets, and then we pick from them the top 10 assets with the highest absolute signals.

Our Global Model Method For Top Brazilian Stocks Market Forecast

The new stock-picking method takes all 10,200 assets that are forecasted by I Know First (the global set). The assets are then filtered by predictability and the top 30 most predictable assets are selected. Then those 30 stocks are scanned to see if there are any Brazilian stocks – if there are such stocks, they are selected and then, based on the number of available assets, the signal filtering is applied at available levels to arrive at the top Brazilian stocks.

As mentioned before, the asset universe under consideration is the full 10,200 set of assets covered by I Know First forecast and includes stocks, commodities and currency pairs. By implementing this new stock-picking method, we are selecting only the most predictable assets and the ones with the highest signals. Therefore, unlike our Daily Forecast picking method, the Global Model method provides a significantly stricter filter which outputs forecasts with narrower assets set. Note, that in case the Global Model filtering procedure is not passed by any of the Brazilian stocks screened on a specific day, the system does not generate the forecast on this date until the stronger predictions come in.

About the I Know First Algorithm

The I Know First self-learning algorithm analyzes, models, and predicts the capital market, including stocks, bonds, currencies, commodities and interest rates markets. The algorithm is based on Artificial Intelligence (AI) and Machine Learning (ML) and incorporates elements of Artificial Neural Networks and Genetic Algorithms.

The I Know First Market Prediction System models and predicts the flow of money between the markets. It separates the predictable information from any “random noise”. It then creates a model that projects the future trajectory of the given market in the multidimensional space of other markets.

Since the model is 100% empirical, the results are based only on factual data, thereby avoiding any biases or emotions that may accompany human derived assumptions. The human factor is only involved in building the mathematical framework and providing the initial set of inputs and outputs to the system. The algorithm produces a stock market forecast with a signal and a predictability indicator. The signal is the number in the middle of the box. The predictability is the number at the bottom of the box. At the top, a specific asset is identified. This format is consistent across all predictions.

Our algorithm provides two independent indicators for each asset – signal and predictability. The signal is the predicted strength and direction of the asset’s movement. This is measured from -inf to +inf. The predictability indicates our confidence in that result. It is a Pearson correlation coefficient between past algorithmic performance and actual market movement. This is measured from -1 to 1.

A detailed description of our heatmap can be found here.

The Stock Market Forecast Performance Evaluation Method

We perform a stock market forecast evaluation on the individual forecast level. This means that we calculate the return of each forecast we have issued for each horizon in the testing period. We then take the average of those results by strategy and forecast horizon.

For example, to evaluate the performance of our 1-month forecasts, we calculate the return of each trade by using this formula:

This simulates a client purchasing the asset on the day we issue our prediction and selling it exactly 1 month in the future from that day.

We iterate this calculation for all trading days in the analyzed period and average the results.

Note that this stock forecast evaluation does not take a set portfolio and follow it. This is a different stock forecast evaluation method at the individual forecast level.

The Hit Ratio Method

The hit ratio helps us to identify the accuracy of our algorithm’s predictions.

Using our Daily Forecast and Global Model asset filtering, we predict the direction of movement of different assets. Our predictions are then compared against actual movements of these assets within the same time horizon.

The hit ratio is then calculated as follows:

For instance, a 90% hit ratio for a top 30 predictability filter with a top 10 signal filter would imply that the algorithm correctly predicted the price movements of 9 out of 10 assets within this particular set of assets.

The Benchmarking Method – Bovespa Index

In order to evaluate our algorithm’s performance in comparison to the Brazilian stock market, we used the Bovespa Index (^BVSP) as a benchmark.

The Bovespa Index, also known as Ibovespa, is composed as a portfolio of the most highly traded stocks in the B3- Brasil Bolsa Balcão (Brazil’s Stock Exchange). The portfolio is revised on a quarterly basis, while its composition must satisfy two conditions: (1) components must be stocks that account for 80% of the volume traded in the last 12 months (2) all portfolio components were traded at least on 80% of the trading days. Although the portfolio composition isn’t determined by companies’ market cap, on average the index components represent approximately 70% of the value traded in the B3 Stock Exchange and therefore qualifies as a benchmark that represents the Brazilian stock market.

For each time horizon, we compare the Ibovespa performance with the performance of our forecasts after the filtering processes described above. For evaluating the performance of the forecasts that are filtered based on the Global Model method, we compare those with Ibovespa’s average performance only when the Global Model filtering resulted in forecasts for Brazilian companies. For example, if after our Global filtering the model recommended buying a given asset and hold it three days on five different occasions, the Ibovespa performance that we use as a benchmark will be the average of returns only in those five intervals of three days.

Performance Evaluation – Overview

We conduct our research for the period from 19 June (date of the first-ever forecast generated for this stock universe with Global Model) to 26 December, 2019 for both Daily Forecast method and for Global Model. Following the methodology, as described in the previous sections, we start our analysis by computing the performance of the algorithm’s signals for time horizons ranging from 3 to 90 days with Daily Forecast model using predictability and signal filtering consecutively applied on the Brazilian stock universe. Then we continue with the evaluation results for stocks picked with Global Model approach and consider what are the advantages and disadvantages of these models.

Performance Evaluation – The Signal Indicator Effect

Once filtering by predictability is done, we utilize the signal indicator in our asset picking method to achieve the maximum forecast performance. It is important to measure it with respect to the benchmark, i.e. how the selected assets out-perform the benchmark, and for that, we will apply the formula:

We further filter and rank the assets based on absolute signal strength to the Top 20 assets under the Daily Forecast model, which were already filtered by predictability.

In this article, we examine the kind of effect the signal filter has. To do so, we have filtered the stocks by predictability, selected the 30 most predictable stocks from the Brazilian stock universe. The stocks are not necessarily all long or short positions, but they are a mix of both. That is because the I Know First algorithm is able to get return for both bull and bear market positions. Therefore, we applied filtering by signal strength to the top 30 assets filtered by predictability.

We further filter and rank the assets based on absolute signal strength to the Top 20 assets under the Daily Forecast model, which were already filtered by predictability.

In parallel, we apply the Global Model method and compare the returns to Ibovespa’s performance – the average Ibovespa’s return measured for the same dates within which stocks passed through Global Model filtering.

Top Brazilian Stocks Results: Average Return, Hit Ratio and Out-performance

As can be seen in Table 1, by applying the Top 30 Predictability filter our algorithm provides not only positive returns, but also significant out-performance over Ibovespa. The data also demonstrates significant positive signal effect on average return: besides few exceptions in cases of longer time horizons (14 days; Top 20, 30 days; Top 10, 90 days; Top 20), we observed higher returns. An exceptionally high performance was observed for 90-days time horizon with Top 5 signal filtering, where the algorithm achieved a return of 12.01%. Those results indicate that the signal effect on forecast return was strong and consistent.

As for the Global Model filtering performance, we observed polarized results. For short time horizons (3, 7 and 14 days) the Global Model filtering resulted in lower returns than the Daily Model (lower, but still positive and above the benchmark). On the other hand, the performance for the 30 and 90-days time horizon was significantly high – 10.56% and 20.25%, respectively.

For all time horizons and filtering methods, hit ratios were higher than 50% accuracy. Besides the initial signal filtering (30-10) in the 30 and 90-days time horizons, the algorithmic forecast was even more accurate with hit ratios ranging from 62% to 72%, and one extremely high hit ratio of 90% achieved by applying the Global Model for 90-days time horizon forecasts. As in the case of average returns, the Global Model’s effect on hit ratio was weak during short time horizons, but extremely strong during the time horizons of 30 and 90-days, when it achieved 71% and 90%, respectively. The signal effect of the Daily Model was consistent and strong, improving hit ratios in almost every case where the filtering by signal was applied (exceptions: 3-days – Top 10, 30-days – Top 20; 90-days – Top 20).

Finally, we present the out-performance measure which reflects the returns compared to Ibovespa benchmark index. As can be seen in Table 4, the highest out-performance was achieved by the top 5 signals filter in the 30-days time horizon (793.81%) and the 7 days time horizon (718.95%). Although returns achieved by the Global Model were significantly higher for the longer time horizons, the out-performance was lower than in the case of the top 5 filters. The reason may come from the differences between Ibovespa’s performance, which accounts for the average return of the index, and the Ibovespa* benchmark, which accounts for the average return in days when the algorithm output forecasts that survived the Global model filtering.

Conclusions

This evaluation report presents the performance of I Know First’s algorithm as reflected in the average returns for all time horizons after the relevant filtering processes. The results of this analysis demonstrated a strong overall performance of forecasts that are filtered by predictability, together with significant and consistent improvement that was obtained by the signal-based filtering method. The analysis also demonstrates the effect of the Global Model method for long time horizons (30 and 90-days).

Although the Global filtering method generated positive returns that outperformed Ibovespa at each time horizon, for short time horizons (3 to 14-days) its effect on returns was mostly insignificant compared to the signal effect. That said, Global Model’s effect was exceptionally strong on longer time horizons, when it delivered returns up to 20.25% and hit ratio up to 90%. I Know First’s research team will continue to monitor the changes in the performance trends of both models in order to evaluate our algorithm’s performance and derive relevant insights that will help provide the best algorithmic trading solutions to our clients.