MIT Researchers Create a Tool for Predicting the Future

Future Prediction is the Need for every Scientist

Nobody knows what the future will bring. Psychics, mediums, and astrological forecasters are examples of non-scientific ways of foretelling the future. However, such practitioners and their methodologies have continually failed to deliver scientifically credible outcomes. Mathematicians and statisticians who specialize in data analysis, on the other hand, frequently use predictive analytic methods to provide a pretty accurate peek of likely future events. Sadly, most of these predictive systems rely on complicated algorithms and substantial computer capacity, rendering them mainly inaccessible to rank and file time-series data analysts.

A system that enables laypeople to forecast stock prices with even higher accuracy without accurate time-series database information is live now; MIT researchers have developed a similar machine. Time-series data is comprised of observations collected over time and become a database that is used to make much of the future predictions.

However, the time-series way of predicting the future involves complex data processing steps with various machine-learning algorithms. The given method cannot be used by un-expert to deploy and predict; to resolve this issue, MIT researchers developed a straightforward interface layer on the complex algorithmic system enabling non-experts to predict the future.

TsbDB – An Exceptional Forecasting Tool for Lay-Persons

After installing tspDB on top of an existing database, a user may execute a prediction query is roughly 0.9 ms, compared to 0.5 ms for a typical exploring question. The confidence intervals (basically a measure of the degree of error permitted in the findings) are also intended to assist non-experts in making more informed decisions by introducing the ambiguity of the forecasts into their decision-making.

MIT researchers “Anish Agarwal, Abdullah Alomar, and Devavrat Shah: On Multivariate Singular Spectrum Analysis and its Variants" developed a technique that promptly incorporates forecasting capability on the head of the old time-series database to make these sophisticated techniques more user-friendly.

Higher Accuracy than Other Tools

Their simplified interface, dubbed “tspDB (time series predict database)”, handles all the intricate modelling on backstage, allowing a non-expert to make a forecast in only a few moments. When executing two tasks, the novel system surpasses state-of-the-art deep learning algorithms in terms of accuracy as well as efficiency: forecasting future outcomes and patching in missing data bits (i.e., imputation of missing values).

The success of tspDB can be attributed to the inclusion of a revolutionary time-series prediction method. This approach is perfect for forecasting multivariate time-series data containing more than one time-dependent parameter. Multiple explanatory variables, such as temperature, dew point, and cloud cover, in a meteorological database, for example, depend on values from prior observations collected throughout time – and hence must be addressed in the model.

The Right Way to Look at Time-Series

Leading author Devavrat Shah, “the Andrew and Erna Viterbi Professor in EECS and a member of the Institute for Data, Systems, and Society and the Laboratory for Information and Decision Systems,” said:

“Even as the time-series data becomes more and more complex, this algorithm can effectively capture any time-series structure out there. It feels like we have found the right lens to look at the model complexity of time-series data.”

For years, Shah and colleagues worked on the challenge of reading “time-series data”, adopting several systems, and incorporating them into tspDB as they constructed the interface. They studied about a remarkably effective classical technique called “singular spectrum analysis (SSA)” around four years ago, which imputes (predicts missing data) and forecasts single time series. In time series analysis, SSA is a nonparametric spectral estimation technique, which implies modeling without pre-parameterizing the action variables.

For a single time series, the technique transformed it to a matrix and employed matrix estimation procedures. The key intellectual challenge was determining how to alter it to use several time series. They identified the answer after several years of struggle: "Stack" the matrices for each individual time series, treat it as a single huge matrix, and then execute the single time-series procedure.

The recent publication also offers intriguing possibilities, such as seeing the multivariate time series as a three-dimensional tensor rather than translating it into a large matrix. A tensor is a multidimensional array of integers, sometimes known as a grid. According to Alomar, this has built a potential link between the traditional time series analysis discipline and the emerging tensor estimation subject.

According to Shah:

“The variant of mSSA that we introduced actually captures all of that beautifully. So, not only does it provide the most likely estimation, but a time-varying confidence interval, as well.”

Automating Prediction

It combines time series analysis, dynamical systems, multivariate geometry, multivariate statistics, and signal processing. The practice of substituting missing values or correcting previous ones is called imputation. While this technique needed human parameter adjustment, the researchers felt it is possible that their interface will be able to generate accurate predictions based on time series data. Previously, they eliminated the necessity for manual intervention in algorithmic implementation.

On real time-series datasets, including inputs from the electrical grid, financial markets, and traffic patterns, they compared the modified mSSA (a variation of SSA they developed) against other cutting-edge algorithms, including deep-learning approaches. Their approach beat all but one of the other algorithms in terms of imputation, and in projecting future values, it surpassed all but one of the other systems. The researchers also demonstrated that their customized version of mSSA is applicable to any time-series data.

Abdullah Alomar said:

“One reason I think this works so well is that the model captures a lot of time series dynamics, but at the end of the day, it is still a simple model. When you are working with something simple like this, instead of a neural network that can easily overfit the data, you can actually perform better.” Shah explained: “The impressive performance of mSSA is what makes tspDB so effective.” They are committed to making their system accessible to everyone.

For the time being, the researchers focus on improving the functionality and user-friendliness of tspDB and investigating additional algorithms that may be added.

Shah added:

“Our interest at the highest level is to make tspDB a success in the form of a broadly utilizable, open-source system. Time-series data are very important, and this is a beautiful concept of actually building prediction functionalities directly into the database. It has never been done before, and so we want to make sure the world uses it.”

Vishal Misra was not involved in this study and is a computer science professor at Columbia University, said:

“This work is very interesting for a number of reasons. It provides a practical variant of mSSA which requires no hand tuning, they provide the first known analysis of mSSA, and the authors demonstrate the real-world value of their algorithm by being competitive with or outperforming several known algorithms for imputations and predictions in (multivariate) time series for several real-world data sets. At the heart of it all is the beautiful modelling work where they cleverly exploit correlations across time (within a time series) and space (across time series) to create a low-rank spatiotemporal factor representation of a multivariate time series. Importantly this model connects the field of time series analysis to that of the rapidly evolving topic of tensor completion, and I expect a lot of follow-on research spurred by this paper.”

Wrapping Discussion

The developed program can assist individuals in several professions to make better forecasts using time-series data and predicting weather, projecting coming stock values, finding lost sales prospects in retail industry, or calculating a patient's probability of acquiring an illness are all examples.

Forecasting is unusual because it enables people to make predictions without first understanding complicated machine-learning methods. The UI is also intended to be user-friendly and straightforward, especially for those who are not data analysis professionals. The system works by dividing data from a time series into individual observations. This data is then used to train a machine-learning algorithm, which may ultimately be used to generate predictions. Users can utilize the interface to enter their data or data that has been pre-loaded into the program.