Recommender systems: How it works
A few years ago, we saw artificial intelligence only in science fiction films, where supercomputers try to enslave the world, and brave heroes save it. Today everyone is talking about AI. But few people know what the magic is behind this and why we need machine learning. We suggest that you sort things out in order.
What have we heard and know about artificial intelligence?
Surely you know the engineering company Boston Dynamics. And you have heard, and perhaps even seen in films and videos, Deepfake technology. The program for playing go, AlphaGo, was also developed on the basis of AI technologies. I guess all of you have seen pictures of “Neural networks” with intertwined circles and arrows.
In fact, everything that is called Machine Learning or Artificial Intelligence is all a lot of mathematics.
Any objects from the real world: people, processes, is transformed into a matrixes and vectors. Since all these are objects of mathematics, you can work with them mathematically: compare, subtract, add, and so on.
Artificial intelligence is the name of an entire field, like chemistry or biology. Machine learning is a subset of AI. Machine learning itself is divided into many areas. Each of these subsections serves for specific tasks: pattern recognition, video recognition, natural language recognition, recommender systems, and so on.
Where it is effectively applied:
- Chat bots;
- autopilots in Tesla;
- Identification by face, by voice;
- Contextual advertising, recommendations in online stores and cinemas;
- And many other directions.
Today we will consider recommender systems.
How does expertise come about?
How do experienced professionals differ from beginners? Why does an experienced sales manager find an approach to a client faster, while an engineer checks exactly those components that could break when starting a production line? It's about experience. Experiences are established patterns in our brains, created from numerous similar situations in the past. We do not even always realize what prompted us to make such a decision: intuition, “feel”, professionalism. In fact, these are well-established connections between neurons in our brain.
This is how artificial neural networks are trained, which are the core of modern expert systems. This is especially important when there is too much data to analyze and their relationship is difficult to establish.
In this case, special mathematical algorithms, after preliminary processing by specialists, use this data to identify implicit dependencies and patterns between the data.
More clearly it looks like this
Of course, this is an extremely simplified model, but the essence remains the same. A computer model studies historical data, determining which parameters (or their relationship) led to the desired results and gives the most important additional weight. After that, it checks the selected data that the model has not yet seen. For example, if weather data for 2010-2018 was loaded into the model for forecasting the harvest, then the check will be carried out on the data of 2019. The model will have to tell what the crop will be based on the weather data. If the accuracy is sufficient, then the model works, if not, then the training will need to be continued.
Using similar methods, we predicted:
- Breakdown of construction equipment in order to carry out maintenance in advance;
- Diseases of colds in people depending on the change in the weather;
- The optimal mode of artificial lung ventilation was determined;
- We predicted the marriage of rolled metal long before its appearance;
- The factors influencing the quality of the metal sheet were identified;
- a number of other important processes.
Recommender systems are one of the most successful and widely used machine learning technologies in business. They are widely used in retail, streaming and news services and many others. A company typically requires a team of expensive data scientists and engineers to develop and maintain such systems. This is why even such large corporations outsource their referral services.
In conclusion of the first part, I would like to highlight several points:
- Machine learning methods are used to find the optimal solution to business problems. For AI to learn to make the right decisions, it needs to be trained on historical data.
- Modern recommender systems are one of the most effective tools based on artificial intelligence.
- Machine learning should be used when a large amount of information needs to be analyzed to make the right decision.
- Machine learning can be used when there is historical data on which to train the system.
In the following articles, we will consider specific examples of the use of recommendation algorithms.
