Machine Learning and Optimization : Relationship

These questions usually asked by the interested group to machine learning. Machine Learning is a numerical optimisation. Mathematical Optimisation includes analytic techniques which can be used to an answer the problem. The difference is very slim between machine learning (ML) and optimization theory. Optimization falls the domain of mathematics. Machine learning falls in the domain of engineering. In real life, numerical and analytical techniques are often used in a hybrid way. One can numerically estimate a lagrangian multiplier, and then use this estimate to deduce an analytic solution. Problems in engineering are often not mathematically well defined, like face recognition. Lot of learning can be seen as optimization. In fact learning is an optimization problem. The goal for machine learning is to optimize the performance of a model given an objective and the training data. There is no precise mathematical formulation that unambiguously describes the problem of face recognition. There is no foolproof way to recognize an unseen photo of person by any method. In real, the chance of failure starts to decline with more data-linked logic.

Machine learning is a suite of tools that can solve a variety of problems. There are some common themes among how these tools work – optimization is one of them. Optimization is a set of tools for finding an element of some set D that minimizes a function f. Every machine learning problem can be cast in that form, but so can many problems that have nothing to do with machine learning at all.

Technically we need to use optimization for machine learning, although most of the time we will find that it is used somewhere in the machine learning algorithm we are looking at.

It is true that machine learning optimisation is a subset of optimsation, there are some differences between classical optimisation and machine learning. Each machine learning problem comes down to an optimization problem. Solving the optimization problem is the last step of machine learning problem. Optimization lies at the heart of machine learning.

We have some long series articles on machine learning, such as Application of Machine Learning in Text Recognition or Approaches of Deep Learning. They may be useful to start with.