Introduction
Regression is a kind of supervised learning algorithm within machine learning. It is an approach to model the relationship between the dependent variable (or target, responses), y, and explanatory variables (or inputs, predictors), X. Its objective is to predict a quantity of the target variable, for example; predicting the stock price, which differs from classification problem, where we want to predict the label of target, for example; predicting the direction of stock (up or down).
Moreover, a regression can be used to answer whether and how several variables are related, or influence each other, for example; determine **_if _**and to what extent the work experience or age impacts salaries.
In this article, I will focus mainly on linear regression and its approaches.
Different approaches to Linear Regression
OLS (Ordinary least squares) goal is to find the best-fitting line (hyperplane) that minimizes the vertical offsets, which can be mean squared error (MSE) or other error metrics (MAE, RMSE) between the target variable and the predicted output.
We can implement a linear regression model using the following approaches:
- Solving model parameters (closed-form equations)
- Using optimization algorithm (gradient descent, stochastic gradient, etc.)
Please note that OLS regression estimates are the best linear unbiased estimator (BLUE, in short). Regression in other forms, the parameter estimates may be biased, for example; ridge regression is sometimes used to reduce the variance of estimates when there is collinearity in the data. However, the discussion of bias and variance is not in the scope of this article (please refer to this great article related to bias and variance).
Closed-form equation
Let’s assume we have inputs of X size n and a target variable, we can write the following equation to represent the linear regression model.
#linear-regression #machine-learning #python #data-science #gradient-descent
