1622550660

Get to know the algorithms that that enables computers to perceive

The word computer vision means the ability of a computer to see and perceive the surrounding. A lot of application holds for computer vision to cover — Object detection and recognition, self driving cars, facial recognition, ball tracking, photo tagging, and many more. Before diving in the technical jargons, first let’s discuss the entire computer vision pipeline.

The entire pipeline is divided into 5 basic steps, each with a specific function. Firstly, the input is needed for the algorithm to process that can be in the form of an image or stream of image (image frames). The next step is pre-processing. In this step, functions are applied to the incoming image(s) so that the algorithm can better understand the image.

Some of the functions involve noise reduction, image scaling, dilation, and erosion, removing color spots, etc. The next step is selecting the area of interest or the region of interest. Under this lies the object detection and image segmentation algorithms. Further, we have feature extraction that means retrieving relevant information/features from the images that are necessary for accomplishing the end goal.

The final step is recognition or prediction, where we recognize objects in a given frame of images or predict the probability of the object in a given image frame.

Let’s look at a real world application of the computer vision pipeline. Facial expression recognition is an application of computer vision that is used by a lot of research labs to get an idea of what effect a particular product has on its users. Again, we have input data to which we apply the pre-processing algorithms.

The next step involves detecting faces in a particular frame and cropping that part of the frame. Once this is achieved, facial landmarks are identified like mouth, eyes, nose, etc. — key features for emotion recognition.

In the end, a prediction model( trained model) classifies the images based on the features extracted in the intermediary steps.

Before I start mentioning the algorithms in computer vision, I want to stress the term ‘Frequency’. The frequency of an image is the rate of change of intensity. High-frequency images have large changes in intensity. A low-frequency image is relatively uniform in brightness or the intensity changes slowly.

On applying Fourier transform to an image we get a magnitude spectrum that yields the information of the image frequency. Concentrated point in the center of the frequency domain image means a lot of low frequency components are present in the image. High frequency components include — edges, corners, stripes, etc. We know that an image is a function of x and y f(x,y). To measure the intensity change, we just take the derivative of the function f(x,y).

The Sobel operator is used in image processing and computer vision for edge detection algorithms. The filter creates an image of emphasizing edges. It computes an approximation of the slope/gradient of the image intensity function. At each pixel in the image, the output of the Sobel operator is both the corresponding gradient vector and the norm of this vector.

The Sobel Operator convolves the image with a small integer-valued filter in the horizontal and vertical directions. This makes the operator inexpensive in terms of computation complexity. The Sx filter detects edges in the horizontal direction and Sy filter detects edges in the vertical direction. It is a high pass filter.

#artificial intelligence #algorithm

1622550660

Get to know the algorithms that that enables computers to perceive

The word computer vision means the ability of a computer to see and perceive the surrounding. A lot of application holds for computer vision to cover — Object detection and recognition, self driving cars, facial recognition, ball tracking, photo tagging, and many more. Before diving in the technical jargons, first let’s discuss the entire computer vision pipeline.

The entire pipeline is divided into 5 basic steps, each with a specific function. Firstly, the input is needed for the algorithm to process that can be in the form of an image or stream of image (image frames). The next step is pre-processing. In this step, functions are applied to the incoming image(s) so that the algorithm can better understand the image.

Some of the functions involve noise reduction, image scaling, dilation, and erosion, removing color spots, etc. The next step is selecting the area of interest or the region of interest. Under this lies the object detection and image segmentation algorithms. Further, we have feature extraction that means retrieving relevant information/features from the images that are necessary for accomplishing the end goal.

The final step is recognition or prediction, where we recognize objects in a given frame of images or predict the probability of the object in a given image frame.

Let’s look at a real world application of the computer vision pipeline. Facial expression recognition is an application of computer vision that is used by a lot of research labs to get an idea of what effect a particular product has on its users. Again, we have input data to which we apply the pre-processing algorithms.

The next step involves detecting faces in a particular frame and cropping that part of the frame. Once this is achieved, facial landmarks are identified like mouth, eyes, nose, etc. — key features for emotion recognition.

In the end, a prediction model( trained model) classifies the images based on the features extracted in the intermediary steps.

Before I start mentioning the algorithms in computer vision, I want to stress the term ‘Frequency’. The frequency of an image is the rate of change of intensity. High-frequency images have large changes in intensity. A low-frequency image is relatively uniform in brightness or the intensity changes slowly.

On applying Fourier transform to an image we get a magnitude spectrum that yields the information of the image frequency. Concentrated point in the center of the frequency domain image means a lot of low frequency components are present in the image. High frequency components include — edges, corners, stripes, etc. We know that an image is a function of x and y f(x,y). To measure the intensity change, we just take the derivative of the function f(x,y).

The Sobel operator is used in image processing and computer vision for edge detection algorithms. The filter creates an image of emphasizing edges. It computes an approximation of the slope/gradient of the image intensity function. At each pixel in the image, the output of the Sobel operator is both the corresponding gradient vector and the norm of this vector.

The Sobel Operator convolves the image with a small integer-valued filter in the horizontal and vertical directions. This makes the operator inexpensive in terms of computation complexity. The Sx filter detects edges in the horizontal direction and Sy filter detects edges in the vertical direction. It is a high pass filter.

#artificial intelligence #algorithm

1624442580

As my knowledge in machine learning grows, so does the number of machine learning algorithms! This article will cover machine learning algorithms that are commonly used in the data science community.

Keep in mind that I’ll be elaborating on some algorithms more than others simply because this article would be as long as a book if I thoroughly explained every algorithm! I’m also going to try to minimize the amount of math in this article because I know it can be pretty daunting for those who aren’t mathematically savvy. Instead, I’ll try to give a concise summary of each and point out some of the key features.

With that in mind, I’m going to start with some of the more fundamental algorithms and then dive into some newer algorithms like CatBoost, Gradient Boost, and XGBoost.

Linear Regression is one of the most fundamental algorithms used to model relationships between a dependent variable and one or more independent variables. In simpler terms, it involves finding the ‘line of best fit’ that represents two or more variables.

The line of best fit is found by minimizing the squared distances between the points and the line of best fit — this is known as minimizing the sum of squared residuals. A residual is simply equal to the predicted value minus the actual value.

In case it doesn’t make sense yet, consider the image above. Comparing the green line of best fit to the red line, notice how the vertical lines (the residuals) are much bigger for the green line than the red line. This makes sense because the green line is so far away from the points that it isn’t a good representation of the data at all!

If you want to learn more about the math behind linear regression, I would start off with Brilliant’s explanation.

#2021 jan tutorials #overviews #algorithms #decision tree #explained #algorithms

1607510226

Some Popular Web App Ideas for 2021

Are you looking for best web application business ideas that make money in 2021?

There are lots of simple web app ideas but all those web application business ideas do not make money.

#trending web app ideas 2021 #trending web application ideas 2021 #web application ideas 2021 #web app ideas 2021 #new web app ideas 2021 #evergreen web app ideas 2021

1593347004

The Greedy Method is an approach for solving certain types of optimization problems. The greedy algorithm chooses the optimum result at each stage. While this works the majority of the times, there are numerous examples where the greedy approach is not the correct approach. For example, let’s say that you’re taking the greedy algorithm approach to earning money at a certain point in your life. You graduate high school and have two options:

#computer-science #algorithms #developer #programming #greedy-algorithms #algorithms

1641805837

The final objective is to estimate the cost of a certain house in a Boston suburb. In 1970, the Boston Standard Metropolitan Statistical Area provided the information. To examine and modify the data, we will use several techniques such as data pre-processing and feature engineering. After that, we'll apply a statistical model like regression model to anticipate and monitor the real estate market.

Project Outline:

- EDA
- Feature Engineering
- Pick and Train a Model
- Interpret
- Conclusion

Before using a statistical model, the EDA is a good step to go through in order to:

- Recognize the data set
- Check to see if any information is missing.
- Find some outliers.
- To get more out of the data, add, alter, or eliminate some features.

- Recognize the data set
- Check to see if any information is missing.
- Find some outliers.
- To get more out of the data, add, alter, or eliminate some features.

# Import the libraries #Dataframe/Numerical libraries import pandas as pd import numpy as np #Data visualization import plotly.express as px import matplotlib import matplotlib.pyplot as plt import seaborn as sns #Machine learning model from sklearn.linear_model import LinearRegression

#Reading the data path='./housing.csv' housing_df=pd.read_csv(path,header=None,delim_whitespace=True)

CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | DIS | RAD | TAX | PTRATIO | B | LSTAT | MEDV | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

0 | 0.00632 | 18.0 | 2.31 | 0 | 0.538 | 6.575 | 65.2 | 4.0900 | 1 | 296.0 | 15.3 | 396.90 | 4.98 | 24.0 |

1 | 0.02731 | 0.0 | 7.07 | 0 | 0.469 | 6.421 | 78.9 | 4.9671 | 2 | 242.0 | 17.8 | 396.90 | 9.14 | 21.6 |

2 | 0.02729 | 0.0 | 7.07 | 0 | 0.469 | 7.185 | 61.1 | 4.9671 | 2 | 242.0 | 17.8 | 392.83 | 4.03 | 34.7 |

3 | 0.03237 | 0.0 | 2.18 | 0 | 0.458 | 6.998 | 45.8 | 6.0622 | 3 | 222.0 | 18.7 | 394.63 | 2.94 | 33.4 |

4 | 0.06905 | 0.0 | 2.18 | 0 | 0.458 | 7.147 | 54.2 | 6.0622 | 3 | 222.0 | 18.7 | 396.90 | 5.33 | 36.2 |

... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |

501 | 0.06263 | 0.0 | 11.93 | 0 | 0.573 | 6.593 | 69.1 | 2.4786 | 1 | 273.0 | 21.0 | 391.99 | 9.67 | 22.4 |

502 | 0.04527 | 0.0 | 11.93 | 0 | 0.573 | 6.120 | 76.7 | 2.2875 | 1 | 273.0 | 21.0 | 396.90 | 9.08 | 20.6 |

503 | 0.06076 | 0.0 | 11.93 | 0 | 0.573 | 6.976 | 91.0 | 2.1675 | 1 | 273.0 | 21.0 | 396.90 | 5.64 | 23.9 |

504 | 0.10959 | 0.0 | 11.93 | 0 | 0.573 | 6.794 | 89.3 | 2.3889 | 1 | 273.0 | 21.0 | 393.45 | 6.48 | 22.0 |

505 | 0.04741 | 0.0 | 11.93 | 0 | 0.573 | 6.030 | 80.8 | 2.5050 | 1 | 273.0 | 21.0 | 396.90 | 7.88 | 11.9 |

**Crime:** It refers to a town's per capita crime rate.

**ZN:** It is the percentage of residential land allocated for 25,000 square feet.

**Indus:** The amount of non-retail business lands per town is referred to as the indus.

**CHAS:** CHAS denotes whether or not the land is surrounded by a river.

**NOX:** The NOX stands for nitric oxide content (part per 10m)

**RM:** The average number of rooms per home is referred to as RM.

**AGE:** The percentage of owner-occupied housing built before 1940 is referred to as AGE.

**DIS:** Weighted distance to five Boston employment centers are referred to as dis.

**RAD:** Accessibility to radial highways index

**TAX:** The TAX columns denote the rate of full-value property taxes per $10,000 dollars.

**B:** B=1000(Bk — 0.63)2 is the outcome of the equation, where Bk is the proportion of blacks in each town.

**PTRATIO:** It refers to the student-to-teacher ratio in each community.

**LSTAT:** It refers to the population's lower socioeconomic status.

**MEDV:** It refers to the 1000-dollar median value of owner-occupied residences.

# Check if there is any missing values. housing_df.isna().sum() CRIM 0 ZN 0 INDUS 0 CHAS 0 NOX 0 RM 0 AGE 0 DIS 0 RAD 0 TAX 0 PTRATIO 0 B 0 LSTAT 0 MEDV 0 dtype: int64

No missing values are found

We examine our data's mean, standard deviation, and percentiles.

housing_df.describe()

CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | DIS | RAD | TAX | PTRATIO | B | LSTAT | MEDV | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

count | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 | 506.000000 |

mean | 3.613524 | 11.363636 | 11.136779 | 0.069170 | 0.554695 | 6.284634 | 68.574901 | 3.795043 | 9.549407 | 408.237154 | 18.455534 | 356.674032 | 12.653063 | 22.532806 |

std | 8.601545 | 23.322453 | 6.860353 | 0.253994 | 0.115878 | 0.702617 | 28.148861 | 2.105710 | 8.707259 | 168.537116 | 2.164946 | 91.294864 | 7.141062 | 9.197104 |

min | 0.006320 | 0.000000 | 0.460000 | 0.000000 | 0.385000 | 3.561000 | 2.900000 | 1.129600 | 1.000000 | 187.000000 | 12.600000 | 0.320000 | 1.730000 | 5.000000 |

25% | 0.082045 | 0.000000 | 5.190000 | 0.000000 | 0.449000 | 5.885500 | 45.025000 | 2.100175 | 4.000000 | 279.000000 | 17.400000 | 375.377500 | 6.950000 | 17.025000 |

50% | 0.256510 | 0.000000 | 9.690000 | 0.000000 | 0.538000 | 6.208500 | 77.500000 | 3.207450 | 5.000000 | 330.000000 | 19.050000 | 391.440000 | 11.360000 | 21.200000 |

75% | 3.677083 | 12.500000 | 18.100000 | 0.000000 | 0.624000 | 6.623500 | 94.075000 | 5.188425 | 24.000000 | 666.000000 | 20.200000 | 396.225000 | 16.955000 | 25.000000 |

max | 88.976200 | 100.000000 | 27.740000 | 1.000000 | 0.871000 | 8.780000 | 100.000000 | 12.126500 | 24.000000 | 711.000000 | 22.000000 | 396.900000 | 37.970000 | 50.000000 |

The crime, area, sector, nitric oxides, 'B' appear to have multiple outliers at first look because the minimum and maximum values are so far apart. In the Age columns, the mean and the Q2(50 percentile) do not match.

We might double-check it by examining the distribution of each column.

- The rate of crime is rather low. The majority of values are in the range of 0 to 25. With a huge value and a value of zero.
- The majority of residential land is zoned for less than 25,000 square feet. Land zones larger than 25,000 square feet represent a small portion of the dataset.
- The percentage of non-retial commercial acres is mostly split between two ranges: 0-13 and 13-23.
- The majority of the properties are bordered by the river, although a tiny portion of the data is not.
- The content of nitrite dioxide has been trending lower from.3 to.7, with a little bump towards.8. It is permissible to leave a value in the range of 0.1–1.
- The number of rooms tends to cluster around the average.
- With time, the proportion of owner-occupied units rises.
- As the number of weights grows, the weight distance between 5 employment centers reduces. It could indicate that individuals choose to live in new high-employment areas.
- People choose to live in places with limited access to roadways (0-10). We have a 30th percentile outlier.
- The majority of dwelling taxes are in the range of $200-450, with large outliers around $700,000.
- The percentage of people with lower status tends to cluster around the median. The majority of persons are of lower social standing.

Because the model is overly generic, removing all outliers will underfit it. Keeping all outliers causes the model to overfit and become excessively accurate. The data's noise will be learned.

The approach is to establish a happy medium that prevents the model from becoming overly precise. When faced with a new set of data, however, they generalise well.

We'll keep numbers below 600 because there's a huge anomaly in the TAX column around 600.

new_df=housing_df[housing_df['TAX']<600]

The overall distribution, particularly the TAX, PTRATIO, and RAD, has improved slightly.

Perfect correlation is denoted by the clear values. The medium correlation between the columns is represented by the reds, while the negative correlation is represented by the black.

With a value of 0.89, we can see that 'MEDV', which is the medium price we wish to anticipate, is substantially connected with the number of rooms 'RM'. The proportion of black people in area 'B' with a value of 0.19 is followed by the residential land 'ZN' with a value of 0.32 and the percentage of black people in area 'ZN' with a value of 0.32.

The metrics that are most connected with price will be plotted.

Gradient descent is aided by feature scaling, which ensures that all features are on the same scale. It makes locating the local optimum much easier.

Mean standardization is one strategy to employ. It substitutes (target-mean) for the target to ensure that the feature has a mean of nearly zero.

def standard(X): '''Standard makes the feature 'X' have a zero mean''' mu=np.mean(X) #mean std=np.std(X) #standard deviation sta=(X-mu)/std # mean normalization return mu,std,sta mu,std,sta=standard(X) X=sta X

CRIM | ZN | INDUS | CHAS | NOX | RM | AGE | DIS | RAD | TAX | PTRATIO | B | LSTAT | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|

0 | -0.609129 | 0.092792 | -1.019125 | -0.280976 | 0.258670 | 0.279135 | 0.162095 | -0.167660 | -2.105767 | -0.235130 | -1.136863 | 0.401318 | -0.933659 |

1 | -0.575698 | -0.598153 | -0.225291 | -0.280976 | -0.423795 | 0.049252 | 0.648266 | 0.250975 | -1.496334 | -1.032339 | -0.004175 | 0.401318 | -0.219350 |

2 | -0.575730 | -0.598153 | -0.225291 | -0.280976 | -0.423795 | 1.189708 | 0.016599 | 0.250975 | -1.496334 | -1.032339 | -0.004175 | 0.298315 | -1.096782 |

3 | -0.567639 | -0.598153 | -1.040806 | -0.280976 | -0.532594 | 0.910565 | -0.526350 | 0.773661 | -0.886900 | -1.327601 | 0.403593 | 0.343869 | -1.283945 |

4 | -0.509220 | -0.598153 | -1.040806 | -0.280976 | -0.532594 | 1.132984 | -0.228261 | 0.773661 | -0.886900 | -1.327601 | 0.403593 | 0.401318 | -0.873561 |

... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |

501 | -0.519445 | -0.598153 | 0.585220 | -0.280976 | 0.604848 | 0.306004 | 0.300494 | -0.936773 | -2.105767 | -0.574682 | 1.445666 | 0.277056 | -0.128344 |

502 | -0.547094 | -0.598153 | 0.585220 | -0.280976 | 0.604848 | -0.400063 | 0.570195 | -1.027984 | -2.105767 | -0.574682 | 1.445666 | 0.401318 | -0.229652 |

503 | -0.522423 | -0.598153 | 0.585220 | -0.280976 | 0.604848 | 0.877725 | 1.077657 | -1.085260 | -2.105767 | -0.574682 | 1.445666 | 0.401318 | -0.820331 |

504 | -0.444652 | -0.598153 | 0.585220 | -0.280976 | 0.604848 | 0.606046 | 1.017329 | -0.979587 | -2.105767 | -0.574682 | 1.445666 | 0.314006 | -0.676095 |

505 | -0.543685 | -0.598153 | 0.585220 | -0.280976 | 0.604848 | -0.534410 | 0.715691 | -0.924173 | -2.105767 | -0.574682 | 1.445666 | 0.401318 | -0.435703 |

For the sake of the project, we'll apply linear regression.

Typically, we run numerous models and select the best one based on a particular criterion.

Linear regression is a sort of supervised learning model in which the response is continuous, as it relates to machine learning.

Form of Linear Regression

y= θX+θ1 or y= θ1+X1θ2 +X2θ3 + X3θ4

y is the target you will be predicting

0 is the coefficient

x is the input

We will Sklearn to develop and train the model

#Import the libraries to train the model from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression

Allow us to utilise the train/test method to learn a part of the data on one set and predict using another set using the train/test approach.

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.4) #Create and Train the model model=LinearRegression().fit(X_train,y_train) #Generate prediction predictions_test=model.predict(X_test) #Compute loss to evaluate the model coefficient= model.coef_ intercept=model.intercept_ print(coefficient,intercept) [7.22218258] 24.66379606613584

In this example, you will learn the model using below hypothesis:

Price= 24.85 + 7.18* Room

It is interpreted as:

For a decided price of a house:

A 7.18-unit increase in the price is connected with a growth in the number of rooms.

As a side note, this is an association, not a cause!

You will need a metric to determine whether our hypothesis was right. The RMSE approach will be used.

Root Means Square Error (RMSE) is defined as the square root of the mean of square error. The difference between the true and anticipated numbers called the error. It's popular because it can be expressed in y-units, which is the median price of a home in our scenario.

def rmse(predict,actual): return np.sqrt(np.mean(np.square(predict - actual))) # Split the Data into train and test set X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.4) #Create and Train the model model=LinearRegression().fit(X_train,y_train) #Generate prediction predictions_test=model.predict(X_test) #Compute loss to evaluate the model coefficient= model.coef_ intercept=model.intercept_ print(coefficient,intercept) loss=rmse(predictions_test,y_test) print('loss: ',loss) print(model.score(X_test,y_test)) #accuracy [7.43327725] 24.912055881970886 loss: 3.9673165450580714 0.7552661033654667 Loss will be 3.96

This means that y-units refer to the median value of occupied homes with 1000 dollars.

This will be less by 3960 dollars.

While learning the model you will have a high variance when you divide the data. Coefficient and intercept will vary. It's because when we utilized the train/test approach, we choose a set of data at random to place in either the train or test set. As a result, our theory will change each time the dataset is divided.

This problem can be solved using a technique called cross-validation.

With 'Forward Selection,' we'll iterate through each parameter to assist us choose the numbers characteristics to include in our model.

- Choose the most appropriate variable (in our case based on high correlation)
- Add the next best variable to the model
- Some predetermined conditions must meet.

We'll use a random state of 1 so that each iteration yields the same outcome.

cols=[] los=[] los_train=[] scor=[] i=0 while i < len(high_corr_var): cols.append(high_corr_var[i]) # Select inputs variables X=new_df[cols] #mean normalization mu,std,sta=standard(X) X=sta # Split the data into training and testing X_train,X_test,y_train,y_test= train_test_split(X,y,random_state=1) #fit the model to the training lnreg=LinearRegression().fit(X_train,y_train) #make prediction on the training test prediction_train=lnreg.predict(X_train) #make prediction on the testing test prediction=lnreg.predict(X_test) #compute the loss on train test loss=rmse(prediction,y_test) loss_train=rmse(prediction_train,y_train) los_train.append(loss_train) los.append(loss) #compute the score score=lnreg.score(X_test,y_test) scor.append(score) i+=1

We have a big 'loss' with a smaller collection of variables, yet our system will overgeneralize in this scenario. Although we have a reduced 'loss,' we have a large number of variables. However, if the model grows too precise, it may not generalize well to new data.

In order for our model to generalize well with another set of data, we might use 6 or 7 features. The characteristic chosen is descending based on how strong the price correlation is.

high_corr_var ['RM', 'ZN', 'B', 'CHAS', 'RAD', 'DIS', 'CRIM', 'NOX', 'AGE', 'TAX', 'INDUS', 'PTRATIO', 'LSTAT']

With 'RM' having a high price correlation and LSTAT having a negative price correlation.

# Create a list of features names feature_cols=['RM','ZN','B','CHAS','RAD','CRIM','DIS','NOX'] #Select inputs variables X=new_df[feature_cols] # Split the data into training and testing sets X_train,X_test,y_train,y_test= train_test_split(X,y, random_state=1) # feature engineering mu,std,sta=standard(X) X=sta # fit the model to the trainning data lnreg=LinearRegression().fit(X_train,y_train) # make prediction on the testing test prediction=lnreg.predict(X_test) # compute the loss loss=rmse(prediction,y_test) print('loss: ',loss) lnreg.score(X_test,y_test) loss: 3.212659865936143 0.8582338376696363

The test set yielded a loss of 3.21 and an accuracy of 85%.

Other factors, such as alpha, the learning rate at which our model learns, could still be tweaked to improve our model. Alternatively, return to the preprocessing section and working to increase the parameter distribution.

For more details regarding scraping real estate data you can contact Scraping Intelligence today

https://www.websitescraper.com/how-to-predict-housing-prices-with-linear-regression.php