Teresa  Jerde

Teresa Jerde

1597969260

K-Nearest Neighbor Algorithm with Amazon Food Reviews Analysis

First We want to know What is Amazon Fine Food Review Analysis?

This dataset consists of reviews of fine foods from amazon. The data span a period of more than 10 years, including all ~500,000 reviews up to October 2012. Reviews include product and user information, ratings, and a plaintext review. We also have reviews from all other Amazon categories.

Amazon reviews are often the most publicly visible reviews of consumer products. As a frequent Amazon user, I was interested in examining the structure of a large database of Amazon reviews and visualizing this information so as to be a smarter consumer and reviewer.

#data-science #machine-learning #applied-ai #towards-data-science #knn-algorithm

What is GEEK

Buddha Community

K-Nearest Neighbor Algorithm with Amazon Food Reviews Analysis
Tyrique  Littel

Tyrique Littel

1604008800

Static Code Analysis: What It Is? How to Use It?

Static code analysis refers to the technique of approximating the runtime behavior of a program. In other words, it is the process of predicting the output of a program without actually executing it.

Lately, however, the term “Static Code Analysis” is more commonly used to refer to one of the applications of this technique rather than the technique itself — program comprehension — understanding the program and detecting issues in it (anything from syntax errors to type mismatches, performance hogs likely bugs, security loopholes, etc.). This is the usage we’d be referring to throughout this post.

“The refinement of techniques for the prompt discovery of error serves as well as any other as a hallmark of what we mean by science.”

  • J. Robert Oppenheimer

Outline

We cover a lot of ground in this post. The aim is to build an understanding of static code analysis and to equip you with the basic theory, and the right tools so that you can write analyzers on your own.

We start our journey with laying down the essential parts of the pipeline which a compiler follows to understand what a piece of code does. We learn where to tap points in this pipeline to plug in our analyzers and extract meaningful information. In the latter half, we get our feet wet, and write four such static analyzers, completely from scratch, in Python.

Note that although the ideas here are discussed in light of Python, static code analyzers across all programming languages are carved out along similar lines. We chose Python because of the availability of an easy to use ast module, and wide adoption of the language itself.

How does it all work?

Before a computer can finally “understand” and execute a piece of code, it goes through a series of complicated transformations:

static analysis workflow

As you can see in the diagram (go ahead, zoom it!), the static analyzers feed on the output of these stages. To be able to better understand the static analysis techniques, let’s look at each of these steps in some more detail:

Scanning

The first thing that a compiler does when trying to understand a piece of code is to break it down into smaller chunks, also known as tokens. Tokens are akin to what words are in a language.

A token might consist of either a single character, like (, or literals (like integers, strings, e.g., 7Bob, etc.), or reserved keywords of that language (e.g, def in Python). Characters which do not contribute towards the semantics of a program, like trailing whitespace, comments, etc. are often discarded by the scanner.

Python provides the tokenize module in its standard library to let you play around with tokens:

Python

1

import io

2

import tokenize

3

4

code = b"color = input('Enter your favourite color: ')"

5

6

for token in tokenize.tokenize(io.BytesIO(code).readline):

7

    print(token)

Python

1

TokenInfo(type=62 (ENCODING),  string='utf-8')

2

TokenInfo(type=1  (NAME),      string='color')

3

TokenInfo(type=54 (OP),        string='=')

4

TokenInfo(type=1  (NAME),      string='input')

5

TokenInfo(type=54 (OP),        string='(')

6

TokenInfo(type=3  (STRING),    string="'Enter your favourite color: '")

7

TokenInfo(type=54 (OP),        string=')')

8

TokenInfo(type=4  (NEWLINE),   string='')

9

TokenInfo(type=0  (ENDMARKER), string='')

(Note that for the sake of readability, I’ve omitted a few columns from the result above — metadata like starting index, ending index, a copy of the line on which a token occurs, etc.)

#code quality #code review #static analysis #static code analysis #code analysis #static analysis tools #code review tips #static code analyzer #static code analysis tool #static analyzer

Exploring The Brute Force K-Nearest Neighbors Algorithm

Did you find any difference between the two graphs?

Both show the accuracy of a classification problem for K values between 1 to 10.

Both of the graphs use the KNN classifier model with ‘Brute-force’ algorithm and ‘Euclidean’ distance metric on same dataset. Then why is there a difference in the accuracy between the two graphs?

Before answering that question, let me just walk you through the KNN algorithm pseudo code.

I hope all are familiar with k-nearest neighbour algorithm. If not, you can read the basics about it at https://www.analyticsvidhya.com/blog/2018/03/introduction-k-neighbours-algorithm-clustering/.

We can implement a KNN model by following the below steps:

  1. Load the data
  2. Initialise the value of k
  3. For getting the predicted class, iterate from 1 to total number of training data points
  4. Calculate the distance between test data and each row of training data. Here we will use Euclidean distance as our distance metric since it’s the most popular method. Some of the other metrics that can be used are Chebyshev, cosine, etc.
  5. Sort the calculated distances in ascending order based on distance values
  6. Get top k rows from the sorted array
  7. Get the most frequent class of these rows
  8. Return the predicted class

#2020 oct tutorials # overviews #algorithms #k-nearest neighbors #machine learning #python

Teresa  Jerde

Teresa Jerde

1597969260

K-Nearest Neighbor Algorithm with Amazon Food Reviews Analysis

First We want to know What is Amazon Fine Food Review Analysis?

This dataset consists of reviews of fine foods from amazon. The data span a period of more than 10 years, including all ~500,000 reviews up to October 2012. Reviews include product and user information, ratings, and a plaintext review. We also have reviews from all other Amazon categories.

Amazon reviews are often the most publicly visible reviews of consumer products. As a frequent Amazon user, I was interested in examining the structure of a large database of Amazon reviews and visualizing this information so as to be a smarter consumer and reviewer.

#data-science #machine-learning #applied-ai #towards-data-science #knn-algorithm

K-Nearest Neighbors

A perfect opening line I must say for presenting the K-Nearest Neighbors. Yes, that’s how simple the concept behind KNN is. It just classifies a data point based on its few nearest neighbors. How many neighbors? That is what we decide.

Looks like you already know a lot of there is to know about this simple model. Let’s dive in to have a much closer look.

Before moving on, it’s important to know that KNN can be used for both classification and regression problems. We will first understand how it works for a classification problem, thereby making it easier to visualize regression.

KNN Classifier

The data we are going to use is the Breast Cancer Wisconsin(Diagnostic) Data Set_. _There are 30 attributes that correspond to the real-valued features computed for a cell nucleus under consideration. A total of 569 such samples are present in this data, out of which 357 are classified as ‘benign’ (harmless) and the rest 212 are classified as _‘malignant’ _(harmful).

The diagnosis column contains ‘M’ or ‘B’ values for malignant and benign cancers respectively. I have changed these values to 1 and 0 respectively, for better analysis.

Also, for the sake of this post, I will only use two attributes from the data → ‘mean radius’ and ‘mean texture’. This will later help us visualize the decision boundaries drawn by KNN. Here’s how the final data looks like (after shuffling):

Let’s code the KNN:

# Defining X and y
X = data.drop('diagnosis',axis=1)
y = data.diagnosis

# Splitting data into train and test
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.25,random_state=42)
# Importing and fitting KNN classifier for k=3
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train,y_train)
# Predicting results using Test data set
pred = knn.predict(X_test)
from sklearn.metrics import accuracy_score
accuracy_score(pred,y_test)

The above code should give you the following output with a slight variation.

0.8601398601398601

What just happened? When we trained the KNN on training data, it took the following steps for each data sample:

  1. Calculate the distance between the data sample and every other sample with the help of a method such as Euclidean.
  2. Sort these values of distances in ascending order.
  3. Choose the top K values from the sorted distances.
  4. Assign the class to the sample based on the most frequent class in the above K values.

Let’s visualize how KNN drew a decision boundary on the train data set and how the same boundary is then used to classify the test data set.

KNN Classification at K=3. Image by Sangeet Aggarwal

With the training accuracy of 93% and the test accuracy of 86%, our model might have shown overfitting here. Why so?

When the value of K or the number of neighbors is too low, the model picks only the values that are closest to the data sample, thus forming a very complex decision boundary as shown above. Such a model fails to generalize well on the test data set, thereby showing poor results.

The problem can be solved by tuning the value of _n_neighbors _parameter. As we increase the number of neighbors, the model starts to generalize well, but increasing the value too much would again drop the performance.

Therefore, it’s important to find an optimal value of K, such that the model is able to classify well on the test data set. Let’s observe the train and test accuracies as we increase the number of neighbors.

#knn-algorithm #data-science #knn #nearest-neighbors #machine-learning #algorithms

Tyrique  Littel

Tyrique Littel

1604023200

Effective Code Reviews: A Primer

Peer code reviews as a process have increasingly been adopted by engineering teams around the world. And for good reason — code reviews have been proven to improve software quality and save developers’ time in the long run. A lot has been written about how code reviews help engineering teams by leading software engineering practitioners. My favorite is this quote by Karl Wiegers, author of the seminal paper on this topic, Humanizing Peer Reviews:

Peer review – an activity in which people other than the author of a software deliverable examine it for defects and improvement opportunities – is one of the most powerful software quality tools available. Peer review methods include inspections, walkthroughs, peer deskchecks, and other similar activities. After experiencing the benefits of peer reviews for nearly fifteen years, I would never work in a team that did not perform them.

It is worth the time and effort to put together a code review strategy and consistently follow it in the team. In essence, this has a two-pronged benefit: more pair of eyes looking at the code decreases the chances of bugs and bad design patterns entering your codebase, and embracing the process fosters knowledge sharing and positive collaboration culture in the team.

Here are 6 tips to ensure effective peer reviews in your team.

1. Keep the Changes Small and Focused

Code reviews require developers to look at someone else’s code, most of which is completely new most of the times. Too many lines of code to review at once requires a huge amount of cognitive effort, and the quality of review diminishes as the size of changes increases. While there’s no golden number of LOCs, it is recommended to create small pull-requests which can be managed easily. If there are a lot of changes going in a release, it is better to chunk it down into a number of small pull-requests.

2. Ensure Logical Coherence of Changes

Code reviews are the most effective when the changes are focused and have logical coherence. When doing refactoring, refrain from making behavioral changes. Similarly, behavioral changes should not include refactoring and style violation fixes. Following this convention prevents unintended changes creeping in unnoticed in the code base.

3. Have Automated Tests, and Track Coverage

Automated tests of your preferred flavor — units, integration tests, end-to-end tests, etc. help automatically ensure correctness. Consistently ensuring that changes proposed are covered by some kind of automated frees up time for more qualitative review; allowing for a more insightful and in-depth conversation on deeper issues.

4. Self-Review Changes Before Submitting for Peer Review

A change can implement a new feature or fix an existing issue. It is recommended that the requester submits only those changes that are complete, and tested for correctness manually. Before creating the pull-request, a quick glance on what changes are being proposed helps ensure that no extraneous files are added in the changeset. This saves tons of time for the reviewers.

5. Automate What Can Be Automated

Human review time is expensive, and the best use of a developer’s time is reviewing qualitative aspects of code — logic, design patterns, software architecture, and so on. Linting tools can help automatically take care of style and formatting conventions. Continuous Quality tools can help catch potential bugs, anti-patterns and security issues which can be fixed by the developer before they make a change request. Most of these tools integrate well with code hosting platforms as well.

6. Be Positive, Polite, and Respectful

Finally, be cognizant of the fact that people on both sides of the review are but human. Offer positive feedback, and accept criticism humbly. Instead of beating oneself upon the literal meaning of words, it really pays off to look at reviews as people trying to achieve what’s best for the team, albeit in possibly different ways. Being cognizant of this aspect can save a lot of resentment and unmitigated negativity.

#agile #code quality #code review #static analysis #code analysis #code reviews #static analysis tools #code review tips #continuous quality #static analyzer