Effect of outliers in classification

Introduction

As we begin working with data, we (generally always) observe that there are few errors in the data, like missing values, outliers, no proper formatting, etc. In nutshell, we call them inconsistency. This consistency, more or less, skews the data and hamper the Machine learning algorithms to predict correctly.

In this article, we will try to see how outliers affect the accuracy of machine learning algorithm and how scaling would have helped or affected our learning. We have used 2 non-parametric algorithms, k-NN and Decision Trees for the simplicity of the objective.

About Data set

We will be using Hepatitis C Virus (HCV) for Egyptian patients Data Set obtained from UCI Machine Learning Repository. Which can be obtained from :

http://archive.ics.uci.edu/ml/datasets/Hepatitis+C+Virus+%28HCV%29+for+Egyptian+patients

This data consists of Egyptian patients who underwent treatment dosages for HCV about 18 months. There are total of 1385 patients with 29 attributes. These attributes ranges from their age, counts of WBC, RBC, plat etc.

Working with the Data

First and foremost thing is to load the data and required libraries in python.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
pd.set_option('display.max_columns', None) # This will help yoy view all the columns rather than the sample when using dataframe.head()
df = pd.read_csv('HCV-Egy-Data.csv')
df.head()

Once we are through the data set it almost advised to check if there are any inconsistencies which we mentioned earlier. To do so we will use python’s info function.

df.info()

df.info() results

Here we observe that we do not have any missing values, and since the data is a numeric data, we can be certain that all the attribute values are numeric i.e, either int64 or float64 type. Additionally, there are no null values, thus we can use our data to model.

We also want to see if there are any outliers, one quick check in pandas library is using _describe() _function. It provides us with the desired statistics like minimum- maximum values, quantiles, Standard deviation, etc.

#outliers #decision-tree #machine-learning #knn #classification #deep learning

What is GEEK

Buddha Community

What are the Best Steps to Effective Data Classification?

Data protection is not only a legal necessity. It is essential for an organization’s survival and profitability. Nowadays, storage has become cheap, and organizations have become data hoarders. And even one day will come when they’ll get around mining all of those data and look for something useful.

But, again, data hoarding causes serious issues. And most of what is collected may become redundant, old, or when it is not touched for years.

Moreover, storage might be cheap, but it is not free. And storing a huge amount of data might cost you and, more importantly, increases your risk.

So, suppose your sensitive data is stored digitally, which includes intellectual property, personally identifiable data on the customers or employees, protected health information or financial account information, and credit card details. In that case, these needs are to be properly secured.

So how to protect your data?

What is data classification?

Here are the seven effective steps to Data Classification

#big data #latest news #what are the best steps to effective data classification? #effective data classification #best #effective

Effect of outliers in classification

Introduction

As we begin working with data, we (generally always) observe that there are few errors in the data, like missing values, outliers, no proper formatting, etc. In nutshell, we call them inconsistency. This consistency, more or less, skews the data and hamper the Machine learning algorithms to predict correctly.

In this article, we will try to see how outliers affect the accuracy of machine learning algorithm and how scaling would have helped or affected our learning. We have used 2 non-parametric algorithms, k-NN and Decision Trees for the simplicity of the objective.

About Data set

We will be using Hepatitis C Virus (HCV) for Egyptian patients Data Set obtained from UCI Machine Learning Repository. Which can be obtained from :

http://archive.ics.uci.edu/ml/datasets/Hepatitis+C+Virus+%28HCV%29+for+Egyptian+patients

This data consists of Egyptian patients who underwent treatment dosages for HCV about 18 months. There are total of 1385 patients with 29 attributes. These attributes ranges from their age, counts of WBC, RBC, plat etc.

Working with the Data

First and foremost thing is to load the data and required libraries in python.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
pd.set_option('display.max_columns', None) # This will help yoy view all the columns rather than the sample when using dataframe.head()
df = pd.read_csv('HCV-Egy-Data.csv')
df.head()

Once we are through the data set it almost advised to check if there are any inconsistencies which we mentioned earlier. To do so we will use python’s info function.

df.info()

df.info() results

Here we observe that we do not have any missing values, and since the data is a numeric data, we can be certain that all the attribute values are numeric i.e, either int64 or float64 type. Additionally, there are no null values, thus we can use our data to model.

We also want to see if there are any outliers, one quick check in pandas library is using _describe() _function. It provides us with the desired statistics like minimum- maximum values, quantiles, Standard deviation, etc.

#outliers #decision-tree #machine-learning #knn #classification #deep learning

What is “effect” or “effectful” mean in Functional Programming?

A lot of the time, when we discuss the effect, we usually talk about side-effect. However, as I study more and more into functional programming and reading more and more functional programming books, I noticed many times “Effect” or “Effectful” had been widely said in the FP community when describing abstract things.

I dig a little deeper into what an “Effect” or “Effectful” means and put that in this blog post for a note to my future self.

It is not Side Effect

Usually, what they meant for “Effect” or “Effectful” is no side effect (sometimes it does). It is Main Effect.

It has something to do with Type Category

A type category is a Math Structure to abstract out representation for all the different fields in Math. When designing a program, we can think in the properties of that program before writing code instead of the other way around. For example, a function sum can be empty (identity law), has the property of combined operation and needs to be associative. (1+2 is equal to 2+1). We can characterize them as and restrict input function to be a Monoid. This way, we can create a solution in a systematic approach that generates fewer bugs.

Within Type Category is a fancy word for a wrapper that produces an “effect” on a given type. I will quote the statement that Alvin Alexander mentioned in Functional and Reactive Domain Modeling:

1. Option models the effects of optionality
2. Future models latency as an effect
3. Try abstract the consequences of failures

Those statements can be rewritten as:

1. Option is a monad that models the effect of optionality (of being something optional)
2. Future is a monad that models the impact of latency
3. Try is a monad that models the impact of failures (manages exception as an effect)

Similarly:

1. Reader is a monad that models the effect of composting operations based on some input.
2. Writer is a monad that models the impact of logging
3. State is a monad that models the impact of State
4. Sync in Cats-effect is a monad that models the effects of synchronous lazy execution.

It is an F[A] instead of A

An effect can be said of what the monad handles.

Quoting from Rob Norris in Functional Programming with Effects — an effectual function returns F[A] rather than A.

#scala #programming #functional-programming #effect #side-effects

What is an Outlier? Algorithms that are affected by outliers.

In statistics, an outlier is an observation point that is distant from other observations.

These extreme values need not necessarily impact the model performance or accuracy, but when they do they are called “Influential” points.

Note: _An _outlier_ is a data point that diverges from an overall pattern in a sample. An influential point is any point that has a large effect on the slope of a regression line._

Now the question arises that how we can detect these outliers and how to handle them?

Well before jumping straight into the solution lets explore that how the outliers being added to our dataset. What is the root cause of it.

#outliers #anomaly-detection #algorithms #outlier-detection #machine-learning

What is a Parallax Effect, and How Does it Help Your WordPress Site? | Grace Themes

The dynamic and the digital world demands more and more intuitive websites on the web and over time, no user wants to interact with the one that is not appealing and engaging to them.

There are certain web design features in the market and the Parallax Effect is one of them. It is a web design technique where the background elements scroll slower than foreground content. It can be found in various premium and free WordPress themes.

If implemented correctly, it induces a fascinating and seamless virtual experience and turns your website into a high-performing website. However, since we are talking about web design, it focuses on a variable perception of an image when someone scrolls down on your website. If you have a parallax effect in your WordPress theme, you have a foreground and a background. The background is usually covered by most of the foreground, but technically this is not a rule.

Hire WordPress developers in India @ up to 60% less cost

With this blog, you will get to know what exactly is the parallax effect and what is its effect on the WordPress site.

Read the full blog here

Best Wordpress development company in India

#parallax-effect #parallax-effect-benefits #parallax-effect-benefits-in-wordpress #wordpress