<strong>Originally published by </strong><a href="https://towardsdatascience.com/@citynorman" target="_blank">Norm Niemer</a><strong> </strong><em>at </em><a href="https://towardsdatascience.com/top-10-coding-mistakes-made-by-data-scientists-bb5bc82faaee" target="_blank">towardsdatascience.com</a>
A data scientist is a “person who is better at statistics than any software engineer and better at software engineering than any statistician”. Many data scientists have a statistics background and little experience with software engineering. I’m a senior data scientist ranked top 1% on Stackoverflow for python coding and work with a lot of (junior) data scientists. Here is my list of 10 common mistakes I frequently see.
Data science needs code AND data. So for someone else to be able to reproduce your results, they need to have access to the data. Seems basic but a lot of people forget to share the data with their code.
import pandas as pd df1 = pd.read_csv('file-i-dont-have.csv') # fails do_stuff(df)
Solution: Use d6tpipe to share data files with your code or upload to S3/web/google drive etc or save to a database so the recipient can retrieve files (but don’t add them to git, see below).
Similar to mistake 1, if you hardcode paths others don’t have access to, they can’t run your code and have to look in lots of places to manually change paths. Booo!
import pandas as pd df = pd.read_csv('/path/i-dont/have/data.csv') # fails do_stuff(df) # or impor os os.chdir('c:\\Users\\yourname\\desktop\\python') # fails
Solution: Use relative paths, global path config variables or d6tpipe to make your data easily accessible.
Since data science code needs data why not dump it in the same directory? And while you are at it, save images, reports and other junk there too. Yikes, what a mess!
├── data.csv ├── ingest.py ├── other-data.csv ├── output.png ├── report.html └── run.py
Most people now version control their code (if you don’t that’s another mistake!! See git). In an attempt to share data, it might be tempting to add data files to version control. That’s ok for very small files but git is not optimized for data, especially large files.
git add data.csv
Enough about data, lets talk about the actual code! Since one of the first things you learn when you learn to code are functions, data science code is mostly organized as a series of functions that are run linearly. That causes several problems, see 4 Reasons Why Your Machine Learning Code is Probably Bad.
def process_data(data, parameter): data = do_stuff(data) data.to_pickle('data.pkl') data = pd.read_csv('data.csv') process_data(data) df_train = pd.read_pickle(df_train) model = sklearn.svm.SVC() model.fit(df_train.iloc[:,:-1], df_train['y'])
Like functions, for loops are the first thing you learn when you learn to code. Easy to understand, but they are slow and excessively wordy, typically indicating you are unaware of vectorized alternatives.
x = range(10) avg = sum(x)/len(x); std = math.sqrt(sum((i-avg)**2 for i in x)/len(x)); zscore = [(i-avg)/std for x] # should be: scipy.stats.zscore(x) # or groupavg =  for i in df['g'].unique(): dfg = df[df[g']==i] groupavg.append(dfg['g'].mean()) # should be: df.groupby('g').mean()
As data, parameters or user input change, your code might break, sometimes without you noticing. That can lead to bad output and if someone makes decisions based on your output, bad data will lead to bad decisions!
assert df['id'].unique().shape == len(ids) # have data for all ids? assert df.isna().sum()<0.9 # catch missing values assert df.groupby(['g','date']).size().max() ==1 # no duplicate values/date? assert d6tjoin.utils.PreJoin([df1,df2],['id','date']).is_all_matched() # all ids matched?
I get it, you’re in a hurry to produce some analysis. You hack things together to get results to your client or boss. Then a week later they come back and say “can you change xyz” or “can you update this please”. You look at your code and can’t remember why you did what you did. And now imagine someone else has to run it.
def some_complicated_function(data): data = data[data['column']!='wrong'] data = data.groupby('date').apply(lambda x: complicated_stuff(x)) data = data[data['value']<0.9] return data
Solution: Take the extra time, even if it’s after you’ve delivered the analysis, to document what you did. You will thank yourself and other will do so even more! You’ll look like a pro!
Back data, it’s DATA science after all. Just like functions and for loops, CSVs and pickle files are commonly used but they are actually not very good. CSVs don’t include a schema so everyone has to parse numbers and dates again. Pickles solve that but only work in python and are not compressed. Both are not good formats to store large datasets.
def process_data(data, parameter): data = do_stuff(data) data.to_pickle('data.pkl') data = pd.read_csv('data.csv') process_data(data) df_train = pd.read_pickle(df_train)
Lets conclude with a controversial one: jupyter notebooks are as common as CSVs. A lot of people use them. That doesn’t make them good. Jupyter notebooks promote a lot of bad software engineering habits mentioned above, notably:
It feels easy to get started but scales poorly.
Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments. Our latest survey report suggests that as the overall Data Science and Analytics market evolves to adapt to the constantly changing economic and business environments, data scientists and AI practitioners should be aware of the skills and tools that the broader community is working on. A good grip in these skills will further help data science enthusiasts to get the best jobs that various industries in their data science functions are offering.
In the programming world, Data types play an important role. Each Variable is stored in different data types and responsible for various functions. Python had two different objects, and They are mutable and immutable objects.
This Data Science with Python Tutorial will help you understand what is Data Science, basics of Python for data analysis, why learn Python, how to install Python, Python libraries for data analysis, exploratory analysis using Pandas, introduction to series and dataframe, loan prediction problem, data wrangling using Pandas, building a predictive model using Scikit-Learn and implementing logistic regression model using Python.
Become a data analysis expert using the R programming language in this [data science](https://360digitmg.com/usa/data-science-using-python-and-r-programming-in-dallas "data science") certification training in Dallas, TX. You will master data...
The agenda of the talk included an introduction to 3D data, its applications and case studies, 3D data alignment and more.