10 Tips and Tricks for Data Scientists Vol.4

We have started a series of articles on tips and tricks for data scientists (mainly in Python and R). In case you missed vol 1, vol 1and vol 3.

Python

1.How To Get Data From Google Drive Into Colab

The Google Colab is becoming more and more popular in Data Science community. Working with Colab Jupyter notebooks, you are able to mount your Google Drive, so that to be able to get data from it directly. Let’s see how we can do it. You should open you notebook and to type the following commands:

It will send you a link to authorize your Google Drive by entering a token and you are done. Then you will see a folder called gdrive under your files:

Now, if you want to get the exact path of the file, you can just right click and choose, Copy path which usually starts with /content/gdrive/MyDrive/

2.How to Impute Missing Values In Pandas

We will provide an example of how you can impute missing values in Pandas following the rules below:

• If the variable is numeric then impute the missing values with the mean
• If the variable is object dtype then impute the missing values with the mode

import pandas as pd
import numpy as np
 
df = pd.DataFrame({'id':list(range(10)),
                   'A':[10,9,8,7,np.nan,np.nan,20,15,12,np.nan], 
                   'B':["A","B","A","A",np.nan,"B","A","B",np.nan,"A"],
                   'C':[np.nan,"BB","CC","BB","BB","CC","AA","BB",np.nan,"AA"],
                   'D':[np.nan,20,18,22,18,17,19,np.nan,17,23]})
df

for i in df.columns:
    if (df[i].dtype=='object'):
        df[i].fillna(df[i].mode()[0], inplace=True)
    elif (df[i].dtype=='float64' or df[i].dtype=='int64'):
        df[i].fillna(df[i].mean(), inplace=True)
 
df

3.How To Read and Write Gzip/Zip Files with Pandas

We can write pandas data frames as gzip and zip files. For example:

#lets use this sample dataframe
df=pd.DataFrame({'A':[1,2,3,4]})

Save it as gzip or zip

# save it as gzip
df.to_csv("dataframe.csv.gz", index=False, compression="gzip")

# save it as gzip
df.to_csv("dataframe.csv.zip", index=False, compression="zip")

Read gzip/zip files Pandas

Pandas can also read gzip/zip files that contain a CSV.

pd.read_csv('dataframe.csv.gz')

   A
0  1
1  2
2  3
3  4

4.How To Read And Write CSV Files Without Headers With Pandas

Read Without Headers

Assume that you have the following CSV file which is without column names. Let’s see how we can read it with pandas. Assume that my_file.csv is:

George,25
Maria,30
John,18

We can read the file as follows:

import pandas as pd
df = pd.read_csv("my_file.csv", header=None)
df

In case we want to give specific names to our column names:

df1 = pd.read_csv("my_file.csv", header=None, names=['colA', 'colB'])
df1 

Write Without Headers

Now, let’s say that we want to write this dataframe to a csv file but without headers.

df.to_csv('filename.csv', header=False, index=False)

As we can see the filname.csv is without headers!

5.Cumulative Count Distinct Values in Python

Sometimes there is a need to do a rolling count of the distinct values of a list/vector. In other words, we want to add up only any new element that appears in our list/vector.

import pandas as pd
 
df = pd.DataFrame({'mylist':["e", "a","a","b","a","b","c", "d", "e"]})
df['CumDistinct'] = (~df.mylist.duplicated()).cumsum()
df
 
# or by using apply
# df['CumDistinct'] = df.mylist.apply(lambda x: (~pd.Series(x).duplicated()).cumsum())

Output:

  mylist  CumDistinct
0      e            1
1      a            2
2      a            2
3      b            3
4      a            3
5      b            3
6      c            4
7      d            5
8      e            5

6.Count The Consecutive Events In Python

We are going to represent a straightforward practical example of how to count consecutive events:

import pandas as pd
df = pd.DataFrame({'Score':['win', 'loss', 'loss', 'loss', 'win', 'win', 'win', 'win', 'win', 'loss', 'win', 'loss', 'loss']})

With the following “hack” we are going to get the column of the “Streak”:

df['Streak'] = df['Score'].groupby((df['Score'] != df['Score'].shift()).cumsum()).cumcount() + 1
df

Output:

   Score  Streak
0    win       1
1   loss       1
2   loss       2
3   loss       3
4    win       1
5    win       2
6    win       3
7    win       4
8    win       5
9   loss       1
10   win       1
11  loss       1
12  loss       2

R

7.Cumulative Count Distinct Values in R

Sometimes there is a need to do a rolling count of the distinct values of a list/vector. In other words, we want to add up only any new element that appears in our list/vector.

# assume that this is our vector
x=c("e", "a","a","b","a","b","c", "d", "e")
 
# we apply the "cumsum(!duplicated(x))" command
data.frame(Vector=x,
CumDistinct=cumsum(!duplicated(x)))

Output:

  Vector CumDistinct
1      e           1
2      a           2
3      a           2
4      b           3
5      a           3
6      b           3
7      c           4
8      d           5
9      e           5

8.Count The Consecutive Events In R

Assume that there is a Roulette Wheel which returns Red (50%) and Black (50%). We are going to simulate N=1,000,000 Rolls and keep track of the streaks of Red and Black respectively. The R function which makes our life easier is the rle but if we want to track the running streak, then we need also to use the seq function. We add also another column, called EndOfStreak which indicates if the Streak has ended or not.

library(tidyverse)
 
# number of simulations
n<-1000000
 
# set a random seed for reproducibility
set.seed(5)
 
# create the data frame
df<-tibble(Rolls=seq(1:n), Outcome=sample(c("Red", "Black"),n,replace = TRUE, prob = c(0.5,0.5)))%>%
  mutate(Streak=sequence(rle(Outcome)$lengths), EndOfStreak=ifelse(lead(Outcome)==Outcome, "No", "Yes"))
 
df%>%print(n=20)

Output:

# A tibble: 1,000,000 x 4
   Rolls Outcome Streak EndOfStreak
   <int> <chr>    <int> <chr>      
 1     1 Black        1 Yes        
 2     2 Red          1 No         
 3     3 Red          2 Yes        
 4     4 Black        1 No         
 5     5 Black        2 Yes        
 6     6 Red          1 No         
 7     7 Red          2 No         
 8     8 Red          3 No         
 9     9 Red          4 Yes        
10    10 Black        1 No         
11    11 Black        2 No         
12    12 Black        3 No         
13    13 Black        4 Yes        
14    14 Red          1 Yes        
15    15 Black        1 No         
16    16 Black        2 No         
17    17 Black        3 Yes        
18    18 Red          1 No         
19    19 Red          2 No         
20    20 Red          3 No  

9.Replace Missing Values With The Mean Value

Assume that you are dealing with a Data Frame with missing values, and you want to replace them with the mean value of the corresponding column. Let’s consider the following Data Frame:

df<-data.frame(id=seq(1,10), ColumnA=c(10,9,8,7,NA,NA,20,15,12,NA), 
           ColumnB=factor(c("A","B","A","A","","B","A","B","","A")),
           ColumnC=factor(c("","BB","CC","BB","BB","CC","AA","BB","","AA")),
           ColumnD=c(NA,20,18,22,18,17,19,NA,17,23)
)
 
df

Clearly, we want to consider only the numeric variables. Let’s see how we can do it in one-line code.

df[sapply(df, is.numeric)] <- lapply(df[sapply(df, is.numeric)], function(x) ifelse(is.na(x), mean(x, na.rm = TRUE), x))
df

10.Replace Categorical Variables With Mode

In Data Science project, it is common to replace the missing values of the categorical variables with the mode. Let’s see the following example:

df<-data.frame(id=seq(1,10), ColumnA=c(10,9,8,7,NA,NA,20,15,12,NA), 
           ColumnB=c("A","B","A","A","","B","A","B","","A"),
           ColumnC=c("","BB","CC","BB","BB","CC","AA","BB","","AA"),
           ColumnD=c(NA,20,18,22,18,17,19,NA,17,23)
)
 
df

Note that the ColumnB and ColumnC are Character columns. Note also that there is no mode function in R. So let’s build it:

getmode <- function(v){
  v=v[nchar(as.character(v))>0]
  uniqv <- unique(v)
  uniqv[which.max(tabulate(match(v, uniqv)))]
}

Now let’s replace all the empty strings of the Character variables with their corresponding column mode. Finally, we should convert the character variables to factors.

df[sapply(df, is.character)] <- lapply(df[sapply(df, is.character)], function(x) ifelse(x=="", getmode(x), x))
df[sapply(df, is.character)] <- lapply(df[sapply(df, is.character)], as.factor)
df

As we can see, we replaced the empty strings with the corresponding mode.