# 1. Tips

These are some tips for the data scientist practioner.

## 1.1. Control ticks

It might be annoying that the default x- and y-axis tick labels are coarsed grained. Take the example below; the x- and y-axis ticks are even numbered. What if we want more granularity and also show all whole numbers?

:

from scipy.special import expit as logistic
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(37)
plt.style.use('ggplot')

x = np.arange(-6, 6.1, 0.1)
y = logistic(x)
s = pd.Series(y, x)

fig, ax = plt.subplots(figsize=(10, 3))

_ = s.plot.line(x, y, ax=ax)
_ = ax.set_title('Basic line plot') Use the set_xticks() and set_yticks() functions to control the ticks on the x- and y-axis.

:

fig, ax = plt.subplots(figsize=(10, 3))

_ = s.plot.line(x, y, ax=ax)
_ = ax.set_title('Basic line plot')
_ = ax.set_xticks(np.arange(-6, 6.1, 1))
_ = ax.set_yticks(np.arange(0, 1.1, 0.1)) ## 1.2. Multi-class, average precision score

In multi-class classification, your y_true (truth labels) might be a 1-dimensional vector, but your predictions y_pred (especially if you use predict_proba()) will be multi-dimensional. The average_precision_score(y_true, y_pred) expects that both y_true and y_pred are multi-dimensional. For example, the following will fail.

y_true = [1, 1, 0, 0, 2, 2]
y_pred = [
[0.0, 1.0, 0.0],
[0.0, 1.0, 0.0],
[1.0, 0.0, 0.0],
[1.0, 0.0, 0.0],
[0.0, 0.0, 1.0],
[0.0, 0.0, 1.0]
]
average_precision_score(y_true, y_pred) # fails


You will need to one-hot encode y_true.

:

from sklearn.preprocessing import OneHotEncoder
from sklearn.metrics import average_precision_score

y_true = np.array([1, 1, 0, 0, 2, 2])

encoder = OneHotEncoder()
Y = encoder.fit_transform(y_true.reshape(-1, 1)).todense()

y_pred = np.array([
[0.0, 1.0, 0.0],
[0.0, 1.0, 0.0],
[1.0, 0.0, 0.0],
[1.0, 0.0, 0.0],
[0.0, 0.0, 1.0],
[0.0, 0.0, 1.0]
])

average_precision_score(Y, y_pred)

/home/super/anaconda3/lib/python3.8/site-packages/sklearn/utils/validation.py:593: FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html
warnings.warn(

:

1.0


## 1.3. Sorting multi-index

You might have a dataframe that has multi-index for the rows and columns. How do you sort by columns or rows?

:

df = pd.DataFrame({
'height': ['tall', 'tall', 'tall', 'tall', 'short', 'short', 'short', 'short'],
'weight': ['heavy', 'heavy', 'light', 'light', 'heavy', 'heavy', 'light', 'light'],
'value': [9.9, 10.0, 7.7, 6.7, 5.5, 6.6, 3.3, 2.3]
})

stats = df.groupby(['height', 'weight']).agg(['mean', 'std'])
stats

:

value
mean std
height weight
short heavy 6.05 0.777817
light 2.80 0.707107
tall heavy 9.95 0.070711
light 7.20 0.707107

To sort based on column multi-index, you have to use sort_values() and supply a list of tuples. Below, we will sort by the mean and then std columns.

:

stats.sort_values([('value', 'mean'), ('value', 'std')])

:

value
mean std
height weight
short light 2.80 0.707107
heavy 6.05 0.777817
tall light 7.20 0.707107
heavy 9.95 0.070711

To sort based on the row multi-index, it is not as complicated, simply supply a list of index names to sort_values().

:

stats.sort_values(['weight'], axis=0)

:

value
mean std
height weight
short heavy 6.05 0.777817
tall heavy 9.95 0.070711
short light 2.80 0.707107
tall light 7.20 0.707107

If you need to sort descendingly, pass in ascending=False.

:

stats.sort_values(['weight'], axis=0, ascending=False)

:

value
mean std
height weight
short light 2.80 0.707107
tall light 7.20 0.707107
short heavy 6.05 0.777817
tall heavy 9.95 0.070711

## 1.4. One-Hot Encoding Pipeline

When you use OHE in a pipeline, you will have to deal with missing values. If you use SimpleImputer and fill missing values with a constant value, you will always end up with an extra column. What’s worse, the columns corresponding to the categorical values will NOT be missing and instead have 0’s. Additionally, your output is a matrix, and you lose the field names. Here’s an example dataframe with 2 numeric field (height and age) and 2 categorical fields (pet and color).

:

df = pd.DataFrame({
'height': [2.3, 3.3, 2.4, 5.5, np.nan],
'pet': ['cat', 'dog', 'cat', np.nan, 'dog'],
'color': ['blue', 'black', np.nan, 'brown', 'black'],
'age': [10, 5, np.nan, 4, 8]
})

df

:

height pet color age
0 2.3 cat blue 10.0
1 3.3 dog black 5.0
2 2.4 cat NaN NaN
3 5.5 NaN brown 4.0
4 NaN dog black 8.0

Let’s say we want to OHE the pet and color fields. See the problems here?

• What happened to the headers?

• The output is a matrix and not a dataframe.

• We have an additional column.

:

from sklearn.impute import SimpleImputer
from imblearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

cat_pipeline = Pipeline(steps=[
('imputer', SimpleImputer(strategy='constant', fill_value=np.nan)),
('ohe', OneHotEncoder(handle_unknown='ignore', sparse=False))
])
transformer = ColumnTransformer(transformers=[('cat', cat_pipeline, ['pet', 'color'])])

pipeline = Pipeline(steps=[
('preprocessing', transformer)
])

pipeline.fit_transform(df)

:

array([[1., 0., 0., 0., 1., 0., 0.],
[0., 1., 0., 1., 0., 0., 0.],
[1., 0., 0., 0., 0., 0., 1.],
[0., 0., 1., 0., 0., 1., 0.],
[0., 1., 0., 1., 0., 0., 0.]])


We can try to recover the field names as follows. If we specified for the transformer to passthrough (eg remainder='passthrough' the numeric fields, then we have to do even more work to align the field names. It should be apparent that the _nan fields are a nuisance and the generic field prefixes of x0_ and x1_ are not very helpful.

:

columns = transformer.named_transformers_['cat'] \
.named_steps['ohe'] \
.get_feature_names_out()
pd.DataFrame(pipeline.transform(df), columns=columns)

:

x0_cat x0_dog x0_nan x1_black x1_blue x1_brown x1_nan
0 1.0 0.0 0.0 0.0 1.0 0.0 0.0
1 0.0 1.0 0.0 1.0 0.0 0.0 0.0
2 1.0 0.0 0.0 0.0 0.0 0.0 1.0
3 0.0 0.0 1.0 0.0 0.0 1.0 0.0
4 0.0 1.0 0.0 1.0 0.0 0.0 0.0

So, what can we do? We can create a custom transformer to fix the problem. Take note of the pattern below. The OheAdjustmentTransformer is the custom transformer that takes in the ColumnTransformer; the former needs the latter to retrieve the OHE fields. Also, notice the dizzying nested structures of

• pipelines within pipelines,

• transformers within pipelines, and

• pipelines within transformers.

The final output is a little bit better since

• the field names are more meaningful (they have the original field names as prefixes instead of x0_),

• the _nan columns are dropped,

• the records for which there should be missing values instead of 0’s are corrected, and

• if we choose to pass through the numeric fields, then they are also available.

:

class OheAdjustmentTransformer:
def __init__(self, transformer, num_columns, cat_columns, missing_suffix='nan'):
self.transformer = transformer
self.num_columns = num_columns
self.cat_columns = cat_columns
self.missing_suffix = missing_suffix

def make_null(d, cat_field, nan_field):
n_df = pd.DataFrame()
for field in d.columns:
if field == nan_field:
continue
u_index = field.find('_')
val = field[u_index+1:]
n_field = f'{cat_field}_{val}'
n_df[n_field] = np.select([d[nan_field]==1], [np.nan], default=d[field])

n_df.index = d.index
return n_df

prefixes = [f'x{i}_' for i in range(len(self.cat_columns))]
c2p = {c: p for c, p in zip(self.cat_columns, prefixes)}
c2n = {c: f'{p}nan' for c, p in zip(self.cat_columns, prefixes)}
df_cols = {c: [f for f in df.columns if f.startswith(p)]
for c, p in zip(self.cat_columns, prefixes)}
dfs = {c: df[df_cols[c]] for c in df_cols}
n_dfs = {c: make_null(d, c, c2n[c]) for c, d in dfs.items()}
f_df = pd.concat([d for d in n_dfs.values()], axis=1)

return f_df

def fit(self, X=None, y=None):
return self

def transform(self, X):
cat_columns = self.transformer.named_transformers_['cat'] \
.named_steps['ohe'] \
.get_feature_names_out()
cat_columns = list(cat_columns)
num_columns = self.num_columns

if X.shape == len(cat_columns):
columns = cat_columns
else:
columns = cat_columns + num_columns

df = pd.DataFrame(X, columns=columns)

if X.shape == len(cat_columns):
else:

cat_columns = ['pet', 'color']
num_columns = [c for c in df.columns if c not in cat_columns]

cat_pipeline = Pipeline(steps=[
('imputer', SimpleImputer(strategy='constant', fill_value=np.nan)),
('ohe', OneHotEncoder(handle_unknown='ignore', sparse=False))
])

transformer = ColumnTransformer(
transformers=[('cat', cat_pipeline, cat_columns)],
remainder='passthrough')

df_pipeline = Pipeline(steps=[
])

pipeline = Pipeline(steps=[
('preprocessing', transformer),
('postprocessing', df_pipeline)
])

pipeline.fit_transform(df)

:

pet_cat pet_dog color_black color_blue color_brown height age
0 1.0 0.0 0.0 1.0 0.0 2.3 10.0
1 0.0 1.0 1.0 0.0 0.0 3.3 5.0
2 1.0 0.0 NaN NaN NaN 2.4 NaN
3 NaN NaN 0.0 0.0 1.0 5.5 4.0
4 0.0 1.0 1.0 0.0 0.0 NaN 8.0

## 1.5. Frequency Encoding

Frequency encoding is a way to turn categorial variables into numeric ones without expanding the dimensionality of the data (as in the case of one-hot encoding). Every categorical value is replaced either with its count or probability. Here’s how to implement frequency encoding with Scikit-Learn. Other implementations are available but break with missing values. This approach here does not. First, let’s create sample data.

:

df = pd.DataFrame({
'x1': [1, 2, 3, 4, np.nan, 6],
'x2': ['cat', 'dog', 'cat', 'cat', 'dog', np.nan],
'y': [1, 3, 4, 5, 2, 6]
})

df

:

x1 x2 y
0 1.0 cat 1
1 2.0 dog 3
2 3.0 cat 4
3 4.0 cat 5
4 NaN dog 2
5 6.0 NaN 6

The frequency encoder is implemented as follows. This implementation will handle with the input X is either a Pandas data frame or numpy array.

:

class FreqEncoder:
def __init__(self, variables):
self.variables_ = variables
self.v2e_ = None
self.e2v_ = None

def fit(self, X=None, y=None):
def get_df_freq(c):
return X[c].value_counts() / X[c].value_counts().sum()

def get_np_freq(i):
s = pd.Series(X[:,i]).value_counts()
return s / s.sum()

if isinstance(X, pd.DataFrame):
self.v2e_ = {c: get_df_freq(c).to_dict() for c in self.variables_}
self.e2v_ = {c: {val: key for key, val in d.items()} for c, d in self.v2e_.items()}
elif isinstance(X, np.ndarray):
self.v2e_ = {i: get_np_freq(i).to_dict() for i in self.variables_}
self.e2v_ = {i: {val: key for key, val in d.items()} for i, d in self.v2e_.items()}
else:
raise Exception(f'X type is not handled: {type(X)}')

return self

def transform(self, X):
if isinstance(X, pd.DataFrame):
return X.assign(**{c: lambda d: d[c].map(self.v2e_[c]) for c in self.variables_})
elif isinstance(X, np.ndarray):
df = pd.DataFrame(X, columns=[f'{i}' for i in range(X.shape)])
df = df.assign(**{f'{i}': lambda d: d[f'{i}'].map(self.v2e_[i]) for i in self.variables_})
return df.values
else:
raise Exception(f'X type is not handled: {type(X)}')

def inverse_transform(self, X):
if isinstance(X, pd.DataFrame):
return X.assign(**{c: lambda d: d[c].map(self.e2v_[c]) for c in self.variables_})
elif isinstance(X, np.ndarray):
df = pd.DataFrame(X, columns=[f'{i}' for i in range(X.shape)])
df = df.assign(**{f'{i}': lambda d: d[f'{i}'].map(self.e2v_[i]) for i in self.variables_})
return df.values
else:
raise Exception(f'X type is not handled: {type(X)}')


Now, let’s use the encoder and inspect its properties. The v2e_ field is the forward mapping of categorial variable values to frequencies and the e2v_ field is the inverse/backward mapping of frequencies to categorial variable values. Obviously, a concern is with ties (we leave it up to the user to add a random fudge factor and renormalize to fix this problem).

:

encoder = FreqEncoder(list(df.select_dtypes(include=['object']).columns))
encoder.fit(df)

encoder.v2e_, encoder.e2v_

:

({'x2': {'cat': 0.6, 'dog': 0.4}}, {'x2': {0.6: 'cat', 0.4: 'dog'}})


Here is the transform.

:

encoder.transform(df)

:

x1 x2 y
0 1.0 0.6 1
1 2.0 0.4 3
2 3.0 0.6 4
3 4.0 0.6 5
4 NaN 0.4 2
5 6.0 NaN 6

Here is the inverse transform.

:

encoder.inverse_transform(encoder.transform(df))

:

x1 x2 y
0 1.0 cat 1
1 2.0 dog 3
2 3.0 cat 4
3 4.0 cat 5
4 NaN dog 2
5 6.0 NaN 6

Let’s use the encoder in a pipeline.

:

from sklearn.linear_model import LinearRegression

X, y = df[['x1', 'x2']], df['y']

model = Pipeline([
('encoder', FreqEncoder(['x2'])),
('imputer', SimpleImputer()),
('estimator', LinearRegression())
])

model.fit(X, y)
model.predict(X)

:

array([1.70620043, 1.89185292, 3.61679885, 4.57209805, 3.03821197,
6.17483778])


We may access the fitted encoder from the pipeline.

:

model.steps.transform(X)

:

x1 x2
0 1.0 0.6
1 2.0 0.4
2 3.0 0.6
3 4.0 0.6
4 NaN 0.4
5 6.0 NaN

And we can see the fitted encoder in the pipeline at action with inverse transformation.

:

model.steps.inverse_transform(model.steps.transform(X))

:

x1 x2
0 1.0 cat
1 2.0 dog
2 3.0 cat
3 4.0 cat
4 NaN dog
5 6.0 NaN

For the sake of completeness, let’s work with numpy arrays.

:

X, y = df[['x1', 'x2']].values, df['y'].values
X, y

:

(array([[1.0, 'cat'],
[2.0, 'dog'],
[3.0, 'cat'],
[4.0, 'cat'],
[nan, 'dog'],
[6.0, nan]], dtype=object),
array([1, 3, 4, 5, 2, 6]))


Notice that since we are using a numpy array for X, we cannot pass in the column names and must pass in the index of the columns that need to be frequency encoded.

:

model = Pipeline([
('encoder', FreqEncoder()),
('imputer', SimpleImputer()),
('estimator', LinearRegression())
])

model.fit(X, y)
model.predict(X)

:

array([1.70620043, 1.89185292, 3.61679885, 4.57209805, 3.03821197,
6.17483778])


Here is the fitted frequency encoded model used to transform the data.

:

model.steps.transform(X[:,:2])

:

array([[1.0, 0.6],
[2.0, 0.4],
[3.0, 0.6],
[4.0, 0.6],
[nan, 0.4],
[6.0, nan]], dtype=object)


Here is the fitted frequency encoded model used to inverse transform the data.

:

model.steps.inverse_transform(model.steps.transform(X[:,:2]))

:

array([[1.0, 'cat'],
[2.0, 'dog'],
[3.0, 'cat'],
[4.0, 'cat'],
[nan, 'dog'],
[6.0, nan]], dtype=object)


## 1.6. Displaying Pandas Series and DataFrames

Often times, we want to display ALL columns and fields of a Pandas Series or DataFrame. We can set the options in a notebook cell as follows.

pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)


But now, these settings take over all subsequent displays. Later, when we need to revert, we can do the following.

# one at a time
pd.reset_option('display.max_rows')
pd.reset_option('display.max_columns')

# or all at once
pd.reset_option('all')


It’s better to use the Pandas option context manager to temporarily make the settings just for a cell.

with pd.option_context('display.max_rows', None, 'display.max_columns', None):
display(some_df)
display(some_series)