Intro¶

Regression is a supervised learning problem: labelled data passed to a model, once model is instantiated some new input data can be passed in to predict what may happen next is a continuious sequence.

The simplest model is a line. A line is a rough generalization that gives the ability to explain and predict variables that have a linear relationship with each other. A line that fits a set of data best is a Linear Regression. In regression problems, we are trying to predict a continuous-valued output: housing price is the most known dataset.

The variable that you are trying to predict is referred to as the dependent variable or response variable: denoted as y in the context of a simple linear regression and Y or yi in the context of multiple linear regression.

The data you can use for regression analysis should be divided into:

Dependent Variable (Response Variable): The variable you are trying to predict or explain.
Independent Variables (Predictor Variables): The variables used to predict or explain the variation in the dependent variable.

Regression, regressionless... Why Should I Care?¶

Load Data, Eyeball¶

For a following code kernel that loads data housing dataset a lota credits to the author of my favorite data science blog Nick H. (NapsterInBlue): https://napsterinblue.github.io/notes/machine_learning/datasets/boston/

Regression: Get The Line, Compute the Loss, Minimize it.¶

A line in maths is determined by its slope and its intercept:

y=mx+b

m (slope) is a measure of how steep the line is, while b (the intercept) is a measure of where the line hits the y-axis. The mecanism of a linear regression is to find the best m and b that represents the line that lies along with points. To understand what goes on if point is far from the best line a concept of loss (squared distance) has to be introduced. Loss, is a measure of how bad the model’s prediction is (referred as error sometimes).To find the loss, sum squared distance from each point to the line.

See the following example to understand slope, intercept and loss of line.

To find the best line that fits the you are minimizing loss, and for this you have two parameters to tune: the slope and intercept to find pair that minimizes loss on average across all of the data. Some datasets have even more than one line of best-fit. As humans we have an intuition where the line should be first, and then enter the parameters to match that image that is what machine learning also tries to achieve.

Gradient Descent for Intercept¶

Fancy name for minimization function (find the gradient of intercept(b) at every point) is gradient descent.

Imagine parabole-shaped curve and your job is to find the bottom of it since lower point than that you cannot get. You iterate through the range of values for parameter you can chage (here intercept) and update coordinates as long as loss is decreasing gradually. When you hit the bottom you need to stop iteration. Gradient descent is a process by which we find the bottom, where word gradient stands for the slope of the curve.

The curve consists of n vertices ($\ x_i, y_i$) we are iterating through all of them and compute n intercept for each of these points.

While iterating let's pay attention at a guess of 5 for the intercept and look at y at the intersection point with the curve, the slope is downward because moving on the intercept a bit, should lower the loss. It means gradient follows downwards - we are still moving to the direction that decreases our loss.

The gradient (derivative) of the loss function with respect to the intercept in a linear regression model, gives the rate of change of the loss. The formula comes out to be for a simple linear regression model (only an intercept and slope parameters given) the mean squared error (MSE):

$\frac{\partial}{\partial b} \text{MSE} = -\frac{2}{N} \sum_{i=1}^{N} (y_i - f(x_i))$

or since we know what $\ f(x_i))$ is:

$\frac{\partial}{\partial b} \text{MSE} = -\frac{2}{N} \sum_{i=1}^{N} (y_i - (mx_i​+b))$

N is the number of points we have in our dataset
m is the current gradient guess
b is the current intercept guess

Notice formula represents the summation over all data points multiplied by -2/N. The negative sign indicates the direction in which the intercept should be adjusted to reduce the mean squared error. The factor of −2/N is a result of the derivative calculation used to update the intercept parameter efficiently. Derivative calculation is heavy. So chose either accept -2/N or wrap your head around following derivation: $ \ \frac{\partial}{\partial b} \text{MSE} = \frac{\partial}{\partial b} \left( \frac{1}{N} \sum_{i=1}^{N} (y_i - f(x_i))^2 \right) = -\frac{2}{N} \sum_{i=1}^{N} (y_i - f(x_i)) \frac{\partial}{\partial b} (y_i - f(x_i)) = -\frac{2}{N} \sum_{i=1}^{N} (y_i - f(x_i)) \$

Look at this one if you prefer python:

def get_gradient_at_b(x,y,m,b):
    # takes in a set of x values, x, 
    # a set of y values, y,
    # a slope m, and an intercept value b
    # computes a diff that has the sum of 
    # all differences
    diff = sum([(y[i] - (m*x[i]+b)) for i in range(0,len(x))])
    b_gradient = -2/len(x) *diff
    return b_gradient

Gradient Descent for Slope¶

This describes the way the loss changes as the slope of our line changes:

$\frac{\partial}{\partial b} \text{MSE} = -\frac{2}{N} \sum_{i=1}^{N} x_i (y_i - (mx_i​+b))$

def get_gradient_at_m(x, y, m, b):
  diff = sum([x[i]*(y[i] - (m*x[i]+b)) for i in range(0,len(x))])
  m_gradient = -2/len(x) *diff
  return m_gradient

Synthesis:Put Together Gradient Descent for Slope and Intercept¶

Given both the m gradient and the b gradient, we’ll be able to follow both of those gradients in right direction (downwards) to the point of lowest loss (both the m value and the b value) wich are the best m and the best b to fit our data!

Steps¶

Remember we need to take some steps (pick/update value b) along with a parabola curve where the size of the step has to be not too big or not too small. A 'step size' determines a size of change to the parameters step.

Implication of a too large or too small steps.¶

Step derived from gradient scaled by learning rate. Let us take small learning rate and see the effect. A new b value in sequence computed like this:

next_b = current_b - (learning_rate * b_gradient)

where current_b is b value on our current iteration step, b_gradient is the gradient of the loss curve for our iteration step, and learning_rate is the size of the step wich is rather small: 0.01.

Convergence¶

Convergence is state when the loss is not changing a lot, the best values for the parameters m and b are supposed to be reached when the program converge. At some iteration step program reaches plateau: stabilized value of parameter b regardless of iteration. Once the loss is less than some threshold value for a bunch of adjacent steps we can stop iteration. The alternative convergence is divergence (not converge at all) is the result of large learning rate.

Learning Rate¶

The job is iteratively get the best m and b values (pairwise): from state corresponding to each pair we move a little in the direction of gradients. The step size to perform gradient descent (go down the loss curve) is a learning rate, the art of choosing it is not get runtime before getting an answer(gradient descent get to converge efficiently), and not too large to not to skip over the best value. Divergence is a result of large step: it is causing a zig-zag-like pattern due to constantly overshooting the best-value (minimum).

Recap: Gradient Descent From Scratch¶

Now when we know how to calculate the gradient and know how to choose direction let us put it together to reach convergence.

In code that follows let us focus on code after # main call: call a function called gradient_descent() that takes in x, y, learning_rate, and a num_iterations.

Pass the Real Data to Gradient Descent¶

Until now we are figured out way to find b and m values using gradient descent. Let's try it out on another dataset and check whether it does look right.

Regression Out of the Box¶

Python module Scikit-learn (sklearn) among other has the linear_model module, that has a LinearRegression() function. Differences with our own implementation are few: in OOP implementation of LinearRegression() the .fit() method returns two variables:

1. the line_fitter.coef_ (the slope),
2. the line_fitter.intercept_ (the intercept).

num_iterations and the learning_rate have default values within scikit-learn.

Execute Regression Analysis Using Scikitlearn¶

Notice how uncomplicated call to the algorithm is: in 3 lines you can find the slope and intercept that mimimize the loss. Given training input data I can proceed with the following steps:

import pandas as pd
from sklearn.linear_model import LinearRegression
# instantiate regression model
regression_model = LinearRegression()
# pass the training data to the model
regression_model.fit(X_train, y_train)
# Use the trained model to predict the corn yield for the testing set
y_pred = regression_model.predict(X_test)

Regression on Other Input Data¶

Imagine I have a good data to practise regression techniques. What is good data then? A dataset that is particularily relevant for you and where there is a linear relationship between two variables.

One common example is housing price data plotted in the beginning against the proximity to city center (it also could be the number of rooms, square footage etc). Similar comparisons such as the household income vs the education level and some related to time - study hours vs the grade received for an exam.

Important Example of Predicting Future Values¶

There are countless real-world datasets. One of the datasets that I used to work with early at university is the corn yield results in tons for each of 5 varieties across five precositiy groups and ten geographic zones. This article https://www.mdpi.com/2223-7747/12/3/446 and this https://www.pioneer.com/us/agronomy/corn-yield-gains.html covers the problem wich is pretty close to one I tried to solve and I'll try to simulate here. In this example I'll go with corn yeild data from FAO: https://www.kaggle.com/datasets/patelris/crop-yield-prediction-dataset?resource=download&select=yield.csv. If it is not accessible you can get data from: https://www.fao.org/faostat/en/#data/QCL

How to Approach the Multidimensional Data¶

The relationship between two variables is now clear to deal with, but what if there are other variables that factor in to dependent variable.

Relationship: Is there a relationship between variables yield, precocity and geographic zones? Can we predict the value of one variable (yield) based on the values of other variables (features)?

Impact of independent variables: How do changes in one or more independent variables (other geographic zone) hypothetically affect the dependent variable? Which other independent variables may have a significant impact on the dependent variable?

Forecasting: Can we use historical data to accurately forecast values of a dependent variable (yield)?

Causal analysis: Does a change in one variable cause a change in another variable? (Identify potential causal relationships between variables).

Control for confounding factors: Can we control for the influence of other factors and determine the specific impact of a particular variable on the dependent variable? Regression analysis allows for controlling and isolating the effects of different variables.

Model evaluation and improvement: How well does the regression model fit the data? Can the model be improved by including additional variables?

Multiple Linear Regression (MLR)¶

If we have multiple factors to predict the values of dependent variable we'll use Multiple Linear Regression (MLR). For example when making predictions for yield, our dependent variable, we’ll want to use multiple independent variables as variety, geographic zone etc and look at relationships. Notice the independent variables are what you change to get effect on the dependent variable whose variation you are trying to model.

So, looking at the way growth conditions affects the yield of a plant, the independent variables would be:

Geographic zone
Variety
...

The dependent variable would be yield.

MLR is based on the following equation.

$\ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_n X_n + \varepsilon \$

where:

• $ Y $ is the dependent variable,

• $\beta_0 \$ is the y-intercept,

• $ \beta_1, \beta_2, \ldots, \beta_n $ are the coefficients of the independent variables determining which independent variable carries more weight. $ ( X_1, X_2, \ldots, X_n )$ respectively,

• $ \varepsilon $ is the error term,
• $ X_1, X_2, \ldots, X_n $ are the independent variables.

Boston Housing Data for MLR¶

Let's Write MLR for Boston Housing¶

In Boston housing the input rows has an attribute 'MEDV' that represents price in 1000 $ that is the dependent variable.

1. Split the Data into Training & Test¶

As step one, divide the whole data at hand into data that the algorithm will learn from - training set (80%) and test set (20%) - part that you partitioned away that you’re actually interested in classifying. It will be used to evaluate models performance.

For each item we are interested in n features (input variables), and 1 label (output variable) features are the input information, and labels are the output or the target variable that the machine learning model aims to predict.

2. Create a Model, Fit to Your Data (Training the Model), Predict Output Variable¶

The steps for MLR in scikit-learn are identical to the steps for simple linear regression:

3. Visualize Results¶

After performed MLR,there is an array with predicted values in variable y_predict. Create a scatterplot like follows:

4. Correlations¶

To see linear interaction between value of a house and variables, different features plotted against MEDV (median value of a house). 2D scatterplot shows how the independent variables impact it. Before I move on I compute the correlation between the variable price and other variables that price might depend on.

Graphically, when you see a downward or upward trend,then a negative or a positive linear relationship exist respectively. Is there a strong correlation on a preceding plot? To understand how features potentially affect price linearly the different features against 'Median Value' were printed above each subplot.

Correlation coefficients printed above each plot, indicates independent variables relationship to Dependent Variable (Response Variable) 'Median Value'. A variable RM with coefficient of 0.7 is rather strongly correlated with the Response Variable and their positive linear relationship is visible on the plot. A negative linear relationship on the contrary means that as 'Median Value' increase, values of independent variable will decrease, e.g. LSTAT.

5. Tune & Evaluate Accuracy of a Regression Model¶

The equation for MLR of two independent variables is this: $\ Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \epsilon \$

And the general equation with n independent variables looks as following: $\ Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \ldots + \beta_nX_n + \epsilon \$

Y is the dependent variable (response variable), X1, X2 Xn are the independent variables (features), β 0 is the intercept, 1β1, 2β2, nβn are the coefficients associated with 1X1, 2X2, nXn respectively, ϵ represents the error term

The .fit() method used above returns object that has two attributes:

.coef_, - coefficients;
.intercept_, the intercept.

Coefficients help determining which independent variable has highest weight.

In our model we used 10 variables, so theres also 10 coefficients. From printed result it seems that a coefficient for variable NOX -16.2 will impact the value of a house more than a coefficient of RAD 0.2 (the former negatively and latter positively).

Residual analysis. Evaluation Metrics, Effect of New Variable¶

The difference between the actual value and the predicted value (in the latest code the actual value vs predicted value) is the residual. $\ e_i = y_i - \hat{y}_i \$

The real values should be pretty close to predicted y values. There are evaluation metrics to assess the performance of a regression model. Common metrics for regression include:

mean squared error (MSE),
root mean squared error (RMSE),
coefficient of determination (R-squared).

All of those are related to residual analysis in the context of regression models: MSE and RMSE are metrics that directly involve the residuals. R-squared often use alongside residuals to evaluate the overall goodness of fit and is defined as:

\$R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}\$ The coefficient R² is defined as:

R² = 1−u/v

where u is the residual sum of squares:

u = ((y - y_predict) ** 2).sum()

and v is the total sum of squares (TSS)

v = ((y - y.mean()) ** 2).sum()

The numerator is u - the residual sum of squares `((y - y_predict) ** 2).sum()`
The denominator v is the total sum of squares (TSS) measuring variation in the dependent (response) variable y: `((y - y.mean()) ** 2).sum()`

There is a sklearn's .score() method that returns the coefficient of determination (R-squared). For our model, the scores returned:

A higher R-squared suggests that a larger proportion of the variability in the dependent variable is captured by the model. Here is the good explanation on r-squared: https://napsterinblue.github.io/notes/machine_learning/regression/r_squared/

Providing a measure of how well the model predictions align with the actual data points. Lower values for MSE and RMSE indicate better model performance.

Here i used the .score() method from LinearRegression to find the coefficient of determination (R²) for the testing set.

We got the R² for our model is 0.83 - that means that all the x variables together explain 83% variation in dependent variable y.

Speaking of accuracy it is possible to add another x variable, 'sweetness' to our model. The effect of adding this x variable can be that the R² can go up to say 0.98. All x included by now explain 98% of the variation. Om the contrary you can remove some of the features that don’t have strong correlations! Good enough is to get a R² of 0.70.

Logistic Regression¶

In logistic regression, the goal is to model the probability that a given input belongs to a certain category or class. A common example might be classifying whether an email is spam or not based on features such as the frequency of certain words, presence of certain phrases, etc.

Recall linear regression equation was:

$\ Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_n X_n + \varepsilon \$

Now if we use linear regression to solve binary classification problem the predicted outcomes(probability) will range from negative to positive infinity.

How Logistic Regression Works¶

The trick for adapting a linear regression model is to use a link function to go from the straight line (that range from negative to positive infinity) into the curve between 0 and 1.

Log-odds¶

Log-odds is called so because it is actually the natural logarithm (ln) of odds (p/(1-p)). The logit link function is used to map the linear combination of input features to the range [0, 1], which represents probabilities. So we apply a logit link function to the left-hand side of our linear regression function, to get our predictions.

$\ \text{logit}(y) = \ln\left(\frac{y}{1-y}\right) = \ln\left(\frac{p}{1-p}\right) = \ln\left(\frac{P(event occurring)}{P(event not occurring)}\right)\ \$

Mathematically it is the natural logarithm of the odds ratio, y is replaced with the letter p because it represents a probability. That formula is the inverse of the logistic function and is used to transform the probability of belonging to a certain class into a continuous value (probabilities back into the log-odds scale ) that can be modeled using linear regression techniques.

The odds of an event occurring is: $\ \frac{P(event occurring)}{P(event not occurring)}\ \$

For example, given the probability of a mail being the spam 0.6, meaning the probability of being not spam is 1 - 0.6 = 0.4. Thus, the odds of spam are =0.6/0.4=1.5. So mail is 1.5 times more likely to be spam than not being it.

Log odds being spam is numpy.log(1.5) = 0.405465, and it is positive since the probability of a mail being spam > 0.5 (0.6).

If we add to the right-hand side the regular linear regression we get the whole thing called sigmoid_function: $\ \ln\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \ldots + \beta_n X_n + \varepsilon \$

Suppose that we still want to fit a model that predicts whether a mail is spam. We’ll use the amount of special words as a predictor. The following code fits the model:

from sklearn.linear_model import LogisticRegression
model = LogisticRegression()
model.fit(special_words_in_mail, spam)

Next, use ordinary regression equation to make predictions for each datapoint:

log_odds = [model.intercept_ + model.coef_ * word 
            for word in special_words_in_mail]

Notice that result for following five datapoints represents a floating point values ranging from negative to positive infinity: these are log odds.

[[2.123343391]
 [-1.59357205]
 [-0.00350846]
 [-3.28394203]
 [ 2.18476234]]

$\ \ln\left(\frac{p}{1-p}\right) =2.123343391 \$ Look how to convert log odd into a probability:

Once you derived: $\ p = \frac{e^{2.123}}{1+e^{2.123}}\ \$ ... this beautiful expression (called the sigmoid function by the way ) can easily be translated into python: np.exp(log_odds)/(1+ np.exp(log_odds)). Why use it? To convert the predicted log odds for each record into a probability of 'positive' outcome and it produces the S-shaped curve (ranges between 0 and 1) instead of a line.

Sklearn's Logistic Regression Implementation¶

Now we are fitting the curve to our data from spam example instead of a line.

Let us look at the previous code. From scikit learn we create a LogisticRegression object and use its method .fit(), which takes a matrix of features and a matrix of class labels (the outcome to be predicted).

Make Logistic Regression Right¶

Split Into Traing/Test Sets¶

We need to split data into training and test sets as follows: X_train - feature matrix contain one dimension which is the presence of special words. y_train contains the outcome for the train data (1 indicates spam and 0 indicates not spam).

Create Logistic Regression Object, Feed With Data¶

Here we get our model and to interpret the coefficient and itercept think as follows:

• large positive coef - indicates than an increase in that feature is associated with a large increase in the log odds / probability of belonging to group labeled as 1.
• lagre negative indicates negative correlation and the 0 is the absence of the association with the outcome.

Looking back at our spam example we can expect that feature (special words) is strongly associated with mails' probability of being classified as spam.

Classification Thresholding¶

Logistic regression has attribute probablility based on which we classify a datapoint belongs to. If predicted probability of positive outcome >= 0.5 (default threshold classification), then the datapoint is assigned to the positive class. Data scientist can move the classification threshold down to 0.4 or even 0.3, whicg increase the sensitivity of our model to predicting a spam.

Imagine we have a sequence of probabilities for 5 datapoints:[0.2906 0.7912 0.3512 0.5734 0.8712], taking a threshold of 0.5 gives you one classification, and another classification if we use a different threshold (e.g. 0.6).

If we know ground truth we can count misclassified points, and choose the threshold probability that would correctly classify all or most of datapoints.

Confusion Matrix¶

A confusion matrix,is the table with counts of true positives, false positives, true negatives, and false negatives.

For example, imagine that for 5 datapoints:[0.2906 0.7912 0.3512 0.5734 0.8712] the true and predicted classes for a logistic regression model are:

y_true = [0, 1, 0, 1, 1]
y_pred = [0, 1, 1, 0, 1]

Scikit-learn has a method for confusion matrix computation:

Interpret Confusion Matrix¶

This output tells us that there are 1 true negatives, 1 false negative, 2 true positives, and 1 false positives. We strive after model giving confusion_matrix with the main diagonal (indices row0 col0, row1 col1 which are the true negatives and true positives, respectively) to be largest. Incorrect classifications were also there (false positives or false negatives). Below is my illustration to confusion matrix, you have kind of flag which is probability of classification: red - False, green - True, while heads are the true labels same color coding. T = true, F = false, P = positive, N = negative

Summarize Confusion Matrix¶

There are four metrics to summarize confusion matrix. :

- Accuracy = (TP + TN)/(TP + FP + TN + FN)
- Precision = TP/(TP + FP)
- Recall = TP/(TP + FN)  #  (sensitivity or true positive rate)
- F1 score: weighted average of precision and recall

Each taking value on scale between 0 and 1 where closer to 1 is better and closer to 0 is worse.

Logistic Regression Review¶

• Goal is to perform binary classification for a dataset with a bunch of features.
• LR implementation requires the features to beto normalized/standardized so they vary over the same range before fitting LR model (regularization happens under the hood in scikit learn).
• Logistic regression is when a linear regression formula extended at left side with a logit link function to fit a sigmoid curve (rather than line) to the data. Why? Since we need to coerce prediction values to be between 0 and 1, therefore we need a sigmoid function.
• Logistic regression model returns coefficients to estimate the log odds of positive classification. To calculate the log-odds, we multiply each feature coefficient by its respective feature value, sum the products, and add the intercept. LR algorithm find the coefficient and intercept that minimize the log-loss which is cost function. '''When calculating Log-Loss, the penalty for each data sample is a function of the probability the model predicted for that sample. For confident incorrect predictions, the penalty is large, and for confident correct predictions, the penalty is small'''. https://www.codecademy.com/courses/machine-learning-logistic-regression/quizzes/logistic-regression-quiz The log odds casted into a probability to get predictions between 0 and 1.
• These coefficients give an estimate relative feature importance if more than one are used. Always check the smallest/largest magnitude coefficient.
• A classification threshold is the spot(cutoff) from which a data sample is classified as belonging to a positive class: the default is 0.5. Remember in the cancer example decreasing the standard classification threshold of a Logistic Regression model below 0.5 is the good idea since we then has to control more candidates that has lower yet quit high probability to get cancer.
• A confusion matrix or summary statistics (accuracy, precision, recall, and F1 score) are the model quality metrics.

Titanic Disaster¶

In our model we used 4 variables, so theres also 4coefficients: it seems that a coefficient for variable Sex and FirstClass will impact the value of a survivalship more than a coefficient of Age and SecondClass (the former two positively and latter negatively).

Make Predictions!¶

Who will survive? .predict() method on sample_passengers clarify the odds of you being a lucky Titanic passenger. The .predict_proba() method on sample_passengers give details on predictions: perishing/ surviving the sinking. After the final classification it appears that regardless of the threshold the chances are bad to survive for travellers in the second class but higher if you are female even in a second class.

Logistic Regression Deep Dive. Assumptions¶

Assumptions 1, 2, 3¶

Given a data remember there are primary assumptions about the data that goes into LR:

1. Target Variable is binary.
2. Independent Observations (no repeated items): bad if for example individuals included in sample multiple times. Always compare the number of unique IDs with sample size - it should be the same: df.id.nunique() == df.id.count().
3. Large sample size so the model do not fail to converge: lm is fit using maximum likelihood estimation instead of least squares minimization to get convergence big sample needed - a rule of thumb at least 10 samples per feature for the smallest class in the outcome variable: max_features = (the smallest class size) / 10 which is max_features = min(df.my_target variable.value_counts()/10).

Assumption 4¶

No influential outliers: any extremely influential outliers are bad for model building - rule out them using z-scores, scaler of the IQR, Cook's distance/influence/leverage.

Now you can see that fractal_dimension_mean has less extreme outliers number. As per assumption 4 LR is sensitive to extreme outliers causing distorsion of findings. Notice fractal_dimension_mean feature has many extreme outliers above the box distribution(grouping).

Assumption 5 Logit Linearly Related to Feature¶

1. Linear relationship between features and log odds(the logit of the outcome) has to exist. Easy to get the visual proof of it using Seaborn regplot (sns.regplot()), with the parameter logistic= True and the feature of interest as argument x. Look at the plot to see if fit model will resemble a sigmoidal curve for feature area_mean.

The fit model resemble a sigmoidal curve - linear relationship exist between some features and the target variable in example above. On the contrary if we take a feature fractal_dimension_mean to get a sigmoidal curve we get a proof of absence of linear relationship between it and target variable.

Assumption 6¶

Multicollinearity in the data is not a good assumption for use of lr. Eliminate highly correlated features otherwise the coefficients and p-values will be inaccurate.

Identify which features are highly correlated and drop one of the features: heatmap correlation plot is a good tool for identifying and to be able to eliminate features of high correlation.

You can now identify highly correlated features and eliminate them from model correlated_pair = ['compactness_mean', 'concavity_mean'] by simply dropping the highly correlated features df = df.drop(columns=correlated_pair). So our predictor variables that are left to participate in lr analysis against outcome variable are following:

Model Hyperparameters¶

Hyperparameters are set before training a model and tuned later to improve model performance while parameters (not hyper) are the attributes of the model implementation (the intercept and coefficients).

Evaluation Metrics of Classification¶

The valuation metrics are common with those for classification tasks. I mentioned them in this article, brush up some concepts in 'Interpret Confusion Matrix' section above. Accuracy – correct predictions count out of the total is one of them but it is misleading metric for imbalanced classes. For predicting probability for one particular class other metrics may be more appropriate to evaluate models performance: precision, recall, or F1-score.

So which metrics matter most for cancer analysis? For cancer dataset, predicting ALL malignant cases as malignant is important regardless of some false positives (true benign marked as possibly malignant)- discarded by follow-up tests; missing a malignant case (false negative) is an expensive mistake. Pulling down false negative will maximize the recall ratio (true positive rate). Recall = TP/(TP + FN)

Let us explore common evaluation metrics for classification. For this we use the predicted outcomes and ground truth test data (x_test).

Confusion Matrix as Evaluation Metrics¶

There are four possible categories: True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN). The predicted classes goes as columns and the actual classes are represented as rows. Confusion matrix is a nice way to get an overview of model predictions.

| *        | Predicted - | Predicted + |
|----------|-------------|-------------|
| Actual - | TN          | FP          |
| Actual + | FN          | TP          |

How many tumors were predicted correctly? How many people's tumors were predicted as false benign and vise versa?

When looking at the confusion matrix (look Interpret Confusion Matrix section) answer following questions:

1. How many correct predictions there are for true positives?
2. Questions about incorrect classifications were also there (false positives or false negatives): how many were classed as benign while being malignant in fact (vise versa)?

You can take one step further and visualize the confusion matrix.

Thresholds¶

Along with class of a sample there is a probability value. Threshold of lowest probability of positive outcome (was named before) helps to tune a model's predictions. An example of a prediction threshold: a default probability > 50% means predicted positive outcome.This way two samples with predicted probabilities of 51% and 99% are considered positive.

The nice way to visualize overlay between positive and negative classes is the histogram of the predicted probabilities for the lr classifier. A lot of points of negative class are grouped around zero probability value. Higher threshold gives fewer false positives and more false negatives, whereas a lower one is associated with fewer false negatives and more false positives.

Untill now, we’ve trained a logreg model, fit data and calculated the predicted class. We can manage to get probability of true outcome =1 when y_pred_prob > 0.5 and 0 otherwise. The predict_proba method gives us the predicted probabilities out of the box and both results can be compared using array_equal():

Let us try alternative thresholds:

On the latest barplot you see how a number of false negatives increases along with threshold values: with the lowest threshold value all the malignant cases are identified (0 false negatives). A higher value gives more false negatives. Whereas a lower value is associated with fewer false negatives and more false positives. Use threshols to tune a model's prediction accuracy.

ROC Curves and Area Under the Curve¶

We inspected a continuum of predictions as per parameterized model by giving the sequence of thresholds in range (0.0 - 1.0) - remember I used np.linespace for this.

There is another way to look at those metrics using:

the True Positive Rate (TPR): TPR = TP / TP + FN,

the False Positive Rate (FPR):FPR = FP / TP + FP.

Plotting these rates as points (TPR on the y-axis and FPR on the x-axis) gives a Receiver Operating Characteristic (ROC) curve. ROC can help choose a threshold that best fits our problem, here: catch every malignant tumor meaning higher TPR. Once ROC is plotted we can get the

The AUC of 0.98 (close to 1) is an indicator of good model separability: is good at distinguishing between malignant and benign tumors with high TPR (true positive rate) also called a recall.

Managing Class Imbalance to Increase Recall Score¶

In real world individuals deals with conformity which I thought is a good analogy with class imbalance. Class imbalance is when the binary categories of outcome variable isn't evenly distributed by data points.

Classes are more or less imbalanced and the majority class is predicted more often and therefore it may create a bias toward it.

To identify the imbalance use Positivity Rate , then use one of the following to counteract it (get higher recall score):

• Stratified Sampling,
• Under-/Oversampling,
• Class Width Balancing.

Positivity Rate¶

Positivity Rate = Total True Positives / Total Cases, values being around 0.5 indicates balanced classes.

Stratified Sampling and its Effect¶

When small dataset with imbalanced classes are splitted into test/traing datasets it increases differences between classes size even more: a random split using stratification on the class labels ensures that the distribution of the target variable (labels) is maintained across different subsets of the data. If dataset originally has 70% instances of class A and 30% instances of class B, then after stratification the same class distribution is preserved.

So contrary to the default test train split...

x_train, x_test, y_train, y_test = train_test_split(df[predictor_var], df[outcome_var], random_state=0, test_size=0.3)

... we want to test stratified sampling approach, using stratify argument available out of the box in scikit-learn.

After observing positivity rates as per stratified sampling the positivity rates appears similar.

Recall decreased a little in connection with stratification and accuracy increased.

Bonus Lines¶

Gradient descent is useful optimization algorithm not only to improve a regression model. When I was describing loss function it comes to mind the example in a book by François Chollet on loss function. Even if it is described in another context, I will have it here anyways.

Dummy Data for MLR¶

Create a pandas DataFrame with columns for corn yield, variety, precocity, and geographic zone. Here I will generate data for corn production problem using the numpy and pandas.