Modeling Online News Popularity

In my previous article on Studying Online News Popularity we used data collected in 2015 by researchers from the Universities of Porto and Minho in Portugal (using ~39000 articles that had been published on Mashable between January 7 2013 to January 7 2015) to examine the relationship between the type and publishing day of an article to its popularity.

In this article we will examine the use of this dataset to build predictive Linear and Tree based models.

The data is mostly clean but there was some work required to combine columns, that were essentially One Hot Encoded, into Categorical columns for Data Channel and Day of the Week data as well some scripting to fill in some missing Data Channel values.

The main reason for this is that while One Hot Encoding is good for linear models, the tree based models perform better with Ordinal Encoding for Categorical data.

The Dataset

As previously mentioned this is a large dataset with ~39000 observations of some 50 attributes. Two of them, namely url and timedelta (the difference between the date the article was downloaded and the date of its publications) are non-predictive and can be discarded. That leaves 47 predictive and 1 target attribute shares(the number of views for the article).

The problem was changed to one of Classification by creating a new target attribute popularity with 2 classes, popular(1) and unpopular(0)— set to a value of 1 if shares > 1400 and 0 otherwise, where 1400 is the median shares value.

. The distribution of popularity values was reasonably balanced with 53% popular(1) and 47% unpopular(0).

The url, timedelta and shares attributes were then dropped from the dataset.

Data Modeling

Since the distribution of popularity values is balanced, accuracy makes a good evaluation metric with the baseline accuracy value being the percentage of the largest class, expressed as a fraction i.e. the baseline accuracy for the entire dataset would be 0.53.

Partitioning

Since the dataset is large, sklearn.model_selection.train_test_split was used twice to split it into Training(64%:25372), Validation(16%:6343) and Test(20%:7929) datasets.

The baseline accuracy measures for the Training, Validation and Test datasets are 0.54, 0.53 and 0.52 respectively.

Linear Model — Logistic Regression(LogisticRegression with SelectKBest)

For the Linear Model, the data was transformed by using OneHotEncoder() and then scaled using StandardScaler().

The SelectKBest() function was used to compute the best k features which were then used to train a LogisticRegression() model. This process was repeated for k ranging from 1 to 62(the total number of features) and the models evaluated against the Validation dataset.

We get a best k value of 51 and the best model has an accuracy of 0.66 against the Validation dataset.

Evaluation Metrics

Using the best model gives us the following accuracy/auc scores for the Training, Validation and Test datasets:

Here’re the Confusion Matrix and ROC curves for the Test dataset

Understanding the Model

Here’s a visualization of the coefficients associated with the 51 features used for the model.

Tree Based Model — Decision Tree(DecisionTreeClassifier)

For the Decision Tree model, the categorical data was encoded using OrdinalEncoder() and the max_depth hyperparameter for the DecisionTreeClassifier() was tuned to 7using the Validation dataset.

Evaluation Metrics

The Decision Tree model gives us the following accuracy/auc scores for the Training, Validation and Test datasets:

Here’re the Confusion Matrix and ROC curves for the Test dataset

Tree Based Model — Random Forest(RandomForestClassifier)

For the Random Forest model, the OridinalEncoder() was used again for the categorical features and the model hyperparameters were tuned as shown below using the Validation dataset:

rf_model = RandomForestClassifier(n_estimators=103, random_state=42, n_jobs=-1, max_depth=25, min_samples_leaf=3, max_features=0.3)

Evaluation Metrics

The Random Forest model gives us the following accuracy/auc scores for the Training, Validation and Test datasets:

Here’re the Confusion Matrix and ROC curves for the Test dataset

Understanding the Model

Feature Importance

The following table details the weight/importance, computing using PermutationImportance(), of the various features used in the model:

Partial Dependence Plots(PDP)

Since the 2 most important features were kw_avg_avg and is_weekend, let’s examine the Partial Dependence Plots for them in isolation and interacting.

kw_avg_avg in isolation:

is_weekend in isolation:

kw_avg_avg, is_weekend interacting:

SHAP Plots

To get a better understanding of the impact of various features on predictions, here’re SHAP (SHapley Additive exPlanations) plots of

An entry accurately predicted as popular(1)

and

One accurately predicted as unpopular(0)

Tree Based Model — Gradient Boosting(XGBoost)

For the Random Forest model, the OridinalEncoder() was used again for the categorical features and the model hyperparameters were tuned as shown below using the Validation dataset:

xgb_model = XGBClassifier(n_estimators=1000, random_state=42, n_jobs=-1, max_depth=13, learning_rate=0.3)
eval_set = [(X_train_encoded, y_train), 
            (X_val_encoded, y_val)]
            
eval_metric = 'error'
xgb_model.fit(X_train_encoded, y_train, 
          eval_set=eval_set, 
          eval_metric=eval_metric,
          early_stopping_rounds=200)

Here’s the resulting Validation Curve:

Evaluation Metrics

The Gradient Boosting model gives us the following accuracy/auc scores for the Training, Validation and Test datasets:

Here’re the Confusion Matrix and ROC curves for the Test dataset

Conclusion

The Random Forest model exhibited the best behavior, closely followed by Gradient Boosting and Linear models with the Decision Tree model trailing behind.