AutoML Leaderboard
Models in this report were generated and selected automatically by MLJAR AutoML. Review model behavior, data suitability, and decision impact before important use.
| Best model | name | model_type | is_stacked | metric_type | metric_value | train_time |
|---|---|---|---|---|---|---|
| 1_Baseline | Baseline | False | logloss | 0.433171 | 1.9 | |
| 2_DecisionTree | Decision Tree | False | logloss | 0.312209 | 9.52 | |
| 3_Linear | Linear | False | logloss | 0.232629 | 3.56 | |
| 4_Default_LightGBM | LightGBM | False | logloss | 0.134072 | 4.59 | |
| the best | Ensemble | Ensemble | False | logloss | 0.132638 | 2.45 |
AutoML Performance
AutoML Performance Boxplot
Features Importance (Original Scale)
Scaled Features Importance (MinMax per Model)
Spearman Correlation of Models
Summary of 2_DecisionTree
Decision Tree
- n_jobs: 2
- criterion: entropy
- max_depth: 4
- explain_level: 1
Validation
- validation_type: custom
Optimized metric
logloss
Training time
8.6 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.312209 | nan |
| auc | 0.88844 | nan |
| f1 | 0.640605 | 0.209302 |
| accuracy | 0.898447 | 0.52381 |
| precision | 0.947368 | 0.583333 |
| recall | 0.979592 | 0 |
| mcc | 0.573276 | 0.209302 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.312209 | nan |
| auc | 0.88844 | nan |
| f1 | 0.538879 | 0.52381 |
| accuracy | 0.898447 | 0.52381 |
| precision | 0.925466 | 0.52381 |
| recall | 0.380102 | 0.52381 |
| mcc | 0.554808 | 0.52381 |
Confusion matrix (at threshold=0.52381)
| Predicted as 0 | Predicted as 1 | |
|---|---|---|
| Labeled as 0 | 2107 | 12 |
| Labeled as 1 | 243 | 149 |
Learning curves
Decision Tree
Tree #1
Rules
if (Complains <= 0.5) and (Frequency of use > 19.5) and (Seconds of Use <= 6020.5) and (Frequency of SMS <= 37.5) then class: 0 (proba: 88.55%) | based on 524 samples
if (Complains <= 0.5) and (Frequency of use > 19.5) and (Seconds of Use > 6020.5) then class: 0 (proba: 100.0%) | based on 485 samples
if (Complains <= 0.5) and (Frequency of use > 19.5) and (Seconds of Use <= 6020.5) and (Frequency of SMS > 37.5) then class: 0 (proba: 99.22%) | based on 256 samples
if (Complains <= 0.5) and (Frequency of use <= 19.5) and (Subscription Length > 10.5) and (Seconds of Use > 329.0) then class: 0 (proba: 82.24%) | based on 152 samples
if (Complains <= 0.5) and (Frequency of use <= 19.5) and (Subscription Length > 10.5) and (Seconds of Use <= 329.0) then class: 0 (proba: 51.18%) | based on 127 samples
if (Complains > 0.5) and (Seconds of Use <= 6421.5) and (Subscription Length <= 38.5) and (Tariff Plan > 0.5) then class: 1 (proba: 98.82%) | based on 85 samples
if (Complains > 0.5) and (Seconds of Use <= 6421.5) and (Subscription Length > 38.5) and (Seconds of Use > 1785.0) then class: 0 (proba: 53.85%) | based on 13 samples
if (Complains <= 0.5) and (Frequency of use <= 19.5) and (Subscription Length <= 10.5) then class: 1 (proba: 100.0%) | based on 11 samples
if (Complains > 0.5) and (Seconds of Use > 6421.5) then class: 0 (proba: 100.0%) | based on 11 samples
if (Complains > 0.5) and (Seconds of Use <= 6421.5) and (Subscription Length > 38.5) and (Seconds of Use <= 1785.0) then class: 1 (proba: 100.0%) | based on 7 samples
if (Complains > 0.5) and (Seconds of Use <= 6421.5) and (Subscription Length <= 38.5) and (Tariff Plan <= 0.5) then class: 1 (proba: 75.0%) | based on 4 samples
Tree #2
Rules
if (Complains <= 0.5) and (Frequency of use > 19.5) and (Seconds of Use <= 6011.5) and (Frequency of SMS <= 66.5) then class: 0 (proba: 90.03%) | based on 592 samples
if (Complains <= 0.5) and (Frequency of use > 19.5) and (Seconds of Use > 6011.5) then class: 0 (proba: 100.0%) | based on 469 samples
if (Complains <= 0.5) and (Frequency of use > 19.5) and (Seconds of Use <= 6011.5) and (Frequency of SMS > 66.5) then class: 0 (proba: 100.0%) | based on 212 samples
if (Complains <= 0.5) and (Frequency of use <= 19.5) and (Age <= 50.0) and (Subscription Length > 32.5) then class: 1 (proba: 52.38%) | based on 147 samples
if (Complains <= 0.5) and (Frequency of use <= 19.5) and (Age <= 50.0) and (Subscription Length <= 32.5) then class: 0 (proba: 79.07%) | based on 86 samples
if (Complains > 0.5) and (Seconds of Use <= 5983.0) and (Subscription Length <= 37.5) and (Frequency of SMS <= 123.0) then class: 1 (proba: 100.0%) | based on 80 samples
if (Complains <= 0.5) and (Frequency of use <= 19.5) and (Age > 50.0) then class: 0 (proba: 100.0%) | based on 30 samples
if (Complains > 0.5) and (Seconds of Use <= 5983.0) and (Subscription Length > 37.5) and (Frequency of use <= 57.5) then class: 1 (proba: 85.71%) | based on 28 samples
if (Complains > 0.5) and (Seconds of Use > 5983.0) then class: 0 (proba: 100.0%) | based on 19 samples
if (Complains > 0.5) and (Seconds of Use <= 5983.0) and (Subscription Length > 37.5) and (Frequency of use > 57.5) then class: 0 (proba: 100.0%) | based on 6 samples
if (Complains > 0.5) and (Seconds of Use <= 5983.0) and (Subscription Length <= 37.5) and (Frequency of SMS > 123.0) then class: 1 (proba: 66.67%) | based on 3 samples
Tree #3
Rules
if (Complains <= 0.5) and (Seconds of Use > 1004.0) and (Seconds of Use <= 6011.5) and (Frequency of SMS <= 62.5) then class: 0 (proba: 89.92%) | based on 605 samples
if (Complains <= 0.5) and (Seconds of Use > 1004.0) and (Seconds of Use > 6011.5) then class: 0 (proba: 100.0%) | based on 462 samples
if (Complains <= 0.5) and (Seconds of Use > 1004.0) and (Seconds of Use <= 6011.5) and (Frequency of SMS > 62.5) then class: 0 (proba: 100.0%) | based on 211 samples
if (Complains <= 0.5) and (Seconds of Use <= 1004.0) and (Subscription Length > 28.5) and (Age <= 50.0) then class: 0 (proba: 52.17%) | based on 184 samples
if (Complains <= 0.5) and (Seconds of Use <= 1004.0) and (Subscription Length <= 28.5) and (Subscription Length > 12.0) then class: 0 (proba: 95.77%) | based on 71 samples
if (Complains > 0.5) and (Seconds of Use <= 5800.5) and (Frequency of use <= 53.5) and (Frequency of SMS > 0.5) then class: 1 (proba: 100.0%) | based on 64 samples
if (Complains > 0.5) and (Seconds of Use <= 5800.5) and (Frequency of use <= 53.5) and (Frequency of SMS <= 0.5) then class: 1 (proba: 95.24%) | based on 21 samples
if (Complains <= 0.5) and (Seconds of Use <= 1004.0) and (Subscription Length > 28.5) and (Age > 50.0) then class: 0 (proba: 100.0%) | based on 18 samples
if (Complains > 0.5) and (Seconds of Use <= 5800.5) and (Frequency of use > 53.5) and (Call Failure > 14.5) then class: 1 (proba: 100.0%) | based on 15 samples
if (Complains > 0.5) and (Seconds of Use <= 5800.5) and (Frequency of use > 53.5) and (Call Failure <= 14.5) then class: 1 (proba: 58.33%) | based on 12 samples
if (Complains > 0.5) and (Seconds of Use > 5800.5) then class: 0 (proba: 100.0%) | based on 8 samples
if (Complains <= 0.5) and (Seconds of Use <= 1004.0) and (Subscription Length <= 28.5) and (Subscription Length <= 12.0) then class: 1 (proba: 100.0%) | based on 4 samples
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 4_Default_LightGBM
LightGBM
- n_jobs: 2
- objective: binary
- num_leaves: 63
- learning_rate: 0.05
- feature_fraction: 0.9
- bagging_fraction: 0.9
- min_data_in_leaf: 10
- metric: binary_logloss
- custom_eval_metric_name: None
- explain_level: 1
Validation
- validation_type: custom
Optimized metric
logloss
Training time
3.6 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.134072 | nan |
| auc | 0.978581 | nan |
| f1 | 0.837905 | 0.296185 |
| accuracy | 0.948626 | 0.372656 |
| precision | 1 | 0.980465 |
| recall | 1 | 0.00150689 |
| mcc | 0.807431 | 0.222174 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.134072 | nan |
| auc | 0.978581 | nan |
| f1 | 0.833977 | 0.372656 |
| accuracy | 0.948626 | 0.372656 |
| precision | 0.841558 | 0.372656 |
| recall | 0.826531 | 0.372656 |
| mcc | 0.803637 | 0.372656 |
Confusion matrix (at threshold=0.372656)
| Predicted as 0 | Predicted as 1 | |
|---|---|---|
| Labeled as 0 | 2058 | 61 |
| Labeled as 1 | 68 | 324 |
Learning curves
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 1_Baseline
Baseline Classifier (Baseline)
- n_jobs: 2
- explain_level: 1
Validation
- validation_type: custom
Optimized metric
logloss
Training time
1.1 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.433171 | nan |
| auc | 0.498457 | nan |
| f1 | 0.270065 | 0.139952 |
| accuracy | 0.385106 | 0.155502 |
| precision | 0.156113 | 0.139952 |
| recall | 1 | 0.139952 |
| mcc | 0 | 0.139952 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.433171 | nan |
| auc | 0.498457 | nan |
| f1 | 0.251938 | 0.155502 |
| accuracy | 0.385106 | 0.155502 |
| precision | 0.155502 | 0.155502 |
| recall | 0.663265 | 0.155502 |
| mcc | -0.00237526 | 0.155502 |
Confusion matrix (at threshold=0.155502)
| Predicted as 0 | Predicted as 1 | |
|---|---|---|
| Labeled as 0 | 707 | 1412 |
| Labeled as 1 | 132 | 260 |
Learning curves
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of Ensemble
Ensemble structure
| Model | Weight |
|---|---|
| 3_Linear | 1 |
| 4_Default_LightGBM | 18 |
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.132638 | nan |
| auc | 0.979533 | nan |
| f1 | 0.840399 | 0.303426 |
| accuracy | 0.949423 | 0.366877 |
| precision | 1 | 0.975267 |
| recall | 1 | 0.0020547 |
| mcc | 0.810374 | 0.303426 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.132638 | nan |
| auc | 0.979533 | nan |
| f1 | 0.836551 | 0.366877 |
| accuracy | 0.949423 | 0.366877 |
| precision | 0.844156 | 0.366877 |
| recall | 0.829082 | 0.366877 |
| mcc | 0.806682 | 0.366877 |
Confusion matrix (at threshold=0.366877)
| Predicted as 0 | Predicted as 1 | |
|---|---|---|
| Labeled as 0 | 2059 | 60 |
| Labeled as 1 | 67 | 325 |
Learning curves
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
Summary of 3_Linear
Logistic Regression (Linear)
- n_jobs: 2
- explain_level: 1
Validation
- validation_type: custom
Optimized metric
logloss
Training time
2.7 seconds
Metric details
| score | threshold | |
|---|---|---|
| logloss | 0.232629 | nan |
| auc | 0.922727 | nan |
| f1 | 0.650544 | 0.316792 |
| accuracy | 0.900836 | 0.535027 |
| precision | 1 | 0.945145 |
| recall | 1 | 1.09297e-05 |
| mcc | 0.583005 | 0.316792 |
Metric details with threshold from accuracy metric
| score | threshold | |
|---|---|---|
| logloss | 0.232629 | nan |
| auc | 0.922727 | nan |
| f1 | 0.568458 | 0.535027 |
| accuracy | 0.900836 | 0.535027 |
| precision | 0.886486 | 0.535027 |
| recall | 0.418367 | 0.535027 |
| mcc | 0.567498 | 0.535027 |
Confusion matrix (at threshold=0.535027)
| Predicted as 0 | Predicted as 1 | |
|---|---|---|
| Labeled as 0 | 2098 | 21 |
| Labeled as 1 | 228 | 164 |
Learning curves
Coefficients
| feature | Learner_1 | Learner_2 | Learner_3 |
|---|---|---|---|
| Complains | 3.74956 | 3.62032 | 3.76029 |
| Call Failure | 0.816066 | 0.979675 | 1.10142 |
| Seconds of Use | 0.237677 | 0.630099 | 0.344317 |
| Tariff Plan | -0.0924329 | 0.026681 | 0.0466637 |
| Subscription Length | -0.014532 | 0.03184 | -0.045425 |
| Age | -0.0471994 | -0.154128 | -0.0301235 |
| Distinct Called Numbers | -0.308848 | -0.391333 | -0.332332 |
| Charge Amount | -0.983847 | -0.938434 | -0.947583 |
| Frequency of SMS | -1.16466 | -1.2956 | -1.3135 |
| Frequency of use | -1.8622 | -2.44999 | -2.11545 |
| intercept | -3.47494 | -3.74221 | -3.58777 |
Permutation-based Importance
Confusion Matrix
Normalized Confusion Matrix
ROC Curve
Kolmogorov-Smirnov Statistic
Precision-Recall Curve
Calibration Curve
Cumulative Gains Curve
Lift Curve
