Skip to content

Algorithms

Fixed custom parameters

Use custom_params to keep selected parameters fixed during default training, random search and hill climbing:

automl = AutoML(
    mode="Compete",
    algorithms=["Random Forest", "RealMLP"],
    custom_params={
        "random_forest": {"max_depth": 3},
        "realmlp": {"device": "auto", "hidden_width": 64},
    },
)

Keys in custom_params are case-insensitive; spaces, underscores and hyphens are ignored. Values are fixed overrides, not lists of search candidates. Existing algorithms accept parameters from their registered search/default configuration; RealMLP, TabM mini, and Kumo/TabICLv2 also accept their documented training, inference, and memory controls. Unknown names and parameters are rejected. AutoML owns the task, metric, class count and CPU thread settings. Custom overrides are saved with the run, do not select algorithms automatically, and currently cannot be combined with Optuna mode. RealMLP requires the optional [realmlp] or [all] installation extra.

Baseline

Classification

The Baseline algorithm is using scikit-learn algorithm: DummyClassifier. It is using strategy prior which returns most frequent class as label and class prior for predict_proba().

Regression

The Baseline algorithm is using scikit-learn algorithm: DummyRegressor. It is using strategy mean which returns mean of the target from training data.

Baseline is not tuned

There will be only one model for algorithm Baseline. This algorithm has no hyperparameters.

Decision Tree

Classification

The Decision Tree is using scikit-learn DecisionTreeClassifier.

Decision Tree hyperparameters for classification

The allowed values of hyperparameters: 

dt_params = {"criterion": ["gini", "entropy"], 
             "max_depth": [2, 3, 4]}
The default set of hyperparameters: 
classification_default_params = {"criterion": "gini", "max_depth": 3}

Regression

The Decision Tree is using scikit-learn DecisionTreeRegressor.

Decision Tree hyperparameters for regression

The allowed values of hyperparameters: 

dt_params = {
    "criterion": ["squared_error"],
    "max_depth": [2, 3, 4]
}
The default set of hyperparameters: 
regression_default_params = {"criterion": "squared_error", "max_depth": 3}

For Decision Tree a visualization can be created with dtreeviz package (not to have explain_level > 0).

Linear

Classification

The Linear is using scikit-learn LogisticRegression.

Linear hyperparameters for classification

Thera are no hyperparameters for Linear model. The parameters used in LogisticRegression initialization: max_iter=500, tol=5e-4, n_jobs=-1.

Regression

The Linear is using scikit-learn LinearRegression.

Linear hyperparameters for regression

Thera are no hyperparameters for Linear model. The parameters used in LinearRegression initialization: n_jobs=-1.

The coefficients are saved in Markdown report if explain_level > 0.

Random Forest

Classification

The Random Forest is using scikit-learn RandomForestClassifier.

Random Forest hyperparameters for classification

The allowed hyperparameters values: 

rf_params = {
    "criterion": ["gini", "entropy"],
    "max_features": [0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    "min_samples_split": [10, 20, 30, 40, 50],
    "max_depth": [4, 6, 8, 10, 12],
}
The default hyperparameters: 
classification_default_params = {
    "criterion": "gini",
    "max_features": 0.6,
    "min_samples_split": 30,
    "max_depth": 6,
}

Regression

The Random Forest is using scikit-learn RandomForestRegressor.

Random Forest hyperparameters for regression

The allowed hyperparameters values: 

regression_rf_params = {
    "criterion": ["mse"],
    "max_features": [0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    "min_samples_split": [10, 20, 30, 40, 50],
    "max_depth": [4, 6, 8, 10, 12],
}
The default hyperparameters: 
regression_default_params = {
    "criterion": "mse",
    "max_features": 0.6,
    "min_samples_split": 30,
    "max_depth": 6,
}

Extra Trees

Classification

The Extra Trees is using scikit-learn ExtraTreesClassifier.

Extra Trees hyperparameters for classification

The allowed hyperparameters values: 

et_params = {
    "criterion": ["gini", "entropy"],
    "max_features": [0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    "min_samples_split": [10, 20, 30, 40, 50],
    "max_depth": [4, 6, 8, 10, 12],
}
The default hyperparameters: 
classification_default_params = {
    "criterion": "gini",
    "max_features": 0.6,
    "min_samples_split": 30,
    "max_depth": 6,
}

Regression

The Extra Trees is using scikit-learn ExtraTreesRegressor.

Extra Trees hyperparameters for regression

The allowed hyperparameters values: 

regression_et_params = {
    "criterion": ["mse"],
    "max_features": [0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    "min_samples_split": [10, 20, 30, 40, 50],
    "max_depth": [4, 6, 8, 10, 12],
}
The default hyperparameters: 
regression_default_params = {
    "criterion": "mse",
    "max_features": 0.6,
    "min_samples_split": 30,
    "max_depth": 6,
}

Xgboost

The AutoML is using Xgboost package.

Binary Classification

Xgboost hyperparameters for binary classification

The allowed hyperparameters values: 

xgb_bin_class_params = {
    "objective": ["binary:logistic"],
    "eval_metric": ["logloss"],
    "eta": [0.05, 0.075, 0.1, 0.15],
    "max_depth": [1, 2, 3, 4, 5, 6, 7, 8, 9],
    "min_child_weight": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    "subsample": [0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    "colsample_bytree": [0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
}
The default hyperparameters: 
classification_bin_default_params = {
    "objective": "binary:logistic",
    "eval_metric": "logloss",
    "eta": 0.1,
    "max_depth": 6,
    "min_child_weight": 1,
    "subsample": 1.0,
    "colsample_bytree": 1.0,
}

Multi-class Classification

Xgboost hyperparameters for multi-class classification

The allowed hyperparameters values: 

xgb_multi_class_params = {
    "objective": ["multi:softprob"],
    "eval_metric": ["mlogloss"],
    "eta": [0.05, 0.075, 0.1, 0.15],
    "max_depth": [1, 2, 3, 4, 5, 6, 7, 8, 9],
    "min_child_weight": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    "subsample": [0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    "colsample_bytree": [0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
}
The default hyperparameters: 
classification_multi_default_params = {
    "objective": "multi:softprob",
    "eval_metric": "mlogloss",
    "eta": 0.1,
    "max_depth": 6,
    "min_child_weight": 1,
    "subsample": 1.0,
    "colsample_bytree": 1.0,
}

Regression

Xgboost hyperparameters for regression

The allowed hyperparameters values: 

xgb_regression_params = {
    "objective": ["reg:squarederror"],
    "eval_metric": ["rmse"],
    "eta": [0.05, 0.075, 0.1, 0.15],
    "max_depth": [1, 2, 3, 4],
    "min_child_weight": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    "subsample": [0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
    "colsample_bytree": [0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
}
The default hyperparameters: 
regression_default_params = {
    "objective": "reg:squarederror",
    "eval_metric": "rmse",
    "eta": 0.1,
    "max_depth": 4,
    "min_child_weight": 1,
    "subsample": 1.0,
    "colsample_bytree": 1.0,
}

CatBoost

The AutoML is using CatBoost package.

Classification

CatBoost hyperparameters for classification

The allowed hyperparameters values: 

classification_params = {
    "learning_rate": [0.05, 0.1, 0.2],
    "depth": [2, 3, 4, 5, 6],
    "rsm": [0.7, 0.8, 0.9, 1],  # random subspace method
    "subsample": [0.7, 0.8, 0.9, 1],  # random subspace method
    "min_data_in_leaf": [1, 5, 10, 15, 20, 30, 50],
}
The default hyperparameters: 
classification_default_params = {
    "learning_rate": 0.1,
    "depth": 6,
    "rsm": 0.9,
    "subsample": 1.0,
    "min_data_in_leaf": 15,
}

  • for binary classification loss_function=Logloss,
  • for mutliclass classification loss_function=MultiClass.

Regression

CatBoost hyperparameters for regression

The allowed hyperparameters values: 

regression_params = {
    "learning_rate": [0.05, 0.1, 0.2],
    "depth": [2, 3, 4, 5, 6],
    "rsm": [0.7, 0.8, 0.9, 1],  # random subspace method
    "subsample": [0.7, 0.8, 0.9, 1],  # random subspace method
    "min_data_in_leaf": [1, 5, 10, 15, 20, 30, 50],
}
The default hyperparameters: 
regression_default_params = {
    "learning_rate": 0.1,
    "depth": 6,
    "rsm": 0.9,
    "subsample": 1.0,
    "min_data_in_leaf": 15,
}

For regression loss_function=RMSE.

LightGBM

The AutoML is using LightGBM package.

Binary Classification

LightGBM hyperparameters for binary classification

The allowed hyperparameters values: 

lgbm_bin_params = {
    "objective": ["binary"],
    "metric": ["binary_logloss"],
    "num_leaves": [3, 7, 15, 31],  
    "learning_rate": [0.05, 0.075, 0.1, 0.15],
    "feature_fraction": [0.8, 0.9, 1.0],
    "bagging_fraction": [0.8, 0.9, 1.0],
    "min_data_in_leaf": [5, 10, 15, 20, 30, 50],
}
The default hyperparameters: 
classification_bin_default_params = {
    "objective": "binary",
    "metric": "binary_logloss",
    "num_leaves": 31,
    "learning_rate": 0.1,
    "feature_fraction": 0.9,
    "bagging_fraction": 0.9,
    "min_data_in_leaf": 10,
}

Multi-class Classification

LightGBM hyperparameters for multi-class classification

The allowed hyperparameters values: 

lgbm_bin_params = {
    "objective": ["multiclass"],
    "metric": ["multi_logloss"],
    "num_leaves": [3, 7, 15, 31], 
    "learning_rate": [0.05, 0.075, 0.1, 0.15],
    "feature_fraction": [0.8, 0.9, 1.0],
    "bagging_fraction": [0.8, 0.9, 1.0],
    "min_data_in_leaf": [5, 10, 15, 20, 30, 50],
}
The default hyperparameters: 
classification_multi_default_params = {
    "objective": "multiclass",
    "metric": "multi_logloss",
    "num_leaves": 31,
    "learning_rate": 0.1,
    "feature_fraction": 0.9,
    "bagging_fraction": 0.9,
    "min_data_in_leaf": 10,
}

Regression

LightGBM hyperparameters for regression

The allowed hyperparameters values: 

lgbm_bin_params = {
    "objective": ["regression"],
    "metric": ["l2"],
    "num_leaves": [3, 7, 15, 31], 
    "learning_rate": [0.05, 0.075, 0.1, 0.15],
    "feature_fraction": [0.8, 0.9, 1.0],
    "bagging_fraction": [0.8, 0.9, 1.0],
    "min_data_in_leaf": [5, 10, 15, 20, 30, 50],
}
The default hyperparameters: 
regression_default_params = {
    "objective": "regression",
    "metric": "l2",
    "num_leaves": 15,
    "learning_rate": 0.1,
    "feature_fraction": 0.9,
    "bagging_fraction": 0.9,
    "min_data_in_leaf": 10,
}

Neural Network

For Neural Network algorithm the Keras and Tensorflow are used. The same set of hyperparameters are used for all Machine Learning tasks (classification and regression). There is difference in output neurons type and loss function depending on ML task.

Neural Network hyperparameters

The allowed hyperparameters values: 

nn_params = {
    "dense_layers": [2],
    "dense_1_size": [16, 32, 64],
    "dense_2_size": [4, 8, 16, 32],
    "dropout": [0, 0.1, 0.25],
    "learning_rate": [0.01, 0.05, 0.08, 0.1],
    "momentum": [0.85, 0.9, 0.95],
    "decay": [0.0001, 0.001, 0.01],
}
The default hyperparameters: 
default_nn_params = {
    "dense_layers": 2,
    "dense_1_size": 32,
    "dense_2_size": 16,
    "dropout": 0,
    "learning_rate": 0.05,
    "momentum": 0.9,
    "decay": 0.001,
}

Binary Classification

  • There is single output neuron with sigmoid activation.
  • The loss function: binary_crossentropy.

Multi-class Classification

  • The number of output neurons is equal to the number of unique classes in the target. The activation in the output layer is softmax.
  • The loss function: categorical_crossentropy.

Regression

  • There is single output neuron with linear activation.
  • The loss function: mean_squared_error.

Nearest Neighbor

The Nearest Neighbor algorithm is using scikit-learn:

Training uses at most 1,000 rows by default. Regression selects a reproducible random subset. Classification uses stratified sampling when it retains every training class. For singleton classes, a small discarded partition, or an omitted class, a reproducible fallback keeps at least one row per class and allocates the remaining places approximately proportionally. Sampling is without replacement and uses only the training fold. AutoML reports a warning when this fallback is used.

If the row cap cannot retain every training class, Nearest Neighbors raises an error before fitting. The selected sample must also contain at least n_neighbors rows. These errors explain which limit needs changing. This handles the internal subsampling failure reported in issue #794.

Nearest Neighbor hyperparameters

The allowed hyperparameters values: 

knn_params = {
    "n_neighbors": [3, 5, 7], 
    "weights": ["uniform", "distance"]
}
The default hyperparameters: 
default_params = {
    "n_neighbors": 5, 
    "weights": "uniform"
}

Kumo

Kumo uses NVIDIA SDM's pretrained tabular model for binary classification, multiclass classification, and regression. It is available only in Compete mode and supports small, medium, and large checkpoints.

Allowed tuning values:

kumo_params = {
    "size": ["small", "medium", "large"],
    "num_estimators": [4, 8, 16],
    "max_context_size": [1024, 4096, 8192],
}
kumo_default_params = {
    "size": "small",
    "num_estimators": 8,
    "max_context_size": 4096,
}

An estimator is a transformed view of the context using shared pretrained weights. The context is a bounded sample of training-fold rows. We do not train new weights or tune network depth/learning rate. Fix any tuning parameter through custom_params={"kumo": {"size": "large", "num_estimators": 8}}.

TabICLv2

TabICLv2 is a separate Compete algorithm using SDM's fixed v2 checkpoint. It supports the same three tasks and has no small/medium/large setting.

tabiclv2_params = {
    "num_estimators": [4, 8, 16],
    "max_context_size": [1024, 4096, 8192],
}
tabiclv2_default_params = {
    "num_estimators": 8,
    "max_context_size": 4096,
}

Both algorithms require Python 3.11+ and an optional extra ([tfm], [kumo], [tabiclv2], or [all]). These extras include the SDM 0.2.1 PyPI backend in the current source checkout. The published alpha requires its older Git backend; see the installation guide. Both algorithms reject sample weights and sparse input. Permutation importance is supported; SHAP is not enabled. See installation and the full model guide for CPU/GPU behavior, input limits, downloads, and portable saves.

Stacked Algorithm

The stacked algorithms are built with predictions from previous (unstacked) models. The stacked algorithms are reusing hyperparameters of already found good models.

  • During the stacking up to 10 best models from each algorithm are used, except Baseline.
  • The out-of-folds predictions are used to construct extended training data. The stacking only works for validation_strategy="kfold" (k-fold cross-validation).
  • The stacked model can be only: Xgboost, LightGBM, CatBoost. The AutoML algorithm selects the best models from unstacked Xgboost, LightGBM, CatBoost and reuses its hyperparameters to train stacked models.

Ensemble

The Ensemble algorithm is implemented based on Caruana article. The Ensemble is using average method, which does a greedy search over all models and try to add (with repetition) a model to the ensemble to improve ensemble's performance. The ensemble performance is computed based on out-of-folds predictions of used models