deprecated
Hyperopt
Hyperopt is deprecated, replaced by Optuna (deprecated June 2025).
Python library for distributed hyperparameter tuning and model selection that Databricks bundled as its own fork in Databricks Runtime ML, with the SparkTrials class fanning trials out across a cluster and logging each one to MLflow.
- The distributed part was a Databricks invention: SparkTrials is an API Databricks developed so an existing Hyperopt run could spread across a cluster without changing the rest of the code.
- Databricks never shipped stock Hyperopt - Databricks Runtime ML carried the Databricks fork of hyperopt, which is why pip-installing the open-source libraries locally does not reproduce the same package.
- Its exit was partly upstream's doing: the first thing the docs now say about it is that the open-source version of Hyperopt is no longer being maintained.
Limitations: SparkTrials only parallelizes single-machine models such as scikit-learn - with distributed algorithms like MLlib you fall back to the default Trials class on the driver - and on GPU clusters maximum parallelism is reduced because each node runs only one executor thread.
Open in REbricked →- Category
- AI / ML
- Also known as
- SparkTrials, Distributed Hyperopt, hyperopt fmin
- Verified
- 2026-09-11
Sources
- Official Official Databricks / Microsoft docs
- Official Databricks docs: hyperparameter tuning with Optuna (the recommended replacement)
- Official Databricks docs: hyperparameter tuning overview (Optuna, Ray Tune, Hyperopt)
- Official Databricks Runtime 16.4 LTS ML release notes: the last ML runtime shipping hyperopt (0.2.8+db1)
- Official Microsoft Learn: Hyperopt concepts (same deprecation notice for Azure Databricks)