Software

HyperSHAP - Shapley Values and Interactions for Explaining HPO

HyperSHAP is a framework to explain the outcomes of hyperparameter optimization (HPO) by leveraging cooperative game theory concepts such as Shapley values and interaction indices. It is designed to provide actionable insights into the role and interplay of hyperparameters, thereby reducing the manual effort typically required to interpret HPO results. While its primary audience is researchers and practitioners in machine learning and artificial intelligence, its use is not restricted to these target groups.

The analysis of HPO results often involves comparing tuned configurations, assessing hyperparameter importance, and identifying optimizer biases. These tasks are typically performed in an ad-hoc and dataset-specific manner, requiring extensive manual inspection of results. Existing approaches often lack the ability to systematically capture interactions among hyperparameters or to generalize explanations across datasets and optimizers.

HyperSHAP addresses these challenges by formulating HPO explanations as cooperative games, where hyperparameters form coalitions whose contributions to performance are quantified. This unified framework enables fine-grained analyses, such as ablation studies, sensitivity attribution, tunability assessment, and optimizer behavior characterization. All computations are naturally parallelizable, making HyperSHAP scalable to modern HPO scenarios. By automating and standardizing the generation of interpretable explanations, HyperSHAP alleviates much of the overhead in analyzing HPO results and provides practitioners with clear guidance on which hyperparameters to focus on and how optimizers behave across tasks.

DeepCAVE 2.0

DeepCAVE is a visualization and analysis tool for AutoML, with a particular focus on hyperparameter optimization (HPO). Built on the Dash framework, it offers a fully interactive experience. The tool features a variety of plugins that enable efficient insight generation, aiding in understanding and debugging the application of HPO. Additionally, the powerful run interface and the modularized plugin structure allow extending the tool at any time effortlessly.

SMAC3

SMAC is a tool for algorithm configuration to optimize the parameters of arbitrary algorithms, including hyperparameter optimization of Machine Learning algorithms. The main core consists of Bayesian Optimization in combination with an aggressive racing mechanism to efficiently decide which of two configurations performs better.

SMAC3 is written in Python3 and continuously tested with Python 3.8, 3.9, and 3.10. Its Random Forest is written in C++. In the following, SMAC is representatively mentioned for SMAC3.

AILibs

AILibs is a modular collection of Java libraries related to automated decision making. It's highlight functionalities are:

• Graph Search (jaicore-search): (AStar, BestFirst, Branch & Bound, DFS, MCTS, and more)

• Logic (jaicore-logic): Represent and reason about propositional and simple first order logic formulas

• Planning (jaicore-planning): State-space planning (STRIPS, PDDL), and hierarchical planning (HTN, ITN, PTN)

• Reproducible Experiments (jaicore-experiments): Design and efficiently conduct experiments in a highly parallelized manner.

• Automated Software Configuration (HASCO): Hierarchical configuration of software systems.

• Automated Machine Learning (ML-Plan): Automatically find optimal machine learning pipelines in WEKA or sklearn

All algorithms in AILibs are steppable, and their behavior can be analyzed via the algorithm inspector: jaicore-algorithminspector. For example, graph search algorithms send events that allow a graph visualization in the algorithm inspector.

DataDancer

DataDancer is a highly modular and flexible workbench and benchmark platform for developing and evaluating AutoML systems. It is based on the remote procedure calls extension of Google ProtoBuf (gRPC). Thereby, the individual modules can be implemented in different programming languages and still function as 

OpenML

OpenML is a platform fostering open research on machine learning, facilitating work on meta-learning, and busting myths in machine learning. Currently, I am one of the core developers of OpenML and responsible for the Java client of OpenML. If you are interested in OpenML and working on or with OpenML do not hesitate to get in touch with me.

Find out more about OpenML

PyExperimenter

PyExperimenter is a tool to facilitate the setup, documentation, execution, and subsequent evaluation of results from an empirical study of algorithms and in particular is designed to reduce the involved manual effort significantly. It is intended to be used by researchers in the field of artificial intelligence, but is not limited to those.

The empirical analysis of algorithms is often accompanied by the execution of algorithms for different inputs and variants of the algorithms (specified via parameters) and the measurement of non-functional properties. Since the individual evaluations are usually independent, the evaluation can be performed in a distributed manner on an HPC system. However, setting up, documenting, and evaluating the results of such a study is often file-based. Usually, this requires extensive manual work to create configuration files for the inputs or to read and aggregate measured results from a report file. In addition, monitoring and restarting individual executions is tedious and time-consuming.

These challenges are addressed by PyExperimenter by means of a single well defined configuration file and a central database for managing massively parallel evaluations, as well as collecting and aggregating their results. Thereby, PyExperimenter alleviates the aforementioned overhead and allows experiment executions to be defined and monitored with ease.

Academic ALPaCA

Acadamic ALPaCA stands for Academic Administration, knowLedge base, Paper organization and Collaboration Assistant and is an administrative tool for bringing better organization and collaboration to academia and other research labs.