API: Multi-Config Specification

The core value of RapidFire AI is in the ability to launch, compare, and dynamically control multiple configurations (configs) in one go. It is already common practice in AI to do hyperparameter sweeps via grid search, random search, or AutoML heuristics to generate knob values.

RapidFire AI generalizes that notion to any type of config knobs, not just regular hyperparameters, but also base model architectures, prompt schemes, LoRA adapters, and optimizers (for training), as well as data chunking, embedding, retrieval, reranking, generation, and prompt schemes (for RAG) and any other user-defined knobs.

Knob Set Generators

To create a multi-config specification, you need two things: knob set generators for knob values and config group generators that take a config with set-valued knobs to generate groups of full configs.

We currently support two common knob set generators: List() for a discrete set of values and Range() for sampling from a continuous value interval.

List(values: List[Any])

Parameters:

values (List[Any]) – List of discrete values for a knob. We recommend using values of the same python data type within a given List(), although this is not enforced.

Range(start: int | float, end: int | float, dtype: str = None, log: bool = False, step: int | float = None)

Parameters:

• start (int | float) – Lower bound of range interval, inclusive.

• end (int | float) – Upper bound of range interval, inclusive.

• dtype (str, optional) – Data type of value to be sampled, either "int" or "float". Optional; if omitted, it is inferred from start and end, viz., "int" only if both of them are python ints and "float" otherwise.

• log (bool, optional) – Whether to sample in log space instead of uniformly; both start and end must then be greater than 0. Default is False. Mutually exclusive with step.

• step (int | float, optional) – Discretization step; sampled values are then multiples of step counting up from start. Default is None, i.e., the interval is treated as continuous. Mutually exclusive with log.

Notes:

By default, Range() performs uniform sampling within the given interval. Set log=True for log-uniform sampling, which is usually a better fit for scale-free knobs such as learning rate, or set step to restrict sampling to a discrete grid of values within the interval. Giving both log and step raises an error.

These variants line up one-to-one with Optuna’s suggest_int() / suggest_float() distributions; so, the same Range() specification carries over to RFOptuna() below.

Note that the return types of the knob set generators are internal to RapidFire AI and they are usable only within the context of the config group generators below.

Config Group Generators

We currently support three config group generators: RFGridSearch() for grid search, RFRandomSearch() for random search, and RFOptuna() for search driven by the popular AutoML library Optuna, including adaptive pruning of unpromising runs.

More support for AutoML heuristics such as SHA and HyperOpt is coming soon. Likewise for RAG/context engineering, we also plan to support the AutoML heuristic syftr.

All three take the config(s) to expand as their first argument. That argument accepts a single config, a plain python list of configs, or a List() of configs. For run_fit(), the configs must be RFModelConfig instances; for run_evals(), they must be config dictionaries. Passing a trainer_type is what puts a generator in fit mode; leaving it unset puts it in evals mode.

Warning

For RFGridSearch() and RFRandomSearch(), the second positional parameter is a legacy, unused create_model_fn, not trainer_type. So, always pass trainer_type and num_runs as keyword arguments. For example, RFGridSearch(config_set, "SFT") assigns "SFT" to create_model_fn, leaves trainer_type unset, which silently puts the generator in evals mode, and then fails validation with “Evals mode requires dict instances”. Write RFGridSearch(config_set, trainer_type="SFT") instead. RFOptuna() does not have this quirk; its second parameter is trainer_type.

RFGridSearch(configs: RFModelConfig | Dict[str, Any] | list, create_model_fn: Callable = None, trainer_type: str = None, num_runs: int = 1)

Parameters:

• configs (RFModelConfig | Dict[str, Any] | list) – The config(s) to expand, with List() for at least one knob. An RFModelConfig (or a list/List() of them) for run_fit(); a config dictionary (or a list/List() of them) for run_evals().

• create_model_fn (Callable, optional) – Legacy parameter that is unused. Never pass anything here; see the warning above.

• trainer_type (str, optional) – The fine-tuning/post-training control flow to use: "SFT", "DPO", or "GRPO". Skip this argument for run_evals().

• num_runs (int, optional) – Ignored by grid search, which always yields the full cross product of knob values. Default is 1.

RFRandomSearch(configs: RFModelConfig | Dict[str, Any] | list, create_model_fn: Callable = None, trainer_type: str = None, num_runs: int = 1)

Parameters:

• configs (RFModelConfig | Dict[str, Any] | list) – The config(s) to sample from, with List() or Range() for at least one knob. An RFModelConfig (or a list/List() of them) for run_fit(); a config dictionary (or a list/List() of them) for run_evals().

• create_model_fn (Callable, optional) – Legacy parameter that is unused. Never pass anything here; see the warning above.

• trainer_type (str, optional) – The fine-tuning/post-training control flow to use: "SFT", "DPO", or "GRPO". Skip this argument for run_evals().

• num_runs (int, optional) – Number of runs (full combinations of knob values) to sample in total. Default is 1; so, set this explicitly.

Notes:

For RFGridSearch(), each knob can have either a single value or a List() of values. Grid search expands only List() valued knobs. A Range() is not supported here: it is passed through to the run as an unexpanded RapidFire AI object instead of a knob value, which will almost certainly fail downstream. So, use RFRandomSearch() or RFOptuna() if you want to explore a continuous interval.

For RFRandomSearch(), each knob can have either a single value, or a List() of values, or a Range() of values. The semantics of sampling are independently-identically-distributed (IID), i.e., we uniformly randomly pick a value from each discrete set and from each continuous set to construct the knob combination for one run. Then we repeat that sampling process in an IID way to accumulate num_runs distinct combinations.

RFRandomSearch() has no constructor seed; its sampling is seeded by the seed argument of run_fit() / run_evals() (default 42).

In evals mode, the generator recognizes exactly three keys for the inference pipeline in your config dictionary: "api_config", "vllm_config", and "pipeline". Everything else in the dictionary is treated as a regular knob. Also read the Generator Configs page.

Note that the return types of the config group generators are internal to RapidFire AI and they are usable only within the context of run_fit() or run_evals() in the Experiment class.

Examples:

# Example 1: Based on SFT tutorial notebook
from rapidfireai.automl import List, RFGridSearch, RFLoraConfig, RFModelConfig, RFSFTConfig

# 2 LoRA adapter configs with different capacities
peft_configs = List([
        RFLoraConfig(
                r=16, lora_alpha=32, lora_dropout=0.05,
                target_modules=["q_proj", "v_proj"], bias="none"
        ),
        RFLoraConfig(
                r=128, lora_alpha=256, lora_dropout=0.05,
                target_modules=["q_proj","k_proj", "v_proj","o_proj"], bias="none"
        )
])

# 2 base models x 2 LoRA configs = 4 combinations in total
config_set = List([
        RFModelConfig(
                model_name="meta-llama/Llama-3.1-8B-Instruct",
                peft_config=peft_configs,
                training_args=RFSFTConfig(learning_rate=2e-4, num_train_epochs=2, ...),
                model_type="causal_lm",
                ...
        ),
        RFModelConfig(
                model_name="mistralai/Mistral-7B-Instruct-v0.3",
                peft_config=peft_configs,
                training_args=RFSFTConfig(learning_rate=2e-4, num_train_epochs=2, ...),
                model_type="causal_lm",
                ...
        )
])

config_group = RFGridSearch(
        configs=config_set,
        trainer_type="SFT"
)


# Example 2: Based on GSM8K tutorial notebook
from rapidfireai.automl import List, RFAPIModelConfig, RFGridSearch, RFPromptManager

gemini_config1 = RFAPIModelConfig(
        client_config={"max_retries": 2},
        endpoint_config={
                "provider": "gemini",
                "api_key_name": "my_gemini_key",
                "api_key": GOOGLE_API_KEY,
                "endpoint": {
                        "model": "gemini-2.5-flash-lite",
                        "name": "gemini-2.5-flash-lite",
                },
        },
        model_config={"max_completion_tokens": 2048},
        rpm_limit=30_000,
        tpm_limit=30_000_000,
        prompt_manager=fewshot_prompt_manager,
)

gemini_config2 = RFAPIModelConfig(
        client_config={"max_retries": 2},
        endpoint_config={
                "provider": "gemini",
                "api_key_name": "my_gemini_key",  # reusing the same stored secret
                "endpoint": {
                        "model": "gemini-3-flash-preview",
                        "name": "gemini-3-flash-preview",
                },
        },
        model_config={
                "max_completion_tokens": 4096,
                "reasoning_effort": List(["medium", "low"]),  # 2 different reasoning levels
        },
        rpm_limit=20_000,
        tpm_limit=20_000_000,
        prompt_manager=fewshot_prompt_manager,
)

config_set = {
        "api_config": List(
                [gemini_config1, gemini_config2]
        ),
        "batch_size": batch_size,
        "preprocess_fn": sample_preprocess_fn,
        "compute_metrics_fn": sample_compute_metrics_fn,
        ...
}

config_group = RFGridSearch(config_set)

RFOptuna

RFOptuna() is a drop-in replacement for the above two generators that uses Optuna’s ask-and-tell API to pick knob combinations adaptively. It supports single- and multi-objective optimization, and it can prune an underperforming run at a chunk (or shard) boundary and spend the freed GPU slot on a fresh, better-informed config, up to a total budget of runs.

Note

from rapidfireai.automl import RFOptuna always succeeds, but instantiating it raises an ImportError with install instructions if Optuna is not importable. Install it with pip install "rapidfireai[optuna]" (which pins optuna>=4.0.0) or with plain python -m pip install optuna. Optuna must go into the same interpreter as your Jupyter kernel; check with import sys; print(sys.executable) in a notebook cell and restart the kernel afterward.

RFOptuna(configs: RFModelConfig | Dict[str, Any] | list, trainer_type: str = None, n_initial: int = 16, budget: int = 40, objective: str = 'minimize:eval_loss', sampler: str = 'tpe', pruner: str = 'median', seed: int = 42, granularity: str = 'chunk')

Parameters:

• configs (RFModelConfig | Dict[str, Any] | list) – The config template(s) that define the search space, with List() or Range() for at least one knob; otherwise it errors out. When you give more than one template, Optuna treats the choice of template as a categorical knob of its own.

• trainer_type (str, optional) – The fine-tuning/post-training control flow to use: "SFT", "DPO", or "GRPO". Skip this argument for run_evals().

• n_initial (int, optional) – Number of configs to launch up front. Default is 16.

• budget (int, optional) – Maximum total number of runs, viz., the initial ones plus any replacements created after pruning. It is raised to n_initial if you give a smaller number; set it equal to n_initial to disable replacement altogether. Default is 40.

• objective (str, optional) – What to optimize, given as "minimize:<metric>" or "maximize:<metric>", e.g., "maximize:accuracy". Comma-separate two or more of these for multi-objective optimization, e.g., "maximize:rougeL,maximize:bleu". Default is "minimize:eval_loss".

• sampler (str, optional) – Optuna sampler to use: "tpe", "cmaes", or "random". Default is "tpe".

• pruner (str, optional) – Optuna pruner to use: "median", "hyperband", or None to disable pruning. Ignored for a multi-objective study. Default is "median".

• seed (int, optional) – Seed for the Optuna sampler. Default is 42.

• granularity (str, optional) – When pruning decisions get evaluated during run_fit(): "chunk" (after every chunk) or "epoch" (after every epoch). Ignored in evals mode. Default is "chunk".

Notes:

Unlike RFRandomSearch(), the seed given to RFOptuna() takes precedence over the seed argument of run_fit() / run_evals(). Because it defaults to 42 rather than to None, changing the seed on the run_fit() / run_evals() call alone will not change which configs Optuna samples; change it on the RFOptuna() constructor instead.

Pruned runs are marked STOPPED, just like runs you stop yourself with IC Ops, and each replacement run appears alongside them as it is created. See the rf-tutorial-optuna-sft-chatqa-tiny and rf-tutorial-optuna-rag-fiqa tutorial notebooks for end-to-end examples.

Example:

# Based on the Optuna SFT tutorial notebook
from rapidfireai.automl import List, RFLoraConfig, RFModelConfig, RFOptuna, RFSFTConfig

config_template = RFModelConfig(
        model_name="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
        peft_config=RFLoraConfig(
                r=List([8, 32]),
                lora_alpha=List([16, 32, 64, 128]),
                lora_dropout=0.1,
                target_modules=List([
                        ["q_proj", "v_proj"],
                        ["q_proj", "k_proj", "v_proj", "o_proj"],
                ]),
                bias="none",
        ),
        training_args=RFSFTConfig(learning_rate=List([1e-3, 1e-4]), ...),
        model_type="causal_lm",
        ...
)

# 4 configs launched up front; up to 2 more created as poor performers are pruned
config_group = RFOptuna(
        configs=[config_template],
        trainer_type="SFT",
        n_initial=4,
        budget=6,
        objective="maximize:rougeL",
        sampler="tpe",
        pruner="median",
        seed=42,
)