API: Multi-Config Specification
===============

The core value of RapidFire AI is in the ability to launch, compare, and dynamically 
control multiple configurations (configs) in one go. 
It is already common practice in AI to do hyperparameter sweeps via grid search, 
random search, or AutoML heuristics to generate knob values. 

RapidFire AI generalizes that notion to any type of config knobs, not just  
regular hyperparameters, but also base model architectures, prompt schemes, 
LoRA adapters, and optimizers (for training), as well as data chunking, embedding, 
retrieval, reranking, generation, and prompt schemes (for RAG) and any other 
user-defined knobs. 


Knob Set Generators
-------

To create a multi-config specification, you need two things: **knob set generators** for 
knob values and **config group generators** that take a config with set-valued knobs 
to generate groups of full configs.

We currently support two common knob set generators: :func:`List()` for a discrete 
set of values and :func:`Range()` for sampling from a continuous value interval.


.. py:function:: List(values: List[Any])

	:param values: List of discrete values for a knob. We recommend using values of the same python data type within a given :func:`List()`, although this is not enforced.
	:type values: List[Any]


.. py:function:: Range(start: int | float, end: int | float, dtype: str = None, log: bool = False, step: int | float = None)

	:param start: Lower bound of range interval, inclusive.
	:type start: int | float

	:param end: Upper bound of range interval, inclusive.
	:type end: int | float

	:param dtype: Data type of value to be sampled, either :code:`"int"` or :code:`"float"`. Optional; if omitted, it is inferred from :code:`start` and :code:`end`, viz., :code:`"int"` only if both of them are python ints and :code:`"float"` otherwise.
	:type dtype: str, optional

	:param log: Whether to sample in log space instead of uniformly; both :code:`start` and :code:`end` must then be greater than 0. Default is :code:`False`. Mutually exclusive with :code:`step`.
	:type log: bool, optional

	:param step: Discretization step; sampled values are then multiples of :code:`step` counting up from :code:`start`. Default is :code:`None`, i.e., the interval is treated as continuous. Mutually exclusive with :code:`log`.
	:type step: int | float, optional


**Notes:**

By default, :func:`Range()` performs uniform sampling within the given interval. 
Set :code:`log=True` for log-uniform sampling, which is usually a better fit for scale-free knobs 
such as learning rate, or set :code:`step` to restrict sampling to a discrete grid of values within 
the interval. Giving both :code:`log` and :code:`step` raises an error.

These variants line up one-to-one with Optuna's :code:`suggest_int()` / :code:`suggest_float()` 
distributions; so, the same :func:`Range()` specification carries over to :func:`RFOptuna()` below.

Note that the return types of the knob set generators are internal to RapidFire AI and 
they are usable only within the context of the config group generators below.



Config Group Generators
-----

We currently support three config group generators: :func:`RFGridSearch()` for grid search, 
:func:`RFRandomSearch()` for random search, and :func:`RFOptuna()` for search driven by the popular 
AutoML library Optuna, including adaptive pruning of unpromising runs. 

More support for AutoML heuristics such as SHA and HyperOpt is coming soon. 
Likewise for RAG/context engineering, we also plan to support the AutoML heuristic syftr.

All three take the config(s) to expand as their first argument. That argument accepts a single 
config, a plain python :code:`list` of configs, or a :func:`List()` of configs. For 
:func:`run_fit()`, the configs must be :class:`RFModelConfig` instances; for :func:`run_evals()`, 
they must be config dictionaries. Passing a :code:`trainer_type` is what puts a generator in fit 
mode; leaving it unset puts it in evals mode.

.. warning::
   For :func:`RFGridSearch()` and :func:`RFRandomSearch()`, the second positional parameter is a 
   legacy, unused :code:`create_model_fn`, *not* :code:`trainer_type`. So, always pass 
   :code:`trainer_type` and :code:`num_runs` as keyword arguments. 
   For example, :code:`RFGridSearch(config_set, "SFT")` assigns :code:`"SFT"` to 
   :code:`create_model_fn`, leaves :code:`trainer_type` unset, which silently puts the generator in 
   evals mode, and then fails validation with "Evals mode requires dict instances". 
   Write :code:`RFGridSearch(config_set, trainer_type="SFT")` instead. 
   :func:`RFOptuna()` does not have this quirk; its second parameter is :code:`trainer_type`.


.. py:function:: RFGridSearch(configs: RFModelConfig | Dict[str, Any] | list, create_model_fn: Callable = None, trainer_type: str = None, num_runs: int = 1)

	:param configs: The config(s) to expand, with :func:`List()` for at least one knob. An :class:`RFModelConfig` (or a list/:func:`List()` of them) for :func:`run_fit()`; a config dictionary (or a list/:func:`List()` of them) for :func:`run_evals()`.
	:type configs: RFModelConfig | Dict[str, Any] | list

	:param create_model_fn: Legacy parameter that is unused. Never pass anything here; see the warning above.
	:type create_model_fn: Callable, optional

	:param trainer_type: The fine-tuning/post-training control flow to use: :code:`"SFT"`, :code:`"DPO"`, or :code:`"GRPO"`. Skip this argument for :func:`run_evals()`.
	:type trainer_type: str, optional 

	:param num_runs: Ignored by grid search, which always yields the full cross product of knob values. Default is 1.
	:type num_runs: int, optional


.. py:function:: RFRandomSearch(configs: RFModelConfig | Dict[str, Any] | list, create_model_fn: Callable = None, trainer_type: str = None, num_runs: int = 1)

	:param configs: The config(s) to sample from, with :func:`List()` or :func:`Range()` for at least one knob. An :class:`RFModelConfig` (or a list/:func:`List()` of them) for :func:`run_fit()`; a config dictionary (or a list/:func:`List()` of them) for :func:`run_evals()`.
	:type configs: RFModelConfig | Dict[str, Any] | list

	:param create_model_fn: Legacy parameter that is unused. Never pass anything here; see the warning above.
	:type create_model_fn: Callable, optional

	:param trainer_type: The fine-tuning/post-training control flow to use: :code:`"SFT"`, :code:`"DPO"`, or :code:`"GRPO"`. Skip this argument for :func:`run_evals()`.
	:type trainer_type: str, optional

	:param num_runs: Number of runs (full combinations of knob values) to sample in total. Default is 1; so, set this explicitly.
	:type num_runs: int, optional


**Notes:**

For :func:`RFGridSearch()`, each knob can have either a single value or a :func:`List()` of values. 
Grid search expands only :func:`List()` valued knobs. A :func:`Range()` is not supported here: it is 
passed through to the run as an unexpanded RapidFire AI object instead of a knob value, which will 
almost certainly fail downstream. So, use :func:`RFRandomSearch()` or :func:`RFOptuna()` if you want 
to explore a continuous interval.

For :func:`RFRandomSearch()`, each knob can have either a single value, or a :func:`List()` of values, or a 
:func:`Range()` of values. The semantics of sampling are independently-identically-distributed (IID), i.e.,
we uniformly randomly pick a value from each discrete set and from each continuous set to construct the 
knob combination for one run. 
Then we repeat that sampling process in an IID way to accumulate :code:`num_runs` distinct combinations.

:func:`RFRandomSearch()` has no constructor :code:`seed`; its sampling is seeded by the :code:`seed` 
argument of :func:`run_fit()` / :func:`run_evals()` (default 42).

In evals mode, the generator recognizes exactly three keys for the inference pipeline in your config 
dictionary: :code:`"api_config"`, :code:`"vllm_config"`, and :code:`"pipeline"`. Everything else in 
the dictionary is treated as a regular knob. Also read 
:doc:`the Generator Configs page</generators>`.

Note that the return types of the config group generators are internal to RapidFire AI and they are usable only 
within the context of :func:`run_fit()` or :func:`run_evals()` in the :class:`Experiment` class.


**Examples:**

.. code-block:: python

	# Example 1: Based on SFT tutorial notebook
	from rapidfireai.automl import List, RFGridSearch, RFLoraConfig, RFModelConfig, RFSFTConfig

	# 2 LoRA adapter configs with different capacities
	peft_configs = List([
		RFLoraConfig(
			r=16, lora_alpha=32, lora_dropout=0.05, 
			target_modules=["q_proj", "v_proj"], bias="none"
		),
		RFLoraConfig(
			r=128, lora_alpha=256, lora_dropout=0.05,
			target_modules=["q_proj","k_proj", "v_proj","o_proj"], bias="none"
		)
	])

	# 2 base models x 2 LoRA configs = 4 combinations in total
	config_set = List([
		RFModelConfig(
			model_name="meta-llama/Llama-3.1-8B-Instruct",
			peft_config=peft_configs,
			training_args=RFSFTConfig(learning_rate=2e-4, num_train_epochs=2, ...),
			model_type="causal_lm",
			...
		),
		RFModelConfig(
			model_name="mistralai/Mistral-7B-Instruct-v0.3",
			peft_config=peft_configs,
			training_args=RFSFTConfig(learning_rate=2e-4, num_train_epochs=2, ...),
			model_type="causal_lm",
			...
		)
	])

	config_group = RFGridSearch(
		configs=config_set,
		trainer_type="SFT"
	)


	# Example 2: Based on GSM8K tutorial notebook
	from rapidfireai.automl import List, RFAPIModelConfig, RFGridSearch, RFPromptManager

	gemini_config1 = RFAPIModelConfig(
		client_config={"max_retries": 2},
		endpoint_config={
			"provider": "gemini",
			"api_key_name": "my_gemini_key",
			"api_key": GOOGLE_API_KEY,
			"endpoint": {
				"model": "gemini-2.5-flash-lite",
				"name": "gemini-2.5-flash-lite",
			},
		},
		model_config={"max_completion_tokens": 2048},
		rpm_limit=30_000,
		tpm_limit=30_000_000,
		prompt_manager=fewshot_prompt_manager,
	)

	gemini_config2 = RFAPIModelConfig(
		client_config={"max_retries": 2},
		endpoint_config={
			"provider": "gemini",
			"api_key_name": "my_gemini_key",  # reusing the same stored secret
			"endpoint": {
				"model": "gemini-3-flash-preview",
				"name": "gemini-3-flash-preview",
			},
		},
		model_config={
			"max_completion_tokens": 4096,
			"reasoning_effort": List(["medium", "low"]),  # 2 different reasoning levels
		},
		rpm_limit=20_000,
		tpm_limit=20_000_000,
		prompt_manager=fewshot_prompt_manager,
	)

	config_set = {
		"api_config": List(
			[gemini_config1, gemini_config2]
		),
		"batch_size": batch_size,
		"preprocess_fn": sample_preprocess_fn,
		"compute_metrics_fn": sample_compute_metrics_fn,
		...
	}

	config_group = RFGridSearch(config_set)



RFOptuna
^^^^^^^^

:func:`RFOptuna()` is a drop-in replacement for the above two generators that uses 
`Optuna <https://optuna.org/>`__'s ask-and-tell API to pick knob combinations adaptively. 
It supports single- and multi-objective optimization, and it can *prune* an underperforming run at a 
chunk (or shard) boundary and spend the freed GPU slot on a fresh, better-informed config, up to a 
total :code:`budget` of runs.

.. note::
   :code:`from rapidfireai.automl import RFOptuna` always succeeds, but instantiating it raises an 
   :code:`ImportError` with install instructions if Optuna is not importable. Install it with 
   :code:`pip install "rapidfireai[optuna]"` (which pins :code:`optuna>=4.0.0`) or with plain 
   :code:`python -m pip install optuna`. Optuna must go into the *same* interpreter as your Jupyter 
   kernel; check with :code:`import sys; print(sys.executable)` in a notebook cell and restart the 
   kernel afterward.


.. py:function:: RFOptuna(configs: RFModelConfig | Dict[str, Any] | list, trainer_type: str = None, n_initial: int = 16, budget: int = 40, objective: str = "minimize:eval_loss", sampler: str = "tpe", pruner: str = "median", seed: int = 42, granularity: str = "chunk")

	:param configs: The config template(s) that define the search space, with :func:`List()` or :func:`Range()` for at least one knob; otherwise it errors out. When you give more than one template, Optuna treats the choice of template as a categorical knob of its own.
	:type configs: RFModelConfig | Dict[str, Any] | list

	:param trainer_type: The fine-tuning/post-training control flow to use: :code:`"SFT"`, :code:`"DPO"`, or :code:`"GRPO"`. Skip this argument for :func:`run_evals()`.
	:type trainer_type: str, optional

	:param n_initial: Number of configs to launch up front. Default is 16.
	:type n_initial: int, optional

	:param budget: Maximum total number of runs, viz., the initial ones plus any replacements created after pruning. It is raised to :code:`n_initial` if you give a smaller number; set it equal to :code:`n_initial` to disable replacement altogether. Default is 40.
	:type budget: int, optional

	:param objective: What to optimize, given as :code:`"minimize:<metric>"` or :code:`"maximize:<metric>"`, e.g., :code:`"maximize:accuracy"`. Comma-separate two or more of these for multi-objective optimization, e.g., :code:`"maximize:rougeL,maximize:bleu"`. Default is :code:`"minimize:eval_loss"`.
	:type objective: str, optional

	:param sampler: Optuna sampler to use: :code:`"tpe"`, :code:`"cmaes"`, or :code:`"random"`. Default is :code:`"tpe"`.
	:type sampler: str, optional

	:param pruner: Optuna pruner to use: :code:`"median"`, :code:`"hyperband"`, or :code:`None` to disable pruning. Ignored for a multi-objective study. Default is :code:`"median"`.
	:type pruner: str, optional

	:param seed: Seed for the Optuna sampler. Default is 42.
	:type seed: int, optional

	:param granularity: When pruning decisions get evaluated during :func:`run_fit()`: :code:`"chunk"` (after every chunk) or :code:`"epoch"` (after every epoch). Ignored in evals mode. Default is :code:`"chunk"`.
	:type granularity: str, optional


**Notes:**

Unlike :func:`RFRandomSearch()`, the :code:`seed` given to :func:`RFOptuna()` takes precedence over 
the :code:`seed` argument of :func:`run_fit()` / :func:`run_evals()`. Because it defaults to 42 rather 
than to :code:`None`, changing the seed on the :func:`run_fit()` / :func:`run_evals()` call alone will 
not change which configs Optuna samples; change it on the :func:`RFOptuna()` constructor instead.

Pruned runs are marked STOPPED, just like runs you stop yourself with IC Ops, and each replacement 
run appears alongside them as it is created. 
See the :code:`rf-tutorial-optuna-sft-chatqa-tiny` and :code:`rf-tutorial-optuna-rag-fiqa` 
:doc:`tutorial notebooks</tutorials>` for end-to-end examples.


**Example:**

.. code-block:: python

	# Based on the Optuna SFT tutorial notebook
	from rapidfireai.automl import List, RFLoraConfig, RFModelConfig, RFOptuna, RFSFTConfig

	config_template = RFModelConfig(
		model_name="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
		peft_config=RFLoraConfig(
			r=List([8, 32]),
			lora_alpha=List([16, 32, 64, 128]),
			lora_dropout=0.1,
			target_modules=List([
				["q_proj", "v_proj"],
				["q_proj", "k_proj", "v_proj", "o_proj"],
			]),
			bias="none",
		),
		training_args=RFSFTConfig(learning_rate=List([1e-3, 1e-4]), ...),
		model_type="causal_lm",
		...
	)

	# 4 configs launched up front; up to 2 more created as poor performers are pruned
	config_group = RFOptuna(
		configs=[config_template],
		trainer_type="SFT",
		n_initial=4,
		budget=6,
		objective="maximize:rougeL",
		sampler="tpe",
		pruner="median",
		seed=42,
	)