API: Generator Configs

RapidFire AI supports both closed-model LLM APIs and self-hosted open LLMs as the generator. For the former, it provides a single unified RFAPIModelConfig that routes any supported provider — OpenAI, Gemini, Anthropic, Azure, and any other OpenAI-compatible endpoint — through the MLflow AI Gateway, which exposes them all behind one OpenAI-compatible API. For the latter, it wraps around the model config of vLLM with RFvLLMModelConfig.

RFAPIModelConfig

This is a single, unified wrapper for closed-model LLM APIs and other remotely-hosted API-accessed models. It routes any supported provider — OpenAI, Gemini, Anthropic, Azure, and more — through the MLflow AI Gateway, which exposes them all behind one OpenAI-compatible API. To switch providers, you generally only change endpoint_config; the rest of the config stays the same.

The individual arguments (knobs) can be List valued or Range valued in an RFAPIModelConfig. That is how you can specify a base set of knob combinations from which a config group can be produced. Also read the Multi-Config Specification page.

Note

When an RFAPIModelConfig is created, the gateway secret, model definition(s), and endpoint(s) described by endpoint_config are provisioned (or reused if they already exist). Misconfiguration such as an unreachable MLflow server or an invalid API key therefore surfaces immediately at construction rather than mid-evaluation. Note that if an endpoint with the same name already exists, it is reused only if its stored API key matches the supplied api_key_name; otherwise provisioning fails and asks you to resolve it via the MLflow dashboard.

See also

Examples:

All examples below are based on the rf-tutorial-scifact-generators tutorial notebook, which demonstrates the unified config across all major providers. Switching providers usually means changing only endpoint_config.

# OpenAI
openai_config = RFAPIModelConfig(
        client_config={"max_retries": 2},
        endpoint_config={
                "provider": "openai",
                "api_key_name": "my_openai_key",
                "api_key": OPENAI_API_KEY,
                "endpoint": {
                        "model": "gpt-5.4-nano",
                        "name": "gpt-5.4-nano",
                        "usage_tracking": True,
                },
        },
        model_config={"max_completion_tokens": 2048},
        rpm_limit=30_000,
        tpm_limit=180_000_000,
        rag=rag_cpu,
        prompt_manager=None,
)

# Gemini — typically only endpoint_config changes
gemini_config = RFAPIModelConfig(
        client_config={"max_retries": 2},
        endpoint_config={
                "provider": "gemini",
                "api_key_name": "my_gemini_key",
                "api_key": GOOGLE_API_KEY,
                "endpoint": {
                        "model": "gemini-2.5-flash-lite",
                        "name": "gemini-2.5-flash-lite",
                        "usage_tracking": True,
                },
        },
        model_config={"max_completion_tokens": 2048},
        rpm_limit=30_000,
        tpm_limit=30_000_000,
        rag=rag_cpu,
        prompt_manager=None,
)

# Anthropic (Claude) — uses itpm_limit/otpm_limit instead of tpm_limit
anthropic_config = RFAPIModelConfig(
        client_config={"max_retries": 2},
        endpoint_config={
                "provider": "anthropic",
                "api_key_name": "my_anthropic_key",
                "api_key": ANTHROPIC_API_KEY,
                "endpoint": {
                        "model": "claude-sonnet-4-6",
                        "name": "claude-sonnet-4-6",
                        "usage_tracking": True,
                },
        },
        model_config={"max_completion_tokens": 2048},
        rpm_limit=2000,
        itpm_limit=800_000,   # Claude uses ITPM and OTPM for rate limiting
        otpm_limit=160_000,
        rag=rag_cpu,
        prompt_manager=None,
)

# Azure OpenAI — add api_base_url and api_version to endpoint_config
azure_config = RFAPIModelConfig(
        client_config={"max_retries": 2},
        endpoint_config={
                "provider": "azure",
                "api_key_name": "my_azure_key",
                "api_key": AZURE_API_KEY,
                "api_base_url": "https://rf-azure-openai.openai.azure.com/",
                "api_version": "2025-04-01-preview",
                "endpoint": {
                        "model": "gpt-5.4-nano",
                        "name": "azure-gpt-5.4-nano",
                        "usage_tracking": True,
                },
        },
        model_config={"max_completion_tokens": 2048},
        rpm_limit=250,
        tpm_limit=250_000,
        rag=rag_cpu,
        prompt_manager=None,
)

# Any OpenAI-compatible gateway (e.g., a Triton/LiteLLM endpoint):
# use provider="openai" with a custom api_base_url
triton_config = RFAPIModelConfig(
        client_config={"max_retries": 2},
        endpoint_config={
                "provider": "openai",
                "api_key_name": "my_triton_key",
                "api_key": TRITON_API_KEY,
                "api_base_url": "https://tritonai-api.ucsd.edu",  # where the models are actually hosted
                "endpoint": {
                        "model": "api-gpt-oss-120b",        # gateway model name
                        "name": "triton_api-gpt-oss-120b",  # endpoint name registered on MLflow
                        "usage_tracking": True,
                },
        },
        model_config={"max_completion_tokens": 2048},
        rpm_limit=500,
        tpm_limit=200_000,
        rag=rag_cpu,
        prompt_manager=None,
)

You can compare any mix of these providers in a single config group via the "api_config" pipeline key.

from rapidfireai.automl import List, RFGridSearch

config_set = {
        "api_config": List([openai_config, gemini_config, anthropic_config, azure_config, triton_config]),
        "batch_size": 8,
        "preprocess_fn": sample_preprocess_fn,
        "postprocess_fn": sample_postprocess_fn,
        "compute_metrics_fn": sample_compute_metrics_fn,
        "accumulate_metrics_fn": sample_accumulate_metrics_fn,
}
config_group = RFGridSearch(config_set)

RFvLLMModelConfig

This is a wrapper around vLLM’s config and SamplingParams classes. The full list of their arguments are available on this page and this page, respectively.

The difference here is that the individual arguments (knobs) can be List valued or Range valued in an RFvLLMModelConfig. That is how you can specify a base set of knob combinations from which a config group can be produced. Also read the Multi-Config Specification page.

class RFvLLMModelConfig

Parameters:

model_config (dict[str, Any]) –
A dictionary with key-value pairs necessary for vLLM-based generation by a self-hosted LLM. All knobs given in this dictionary are simply passed to vLLM as is. vLLM will use its defaults for unspecified knobs. We recommend listing at least the following knobs.
- "model": Name or path of the Hugging Face model to use, e.g., “Qwen/Qwen2.5-0.5B-Instruct”.
- "dtype": Data type for model weights and activations, e.g., “half”, “float”, “bfloat16”.
- "distributed_executor_backend": Backend to use for distributed model workers, either “ray” or “mp” (multiprocessing). Only “mp” supported for now.
- "max_model_len": Model context length (prompt and output). If unspecified, will be automatically derived from the model config.
sampling_params (dict[str, Any]) –
A dictionary with key-value pairs to control the sampling behavior during text generation by vLLM with a self-hosted LLM. All knobs given in this dictionary are simply passed to vLLM as is. vLLM will use its defaults for unspecified knobs. We recommend listing at least the following knobs.
- "temperature": Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. 0.0 means greedy sampling. Default is 1.0.
- "top_p": Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens. Default is 1.0.
- "max_tokens": Maximum number of tokens to generate per output sequence.
rag (RFLangChainRagSpec, optional) – An instance of a RapidFire AI RAG pipeline spec. Also read the API: RFLangChainRagSpec page.
prompt_manager (PromptManager, optional) – An instance of a RapidFire AI PromptManager. Also read the API: Prompt Manager and Other Eval Config Knobs page.