API: Generator Configs =============== RapidFire AI supports both closed-model LLM APIs and self-hosted open LLMs as the generator. For the former, it provides a single unified :class:`RFAPIModelConfig` that routes any supported provider — OpenAI, Gemini, Anthropic, Azure, and any other OpenAI-compatible endpoint — through the MLflow AI Gateway, which exposes them all behind one OpenAI-compatible API. For the latter, it wraps around the model config of vLLM with :class:`RFvLLMModelConfig`. RFAPIModelConfig ------ This is a single, unified wrapper for closed-model LLM APIs and other remotely-hosted API-accessed models. It routes any supported provider — OpenAI, Gemini, Anthropic, Azure, and more — through the MLflow AI Gateway, which exposes them all behind one OpenAI-compatible API. To switch providers, you generally only change :code:`endpoint_config`; the rest of the config stays the same. The individual arguments (knobs) can be :class:`List` valued or :class:`Range` valued in an :class:`RFAPIModelConfig`. That is how you can specify a base set of knob combinations from which a config group can be produced. Also read :doc:`the Multi-Config Specification page`. .. note:: When an :class:`RFAPIModelConfig` is created, the gateway secret, model definition(s), and endpoint(s) described by :code:`endpoint_config` are provisioned (or reused if they already exist). Misconfiguration such as an unreachable MLflow server or an invalid API key therefore surfaces immediately at construction rather than mid-evaluation. Note that if an endpoint with the same :code:`name` already exists, it is reused only if its stored API key matches the supplied :code:`api_key_name`; otherwise provisioning fails and asks you to resolve it via the MLflow dashboard. .. seealso:: In a **Clone-Modify** IC Op for :func:`run_evals()`, the generator config is editable at query time, but editing :code:`endpoint_config` re-provisions the gateway eagerly, and a swapped vLLM model must be downloadable and fit GPU memory. See :ref:`which knobs can be modified ` on the IC Ops page. .. py:class:: RFAPIModelConfig :param client_config: A dictionary of OpenAI-compatible client kwargs used to initialize the AsyncOpenAI client. The gateway :code:`base_url` and a placeholder :code:`api_key` are injected automatically so the client routes through the gateway. So, do NOT put your real API key here (it goes in :code:`endpoint_config`). We recommend listing at least the following knobs. * :code:`"max_retries"`: Maximum number of retry attempts for failed API calls. Optional. Default is 2. * :code:`"timeout"`: Request timeout in seconds. Optional; if unset, the underlying OpenAI client default of 600 seconds (10 minutes) applies. :type client_config: dict[str, Any] :param endpoint_config: A dictionary describing the MLflow gateway endpoint(s) for the chosen provider. Contains the following keys. * :code:`"provider"`: The model provider, e.g., "openai", "gemini", "anthropic", or "azure". For any other OpenAI-compatible gateway (for example, a LiteLLM/Triton endpoint), use "openai" together with a custom :code:`"api_base_url"`. * :code:`"api_key_name"`: Name under which the API key is stored as a gateway secret. * :code:`"api_key"`: The actual API key value. Note that we are NOT able to provide a publicly visible API key. * :code:`"api_base_url"`: Provider base URL. Optional for first-party providers; required when targeting Azure or any other OpenAI-compatible gateway where the models are remotely hosted. * :code:`"api_version"`: API version string. Optional; used by Azure OpenAI. * :code:`"endpoint"`: Either a single :code:`dict` (one endpoint, i.e., a leaf config) or an :class:`automl.List` of dicts (grid search expands it into one :class:`RFAPIModelConfig` per endpoint dict). A plain Python :code:`list` is NOT supported, as it would silently skip grid-search expansion. Each dict can contain: * :code:`"name"`: The endpoint name registered on the gateway/MLflow. Required. * :code:`"model"`: The provider/gateway model name, e.g., "gpt-5.4-nano". Required unless the endpoint already exists. * :code:`"usage_tracking"`: Whether to record token/cost/latency usage for this endpoint on the AI Gateway dashboard. Optional bool; defaults to True. :type endpoint_config: dict[str, Any] :param model_config: A dictionary of sampling/generation knobs for inference, passed through to the provider via the OpenAI-compatible API. Optional. Common knobs include: * :code:`"temperature"`: Controls randomness of sampling; lower is more deterministic, higher is more random. * :code:`"top_p"`: Nucleus sampling threshold; the model considers only the tokens comprising the top :code:`top_p` probability mass. * :code:`"max_completion_tokens"`: Upper bound on the number of tokens generated per completion. * :code:`"reasoning_effort"`: For reasoning models, constrains reasoning effort, e.g., "minimal", "low", "medium", "high". :type model_config: dict[str, Any] :param rpm_limit: Requests per minute limit, used for throttling to avoid exceeding provider quotas. Required. Check the rate limits published by your provider for details on your tier and the latest per-model limits. :type rpm_limit: int :param tpm_limit: Combined tokens per minute limit. Required for all providers EXCEPT :code:`"anthropic"` (for which it is rejected). Cannot be combined with :code:`itpm_limit`/:code:`otpm_limit`. :type tpm_limit: int, optional :param itpm_limit: Input tokens per minute limit. Only accepted for :code:`provider="anthropic"`, and must be paired with :code:`otpm_limit`. :type itpm_limit: int, optional :param otpm_limit: Output tokens per minute limit. Only accepted for :code:`provider="anthropic"`, and must be paired with :code:`itpm_limit`. :type otpm_limit: int, optional :param max_completion_tokens: Maximum completion tokens per request. If not given, it is extracted from :code:`model_config` (falling back to 150). :type max_completion_tokens: int, optional :param rag: An instance of a RapidFire AI RAG pipeline spec. Also read :doc:`the API: RFLangChainRagSpec page `. :type rag: RFLangChainRagSpec, optional :param prompt_manager: An instance of a RapidFire AI PromptManager. Also read :doc:`the API: Prompt Manager and Other Eval Config Knobs page `. :type prompt_manager: PromptManager, optional :param verbose: Whether to print gateway-provisioning progress to stdout. Default is True. Mainly for internal use; it is set to False automatically during grid/random search expansion to suppress duplicate output. :type verbose: bool, optional .. note:: Providers use one of two rate-limit schemes. Most providers (OpenAI, Gemini, Azure, and other OpenAI-compatible endpoints) use a single combined :code:`tpm_limit`. Anthropic instead enforces separate input and output quotas; so, it requires :code:`itpm_limit` and :code:`otpm_limit` and rejects :code:`tpm_limit`. You must specify either :code:`tpm_limit` OR both :code:`itpm_limit` and :code:`otpm_limit`, never both schemes at once. Note that :code:`rpm_limit` is always required. .. seealso:: - :doc:`API: RFLangChainRagSpec ` - :doc:`API: Prompt Manager and Other Eval Config Knobs page ` **Examples:** All examples below are based on the :code:`rf-tutorial-scifact-generators` tutorial notebook, which demonstrates the unified config across all major providers. Switching providers usually means changing only :code:`endpoint_config`. .. code-block:: python # OpenAI openai_config = RFAPIModelConfig( client_config={"max_retries": 2}, endpoint_config={ "provider": "openai", "api_key_name": "my_openai_key", "api_key": OPENAI_API_KEY, "endpoint": { "model": "gpt-5.4-nano", "name": "gpt-5.4-nano", "usage_tracking": True, }, }, model_config={"max_completion_tokens": 2048}, rpm_limit=30_000, tpm_limit=180_000_000, rag=rag_cpu, prompt_manager=None, ) # Gemini — typically only endpoint_config changes gemini_config = RFAPIModelConfig( client_config={"max_retries": 2}, endpoint_config={ "provider": "gemini", "api_key_name": "my_gemini_key", "api_key": GOOGLE_API_KEY, "endpoint": { "model": "gemini-2.5-flash-lite", "name": "gemini-2.5-flash-lite", "usage_tracking": True, }, }, model_config={"max_completion_tokens": 2048}, rpm_limit=30_000, tpm_limit=30_000_000, rag=rag_cpu, prompt_manager=None, ) # Anthropic (Claude) — uses itpm_limit/otpm_limit instead of tpm_limit anthropic_config = RFAPIModelConfig( client_config={"max_retries": 2}, endpoint_config={ "provider": "anthropic", "api_key_name": "my_anthropic_key", "api_key": ANTHROPIC_API_KEY, "endpoint": { "model": "claude-sonnet-4-6", "name": "claude-sonnet-4-6", "usage_tracking": True, }, }, model_config={"max_completion_tokens": 2048}, rpm_limit=2000, itpm_limit=800_000, # Claude uses ITPM and OTPM for rate limiting otpm_limit=160_000, rag=rag_cpu, prompt_manager=None, ) # Azure OpenAI — add api_base_url and api_version to endpoint_config azure_config = RFAPIModelConfig( client_config={"max_retries": 2}, endpoint_config={ "provider": "azure", "api_key_name": "my_azure_key", "api_key": AZURE_API_KEY, "api_base_url": "https://rf-azure-openai.openai.azure.com/", "api_version": "2025-04-01-preview", "endpoint": { "model": "gpt-5.4-nano", "name": "azure-gpt-5.4-nano", "usage_tracking": True, }, }, model_config={"max_completion_tokens": 2048}, rpm_limit=250, tpm_limit=250_000, rag=rag_cpu, prompt_manager=None, ) # Any OpenAI-compatible gateway (e.g., a Triton/LiteLLM endpoint): # use provider="openai" with a custom api_base_url triton_config = RFAPIModelConfig( client_config={"max_retries": 2}, endpoint_config={ "provider": "openai", "api_key_name": "my_triton_key", "api_key": TRITON_API_KEY, "api_base_url": "https://tritonai-api.ucsd.edu", # where the models are actually hosted "endpoint": { "model": "api-gpt-oss-120b", # gateway model name "name": "triton_api-gpt-oss-120b", # endpoint name registered on MLflow "usage_tracking": True, }, }, model_config={"max_completion_tokens": 2048}, rpm_limit=500, tpm_limit=200_000, rag=rag_cpu, prompt_manager=None, ) You can compare any mix of these providers in a single config group via the :code:`"api_config"` pipeline key. .. code-block:: python from rapidfireai.automl import List, RFGridSearch config_set = { "api_config": List([openai_config, gemini_config, anthropic_config, azure_config, triton_config]), "batch_size": 8, "preprocess_fn": sample_preprocess_fn, "postprocess_fn": sample_postprocess_fn, "compute_metrics_fn": sample_compute_metrics_fn, "accumulate_metrics_fn": sample_accumulate_metrics_fn, } config_group = RFGridSearch(config_set) RFvLLMModelConfig ------ This is a wrapper around vLLM's :class:`config` and :class:`SamplingParams` classes. The full list of their arguments are available on `this page `__ and `this page `__, respectively. The difference here is that the individual arguments (knobs) can be :class:`List` valued or :class:`Range` valued in an :class:`RFvLLMModelConfig`. That is how you can specify a base set of knob combinations from which a config group can be produced. Also read :doc:`the Multi-Config Specification page`. .. py:class:: RFvLLMModelConfig :param model_config: A dictionary with key-value pairs necessary for vLLM-based generation by a self-hosted LLM. All knobs given in this dictionary are simply passed to vLLM as is. vLLM will use its defaults for unspecified knobs. We recommend listing at least the following knobs. * :code:`"model"`: Name or path of the Hugging Face model to use, e.g., "Qwen/Qwen2.5-0.5B-Instruct". * :code:`"dtype"`: Data type for model weights and activations, e.g., "half", "float", "bfloat16". * :code:`"distributed_executor_backend"`: Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). Only "mp" supported for now. * :code:`"max_model_len"`: Model context length (prompt and output). If unspecified, will be automatically derived from the model config. :type model_config: dict[str, Any] :param sampling_params: A dictionary with key-value pairs to control the sampling behavior during text generation by vLLM with a self-hosted LLM. All knobs given in this dictionary are simply passed to vLLM as is. vLLM will use its defaults for unspecified knobs. We recommend listing at least the following knobs. * :code:`"temperature"`: Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. 0.0 means greedy sampling. Default is 1.0. * :code:`"top_p"`: Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens. Default is 1.0. * :code:`"max_tokens"`: Maximum number of tokens to generate per output sequence. :type sampling_params: dict[str, Any] :param rag: An instance of a RapidFire AI RAG pipeline spec. Also read :doc:`the API: RFLangChainRagSpec page `. :type rag: RFLangChainRagSpec, optional :param prompt_manager: An instance of a RapidFire AI PromptManager. Also read :doc:`the API: Prompt Manager and Other Eval Config Knobs page `. :type prompt_manager: PromptManager, optional .. seealso:: - `vLLM Config API Reference `_ - `vLLM Sampling Params API Reference `_ - :doc:`RapidFire AI API: RFLangChainRagSpec ` **Examples:** .. code-block:: python # Based on FiQA chatbot tutorial notebook vllm_config1 = RFvLLMModelConfig( model_config={ "model": "Qwen/Qwen2.5-0.5B-Instruct", "dtype": "half", "gpu_memory_utilization": 0.7, "tensor_parallel_size": 1, "distributed_executor_backend": "mp", "enable_chunked_prefill": True, "enable_prefix_caching": True, "max_model_len": 2048, "disable_log_stats": True, # Disable vLLM progress logging }, sampling_params={ "temperature": 0.8, "top_p": 0.95, "max_tokens": 512, }, rag=rag_gpu, prompt_manager=None, )