API: LoRA and Model Configs =============== RapidFire AI's core APIs for model and adapter specifications are all thin wrappers around the corresponding APIs of the Hugging Face's libraries `transformers `__ and `PEFT LoRA `__. .. seealso:: In a **warm-start Clone-Modify** IC Op, model-architecture and LoRA-structural knobs (base model, :code:`dtype`/quantization, LoRA :code:`rank`, :code:`target_modules`) cannot change, or the parent checkpoint/adapter will fail to load. See :ref:`which knobs can be modified ` on the IC Ops page. RFLoraConfig ------ This is a wrapper around :class:`LoraConfig` in HF PEFT. The full signature and list of arguments are available on `this page `__. The difference here is that the individual arguments (knobs) can be :class:`List` valued or :class:`Range` valued in a :class:`RFLoraConfig`. That is how you can specify a base set of knob combinations from which a config group can be produced. Also read :doc:`the Multi-Config Specification page`. **Example:** .. code-block:: python # Singleton config RFLoraConfig( r=128, lora_alpha=256, lora_dropout=0.05, target_modules=["q_proj", "v_proj"], bias="none" ) # 4 combinations RFLoraConfig( r=List([16, 32]), lora_alpha=List([16,32]), lora_dropout=0.05, target_modules=["q_proj", "v_proj"], bias="none" ) # 2 combinations RFLoraConfig( r=64, lora_alpha=128, target_modules=List([["q_proj", "v_proj"], ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]]), bias="none" ) **Notes:** In terms of impact on LLM behavior, the LoRA knobs that are usually experimented with are as follows: * :code:`r` (rank): The most critical knob. Typically a power of 2 between 8 and 128. Higher rank means higher adapter learning capacity but slightly higher GPU memory footprint and compute time. * `lora_alpha`: Controls adaptation strength. Typical values are 16, 32, and 64. Usually set to 2x :code:`r`, but this can be varied too. * :code:`target_modules`: Which layers to apply LoRA to. Common options: * Query/Value only: `["q_proj", "v_proj"]` * Query/Key/Value: `["q_proj", "k_proj", "v_proj"]` * All linear layers: `["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]` * :code:`lora_dropout`: Controls regularization. Typically between 0.0 to 0.05; often set at 0.0 unless there is overfitting. * :code:`init_lora_weights`: Initialization strategy. Most use default :code:`True` but some experiment with :code:`"gaussian"` or newer methods such as :code:`"pissa"`. Most other knobs such as `bias`, `use_rslora`, `modules_to_save`, etc. can be left as PEFT defaults unless you want to explore specific advanced variations. A common combination is to start with rank 16 and 32 (resp. alpha 32 and 64) and only target query/value projection modules. Then expand to more based on observed loss and eval metrics behavior (overfitting or underfitting) and your time/compute constraints. RFModelConfig ------ This is a core class in the RapidFire AI API that abstracts multiple Hugging Face APIs under the hood to simplify and unify all model-related specifications. In particular, it unifies model loading, training configurations, and LoRA settings into one class. It gives you flexibility to try out variations of LoRA adapter structures, training arguments for multiple control flows (SFT, DPO, and GRPO), formatting and metrics functions, and generation specifics. Some of the arguments (knobs) here can also be :class:`List` valued or :class:`Range` valued depending on its data type, as explained below. All this helps form the base set of knob combinations from which a config group can be produced. Also read :doc:`the Multi-Config Specification page`. .. py:class:: RFModelConfig :param model_name: Model identifier for use with Hugging Face's :code:`AutoModel.from_pretrained()`. Can be a Hugging Face model hub name (e.g., ``"Qwen/Qwen2.5-7B-Instruct"``) or local path to a checkpoint. :type model_name: str :param tokenizer: Hugging Face Tokenizer identifier, typically same as :code:`model_name` string but can be different. :type tokenizer: str, optional :param tokenizer_kwargs: Additional keyword arguments passed to tokenizer's :code:`from_pretrained()` method (e.g., :code:`padding_side`, :code:`truncation`, and :code:`model_max_length`). :type tokenizer_kwargs: Dict[str, Any], optional :param formatting_func: Custom user-given data preprocessing function for preparing a single example with system prompt, roles, etc. Can be a :class:`List` for multi-config. :type formatting_func: Callable | :class:`List` of Callable, optional :param compute_metrics: Custom user-given evaluation function passed to Hugging Face's :code:`Trainer.compute_metrics` for use during evaluation phases. Can be a :class:`List` for multi-config. :type compute_metrics: Callable | :class:`List` of Callable, optional :param peft_config: RFLoraConfig as described above; thin wrapper around :code:`peft.LoraConfig` for LoRA fine-tuning. Can be a :class:`List` for multi-config. :type peft_config: RFLoraConfig | :class:`List` of RFLoraConfig, optional :param training_args: RF trainer configuration object specifying the training flow and its parameters. Also read :doc:`the Trainer Configs page`. :type training_args: :class:`RFSFTConfig` | :class:`RFDPOConfig` | :class:`RFGRPOConfig` :param model_type: Custom user-defined string to identify model type inside your :code:`create_model_fn()` given to :func:`run_fit()`; default: ``"causal_lm"`` :type model_type: str, optional :param model_kwargs: Additional parameters for model initialization, passed to :code:`AutoModel.from_pretrained()` (e.g., :code:`torch_dtype`, :code:`device_map`, :code:`trust_remote_code`). :type model_kwargs: Dict[str, Any], optional :param ref_model_name: For DPO and GRPO only; akin to :code:`model_name` above but for the frozen reference model. :type ref_model_name: str, optional :param ref_model_type: For DPO and GRPO only; akin :code:`model_type` above but for the frozen reference model. :type ref_model_type: str, optional :param ref_model_kwargs: For DPO and GRPO only; akin :code:`model_kwargs` above but for the frozen reference model. :type ref_model_kwargs: Dict[str, Any], optional :param reward_funcs: Reward functions for evaluating generated outputs during training (mainly for GRPO but can be used for DPO too). Can be a :class:`List` for multi-config. :type reward_funcs: Callable | [Callable] | :class:`List` of Callable | :class:`List` of [Callable], optional :param generation_config: Arguments for text generation passed to :code:`model.generate()` (e.g., :code:`max_new_tokens`, :code:`temperature`, :code:`top_p`). Three of its keys, viz., :code:`max_input_length`, :code:`add_generation_prompt`, and :code:`truncation_side`, are consumed by RapidFire AI instead of being forwarded to :code:`model.generate()`; they are explained under "Prompt-Side Knobs Inside generation_config" right after this parameter list. :type generation_config: Dict[str, Any], optional :param num_gpus: Number of GPUs to use for each run/config produced from this :code:`RFModelConfig`. Can skip this and specify :code:`num_gpus` in :func:`run_fit()` if all runs must use same number of GPUs. :type num_gpus: int, optional .. seealso:: - :doc:`Hugging Face Transformers documentation ` - :doc:`Hugging Face PEFT library documentation ` - :class:`RFSFTConfig`, :class:`RFDPOConfig`, :class:`RFGRPOConfig` for training argument configurations **Prompt-Side Knobs Inside generation_config** During :func:`run_fit()`, if you set *both* :code:`compute_metrics` and :code:`generation_config` on an :class:`RFModelConfig`, RapidFire AI attaches a generation callback that produces text at each evaluation phase and hands it to your :code:`compute_metrics` function. That callback tokenizes the eval prompts itself; so, three keys placed in :code:`generation_config` configure the *prompt* side and are popped off before the rest of the dictionary is forwarded to :code:`model.generate()`. **max_input_length** : int, optional Token length at which eval prompts are truncated. If unset, it falls back to the tokenizer's :code:`model_max_length` minus :code:`max_new_tokens` so the response has room inside the model's context window, and to 4096 when the tokenizer reports no real maximum. A warning is logged once per run when prompts actually get truncated. **add_generation_prompt** : bool, optional Passed to the tokenizer's :code:`apply_chat_template()` for chat-formatted eval data. Default is :code:`True`, which is what makes an instruction-tuned model start its assistant turn. **truncation_side** : str, optional Which end of an over-long prompt to drop, :code:`"left"` or :code:`"right"`. Default is :code:`"left"`, so the most recent tail of a chat-style prompt is preserved. The tokenizer's original setting is restored afterward. Note that :code:`use_cache` is also popped and ignored by this callback, leaving the KV-cache decision to the model/caller defaults, since enabling it here can cause GPU OOM. **Examples:** .. code-block:: python # Based on the SFT tutorial notebook RFModelConfig( model_name="meta-llama/Llama-3.1-8B-Instruct", peft_config=rfloraconfig1, training_args=rfsftconfig1, model_type="causal_lm", model_kwargs={"device_map": "auto", "torch_dtype": "auto","use_cache":False}, formatting_func=sample_formatting_function, compute_metrics=sample_compute_metrics, generation_config = { "max_new_tokens": 256, "temperature": 0.6, "top_p": 0.9, "top_k": 40, "repetition_penalty": 1.18, } ) # Based on the GRPO tutorial notebook RFModelConfig( model_name="Qwen/Qwen2.5-7B-Instruct", peft_config=rfloraconfig, training_args=rfgrpoconfig1, formatting_func=sample_formatting_function, reward_funcs=reward_funcs, model_kwargs={"load_in_4bit": True, "device_map": "auto", "torch_dtype": "auto", "use_cache": False}, tokenizer_kwargs={"model_max_length": 2048, "padding_side": "left", "truncation": True} ) **Notes:** Note that one :class:`RFModelConfig` object can have only one base model configuration and one training control flow arguments dictionary. But you can specify a :class:`List` of PEFT configs, formatting functions, eval metrics functions, and reward functions list as part of your multi-config specification.