API: Experiment ================ Overview -------- An "Experiment" is the central concept in RapidFire AI to organize and streamline your multi-config AI experimentation. Every experiment must have a unique user-given name that is used to collate plots on the metrics dashboard and for saving its artifacts. If you (mistakenly) reuse a previous experiment name, RapidFire AI will append a suffix akin to what filesystems do. The Experiment class has the functions and semantics detailed below. As of this writing, an experiment can operate in either :code:`"evals"` mode or :code:`"fit"` mode but not both. We plan to allow both modes in the same experiment very soon. .. note:: Only one function in this class can be run at a time. If you interrupt a long-running function, say, :func:`run_evals()` or :func:`run_fit()`, wait for up to 2min for the Python threads to be cleaned up before running another function. Experiment Constructor ------ Constructor to instantiate a new experiment. .. py:function:: __init__(self, experiment_name: str, mode: str = "evals", experiment_path: str = "$RF_HOME/rapidfire_experiments", num_cpus: int = None, num_gpus: int = None) -> None :param experiment_name: Unique name for this experiment :type experiment_name: str :param mode: Mode of this experiment, either :code:`"fit"` or :code:`"evals"`; default is :code:`"evals"`. The default matches the :code:`rapidfireai init` install default; :code:`"fit"` is the :code:`--train` opt-in. :type mode: str :param experiment_path: Path to a folder to store this experiment's artifacts. Default is ``"$RF_HOME/rapidfire_experiments"`` :type experiment_path: str, optional :param num_cpus: Cap on the number of logical CPUs handed to the execution backend (Ray). Must be greater than 0 and no greater than the number available. If unset, defaults to 90% of available logical CPUs. Used in :code:`"evals"` mode only; validated but otherwise ignored in :code:`"fit"` mode. :type num_cpus: int, optional :param num_gpus: Cap on the number of GPUs handed to the execution backend (Ray). Must be non-negative and no greater than the number available. If unset, defaults to all available GPUs. Used in :code:`"evals"` mode only; validated but otherwise ignored in :code:`"fit"` mode. :type num_gpus: int, optional :return: None :rtype: None **Example:** .. code-block:: python # Based on FiQA RAG chatbot tutorial notebook (evals is the default mode) >>> experiment = Experiment(experiment_name="exp1-fiqa") Allocating 8 CPUs and 1 GPUs to the experiment Using 1 actors, 1.0 GPUs per actor, 8.0 CPUs per actor Experiment exp1-fiqa created ... # Based on SFT chatbot tutorial notebook >>> experiment = Experiment(experiment_name="exp1-chatqa", mode="fit") Experiment exp1-chatqa created ... **Notes:** You can instantiate as many experiment objects as you want. We recommend explicitly ending a previous experiment (see :func:`end()` below) before starting a new one so that you are cognizant of your code and/or config changes across them. If you are using Jupyter and if its kernel restarts or gets interrupted for whatever reason, you can just reconnect the kernel and pick up that experiment from where you left off by just rerunning its constructor cell as is (unless you ended the experiment explicitly). Using that object you can continue that experiment as before. .. note:: The :code:`mode` you request is validated against the mode RapidFire AI was installed with (:code:`rapidfireai init` installs :code:`"evals"`; :code:`rapidfireai init --train` installs :code:`"fit"`). This check runs before any backend setup, so a mismatch fails immediately with a message telling you which install mode is active, rather than partway through the experiment. Resource Allocation Defaults and Override Semantics ------ In :code:`"evals"` mode, the constructor auto-detects the available hardware at creation time and computes a default actor allocation. It then prints the allocation (CPUs/GPUs handed to the backend, number of actors, and per-actor budgets). These computed defaults are then used by :func:`run_evals()`, and you can override them per call. The defaults are derived as follows: - **GPU machines:** One actor per GPU, with 1 GPU per actor. - **CPU-only machines:** By default 90% of logical CPUs are handed to the backend and divided evenly across actors. The actor count is tiered by the number of CPUs allocated to the backend: 2 for up to 2 CPUs, 4 for up to 4, 8 for up to 32, 16 for up to 128, and 32 beyond that. The :code:`num_cpus` and :code:`num_gpus` constructor arguments cap what is handed to the backend; :code:`num_cpus` must be greater than 0; :code:`num_gpus` must be non-negative; neither may exceed what is available. A host with fewer than 2 logical CPUs is rejected outright: the constructor errors out with a "Minimum required: 2 CPUs" message before allocating anything. :func:`run_evals()` exposes :code:`num_actors`, :code:`gpus_per_actor`, and :code:`cpus_per_actor`; any value you pass overrides the auto-detected default for that call, while any left as :code:`None` falls back to the constructor's computed value. The override is validated: :code:`num_actors * gpus_per_actor` may not exceed the available GPUs and :code:`num_actors * cpus_per_actor` may not exceed the available CPUs, otherwise the call errors out. If :code:`gpus_per_actor` resolves to 0 (a CPU-only allocation), a warning reminds you to use external model APIs for inference. In :code:`"fit"` mode, this auto-allocation does not apply; control GPU usage per config via the :code:`num_gpus` argument of :func:`run_fit()` (and :code:`RFModelConfig`) instead. Run Fit ------ The main function to launch training (including LLM fine-tuning and post-training) and evaluation for a given config group in one go. See :doc:`the Multi-Config Specification page` for more details on how to construct a config group. .. py:function:: run_fit(self, param_config: Any, create_model_fn: Callable, train_dataset: Dataset, eval_dataset: Dataset, num_chunks: int, seed: int=42, num_gpus: int=1, monte_carlo_simulations: int=1000) -> None: :param param_config: A train config knob dictionary, a generated config group, or a :code:`list` of configs or config groups :type param_config: Train config-group or list as described in :doc:`the Multi-Config Specification page` :param create_model_fn: User-given function to create a model instance; a single cfg is passed as input by the system :type create_model_fn: Callable :param train_dataset: Training dataset :type train_dataset: Dataset :param eval_dataset: Evaluation dataset to measure eval metrics :type eval_dataset: Dataset :param num_chunks: Number of logical splits of data to control degree of concurrency for multi-config execution (recommended: at least 4) :type num_chunks: int :param seed: Seed for any randomness used in your code (default: 42) :type seed: int, optional :param num_gpus: Number of GPUs to use per run/config for each config represented in :code:`param_config` (default: 1); overriden by any :code:`num_gpus` given in :code:`RFModelConfig` for those associated configs. :type num_gpus: int, optional :param monte_carlo_simulations: Number of Monte Carlo rollouts the adaptive scheduler performs before each scheduling decision to pick the run-to-worker assignment with the lowest estimated makespan (default: 1000). Raising it can yield slightly better schedules at the cost of more scheduling overhead; lowering it does the opposite. :type monte_carlo_simulations: int, optional :return: None :rtype: None **Example:** .. code-block:: python # Based on SFT chatbot tutorial notebook >>> experiment.run_fit(config_group, sample_create_model, train_dataset, eval_dataset, num_chunks=4, seed=42) Started 4 worker processes successfully ... **Notes:** This method auto-generates the metrics files as per user specification and auto-plots them on the dashboard. Within an experiment, you can rerun :func:`run_fit()` as many times as you want. All of them will be overlaid on the same plots on the metrics dashboard. Note that :func:`run_fit()` must be actively running for you to be able to use Interactive Control (IC) ops on the dashboard. The :code:`param_config` argument is very versatile in allowing you to construct various knob combinations and launch them in one go. It can be a single config dictionary, a :code:`list` of config dictionaries, a config group generator output (:func:`RFGridSearch()`, :func:`RFRandomSearch()`, or :func:`RFOptuna()`), or even a :code:`list` with mix of configs or config group generator outputs as its elements. Please see the :doc:`the Multi-Config Specification page` for more details. Each individual config is passed as input to your :func:`create_model_fn()`. Inside it you can use whatever knob you set in the config group, e.g., model type or name to instantiate a model accordingly. You can import models from libraries such as HuggingFace transformers or load your own PyTorch checkpoints. The :code:`num_chunks` argument is a critical one that enables you to balance a higher degree of concurrency you desire for cross-config comparisons against the (relatively minor) extra swapping overhead incurred. We recommend at least 4, which means you will see results for all runs on 1/4th of the data at a time. Run Evals ------ The main function to launch LLM evaluation (evals), including with optional RAG, for a given config group in one go. See :doc:`the Multi-Config Specification page` for more details on how to construct a config group. .. py:function:: run_evals(self, config_group: Any, dataset: Dataset, num_shards: int=4, seed: int=42, num_actors: int=None, gpus_per_actor: float=None, cpus_per_actor: float=None) -> dict[int, tuple[dict, dict]]: :param config_group: Single evals config knob dictionary, a generated config group, or a :code:`list` of configs or config groups :type config_group: Evals config-group or list as described in :doc:`the Multi-Config Specification page` :param dataset: Evaluation dataset to measure eval metrics :type dataset: Dataset :param num_shards: Number of logical splits of data to control degree of concurrency for multi-config execution (recommended: at least 4) :type num_shards: int :param seed: Seed to control randomness for online aggregation (default: 42) :type seed: int, optional :param num_actors: Number of parallel worker actors to control degree of concurrency. Defaults to :code:`None`, in which case the value auto-detected by the constructor is used (one actor per GPU on GPU machines; a CPU-count-based tier otherwise). See the Resource Allocation section above. :type num_actors: int, optional :param gpus_per_actor: Number of GPUs assigned to each actor. Defaults to :code:`None` (auto-detected: 1.0 on GPU machines, 0.0 on CPU-only machines). :type gpus_per_actor: float, optional :param cpus_per_actor: Number of CPUs assigned to each actor. Defaults to :code:`None` (auto-detected from the constructor's resource allocation). :type cpus_per_actor: float, optional :return: Dictionary keyed by run/config ID; see the description of the returned structure in the Notes below. :rtype: dict[int, tuple[dict, dict]] **Example:** .. code-block:: python # Based on FiQA RAG chatbot tutorial notebook >>> experiment.run_evals(config_group=config_group, dataset=fiqa_dataset, num_shards=4, num_actors=8, seed=42) Started 8 actor processes ... **Notes:** This method auto-generates the metrics as per user specification and lists them in an auto-updated table shown on the notebook itself (and soon, on the metrics dashboard also). Alongside the metrics table, the Interactive Control (IC) Ops panel will also appear on the notebook itself. Note that :func:`run_evals()` must be actively running for you to be able to use IC Ops. Within an experiment, you can rerun :func:`run_evals()` as many times as you want. All of them will be overlaid on the same plots on the metrics dashboard. The :code:`config_group` argument allows you to construct various knob combinations for inference pipelines and launch them in one go. These pipelines can involve LLMs running on your GPUs, or OpenAI API calls, or both. Just like with :func:`run_fit()` above, you can provide a single config dictionary, a :code:`list` of config dictionaries, a config group generator output (:func:`RFGridSearch()`, :func:`RFRandomSearch()`, or :func:`RFOptuna()`), or even a :code:`list` with mix of configs or config group generator outputs as its elements. Please see the :doc:`the Multi-Config Specification page` for more details. The :code:`num_shards` argument is identical to the :code:`num_chunks` argument of :func:`run_fit()` above. That is, it let you balance the degree of concurrency for cross-config comparisons against the (minor) extra swapping overhead incurred. Again, we recommend at least 4, which means you will see results being updated for all runs on 1/4th of the data at a time. For resource control via :code:`num_actors`, :code:`gpus_per_actor`, and :code:`cpus_per_actor`, see the Resource Allocation Defaults and Override Semantics section above. **Return value:** Unlike :func:`run_fit()`, this function returns a value: a dictionary keyed by run/config ID. Each value is a 2-tuple :code:`(results, metrics)`: - **results** — A dictionary of the raw per-query outputs accumulated across all processed shards (e.g., predictions, references, and any keys your postprocess/compute functions emit). - **metrics** — An ordered dictionary that leads with the run's identity and configuration and then lists the eval metrics, in this order: - :code:`run_id` - :code:`model_name` — the model identifier - the configuration knobs set on the config (listed below) - :code:`Samples Processed` - each of your eval metrics, carrying confidence-interval fields (:code:`value`, :code:`lower_bound`, :code:`upper_bound`, :code:`margin_of_error`) per :doc:`Online Aggregation for Evals` The configuration knobs surfaced in the **metrics** dictionary — each present only if set on the config, and each wrapped as :code:`{"value": ...}` — are: - :code:`text_splitter_cfg` - :code:`embedding_cfg` - :code:`vector_store_cfg` - :code:`search_cfg` - :code:`reranker_cfg` - :code:`sampling_params` - :code:`prompt_manager_k` - :code:`model_config` The model identifier appears under :code:`model_name` (the :code:`model` key is removed from :code:`model_config` to avoid redundancy). Because every entry is wrapped, read a knob as, e.g., :code:`metrics["search_cfg"]["value"]`. This gives you a config-to-metrics mapping straight off the return value (join the **results** / **metrics** dictionaries by run/config ID). Note that this surfaced set is a curated subset of knobs, not the full config dictionary. The returned dictionary includes runs that COMPLETED, were STOPPED (via IC Ops or Optuna pruning), or are ONGOING with partial results. It excludes runs that were DELETED or FAILED, as well as any run that produced no results (for example, a clone that was created but never processed a shard). End ------- End the current experiment to clear out relevant system state and allow you to move on to a new experiment. Please do *not* run this when a :func:`run_fit()` is still running. .. py:function:: end(self) -> None :return: None :rtype: None Cancel Current ------- Cancel the currently running task. Works in both :code:`"evals"` and :code:`"fit"` modes. .. py:function:: cancel_current(self) -> None :return: None :rtype: None Get Runs Information ------- Returns metadata about all the runs from across all :func:`run_fit()` invocations in the current experiment. .. note:: This function is available in :code:`"fit"` mode only; calling it in :code:`"evals"` mode raises a :code:`ValueError`. In :code:`"evals"` mode, per-run configuration and metrics are returned directly by :func:`run_evals()` (see its Return value above). .. py:function:: get_runs_info(self) -> pd.DataFrame: :return: A DataFrame with one row per run and the following columns: - :code:`run_id` - :code:`status` - :code:`metric_run_id` — the run's ID on the tracking backend (e.g., MLflow) - :code:`completed_steps` - :code:`total_steps` - :code:`num_chunks_visited_curr_epoch` - :code:`num_epochs_completed` - :code:`chunk_offset` — the chunk index this run started from - :code:`error` - :code:`source` - :code:`ended_by` - :code:`warm_started_from` - :code:`cloned_from` — :code:`run_id` of the parent run for a Clone-Modify clone; :code:`None` otherwise - :code:`estimated_runtime` — the scheduler's runtime estimate for this run - :code:`required_workers` — number of workers (GPUs) this run needs per chunk - :code:`config` — the run's leaf config, minus its :code:`additional_kwargs` entry :rtype: pandas.DataFrame **Examples:** .. code-block:: python # Get metadata of all runs from this experiments so far; based on SFT notebook all_runs_info = experiment.get_runs_info() all_runs_info # Screenshot of output below .. raw:: html Outputs of get runs info .. note:: The screenshot above was captured on an earlier release. It still shows the older column names :code:`mlflow_run_id` and :code:`start_chunk_id`, and it predates the :code:`cloned_from`, :code:`estimated_runtime`, and :code:`required_workers` columns. The column list above is the current one. **Notes:** This function is useful for programmatic post-processing and/or pre-processing of runs and their config knobs. For instance, you can use it as part of new custom AutoML procedure to launch a new :func:`run_fit()` with new config knob values based on :func:`get_results()` from past :func:`run_fit()` invocations. We plan to expand this API in the future to return other details about runs such as total runtime, GPU utilization, etc. based on feedback. Get Results ------- Returns all metrics (including loss, eval loss, and any eval metrics defined) for all steps for all runs from across all :func:`run_fit()` in the current experiment. .. note:: This function is available in :code:`"fit"` mode only; calling it in :code:`"evals"` mode raises a :code:`ValueError`. In :code:`"evals"` mode, results are returned directly by :func:`run_evals()` (see its Return value above). .. py:function:: get_results(self) -> pd.DataFrame :return: A DataFrame with the following columns: run ID, step number, loss, and one column per metric plot displayed on the dashboard :rtype: pandas.DataFrame **Examples:** .. code-block:: python # Get results of all runs from this experiments so far; based on SFT notebook all_results = experiment.get_results() print(all_results.columns) # Screenshot of output below all_results # Screenshot of output below .. raw:: html Columns in DataFrame returned by get results .. raw:: html Outputs of get results **Notes:** This function can be useful for programmatic post-processing of the results of your experiments. For instance, you can use it as part of new custom AutoML procedure if you'd like to adjust your config for a new :func:`run_fit()` based on the results of your last :func:`run_fit()`. Get Log File Path ------- Returns the path to a log file for the current experiment. .. py:function:: get_log_file_path(self, log_type: str = None) -> Path :param log_type: Which log to return. :code:`None`, :code:`"main"`, or :code:`"experiment"` returns the main experiment log; :code:`"training"` returns the training log. Any other value raises a :code:`ValueError`. :type log_type: str, optional :return: Path to the requested log file :rtype: pathlib.Path