API: Experiment
Overview
The concept of an “Experiment” in RapidFire AI helps organize and streamline your AI experimentation.
Every experiment must have a unique user-given name that is used to collate plots on the ML metrics dashboard and for saving its artifacts. If you (mistakenly) reuse a previous experiment name, RapidFire AI will append a suffix akin to what filesystems do.
If you are using Jupyter and if its kernel restarts or gets interrupted for whatever reason, you can just reconnect the kernel and pick up that experiment from where you left off by just rerunning its constructor cell as is. Using that object you can continue your experiment as before.
The Experiment class has the functions and semantics detailed below.
Experiment Constructor
Constructor to instantiate a new experiment.
- __init__(self, ml_spec: str, experiment_name: str, experiments_path: str [ = "./rapidfire_experiments"]) None
Example:
# Based on SFT chatbot tutorial notebook
>>> experiment = Experiment(experiment_name="exp1-chatqa")
Experiment exp1-chatqa created ...
Notes:
You can instantiate as many experiment objects as you want.
We recommend explicitly ending a previous experiment (see end() below) before starting
a new one so that you are cognizant of your code and/or config changes across them.
Run Fit
Main function to launch training and evaluation for a given config group in one go. See the Multi-Config Specification page for more details on how to construct a config group.
- run_fit(self, param_config: Any, create_model_fn: Callable, train_dataset: Dataset, eval_dataset: Dataset, num_chunks: int, seed: int = 42) None:
- Parameters:
param_config (Config-group or list as described in the Multi-Config Specification page) – Single config knob dictionary, a generated config group, or a
Listof configs or config groupscreate_model_fn (Callable) – User-given function to create a model instance; a single cfg is passed as input by the system
train_dataset (Dataset) – Training dataset
eval_dataset (Dataset) – Evaluation dataset to measure eval metrics
num_chunks (int) – Number of logical splits of data to control degree of concurrency for multi-config execution (recommended: at least 4)
seed (int, optional) – Seed for any randomness used in your code (default: 42)
- Returns:
None
- Return type:
None
Example:
# Based on SFT chatbot tutorial notebook
>>> experiment.run_fit(config_group, sample_create_model, train_dataset, eval_dataset, num_chunks=4, seed=42)
Started 4 worker processes successfully ...
Notes:
It auto-generates the ML metrics files as per user specification and auto-plots them on the dashboard.
Within an experiment, you can rerun run_fit() as many times as you want. All of them
will be overlaid on the same plots on the ML metrics dashboard.
Note that run_fit() must be actively running for you to be able to use Interactive Control (IC)
ops on the dashboard.
The param_config argument is very versatile in allowing you to construct various knob combinations
and launch them in one go.
It can be a single config dictionary, a List of config dictionaries, a config group generator output
(RFGridSearch() or RFRandomSearch() for now), or even a List with mix of configs or
config group generator outputs as its elements.
Please see the the Multi-Config Specification page for more details.
Each individual config is passed as input to your create_model_fn(). Inside it you can use whatever
knob you set in the config group, e.g., model type or name to instantiate a model accordingly.
You can import models from libraries such as HuggingFace transformers or load your own PyTorch checkpoints.
The num_chunks argument is a critical one that enables you to balance the higher degree of concurrency
you desire for cross-config comparisons against the (relatively minor) extra swapping overhead incurred.
We recommend at least 4, which means you will see results for all runs on 1/4th of the data at a time.
End
End the current experiment to clear out relevant system state and allow you to move on to a new experiment.
Please do not run this when a run_fit() is still running.
Get Runs Information
Returns metadata about all the runs from across all run_fit() invocations in the current experiment.
- get_runs_info(self) pd.DataFrame:
- Returns:
A DataFrame with the following columns: run_id, status, mlflow_run_id, completed_steps, total_steps, start_chunk_id, num_chunks_visited_curr_epoch, num_epochs_completed, error, source, ended_by, warm_started_from, config (full config dictionary)
- Return type:
pandas.DataFrame
Examples:
# Get metadata of all runs from this experiments so far; based on SFT notebook
all_runs_info = experiment.get_runs_info()
all_runs_info # Screenshot of output below
Notes:
This function is useful for programmatic post-processing and/or pre-processing of runs and their config knobs.
For instance, you can use it as part of new custom AutoML procedure to launch a new run_fit() with new config
knob values based on get_results() from past run_fit() invocations.
We plan to expand this API in the future to return other details about runs such as total runtime, GPU utilization, etc. based on feedback.
Get Results
Returns all metrics (including loss, eval loss, and any eval metrics defined) for all steps
for all runs from across all run_fit() in the current experiment.
- get_results(self) pd.DataFrame
- Returns:
A DataFrame with the following columns: run ID, step number, loss, and one column per metric plot displayed on the dashboard
- Return type:
pandas.DataFrame
Examples:
# Get results of all runs from this experiments so far; based on SFT notebook
all_results = experiment.get_results()
print(all_results.columns) # Screenshot of output below
all_results # Screenshot of output below
Notes:
This function can be useful for programmatic post-processing of the results of your experiments.
For instance, you can use it as part of new custom AutoML procedure if you’d like to adjust your
config for a new run_fit() based on the results of your last run_fit().