Troubleshooting

Use this page to diagnose and resolve common issues when installing and running RapidFire AI.

Note

RapidFire AI requires Python 3.12+. Verify your shell’s python3 is 3.12 before creating/activating the venv.

Quick diagnostics

If you encounter any error, run the doctor command to get a complete diagnostic report (Python env, relevant packages, GPU/CUDA, and key environment variables):

rapidfireai doctor

The report ends with the tail of every log file under the RapidFire AI log directory, 10 lines each by default. Change that with --log-lines: a larger number for more context, 0 to list the log files without their contents, or -1 for the files in full.

rapidfireai doctor --log-lines 100

Managing the services

These are the commands the CLI accepts; start is used if you give none.

Command	What it does
rapidfireai start	Starts the services, viz., the dashboard frontend, the tracking server (MLflow by default; see --tracking-backends), and the API server/dispatcher. Stays in the foreground and monitors them.
rapidfireai start --no-frontend	Same, minus the dashboard frontend; the tracking and API servers still start.
rapidfireai status	Prints the version, the installed mode (fit or evals), and the state of each service.
rapidfireai restart	Stops everything, waits a couple of seconds, then starts it again.
rapidfireai stop	Stops all services and frees their ports.
rapidfireai jupyter	Starts a Jupyter server on port 8850, preconfigured with the cross-origin settings that the in-notebook IC Ops panel for run_evals() needs.
rapidfireai doctor	Prints the diagnostic report described above.
rapidfireai init	Installs the mode-specific dependencies and copies the tutorial notebooks; see the walkthrough.
rapidfireai --version	Prints the installed version.

Init cannot detect CUDA

rapidfireai init detects your CUDA version through nvcc or nvidia-smi. On a host where neither is available, it says it is disabling CUDA usage and installs a CPU-only dependency set. If that host does have a usable GPU, pin the versions yourself:

# fit or evals install
rapidfireai init --cudaversion 12.4

# evals install on a host without nvidia-smi
rapidfireai init --cudaversion 12.4 --computecapabilityversion 8.0

Both accept major.minor (e.g. 12.4, 8.0); --cudaversion also accepts a bare major version (e.g. 12) or 0.0 to disable CUDA explicitly.

Hugging Face permission errors (login not picked up)

Run the Hugging Face login from the SAME virtual environment where you installed RapidFire AI.

Activate your venv and log in:

source .venv/bin/activate
pip install huggingface-hub
huggingface-cli login
huggingface-cli whoami  # Prints the HF account/orgs for the credentials this venv sees

Using Jupyter notebooks:

• If you logged in while a notebook was already running, restart the notebook kernel so it picks up the new Hugging Face credentials.

• Ensure the notebook uses the same venv kernel.

Port conflicts (services already running)

RapidFire AI uses these local ports by default:

Port	Service	Override env var
8850	Jupyter (RAG evals in-notebook)	RF_JUPYTER_PORT
8851	API server / dispatcher	RF_API_PORT
8852	MLflow dashboard / Traces logging	RF_MLFLOW_PORT
8853	Frontend dashboard	RF_FRONTEND_PORT
8855	Ray	RF_RAY_PORT

If you encounter port conflicts, either kill the processes already using these ports:

lsof -t -i:8850 | xargs kill -9  # jupyter
lsof -t -i:8851 | xargs kill -9  # dispatcher / API server
lsof -t -i:8852 | xargs kill -9  # mlflow / Traces
lsof -t -i:8853 | xargs kill -9  # frontend dashboard
lsof -t -i:8855 | xargs kill -9  # ray

Or change the port a service binds to by setting its environment variable before rapidfireai start (or in your notebook before importing rapidfireai):

export RF_JUPYTER_PORT=8860    # default 8850
export RF_API_PORT=8861        # default 8851
export RF_MLFLOW_PORT=8862     # default 8852 (MLflow / Traces)
export RF_FRONTEND_PORT=8863   # default 8853 (dashboard)
export RF_RAY_PORT=8865        # default 8855
rapidfireai start

Note

On a remote machine, remember to forward whichever ports you end up using (see the port-forwarding step in the walkthrough).

Select specific GPU(s) to use

Set the CUDA_VISIBLE_DEVICES environment variable BEFORE running rapidfireai start to control which GPU(s) RapidFire can see and use.

export CUDA_VISIBLE_DEVICES=2   # use GPU index 2 only
rapidfireai start

Multiple GPUs (example: GPUs 0 and 2):

export CUDA_VISIBLE_DEVICES=0,2
rapidfireai start

From a Python script (set before importing/starting RapidFire):

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "2"
# then start your RapidFire workflow

See also

• For known limitations and workarounds, see Known Issues.

• If you are just getting started, follow the Walkthrough.