Troubleshooting
===============

Use this page to diagnose and resolve common issues when installing and running RapidFire AI.

.. note::

  RapidFire AI requires Python 3.12+. Verify your shell's ``python3`` is 3.12 before creating/activating the venv.

Quick diagnostics
-----------------

If you encounter any error, run the doctor command to get a complete diagnostic report (Python env, relevant packages, GPU/CUDA, and key environment variables):

.. code-block:: bash

   rapidfireai doctor

The report ends with the tail of every log file under the RapidFire AI log directory, 10 lines each by default. Change that with ``--log-lines``: a larger number for more context, ``0`` to list the log files without their contents, or ``-1`` for the files in full.

.. code-block:: bash

   rapidfireai doctor --log-lines 100


Managing the services
---------------------

These are the commands the CLI accepts; ``start`` is used if you give none.

.. list-table::
   :header-rows: 1
   :widths: 34 66

   * - Command
     - What it does
   * - ``rapidfireai start``
     - Starts the services, viz., the dashboard frontend, the tracking server (MLflow by default; see ``--tracking-backends``), and the API server/dispatcher. Stays in the foreground and monitors them.
   * - ``rapidfireai start --no-frontend``
     - Same, minus the dashboard frontend; the tracking and API servers still start.
   * - ``rapidfireai status``
     - Prints the version, the installed mode (``fit`` or ``evals``), and the state of each service.
   * - ``rapidfireai restart``
     - Stops everything, waits a couple of seconds, then starts it again.
   * - ``rapidfireai stop``
     - Stops all services and frees their ports.
   * - ``rapidfireai jupyter``
     - Starts a Jupyter server on port 8850, preconfigured with the cross-origin settings that the in-notebook IC Ops panel for ``run_evals()`` needs.
   * - ``rapidfireai doctor``
     - Prints the diagnostic report described above.
   * - ``rapidfireai init``
     - Installs the mode-specific dependencies and copies the tutorial notebooks; see the :doc:`walkthrough </walkthrough>`.
   * - ``rapidfireai --version``
     - Prints the installed version.


Init cannot detect CUDA
-----------------------

``rapidfireai init`` detects your CUDA version through ``nvcc`` or ``nvidia-smi``. On a host where neither is available, it says it is disabling CUDA usage and installs a CPU-only dependency set. If that host does have a usable GPU, pin the versions yourself:

.. code-block:: bash

   # fit or evals install
   rapidfireai init --cudaversion 12.4

   # evals install on a host without nvidia-smi
   rapidfireai init --cudaversion 12.4 --computecapabilityversion 8.0

Both accept ``major.minor`` (e.g. ``12.4``, ``8.0``); ``--cudaversion`` also accepts a bare major version (e.g. ``12``) or ``0.0`` to disable CUDA explicitly.


Hugging Face permission errors (login not picked up)
---------------------------------------------------

Run the Hugging Face login from the SAME virtual environment where you installed RapidFire AI.

Activate your venv and log in:

.. code-block:: bash

   source .venv/bin/activate
   pip install huggingface-hub
   huggingface-cli login
   huggingface-cli whoami  # Prints the HF account/orgs for the credentials this venv sees


Using Jupyter notebooks:

- If you logged in while a notebook was already running, restart the notebook kernel so it picks up the new Hugging Face credentials.
- Ensure the notebook uses the same venv kernel. 


Port conflicts (services already running)
----------------------------------------

RapidFire AI uses these local ports by default:

.. list-table::
   :header-rows: 1
   :widths: 16 44 40

   * - Port
     - Service
     - Override env var
   * - 8850
     - Jupyter (RAG evals in-notebook)
     - ``RF_JUPYTER_PORT``
   * - 8851
     - API server / dispatcher
     - ``RF_API_PORT``
   * - 8852
     - MLflow dashboard / Traces logging
     - ``RF_MLFLOW_PORT``
   * - 8853
     - Frontend dashboard
     - ``RF_FRONTEND_PORT``
   * - 8855
     - Ray
     - ``RF_RAY_PORT``

If you encounter port conflicts, either kill the processes already using these ports:

.. code-block:: bash

   lsof -t -i:8850 | xargs kill -9  # jupyter
   lsof -t -i:8851 | xargs kill -9  # dispatcher / API server
   lsof -t -i:8852 | xargs kill -9  # mlflow / Traces
   lsof -t -i:8853 | xargs kill -9  # frontend dashboard
   lsof -t -i:8855 | xargs kill -9  # ray

Or change the port a service binds to by setting its environment variable before ``rapidfireai start`` (or in your notebook before importing rapidfireai):

.. code-block:: bash

   export RF_JUPYTER_PORT=8860    # default 8850
   export RF_API_PORT=8861        # default 8851
   export RF_MLFLOW_PORT=8862     # default 8852 (MLflow / Traces)
   export RF_FRONTEND_PORT=8863   # default 8853 (dashboard)
   export RF_RAY_PORT=8865        # default 8855
   rapidfireai start

.. note::
  On a remote machine, remember to forward whichever ports you end up using (see the port-forwarding step in the walkthrough).



Select specific GPU(s) to use
-----------------------------

Set the ``CUDA_VISIBLE_DEVICES`` environment variable BEFORE running ``rapidfireai start`` to control which GPU(s) RapidFire can see and use.

.. code-block:: bash

   export CUDA_VISIBLE_DEVICES=2   # use GPU index 2 only
   rapidfireai start

Multiple GPUs (example: GPUs 0 and 2):

.. code-block:: bash

   export CUDA_VISIBLE_DEVICES=0,2
   rapidfireai start

From a Python script (set before importing/starting RapidFire):

.. code-block:: python

   import os
   os.environ["CUDA_VISIBLE_DEVICES"] = "2"
   # then start your RapidFire workflow


See also
--------

- For known limitations and workarounds, see :doc:`Known Issues </issues>`.
- If you are just getting started, follow the :doc:`Walkthrough </walkthrough>`.