Premise: A Decision model with n classes for each input the model generates the a probability( ($ p_i $) )for each of n classes
when the class with ($ p_i \approx 1 $) decision is probably correct and error is less.

when classes have equal probability ($1/n$) answer is close to guess and errors come from uncertain inputs it needs non-ambiguous judgement call typically via human interference or nuanced fallback engineering

Obvious way to use threshold T for each ($ n $) look at top probability ($ p_{max} $) if ($p_{max}$) \leq ($ T $) reject else we answer with top probability outcome it dials ($ T = 1/n $) to ($ T =1 $)

Rejectng answer cost ($ F$) and cost of answering wrong at a level ($1 - p_k$) , k is classes and answer only if ($ 1-p_{max}$) < ($ F$)

Let Rejection cost(F) cost of error at level ($ NX_{errcost} $) and proce paid during input ($ NX_{price} $) and error rate on next level ($ NX_{err} $)

($ F = NX_{err} + NX_{price} / NX_{errcost}$)

Chow's rule:

($ 1 - p_{max} < NX_{err} + NX_{price} / NX_{errcost}$)

when errors are expensive llm price barely matters however when errors are cheap llm's price matters more than its mistakes when error under certain amount

Act only when what we gain from it is more than what we pay in return. - Adam smith

judgeing LLM -> price , error rate and cost of each error and a break even point where next level saves equals the price

import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import interact, FloatSlider, Checkbox

p_max = np.linspace(0, 1, 500)
lhs = 1 - p_max

def plot(NX_err=0.05, NX_price=2.0, NX_errcost=100.0):
    rhs = NX_err + NX_price / NX_errcost      # constant threshold
    p_star = 1 - rhs                          # inequality holds for p_max > p_star

    fig, ax = plt.subplots(figsize=(6, 4))
    ax.plot(p_max, lhs, label=r"$1 - p_{max}$")
    ax.axhline(rhs, color="C1", ls="--",
               label=r"$NX_{err} + NX_{price}/NX_{errcost}$")
    ax.fill_between(p_max, 0, 1, where=p_max > p_star,
                    color="green", alpha=0.15, label="inequality holds")
    ax.axvline(p_star, color="gray", ls=":", lw=1)

    ax.set(xlabel=r"$p_{max}$", ylabel="value", xlim=(0, 1), ylim=(0, 1),
           title=rf"threshold = {rhs:.3f},  $p^*$ = {p_star:.3f}")
    ax.legend(loc="upper right")
    plt.tight_layout()
    plt.show()

interact(
    plot,
    NX_err=FloatSlider(value=0.05, min=0, max=1, step=0.005,
                       description="NX_err", readout_format=".3f",
                       continuous_update=True),
    NX_price=FloatSlider(value=2.0, min=0, max=50, step=0.1,
                         description="NX_price"),
    NX_errcost=FloatLogSlider(value=100.0, base=10, min=0, max=4, step=0.05,
                              description="NX_errcost"),
);

when error is not same in real world during random audit

($ 1 - p_{max} * NX_{errcost}(max)/C'< NX_{err} + NX_{price} / C'$)

where ($ C' = NX_{errcost}$)

p_max = 0.8
r = 1.0
clamp_top = True

def plot_chow(a=0.05, b=0.25):
    # a = NX_price, b = NX_err, r = NX_errcost(max)/C'
    x = np.linspace(0.05, 5, 600)            # x = C'
    y = np.maximum(0, 1 - a - b / x)         # hyperbola, clamped at 0
    if clamp_top:
        y = np.minimum(y, 1)

    h = 1 - p_max * r
    ymax = 1.05
    upper = np.maximum(y, h)                 # top of the Abstain region / bottom of Answer

    fig, ax = plt.subplots(figsize=(7, 5))

    # Abstain: between the line and the curve, only where the curve is above the line
    ax.fill_between(x, h, y, where=(y > h), interpolate=True,
                    color="lightpink", alpha=0.6, label="Abstain")

    # Answer: everything above both the line and the curve (nothing below h)
    ax.fill_between(x, upper, ymax,
                    color="palegreen", alpha=0.6, label="Answer")

    ax.plot(x, y, lw=2, label=r"threshold $y = 1 - a - b/x$")
    ax.axhline(h, color="k", ls="--",
               label=r"$1 - p_{max}\,r$ (left side)")

    ax.set_xlabel("x  ($C'$)")
    ax.set_ylabel("y  ($p_{max} * NX_{errcost}(max)/C'$)")
    ax.set_xlim(x[0], x[-1])
    ax.set_ylim(0, ymax)
    ax.legend()
    ax.set_title("Modified Chow's rule")
    plt.show()

interact(plot_chow,
         a=FloatSlider(min=0, max=2, step=0.01, value=0.2, description="NX_price"),
         b=FloatSlider(min=0, max=1, step=0.01, value=0.1, description="NX_err"))

<function __main__.plot_chow(a=0.05, b=0.25)>

if wait cost and latency of the model is considered

($ 1 - p_{max} < NX_{err} + ( NX_{price} + waitcost * NX_{latency}) / NX_{errcost}$)

Non Determinism in Decision Models