llm-platform

llm-platform is a capability-agnostic LLM tool-calling runtime with a plugin host.

It is written in Python, calls LiteLLM Completions, and is configured through YAML and plugins. You can read the code on GitHub or skip ahead and see the proof.

View on GitHub See the proof

The problem

Most tool-calling apps bury Slack, Jira, or calc logic inside the same module that runs the model loop, so the next product copies half a host by accident.

I needed a reusable solver that owns completion, session, compaction, and plugin composition, while each app owns which capabilities load and how the product UI looks. What was missing was a host extracted from a real app (slack-search) that stays agnostic about domains and still runs a disciplined tool loop.

Why I built it

Day to day, I wanted to define a capability once as a plugin and load it from YAML in whichever app needed it, with the same runtime shared across local projects through an editable install.

A Python package was enough for that. Apps call build_runtime(load_config()) and the host runs the loop, while plugins supply schemas, callables, and optional prompt sections.

What I built

llm-platform is a generic tool-calling runtime. The package owns the completion path (injected completion_fn, default LiteLLM Completions), session handling, sliding-window compaction, timeouts, structured events, tool allow and deny lists, YAML config validation, and a plugin host. Apps own capabilities and the product UI. Opt-in builtins cover time and todo (session-scoped), and domain APIs such as Slack and Jira stay in the app, outside the builtins.

You install it with pip install -e /path/to/llm-platform (Python 3.10+), point it at llm_config.yaml or LLM_CONFIG_PATH, and call runtime.query(...). The sample apps under sample_apps/simple and sample_apps/complex show a no-plugin math path and an app-local calculate plugin. Its status is local multi-project use through an editable install, and I do not position it as a multi-tenant production platform. Completions is supported, and Responses (api.kind: responses) is experimental.

Three decisions

These are the three choices I would defend in a design review.

1. Host, plugin, and app stay separate.

The host assembles the prompt, calls the model, executes tools, manages session and compaction, and enforces depth and timeout limits. A plugin is one problem class, made of schemas, callables, and an optional system_prompt_section(). The app chooses which plugins to load and owns the product UI. I was counting on domain logic living in plugins and apps so the next product does not fork the solver. That split is why the same package can power sample_apps/simple and a Slack-shaped host without rewriting the loop.

2. The model chooses tools from schemas.

The host sends the system prompt, tool schemas, and session messages. If the model returns tool calls, the host runs them and continues until it gets a normal answer or hits max_tool_call_depth. The host does not pick tools in application code. I treated schemas and prompt sections as the control surface, because the failure I care about is a hardcoded if-ladder that reimplements routing the model already does. Allow and deny lists still filter what is loaded, but they do not replace model selection with host guesswork.

3. Completions is the supported default, and Responses stays experimental.

api.kind: completions is the supported transport. api.kind: responses maps Completions-shaped sessions through litellm.responses with function tools only, and the README marks it experimental. The bet was that a reusable host needs one path peers can trust in local apps first. Experimenting with Responses is fine behind a config flag, but shipping it as the default would oversell stability the status table does not claim.

Proof

You can follow the same paths from the public README and HOWTO.md. 

from llm_platform import configure_litellm_api_key, load_config, build_runtime

configure_litellm_api_key()
runtime = build_runtime(load_config())
print(runtime.query("Hello"))

The README gives this minimal config shape. 

model: openai/gpt-4.1-2025-04-14
plugins: []
max_tool_call_depth: 3
api:
  kind: completions

The public repo also confirms plugin registration via @register_plugin, the time / todo builtins, and validate_config. It shows structured events (turn.start, tool.call, and related), the sample apps under sample_apps/, and the status table for Completions versus experimental Responses.

llm-platform runtime architecture: App entry through YAML and plugins into host tool loop; model chooses tools from schemas; Completions default
The runtime map runs from app entry through YAML config and plugin load into the host tool loop (session, compaction, timeouts, events). The model sees schemas and chooses tools. Completions is the supported transport, and Responses remains experimental.

Why I built it this way

I hold these boundaries whenever more than one product needs a tool loop. I want clear ownership and a model-driven schema surface, and I want a default transport I am willing to support.

Keeping domain tools out of the host lets plugins stay swappable, and letting the model choose from schemas keeps routing in the prompt contract. Completions stays the supported default so local apps have a stable path, while Responses stays labeled experimental. If I am going to reuse a solver across projects, I want the split between host, plugin, and app to be something I can draw on a whiteboard.