Skip to content

Add -u/--usage to llm chat - #1754

Merged
simonw merged 3 commits into
simonw:mainfrom
bianyh:feat/chat-token-usage
Oct 21, 2026
Merged

simonw merged 3 commits into
simonw:mainfrom
bianyh:feat/chat-token-usage

Conversation

@bianyh

@bianyh bianyh commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Closes #980

Adds -u / --usage to llm chat. After each completed turn it prints the provider-reported token usage to stderr, using the same yellow output and format as llm prompt -u:

Token usage: 10 input, 20 output

For tool chains, each underlying model response gets its own usage line. Continuing a conversation reports only newly generated responses, not the stored history. Without the flag, output is unchanged. This reports token counts and any provider usage details, not a cumulative total or dollar cost.

The existing prompt-command display code is extracted into a shared helper. Documentation and generated CLI help are updated.

Validation

On Linux, Python 3.12.3:

  • Before implementation: the new tests yielded 9 failures and 2 passing no-flag controls, because chat did not recognize either usage flag.
  • After implementation: 11 new cases pass, covering both flag forms, streaming and non-streaming, multiple turns, continued conversations, tool chains, missing/zero counts, and additional usage details.
  • Full suite: pytest -q — 1219 passed.
  • Final chat suite: pytest tests/test_chat.py -q — 21 passed.
  • black --check ., ruff check ., mypy llm, and git diff --check pass.
  • Full Cog check (including README) and Sphinx HTML build pass.

New tests use mock models and make no provider calls. Windows/macOS were not run locally; chat tests retain the existing Windows xfail convention.

AI assistance was used for investigation, implementation, and tests.

@simonw

simonw commented Oct 18, 2026

Copy link
Copy Markdown
Owner

I tested this like so:

~/dev/llm % uv run --with-editable . --with 'llm-tools-datasette>=0.1.1' llm chat --td -u -T 'Datasette("https://datasette.simonwillison.net/simonwillisonblog")'
Chatting with gpt-6-luna
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
> list tables

Tool call: Datasette_schema({})
  [
    {
      "group_concat(sql, ';')": "CREATE TABLE \"comprehend_entity_types\" (\n   \"id\" ..."
    }
  ]

**Listing tables concisely**

I need to create a list of tables as per the user's request. Should I include all tables, both actual and virtual, or just the main ones? It seems like they might only want the schema-related tables without any internal ones. We have 27 regular app/auth tables, 6 virtual FTS, and 24 internal ones. I think I'll query sqlite_master while excluding any internal shadow tables to gather the actual table names. This way, I’ll present a clear list.
Tool call: Datasette_query({'sqlite_sql': "SELECT name FROM sqlite_master WHERE type IN ('table','view') AND name NOT LIKE '%_fts_%' AND name NOT LIKE 'sqlite_%' ORDER BY name;"})
  [
    {
      "name": "auth_permission"
    },
    {
      "name": "auth_user"
    },
    {
      "name": "blog_beat"
    },
    {
      "name": "blog_beat_tags"
    },
    {
      "name": "blog_blogmark"
    },
    {
      "name": "blog_blogmark_fts"
    },
    {
      "name": "blog_blogmark_tags"
    },
    {
      "name": "blog_entry"
    },
    {
      "name": "blog_entry_comprehend_entities"
    },
    {
      "name": "blog_entry_comprehend_entities_done"
    },
    {
      "name": "blog_entry_embeddings"
    },
    {
      "name": "blog_entry_fts"
    },
    {
      "name": "blog_entry_tags"
    },
    {
      "name": "blog_note"
    },
    {
      "name": "blog_note_fts"
    },
    {
      "name": "blog_note_tags"
    },
    {
      "name": "blog_previoustagname"
    },
    {
      "name": "blog_quotation"
    },
    {
      "name": "blog_quotation_fts"
    },
    {
      "name": "blog_quotation_tags"
    },
    {
      "name": "blog_series"
    },
    {
      "name": "blog_series_fts"
    },
    {
      "name": "blog_tag"
    },
    {
      "name": "blog_tag_fts"
    },
    {
      "name": "comprehend_entities"
    },
    {
      "name": "comprehend_entity_types"
    },
    {
      "name": "django_content_type"
    },
    {
      "name": "guides_chapter"
    },
    {
      "name": "guides_guide"
    },
    {
      "name": "guides_guidesection"
    },
    {
      "name": "redirects_redirect"
    },
    {
      "name": "til"
    }
  ]


Tables (including full-text search tables):

- `auth_permission`
- `auth_user`
- `blog_beat`
- `blog_beat_tags`
- `blog_blogmark`
- `blog_blogmark_fts`
- `blog_blogmark_tags`
- `blog_entry`
- `blog_entry_comprehend_entities`
- `blog_entry_comprehend_entities_done`
- `blog_entry_embeddings`
- `blog_entry_fts`
- `blog_entry_tags`
- `blog_note`
- `blog_note_fts`
- `blog_note_tags`
- `blog_previoustagname`
- `blog_quotation`
- `blog_quotation_fts`
- `blog_quotation_tags`
- `blog_series`
- `blog_series_fts`
- `blog_tag`
- `blog_tag_fts`
- `comprehend_entities`
- `comprehend_entity_types`
- `django_content_type`
- `guides_chapter`
- `guides_guide`
- `guides_guidesection`
- `redirects_redirect`
- `til`
Token usage: 71 input, 16 output, {"input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 0}, "output_tokens_details": {"reasoning_tokens": 0}}
Token usage: 4,111 input, 246 output, {"input_tokens_details": {"cache_write_tokens": 4108, "cached_tokens": 0}, "output_tokens_details": {"reasoning_tokens": 190}}
Token usage: 4,648 input, 224 output, {"input_tokens_details": {"cache_write_tokens": 537, "cached_tokens": 4108}, "output_tokens_details": {"reasoning_tokens": 0}}

@simonw simonw added the enhancement New feature or request label Oct 21, 2026
@simonw
simonw merged commit aebaca2 into simonw:main Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add -u usage flag to llm chat

2 participants

⌂