A business agent can lose essential instructions when its input builder cuts a long prompt at an arbitrary character or token boundary. A small local budget builder gives you a more inspectable rule: keep mandatory context, add complete optional records only when they fit, and return a list of what was excluded.

This tutorial builds that input-preparation step with tiktoken. It does not call a model or establish the token count of a complete API request. The fictional Northstar Repairs example uses sample service information, not a client case study.

Choose the boundary you are actually measuring

tiktoken is a byte-pair encoding tokenizer. Here we select o200k_base explicitly and measure one rendered text string. This makes the rule reproducible; it does not make the encoding appropriate for every model. Select and verify the encoding used by your target integration before adapting this example. The pinned README explains encoding selection.

The illustrative budget is 150 tokens for this text builder. It is not a model context-window specification. Roles, tools, conversation history, multimodal inputs and request formatting can change the complete input count. A model also needs room for output and, where applicable, reasoning tokens. Check the context guidance and use the documented input-token count endpoint when that is appropriate for your actual Responses request. That endpoint was not called in this tutorial.

Set up an isolated local environment

The example was executed on Windows with Python 3.14.3 and tiktoken 0.14.0. Install in a virtual environment. No API key is required for the local counting example. The release metadata records the dependencies and available distributions; tiktoken itself uses the MIT license.

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install tiktoken==0.14.0
$env:TIKTOKEN_CACHE_DIR = Join-Path $PWD "encoding-cache"

Those last commands are PowerShell syntax. On another operating system, use its virtual-environment executable and environment-variable syntax. The first encoding load can fetch its vocabulary over HTTPS. The encoding definition specifies the vocabulary and expected hash; the loader implements caching and hash verification. A cached vocabulary is useful for repeat local runs, but first-run offline installation was not tested.

Keep required records and count the whole rendered candidate

Save this as budget.py. Required records are rendered first in their supplied relative order. Optional records are considered in their supplied order and appended when the entire candidate fits. A large optional record can be omitted while a later smaller record still fits. This is a greedy capacity rule, not relevance ranking or an optimal packing algorithm.

"""Count the complete rendered text; never truncate a source record."""
from dataclasses import dataclass
import tiktoken

ENC = tiktoken.get_encoding('o200k_base')

@dataclass(frozen=True)
class Record:
    id: str
    text: str
    required: bool = False

def render(instructions, records):
    return instructions + '\n\nSOURCE RECORDS\n' + ''.join(
        f'\n--- {r.id} ---\n{r.text}\n' for r in records)

def count(text):
    return len(ENC.encode_ordinary(text))

def build(instructions, records, text_budget):
    if type(text_budget) is not int or text_budget < 1:
        raise ValueError('text_budget must be a positive integer')
    if len({r.id for r in records}) != len(records):
        raise ValueError('Duplicate record ID')
    if any(not r.id or '\n' in r.id or '\r' in r.id for r in records):
        raise ValueError('Record IDs must be nonempty single lines')
    selected = [r for r in records if r.required]
    if count(render(instructions, selected)) > text_budget:
        raise ValueError('Required context exceeds text budget; revise scope')
    omitted = []
    for record in records:
        if record.required:
            continue
        candidate = selected + [record]
        if count(render(instructions, candidate)) <= text_budget:
            selected = candidate
        else:
            omitted.append(record.id)
    text = render(instructions, selected)
    return {'text': text, 'tokens': count(text), 'included': [r.id for r in selected],
            'omitted': omitted, 'encoding': 'o200k_base', 'text_budget': text_budget}

Counting the whole candidate includes separators and avoids assuming that independently counted pieces add up exactly after concatenation. If required context alone exceeds the budget, the function refuses to build an input. Resolve that explicitly by narrowing the task, changing the required material or choosing an appropriate larger allowance.

encode_ordinary treats special-token spellings as ordinary text; this prevents a literal spelling in a source record from raising the special-token exception associated with the default encode behavior. That behavior comes from the encoding implementation. It does not sanitize content or stop prompt injection.

Run a fictional service handoff

Save the following beside the module as example.py. The scope and human-review rule are mandatory. The repetitive history record is optional, and a smaller hours record can fit after that history is omitted.

from budget import Record, build
import json

instructions = "Draft a handoff from these fictional records. Flag missing facts."
records = [
    Record("scope", "Northstar Repairs is fictional. Do not invent prices.", True),
    Record("history", "Older fictional maintenance notes. " * 200),
    Record("hours", "Sample opening hours: Monday-Friday, 9-5."),
    Record("review", "A human must approve the final customer handoff.", True),
]
result = build(instructions, records, text_budget=150)
print(json.dumps(result, indent=2, ensure_ascii=False))

.\.venv\Scripts\python.exe example.py

The executed example produced this local result:

{
  "text": "Draft a handoff from these fictional records. Flag missing facts.\n\nSOURCE RECORDS\n\n--- scope ---\nNorthstar Repairs is fictional. Do not invent prices.\n\n--- review ---\nA human must approve the final customer handoff.\n\n--- hours ---\nSample opening hours: Monday-Friday, 9-5.\n",
  "tokens": 59,
  "included": [
    "scope",
    "review",
    "hours"
  ],
  "omitted": [
    "history"
  ],
  "encoding": "o200k_base",
  "text_budget": 150
}

Keep the omission report alongside the input. In an application, show omitted record IDs to the operator or store them in a restricted diagnostic record. Do not silently describe the resulting prompt as containing all business information. This example does not send the text anywhere.

Diagram: required scope and review plus optional hours included; history omitted intact; final rendered text 59 of 150 local tokens.
Diagram of the executed fictional example: 59 of 150 local text tokens; scope, review and hours included; history omitted intact. This is not a complete model-request count.

Verification and operating limits

Twelve isolated tests passed: required content and instructions remain intact; mandatory overflow is refused; an exact-fit boundary succeeds; a large record is omitted rather than cut; later small records can fit; Unicode and literal special-token text work; results repeat deterministically; duplicate IDs and invalid budgets are refused; and reported tokens equal a recount of the complete output. These tests validate input construction, not model answer quality, savings, or production reliability.

For a regression check, keep test_budget.py beside the module and run:

.\.venv\Scripts\python.exe -m unittest test_budget -v
import unittest
from budget import Record, build, count, render

class BudgetTests(unittest.TestCase):
    def setUp(self):
        self.i = 'Use only supplied records. Report missing information.'
        self.r = Record('scope', 'Fictional Northstar Repairs: draft a service handoff.', True)
    def test_required_preserved(self):
        self.assertIn(self.r.text, build(self.i, [self.r], 100)['text'])
    def test_instructions_preserved(self):
        self.assertTrue(build(self.i, [], 100)['text'].startswith(self.i))
    def test_required_overflow_refused(self):
        with self.assertRaises(ValueError): build(self.i, [self.r], 1)
    def test_exact_boundary(self):
        n = count(render(self.i, [self.r]))
        self.assertEqual(build(self.i, [self.r], n)['tokens'], n)
    def test_record_never_truncated(self):
        r = Record('large', 'Do not cut this record. ' * 400)
        a = build(self.i, [self.r, r], 100)
        self.assertEqual(a['omitted'], ['large'])
        self.assertNotIn(r.text[:40], a['text'])
    def test_later_small_can_fit(self):
        a = build(self.i, [self.r, Record('big', 'large '*500), Record('small', 'Hours: 9–5')], 100)
        self.assertEqual(a['included'], ['scope', 'small'])
    def test_unicode(self):
        r = Record('unicode', 'Café 東京 🔧')
        self.assertIn(r.text, build(self.i, [r], 100)['text'])
    def test_special_spelling_is_plain_text(self):
        self.assertGreater(count('<|endoftext|>'), 0)
    def test_deterministic(self):
        self.assertEqual(build(self.i, [self.r], 100), build(self.i, [self.r], 100))
    def test_duplicate_ids(self):
        with self.assertRaises(ValueError): build(self.i, [self.r, self.r], 100)
    def test_invalid_budget(self):
        for n in [0, -1, True, 1.5]:
            with self.assertRaises(ValueError): build(self.i, [], n)
    def test_full_text_recount(self):
        a = build(self.i, [self.r, Record('note', 'Fictional sample note')], 100)
        self.assertEqual(a['tokens'], count(a['text']))
        self.assertLessEqual(a['tokens'], a['text_budget'])

if __name__ == '__main__': unittest.main(verbosity=2)

Before integrating this into a business agent, decide who marks a record as required, how optional records are ordered and how missing material is surfaced. Untrusted documents must not choose their own priority. The delimiters here are readability aids, not security boundaries: source text can contain misleading instructions or delimiter-like text. The code performs no authorization, redaction, relevance assessment, model execution or output validation.

Repeatedly tokenizing the full candidate is simple to inspect but can be expensive for very large record sets. No throughput benchmark or large-scale memory test was run. The test environment was Windows only. Keep budgets conservative until your actual request formatting and output allowance are verified.

Connect the input builder to a useful workflow

A practical next step is to retrieve relevant records first, apply this capacity rule second, then validate the resulting model output before a downstream write. AlgoThoughts covers local procedure retrieval and structured output validation as separate stages. Combining stages requires its own integration tests; this tutorial did not run that complete pipeline.

Published by Jason Ead for AlgoThoughts AI News. My Biz Heroes connects this kind of practical engineering to business websites, automation, workflows, AI tools and training. The scenario is fictional, and the reported checks were executed for this article’s example; they are not claims about Jason’s personal use or client results.

Updated October 10, 2026: future-tech conceptual cover and a diagram of the recorded local example added. Backslashes lost during the visual revision were restored and the public code was reproduced again.