The llama.cpp project released build b10003, which includes a significant refactoring of the `tokenize` example. The tool has been migrated to use `common_params_parse`, replacing its previous hand-rolled argument parsing logic with shared helpers.

  • Exposes model-sourcing flags (-m, -mu, -dr, -hf, -hff, --offline, HF_TOKEN) to LLAMA_EXAMPLE_TOKENIZE.
  • Registers --ids, --stdin, --no-bos, --no-parse-special, and --show-count as common args.
  • Sets parse_special default to true for TOKENIZE to preserve existing behavior.
  • Routes errors through LOG_ERR instead of fprintf(stderr).

This update standardizes the tokenize tool's interface with other llama.cpp examples, improving consistency and maintainability across the codebase.