Token counting on the bedrock-mantle endpoint: Documented how to count input tokens through the Anthropic count_tokens path on the bedrock-mantle endpoint, including for Claude models that are not supported by CountTokens on bedrock-runtime .

Monitor your token usage by counting tokens before running inference

When you run model inference, the number of tokens that you send in the input contributes to the cost of the request and towards the quota of tokens that you can use per minute and per day. The CountTokens API helps you estimate token usage before sending requests to foundation models by returning the token count that would be used if the same input were sent to the model in an inference request.

Using the CountTokens API doesn't incur charges. Some Anthropic Claude models don't support CountTokens on bedrock-runtime. For these models, count input tokens by calling Anthropic's count_tokens API on the bedrock-mantle endpoint instead.

What to do

• Estimate costs before sending inference requests.
• Optimize prompts to fit within token limits.
• Plan for token usage in your applications.

Supported models and Regions for token counting

To see which models support token counting, please visit models at a glance and pick the model you are interested in.

Count tokens using the bedrock-runtime endpoint

To count the number of input tokens in an inference request, send a CountTokens request with an Amazon Bedrock runtime endpoint, specify the model in the header and the input to count tokens for in the body field. The value of the body field depends on whether you're counting input tokens for an InvokeModel or Converse request.

Example: count tokens for a bedrock-runtime request

The examples in this section let you count tokens for an InvokeModel and Converse request with Anthropic Claude 3 Haiku.

Count tokens using the bedrock-mantle endpoint

The bedrock-mantle endpoint exposes Anthropic's count_tokens API at /anthropic/v1/messages/count_tokens. Use it to count input tokens for Anthropic Claude models that don't support CountTokens on bedrock-runtime. The /anthropic/v1/messages path is Claude-specific; non-Anthropic models on bedrock-mantle return The model 'X' does not support the '/anthropic/v1/messages' API.

Source: AWS release notes

If you need further guidance on AWS, our experts are available at AWS@westloop.io. You may also reach us by submitting the Contact Us form.