ai.encode

Returns the tokenized representation of a text.

Syntax

encode(text, options?)

Parameters

• text

  The string to encode.

• options optional

  An optional object containing the following optional properties:

  • model optional

  The name of the model to use. Defaults to the current codebook model. For example, "llama2-7b-chat" or "gpt-3.5-turbo".

  • specials optional

  An object that maps additional special tokens to their integer values. For example: { "<|endoftext|>": 50256 }. The additional tokens are merged with the default special tokens for the model. (Note that not all models support additional special tokens.)

Return value

An array of numbers representing the encoded text.

Examples

ai.encode('hello', { model: 'llama2-7b-chat' })

[1, 29871, 22172]

ai.encode("<|im_start|>hello world!", { model: "gpt-4", specials: {"<|im_start|>": 100264 }})

[100264, 15339, 1917, 0]

See also

• ai.decode