TERMIUM Plus®

The Government of Canada’s terminology and linguistic data bank.

TOKENISER [1 record]

Filter results by subject field Alphabetical list of terms

Filter results by subject field

Alphabetical list of terms

Record 1 2024-05-29

English

Subject field(s)
  • Computer Processing of Language Data
  • Lexicology, Lexicography, Terminology
  • Applications of Automation
  • Artificial Intelligence
DEF

A software program that … determines boundaries for individual tokens [such as words, numbers or punctuation] in text.

Key term(s)
  • tokeniser

French

Domaine(s)
  • Informatisation des données linguistiques
  • Lexicologie, lexicographie et terminologie
  • Automatisation et applications
  • Intelligence artificielle
CONT

En traitement automatique de la langue naturelle, le segmenteur réalise la segmentation d’un texte en une liste de mots, une liste de segments ou de parties de mots (ou symboles), ou encore une suite de caractères.

Spanish

Save record 1

Copyright notice for the TERMIUM Plus® data bank

© Public Services and Procurement Canada, 2024
TERMIUM Plus®, the Government of Canada's terminology and linguistic data bank
A product of the Translation Bureau

Display options

Change the order of display of the official languages of Canada
Option to display the non-official languages (Spanish or Portuguese)
Display definitions, contexts, etc., on records

Features

Language Portal of Canada

Access a collection of Canadian resources on all aspects of English and French, including quizzes.

Writing tools

The Language Portal’s writing tools have a new look! Easy to consult, they give you access to a wealth of information that will help you write better in English and French.

Glossaries and vocabularies

Access Translation Bureau glossaries and vocabularies.

Date Modified: