TERMIUM Plus®
The Government of Canada’s terminology and linguistic data bank.
TOKENISER [2 records]
Record 1 - internal organization data 2020-04-30
Record 1, English
Record 1, Subject field(s)
- Computer Programs and Programming
- Computer Processing of Language Data
- Lexicology, Lexicography, Terminology
- Applications of Automation
Record 1, Main entry term, English
- tokenizer
1, record 1, English, tokenizer
correct
Record 1, Abbreviations, English
Record 1, Synonyms, English
Record 1, Textual support, English
Record number: 1, Textual support number: 1 DEF
A software program that … determines boundaries for individual tokens [such as words, numbers or punctuation] in text. 2, record 1, English, - tokenizer
Record 1, Key term(s)
- tokeniser
Record 1, French
Record 1, Domaine(s)
- Programmes et programmation (Informatique)
- Informatisation des données linguistiques
- Lexicologie, lexicographie et terminologie
- Automatisation et applications
Record 1, Main entry term, French
- segmenteur
1, record 1, French, segmenteur
masculine noun
Record 1, Abbreviations, French
Record 1, Synonyms, French
Record 1, Textual support, French
Record 1, Spanish
Record 1, Textual support, Spanish
Record 2 - internal organization data 2003-12-12
Record 2, English
Record 2, Subject field(s)
- Computer Programs and Programming
- Programming Languages
Record 2, Main entry term, English
- tokeniser
1, record 2, English, tokeniser
correct
Record 2, Abbreviations, English
Record 2, Synonyms, English
Record 2, Textual support, English
Record number: 2, Textual support number: 1 DEF
A utility program that translates lines of ... source code ... 1, record 2, English, - tokeniser
Record number: 2, Textual support number: 1 OBS
In general, tokenising a text means merely identifying word forms and sentences. However, in highly technical documents such as the Unix man pages, this may become a formidable task. Apart from regular word forms, the ... tokeniser has to recognise ... Path names and absolute file names: /usr/bin/X11; usr/5bin/ls, /etc/hostname.le ... 1, record 2, English, - tokeniser
Record 2, French
Record 2, Domaine(s)
- Programmes et programmation (Informatique)
- Langages de programmation
Record 2, Main entry term, French
- traducteur de codes sources
1, record 2, French, traducteur%20de%20codes%20sources
masculine noun
Record 2, Abbreviations, French
Record 2, Synonyms, French
Record 2, Textual support, French
Record 2, Spanish
Record 2, Textual support, Spanish
Copyright notice for the TERMIUM Plus® data bank
© Public Services and Procurement Canada, 2024
TERMIUM Plus®, the Government of Canada's terminology and linguistic data bank
A product of the Translation Bureau
Features
Language Portal of Canada
Access a collection of Canadian resources on all aspects of English and French, including quizzes.
Writing tools
The Language Portal’s writing tools have a new look! Easy to consult, they give you access to a wealth of information that will help you write better in English and French.
Glossaries and vocabularies
Access Translation Bureau glossaries and vocabularies.
- Date Modified: