|
Last change
on this file since 243d027 was
243d027,
checked in by Tomasz Obrebski <to@…>, 11 years ago
|
|
tok accepts options and prints help message now
|
-
Property mode set to
100644
|
|
File size:
865 bytes
|
| Line | |
|---|
| 1 | package "tok" |
|---|
| 2 | version "0.1" |
|---|
| 3 | usage "tok [OPTIONS]" |
|---|
| 4 | purpose "tok transforms raw text into UTT format." |
|---|
| 5 | |
|---|
| 6 | description "OPTIONS" |
|---|
| 7 | |
|---|
| 8 | option "interactive" i "Interactive mode (no output buffering)." flag off |
|---|
| 9 | |
|---|
| 10 | text " |
|---|
| 11 | DESCRIPTION |
|---|
| 12 | |
|---|
| 13 | tok reads from standard input, identifies tokens on the basis of their orthographic form and writes a sequence of segments in UTT format to |
|---|
| 14 | the standard output. The type of the token is printed as the type field. |
|---|
| 15 | |
|---|
| 16 | OUTPUT FORMAT |
|---|
| 17 | |
|---|
| 18 | UTT-file with four fields: start, length, type, and form. In the type field five types of tokens are distinguished: |
|---|
| 19 | |
|---|
| 20 | W (word) - continuous sequence of letters |
|---|
| 21 | N (number) - continuous sequence of digits |
|---|
| 22 | S (space) - continuous sequence of space characters |
|---|
| 23 | P (punctuation) - single printable character other than W, N, S |
|---|
| 24 | B (unprintable character) - single unprintable character |
|---|
| 25 | |
|---|
| 26 | USAGE EXAMPLE |
|---|
| 27 | |
|---|
| 28 | tok |
|---|
| 29 | " |
|---|
Note: See
TracBrowser
for help on using the repository browser.
