Last change
on this file since 243d027 was
243d027,
checked in by Tomasz Obrebski <to@…>, 11 years ago
|
tok accepts options and prints help message now
|
-
Property mode set to
100644
|
File size:
865 bytes
|
Rev | Line | |
---|
[5f4d9c3] | 1 | package "tok" |
---|
| 2 | version "0.1" |
---|
[243d027] | 3 | usage "tok [OPTIONS]" |
---|
| 4 | purpose "tok transforms raw text into UTT format." |
---|
[5f4d9c3] | 5 | |
---|
[243d027] | 6 | description "OPTIONS" |
---|
| 7 | |
---|
| 8 | option "interactive" i "Interactive mode (no output buffering)." flag off |
---|
| 9 | |
---|
| 10 | text " |
---|
| 11 | DESCRIPTION |
---|
| 12 | |
---|
| 13 | tok reads from standard input, identifies tokens on the basis of their orthographic form and writes a sequence of segments in UTT format to |
---|
| 14 | the standard output. The type of the token is printed as the type field. |
---|
| 15 | |
---|
| 16 | OUTPUT FORMAT |
---|
| 17 | |
---|
| 18 | UTT-file with four fields: start, length, type, and form. In the type field five types of tokens are distinguished: |
---|
| 19 | |
---|
| 20 | W (word) - continuous sequence of letters |
---|
| 21 | N (number) - continuous sequence of digits |
---|
| 22 | S (space) - continuous sequence of space characters |
---|
| 23 | P (punctuation) - single printable character other than W, N, S |
---|
| 24 | B (unprintable character) - single unprintable character |
---|
| 25 | |
---|
| 26 | USAGE EXAMPLE |
---|
| 27 | |
---|
| 28 | tok |
---|
| 29 | " |
---|
Note: See
TracBrowser
for help on using the repository browser.