Introduction
The function of this class is reading the characters into token from reader class or it reads the content as a tokens from the input stream. The token reads words, numbers. The process of breaking the sentence or parsing files into tokens is called tokenizing or lexing. It recognize all identifiers, words, numbers, strings.
Description
There are fields to represent that which types of tokens are reading especially they are,
nval(It represents the number value of the token)
sval(It represents the string value of the token)
ttype(It represents the token reads type like numbers,words )
After completion of reading tokens then close Java.io StreamTokenizer which also close the reader instance. It can close by calling close() method or using try with resources.
Declaration
Java.io StreamTokenizer class is declared below:
Public class StreamTokenizer extends Object Fields
Java.io StreamTokenizer class fields are:
- String sval : The field represents that the current token is a word, it contains the value of the word.
- double nval : The field represents the current token is a number, it contains the value of that number.
- int ttype : The field represents the type of token reads when calling next token method.
- static int TT_NUMBER : This field represents that number token to read.
- static int TT_WORD : This field represents that word token to read.
- static int TT_EOF : This field represents that end of stream token to read.
- static int TT_EOL : This field represents that end of line the token to read.
Table
| Constructor | Description |
|---|---|
| StreamTokenizer (Reader r) | The function of this constructor is to create Java.io StreamTokenizer to parse the stream of characters. |
Table
| Methods | Description |
|---|---|
| int lineno() | The function of this method is to return the current line number. |
| int nextToken() | The function of this method is to parse the next token from input stream. |
| void parseNumbers() | The function of this method is to represent the numbers parsed by token. |
| void wordChars(int low, int hi) | The function of this method is to represent the range of characters that are word constituents. |
| void slashslashComments(boolean flag) | The function of this method is to check whether the comments of CPP are recognized by tokenizer. |
| void resetSyntax() | The function of this method is to reset the syntax of tokenizer, then all the characters in ordinary position. |
| void CommentChar(int ch) | The function of this method is to represent an single line comment from given argument. |
| void ordinaryChar(int ch) | The function of this method is to represent that the characters in tokenizer are ordinary. |
Example
Following is an example.
[c]package java2;
import java.io.FileReader;
import java.io.IOException;
import java.io.StreamTokenizer;
public class StreamTokenizerDemoTwo {
public static void main(String args[]) throws IOException {
//Using ordinaryChars method
FileReader fileReader = new FileReader("E:/splesson.txt");
StreamTokenizer st = new StreamTokenizer(fileReader);
System.out.println("After Using ordinaryChar() method \n");
st.ordinaryChar('A');
printStreamTokenizer(st);
fileReader.close();
//Using resetSyntax method
fileReader = new FileReader("E:/splesson.txt");
st = new StreamTokenizer(fileReader);
System.out.println("\nAfter Using resetSyntax() method \n");
st.resetSyntax();
printStreamTokenizer(st);
fileReader.close();
}
static void printStreamTokenizer(StreamTokenizer st) throws IOException{
int token =0;
while((token = st.nextToken()) != StreamTokenizer.TT_EOF) {
if(st.ttype == StreamTokenizer.TT_NUMBER) {
System.out.println("Number: "+st.nval);
} else if(st.ttype == StreamTokenizer.TT_WORD) {
System.out.println("Word: "+st.sval);
}else {
System.out.println("Ordinary Char: "+(char)token);
}
}
}
} [/c]
ordinaryChar() specifies that tokenizer has "ordinary" character arguments. resetSyntax () resets the tokenizer syntax table.
Output
The result will be as follows.
[c]
After Using ordinaryChar() method
Word: Hello
Word: Java.Java
Word: IO
Word: is
Word: fun.
After Using resetSyntax() method
Ordinary Char: H
Ordinary Char: e
Ordinary Char: l
Ordinary Char: l
Ordinary Char: o
Ordinary Char:
Ordinary Char: J
Ordinary Char: a
Ordinary Char: v
Ordinary Char: a
Ordinary Char: .
Ordinary Char: J
Ordinary Char: a
Ordinary Char: v
Ordinary Char: a
Ordinary Char:
Ordinary Char: I
Ordinary Char: O
Ordinary Char:
Ordinary Char: i
Ordinary Char: s
Ordinary Char:
Ordinary Char: f
Ordinary Char: u
Ordinary Char: n
Ordinary Char: .
[/c]
Key Points
- The function of this class is reading the characters into token from reader class or it reads the content as a tokens from the input stream.
- The token reads words, numbers.
- The process of breaking the sentence or parsing files into tokens is called tokenizing or lexing.
- It recognize all identifiers, words, numbers, strings.