Java - StreamTokenizer nextToken() method

Description

The Java StreamTokenizer nextToken() method parses the next token from the input stream of this tokenizer. The type of the next token is returned in the ttype field. Additional information about the token may be in the nval field or the sval field of this tokenizer. Typical clients of this class first set up the syntax tables and then sit in a loop calling nextToken to parse successive tokens until TT_EOF is returned.

Declaration

Following is the declaration for java.io.StreamTokenizer.nextToken() method.

public int nextToken()

Parameters

fl − true indicates that all word tokens should be lowercased.

Return Value

This method returns the value of the ttype field.

Exception

NA

Example - Usage of StreamTokenizer nextToken() method

The following example shows the usage of StreamTokenizer nextToken() method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStreamReader;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Reader;
import java.io.StreamTokenizer;

public class StreamTokenizerDemo {
   public static void main(String[] args) {
      String text = "Hello. This is a text \n that will be split "
         + "into tokens. 1 + 1 = 2";
         
      try {
         // create a new file with an ObjectOutputStream
         FileOutputStream out = new FileOutputStream("test.txt");
         ObjectOutputStream oout = new ObjectOutputStream(out);

         // write something in the file
         oout.writeUTF(text);
         oout.flush();

         // create an ObjectInputStream for the file we created before
         ObjectInputStream ois = new ObjectInputStream(new FileInputStream("test.txt"));

         // create a new tokenizer
         Reader r = new BufferedReader(new InputStreamReader(ois));
         StreamTokenizer st = new StreamTokenizer(r);

         // print the stream tokens
         boolean eof = false;
         
         do {
            int token = st.nextToken();
            
            switch (token) {
               case StreamTokenizer.TT_EOF:
                  System.out.println("End of File encountered.");
                  eof = true;
                  break;
                  
               case StreamTokenizer.TT_EOL:
                  System.out.println("End of Line encountered.");
                  break;
                  
               case StreamTokenizer.TT_WORD:
                  System.out.println("Word: " + st.sval);
                  break;
                  
               case StreamTokenizer.TT_NUMBER:
                  System.out.println("Number: " + st.nval);
                  break;
                  
               default:
                  System.out.println((char) token + " encountered.");
                  
                  if (token == '!') {
                     eof = true;
                  }
            }
         } while (!eof);

      } catch (Exception ex) {
         ex.printStackTrace();
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result −

Word: Hello.
Word: This
Word: is
Word: a
Word: text
Word: that
Word: will
Word: be
Word: split
Word: into
Word: tokens.
Number: 1.0
+ encountered.
Number: 1.0
= encountered.
Number: 2.0
End of File encountered.

Example - Basic usage with words and numbers

The following example shows the usage of StreamTokenizer nextToken() method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "Java 101 Python 202";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         switch (tokenizer.ttype) {
            case StreamTokenizer.TT_WORD:
               System.out.println("Word: " + tokenizer.sval);
               break;
            case StreamTokenizer.TT_NUMBER:
               System.out.println("Number: " + tokenizer.nval);
               break;
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: Java
Number: 101.0
Word: Python
Number: 202.0

Explanation

• nextToken() reads one token at a time.

• If it's a word, it's stored in sval; if it's a number, in nval.

Example - Handling symbols and punctuation

The following example shows the usage of StreamTokenizer nextToken() method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "x = 10 + 20";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.ordinaryChar('='); // Treat '=' as a single-char token
      tokenizer.ordinaryChar('+'); // Treat '+' as a single-char token

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         if (tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println("Word: " + tokenizer.sval);
         } else if (tokenizer.ttype == StreamTokenizer.TT_NUMBER) {
            System.out.println("Number: " + tokenizer.nval);
         } else {
            System.out.println("Symbol: " + (char) tokenizer.ttype);
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: x
Symbol: =
Number: 10.0
Symbol: +
Number: 20.0

Explanation

• We use ordinaryChar() to treat = and + as symbols.

• nextToken() handles each token accordingly.