Locale-Sensitive Java Method

`java.io.StreamTokenizer`

public StreamTokenizer(InputStream is)

public StreamTokenizer(Reader r)

Internationalization (I18n) Method Overview

The StreamTokenizer class takes an input stream and parses it into "tokens," allowing the tokens to be read one at a time. This class should not be used to parse natural language text for the purpose of detecting character tokens or boundaries between words, sentences or lines.

I18n Issues

Such boundaries vary according to the rules of the language in which the text was written. Java provides locale-sensitive functionality to perform this type of parsing within the BreakIterator class.

Globalyzer will be unable to detect whether an instance of StreamTokenizer is being used to parse natural language or not. If you determine that an instance of StreamTokenizer does not pose I18n problems, you can use Globalyzer's Ignore Comment functionality to ensure that it isn't picked up in a subsequent scan.

Suggested Replacement

public static BreakIterator.getLineInstance(Locale locale) public static BreakIterator.getSentenceInstance(Locale locale) public static BreakIterator.getWordInstance(Locale locale) public static BreakIterator.getCharacterInstance(Locale locale)

Instead of:

InputStream in; //instantiate input stream StreamTokenizer tokenizer = new StreamTokenizer(in);

Use:

//retrieve the user's locale Locale locale = getUserLocale(); //pass the user's locale as an argument BreakIterator boundary = BreakIterator.getWordInstance(locale);

Please see the Java Internationalization Trail for more information about the BreakIterator class.

Locale-Sensitive Java Methods