lucene::analysis::CharTokenizer Class Reference

An abstract base class for simple, character-oriented tokenizers. More...

#include <Analyzers.h>

Inheritance diagram for lucene::analysis::CharTokenizer:

List of all members.

Public Member Functions
	CharTokenizer (lucene::util::Reader *in)
virtual	~CharTokenizer ()
bool	next (Token *token)
	Sets token to the next token in the stream, returns false at the EOS.
Protected Member Functions
virtual bool	isTokenChar (const TCHAR c) const =0
	Returns true iff a character should be included in a token.
virtual TCHAR	normalize (const TCHAR c) const
	Called on each token character to normalize it before it is added to the token.

---

Detailed Description

An abstract base class for simple, character-oriented tokenizers.

---

Constructor & Destructor Documentation

lucene::analysis::CharTokenizer::CharTokenizer

(

lucene::util::Reader *

in

)

virtual lucene::analysis::CharTokenizer::~CharTokenizer

(

)

[inline, virtual]

---

Member Function Documentation

virtual bool lucene::analysis::CharTokenizer::isTokenChar

(

const TCHAR

c

)

const [protected, pure virtual]

Returns true iff a character should be included in a token.

This tokenizer generates as tokens adjacent sequences of characters which satisfy this predicate. Characters for which this is false are used to define token boundaries and are not included in tokens.

Implemented in lucene::analysis::LetterTokenizer, and lucene::analysis::WhitespaceTokenizer.

bool lucene::analysis::CharTokenizer::next

(

Token *

token

)

[virtual]

Sets token to the next token in the stream, returns false at the EOS.

Implements lucene::analysis::TokenStream.

virtual TCHAR lucene::analysis::CharTokenizer::normalize

(

const TCHAR

c

)

const [protected, virtual]

Called on each token character to normalize it before it is added to the token.

The default implementation does nothing. Subclasses may use this to, e.g., lowercase tokens.

Reimplemented in lucene::analysis::LowerCaseTokenizer.

---

The documentation for this class was generated from the following file:

• Analyzers.h