lucene::analysis::Token Class Reference

A Token is an occurence of a term from the text of a field. More...

#include <AnalysisHeader.h>

Inheritance diagram for lucene::analysis::Token:

Classes
class	OrderCompare
	Compares the Token for their order. More...
Public Member Functions
	Token ()
	~Token ()
	Token (const TCHAR text, const int32_t start, const int32_t end, const TCHAR typ=defaultType)
void	set (const TCHAR text, const int32_t start, const int32_t end, const TCHAR typ=defaultType)
size_t	bufferLength ()
void	growBuffer (size_t size)
void	setPositionIncrement (int32_t posIncr)
int32_t	getPositionIncrement () const
const TCHAR *	termText () const
size_t	termTextLength ()
void	resetTermTextLen ()
void	setText (const TCHAR *txt)
int32_t	startOffset () const
	Returns this Token's starting offset, the position of the first character corresponding to this token in the source text.
void	setStartOffset (int32_t val)
int32_t	endOffset () const
	Returns this Token's ending offset, one greater than the position of the last character corresponding to this token in the source text.
void	setEndOffset (int32_t val)
const TCHAR *	type () const
	returns reference
void	setType (const TCHAR *val)
	returns reference
TCHAR *	toString () const
Public Attributes
TCHAR *	_termText
int32_t	_termTextLen
Static Public Attributes
static const TCHAR *	defaultType

Detailed Description

A Token is an occurence of a term from the text of a field.

It consists of a term's text, the start and end offset of the term in the text of the field, and a type string.

The start and end offsets permit applications to re-associate a token with its source text, e.g., to display highlighted query terms in a document browser, or to show matching text fragments in a KWIC (KeyWord In Context) display, etc.

The type is an interned string, assigned by a lexical analyzer (a.k.a. tokenizer), naming the lexical or syntactic class that the token belongs to. For example an end of sentence marker token might be implemented with type "eos". The default token type is "word".

Constructor & Destructor Documentation

lucene::analysis::Token::Token ( )

lucene::analysis::Token::~Token ( )

lucene::analysis::Token::Token	(	const TCHAR *	text,
		const int32_t	start,
		const int32_t	end,
		const TCHAR *	typ = `defaultType`
	)

Member Function Documentation

size_t lucene::analysis::Token::bufferLength ( ) [inline]

int32_t lucene::analysis::Token::endOffset ( ) const [inline]

Returns this Token's ending offset, one greater than the position of the last character corresponding to this token in the source text.

int32_t lucene::analysis::Token::getPositionIncrement ( ) const

void lucene::analysis::Token::growBuffer ( size_t size )

void lucene::analysis::Token::resetTermTextLen ( )

void lucene::analysis::Token::set	(	const TCHAR *	text,
		const int32_t	start,
		const int32_t	end,
		const TCHAR *	typ = `defaultType`
	)

void lucene::analysis::Token::setEndOffset ( int32_t val ) [inline]

void lucene::analysis::Token::setPositionIncrement ( int32_t posIncr )

void lucene::analysis::Token::setStartOffset ( int32_t val ) [inline]

void lucene::analysis::Token::setText ( const TCHAR * txt )

void lucene::analysis::Token::setType ( const TCHAR * val ) [inline]

returns reference

int32_t lucene::analysis::Token::startOffset ( ) const [inline]

Returns this Token's starting offset, the position of the first character corresponding to this token in the source text.

Note that the difference between endOffset() and startOffset() may not be equal to termText.length(), as the term text may have been altered by a stemmer or some other filter.

const TCHAR* lucene::analysis::Token::termText ( ) const

size_t lucene::analysis::Token::termTextLength ( )

TCHAR* lucene::analysis::Token::toString ( ) const

const TCHAR* lucene::analysis::Token::type ( ) const [inline]

returns reference