lucene::analysis::TokenStream Class Reference
A TokenStream enumerates the sequence of tokens, either from fields of a document or from query text. More...
#include <AnalysisHeader.h>
Public Member Functions | |
| virtual bool | next (Token *token)=0 |
| Sets token to the next token in the stream, returns false at the EOS. | |
| virtual void | close ()=0 |
| Releases resources associated with this stream. | |
| virtual | ~TokenStream () |
| Token * | next () |
| This is for backwards compatibility only. | |
Detailed Description
A TokenStream enumerates the sequence of tokens, either from fields of a document or from query text.This is an abstract class. Concrete subclasses are:
- Tokenizer, a TokenStream whose input is a Reader; and
- TokenFilter, a TokenStream whose input is another TokenStream.
Constructor & Destructor Documentation
| virtual lucene::analysis::TokenStream::~TokenStream | ( | ) | [inline, virtual] |
Member Function Documentation
| virtual bool lucene::analysis::TokenStream::next | ( | Token * | token | ) | [pure virtual] |
Sets token to the next token in the stream, returns false at the EOS.
Implemented in lucene::analysis::CharTokenizer, lucene::analysis::LowerCaseFilter, lucene::analysis::StopFilter, lucene::analysis::ISOLatin1AccentFilter, lucene::analysis::KeywordTokenizer, lucene::analysis::LengthFilter, lucene::analysis::standard::StandardFilter, and lucene::analysis::standard::StandardTokenizer.
| virtual void lucene::analysis::TokenStream::close | ( | ) | [pure virtual] |
Releases resources associated with this stream.
Implemented in lucene::analysis::Tokenizer, and lucene::analysis::TokenFilter.
| Token* lucene::analysis::TokenStream::next | ( | ) |
This is for backwards compatibility only.
You should pass the token you want to fill to next(), this will save a lot of object construction and destructions.
- Deprecated:
- . use next(token). Kept only to avoid breaking existing code.
The documentation for this class was generated from the following file: