Uses of Class
org.apache.lucene.analysis.TokenFilter
Packages that use TokenFilter
Package
Description
Text analysis.
Analyzer for Arabic.
Analyzer for Bulgarian.
Analyzer for Bengali Language.
Provides various convenience classes for creating boosts on Tokens.
Analyzer for Brazilian Portuguese.
Analyzer for Chinese, Japanese, and Korean, which indexes bigrams.
Analyzer for Sorani Kurdish.
Fast, general-purpose grammar-based tokenizers.
Construct n-grams for frequently occurring terms and phrases.
A filter that decomposes compound words you find in many Germanic languages into the word parts.
Basic, general-purpose analysis components.
Analyzer for Czech.
Analyzer for German.
Analyzer for Greek.
Analyzer for English.
Analyzer for Spanish.
Analyzer for Persian.
Analyzer for Finnish.
Analyzer for French.
Analyzer for Irish.
Analyzer for Galician.
Analyzer for Hindi.
Analyzer for Hungarian.
A Java implementation of Hunspell stemming and
spell-checking algorithms (
Hunspell
), and a stemming
TokenFilter (HunspellStemFilter
) based on it.Analysis components based on ICU
Analyzer for Indonesian.
Analyzer for Indian languages.
Analyzer for Italian.
Analyzer for Japanese.
Analyzer for Korean.
Analyzer for Latvian.
MinHash filtering (for LSH).
Miscellaneous Tokenstreams.
Character n-gram tokenizers and filters.
Analyzer for Norwegian.
Set of components for pattern-based (regex) analysis.
Provides various convenience classes for creating payloads on Tokens.
Analysis components for phonetic search.
Analyzer for Portuguese.
Filter to reverse token text.
Analyzer for Russian.
Word n-gram filters.
Analyzer for Serbian.
Stempel: Algorithmic Stemmer
Analyzer for Swedish.
Analysis components for Synonyms.
Analysis components for Synonyms using Word2Vec model.
Analyzer for Telugu Language.
Analyzer for Turkish.
Utility functions for text analysis.
Monitoring framework
Highlighting search terms.
Analyzer based autosuggest.
Support for document suggestion
The UnifiedHighlighter -- a flexible highlighter that can get offsets from postings, term
vectors, or analysis.
-
Uses of TokenFilter in org.apache.lucene.analysis
Subclasses of TokenFilter in org.apache.lucene.analysisModifier and TypeClassDescriptionfinal class
This class can be used if the token attributes of a TokenStream are intended to be consumed more than once.class
Abstract base class for TokenFilters that may remove tokens.class
An abstract TokenFilter that exposes its input stream as a graphclass
Normalizes token text to lower case.class
Removes stop words from a token stream. -
Uses of TokenFilter in org.apache.lucene.analysis.ar
Subclasses of TokenFilter in org.apache.lucene.analysis.arModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesArabicNormalizer
to normalize the orthography.final class
ATokenFilter
that appliesArabicStemmer
to stem Arabic words.. -
Uses of TokenFilter in org.apache.lucene.analysis.bg
Subclasses of TokenFilter in org.apache.lucene.analysis.bgModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesBulgarianStemmer
to stem Bulgarian words. -
Uses of TokenFilter in org.apache.lucene.analysis.bn
Subclasses of TokenFilter in org.apache.lucene.analysis.bnModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesBengaliNormalizer
to normalize the orthography.final class
ATokenFilter
that appliesBengaliStemmer
to stem Bengali words. -
Uses of TokenFilter in org.apache.lucene.analysis.boost
Subclasses of TokenFilter in org.apache.lucene.analysis.boostModifier and TypeClassDescriptionfinal class
Characters before the delimiter are the "token", those after are the boost. -
Uses of TokenFilter in org.apache.lucene.analysis.br
Subclasses of TokenFilter in org.apache.lucene.analysis.br -
Uses of TokenFilter in org.apache.lucene.analysis.cjk
Subclasses of TokenFilter in org.apache.lucene.analysis.cjkModifier and TypeClassDescriptionfinal class
Forms bigrams of CJK terms that are generated from StandardTokenizer or ICUTokenizer.final class
ATokenFilter
that normalizes CJK width differences: Folds fullwidth ASCII variants into the equivalent basic latin Folds halfwidth Katakana variants into the equivalent kana -
Uses of TokenFilter in org.apache.lucene.analysis.ckb
Subclasses of TokenFilter in org.apache.lucene.analysis.ckbModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesSoraniNormalizer
to normalize the orthography.final class
ATokenFilter
that appliesSoraniStemmer
to stem Sorani words. -
Uses of TokenFilter in org.apache.lucene.analysis.classic
Subclasses of TokenFilter in org.apache.lucene.analysis.classicMethods in org.apache.lucene.analysis.classic that return TokenFilter -
Uses of TokenFilter in org.apache.lucene.analysis.commongrams
Subclasses of TokenFilter in org.apache.lucene.analysis.commongramsModifier and TypeClassDescriptionfinal class
Construct bigrams for frequently occurring terms while indexing.final class
Wrap a CommonGramsFilter optimizing phrase queries by only returning single words when they are not a member of a bigram.Methods in org.apache.lucene.analysis.commongrams that return TokenFilterModifier and TypeMethodDescriptionCommonGramsFilterFactory.create
(TokenStream input) CommonGramsQueryFilterFactory.create
(TokenStream input) Create a CommonGramsFilter and wrap it with a CommonGramsQueryFilter -
Uses of TokenFilter in org.apache.lucene.analysis.compound
Subclasses of TokenFilter in org.apache.lucene.analysis.compoundModifier and TypeClassDescriptionclass
Base class for decomposition token filters.class
ATokenFilter
that decomposes compound words found in many Germanic languages.class
ATokenFilter
that decomposes compound words found in many Germanic languages.Methods in org.apache.lucene.analysis.compound that return TokenFilterModifier and TypeMethodDescriptionHyphenationCompoundWordTokenFilterFactory.create
(TokenStream input) -
Uses of TokenFilter in org.apache.lucene.analysis.core
Subclasses of TokenFilter in org.apache.lucene.analysis.coreModifier and TypeClassDescriptionfinal class
Folds all Unicode digits in[:General_Category=Decimal_Number:]
to Basic Latin digits (0-9
).final class
Converts an incoming graph token stream, such as one fromSynonymGraphFilter
, into a flat form so that all nodes form a single linear chain with no side paths.final class
Normalizes token text to lower case.final class
Removes stop words from a token stream.final class
Removes tokens whose types appear in a set of blocked types from a token stream.final class
Normalizes token text to UPPER CASE. -
Uses of TokenFilter in org.apache.lucene.analysis.cz
Subclasses of TokenFilter in org.apache.lucene.analysis.czModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesCzechStemmer
to stem Czech words. -
Uses of TokenFilter in org.apache.lucene.analysis.de
Subclasses of TokenFilter in org.apache.lucene.analysis.deModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesGermanLightStemmer
to stem German words.final class
ATokenFilter
that appliesGermanMinimalStemmer
to stem German words.final class
Normalizes German characters according to the heuristics of the German2 snowball algorithm.final class
ATokenFilter
that stems German words. -
Uses of TokenFilter in org.apache.lucene.analysis.el
Subclasses of TokenFilter in org.apache.lucene.analysis.elModifier and TypeClassDescriptionfinal class
Normalizes token text to lower case, removes some Greek diacritics, and standardizes final sigma to sigma.final class
ATokenFilter
that appliesGreekStemmer
to stem Greek words. -
Uses of TokenFilter in org.apache.lucene.analysis.en
Subclasses of TokenFilter in org.apache.lucene.analysis.enModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesEnglishMinimalStemmer
to stem English words.final class
TokenFilter that removes possessives (trailing 's) from words.final class
A high-performance kstem filter for english.final class
Transforms the token stream as per the Porter stemming algorithm.Methods in org.apache.lucene.analysis.en that return TokenFilter -
Uses of TokenFilter in org.apache.lucene.analysis.es
Subclasses of TokenFilter in org.apache.lucene.analysis.esModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesSpanishLightStemmer
to stem Spanish words.final class
Deprecated.final class
ATokenFilter
that appliesSpanishPluralStemmer
to stem Spanish words. -
Uses of TokenFilter in org.apache.lucene.analysis.fa
Subclasses of TokenFilter in org.apache.lucene.analysis.faModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesPersianNormalizer
to normalize the orthography.final class
ATokenFilter
that appliesPersianStemmer
to stem Persian words. -
Uses of TokenFilter in org.apache.lucene.analysis.fi
Subclasses of TokenFilter in org.apache.lucene.analysis.fiModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesFinnishLightStemmer
to stem Finnish words. -
Uses of TokenFilter in org.apache.lucene.analysis.fr
Subclasses of TokenFilter in org.apache.lucene.analysis.frModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesFrenchLightStemmer
to stem French words.final class
ATokenFilter
that appliesFrenchMinimalStemmer
to stem French words. -
Uses of TokenFilter in org.apache.lucene.analysis.ga
Subclasses of TokenFilter in org.apache.lucene.analysis.gaModifier and TypeClassDescriptionfinal class
Normalises token text to lower case, handling t-prothesis and n-eclipsis (i.e., that 'nAthair' should become 'n-athair') -
Uses of TokenFilter in org.apache.lucene.analysis.gl
Subclasses of TokenFilter in org.apache.lucene.analysis.glModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesGalicianMinimalStemmer
to stem Galician words.final class
ATokenFilter
that appliesGalicianStemmer
to stem Galician words. -
Uses of TokenFilter in org.apache.lucene.analysis.hi
Subclasses of TokenFilter in org.apache.lucene.analysis.hiModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesHindiNormalizer
to normalize the orthography.final class
ATokenFilter
that appliesHindiStemmer
to stem Hindi words. -
Uses of TokenFilter in org.apache.lucene.analysis.hu
Subclasses of TokenFilter in org.apache.lucene.analysis.huModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesHungarianLightStemmer
to stem Hungarian words. -
Uses of TokenFilter in org.apache.lucene.analysis.hunspell
Subclasses of TokenFilter in org.apache.lucene.analysis.hunspellModifier and TypeClassDescriptionfinal class
TokenFilter that uses hunspell affix rules and words to stem tokens. -
Uses of TokenFilter in org.apache.lucene.analysis.icu
Subclasses of TokenFilter in org.apache.lucene.analysis.icuModifier and TypeClassDescriptionfinal class
A TokenFilter that applies search term folding to Unicode text, applying foldings from UTR#30 Character Foldings.class
Normalize token text with ICU'sNormalizer2
final class
ATokenFilter
that transforms text with ICU. -
Uses of TokenFilter in org.apache.lucene.analysis.id
Subclasses of TokenFilter in org.apache.lucene.analysis.idModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesIndonesianStemmer
to stem Indonesian words. -
Uses of TokenFilter in org.apache.lucene.analysis.in
Subclasses of TokenFilter in org.apache.lucene.analysis.inModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesIndicNormalizer
to normalize text in Indian Languages. -
Uses of TokenFilter in org.apache.lucene.analysis.it
Subclasses of TokenFilter in org.apache.lucene.analysis.itModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesItalianLightStemmer
to stem Italian words. -
Uses of TokenFilter in org.apache.lucene.analysis.ja
Subclasses of TokenFilter in org.apache.lucene.analysis.jaModifier and TypeClassDescriptionfinal class
Replaces term text with theBaseFormAttribute
.final class
ATokenFilter
that adds Japanese romanized tokens to the term attribute.final class
ATokenFilter
that normalizes common katakana spelling variations ending in a long sound character by removing this character (U+30FC).class
ATokenFilter
that normalizes Japanese numbers (kansūji) to regular Arabic decimal numbers in half-width characters.final class
Removes tokens that match a set of part-of-speech tags.final class
ATokenFilter
that replaces the term attribute with the reading of a token in either katakana or romaji form. -
Uses of TokenFilter in org.apache.lucene.analysis.ko
Subclasses of TokenFilter in org.apache.lucene.analysis.koModifier and TypeClassDescriptionclass
ATokenFilter
that normalizes Korean numbers to regular Arabic decimal numbers in half-width characters.final class
Removes tokens that match a set of part-of-speech tags.final class
Replaces term text with theReadingAttribute
which is the Hangul transcription of Hanja characters. -
Uses of TokenFilter in org.apache.lucene.analysis.lv
Subclasses of TokenFilter in org.apache.lucene.analysis.lvModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesLatvianStemmer
to stem Latvian words. -
Uses of TokenFilter in org.apache.lucene.analysis.minhash
Subclasses of TokenFilter in org.apache.lucene.analysis.minhashModifier and TypeClassDescriptionclass
Generate min hash tokens from an incoming stream of tokens. -
Uses of TokenFilter in org.apache.lucene.analysis.miscellaneous
Subclasses of TokenFilter in org.apache.lucene.analysis.miscellaneousModifier and TypeClassDescriptionfinal class
This class converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into their ASCII equivalents, if one exists.final class
A filter to apply normal capitalization rules to Tokens.final class
Removes words that are too long or too short from the stream.class
Allows skipping TokenFilters based on the current set of attributes.class
Filters all tokens that cannot be parsed to a date, using the providedDateFormat
.final class
Characters before the delimiter are the "token", the textual integer after is the term frequency.final class
Allows Tokens with a given combination of flags to be dropped.class
Filter outputs a single token which is a concatenation of the sorted and de-duplicated set of input tokens.final class
Deprecated.Fix the token filters that create broken offsets in the first place.final class
When the plain text is extracted from documents, we will often have many words hyphenated and broken into two lines.final class
A TokenFilter that only keeps tokens with text contained in the required words.class
Marks terms as keywords via theKeywordAttribute
.final class
This TokenFilter emits each incoming token twice once as keyword and once non-keyword, in other words once withKeywordAttribute.setKeyword(boolean)
set totrue
and once set tofalse
.final class
Removes words that are too long or too short from the stream.final class
This TokenFilter limits the number of tokens while indexing.final class
Lets all tokens pass through until it sees one with a start offset <= a configured limit, which won't pass and ends the stream.final class
This TokenFilter limits its emitted tokens to those with positions that are not greater than the configured limit.final class
Marks terms as keywords via theKeywordAttribute
.class
A ConditionalTokenFilter that only applies its wrapped filters to tokens that are not contained in a protected set.final class
A TokenFilter which filters out Tokens at the same position and Term text as the previous token in the stream.final class
This filter folds Scandinavian characters åÅäæÄÆ->a and öÖøØ->o.final class
This filter normalize use of the interchangeable Scandinavian characters æÆäÄöÖøØ and folded variants (aa, ao, ae, oe and oo) by transforming them to åÅæÆøØ.final class
Marks terms as keywords via theKeywordAttribute
.final class
Provides the ability to override anyKeywordAttribute
aware stemmer with custom dictionary-based stemming.final class
Trims leading and trailing whitespace from Tokens in the stream.final class
A token filter for truncating the terms into a specific length.final class
Adds theTypeAttribute.type()
as a synonym, i.e.final class
Deprecated.UseWordDelimiterGraphFilter
instead: it produces a correct token graph so that e.g.final class
Splits words into subwords and performs optional transformations on subword groups, producing a correct token graph so that e.g.Methods in org.apache.lucene.analysis.miscellaneous that return TokenFilterModifier and TypeMethodDescriptionWordDelimiterFilterFactory.create
(TokenStream input) Deprecated.WordDelimiterGraphFilterFactory.create
(TokenStream input) -
Uses of TokenFilter in org.apache.lucene.analysis.ngram
Subclasses of TokenFilter in org.apache.lucene.analysis.ngramModifier and TypeClassDescriptionfinal class
Tokenizes the given token into n-grams of given size(s).final class
Tokenizes the input into n-grams of the given size(s).Methods in org.apache.lucene.analysis.ngram that return TokenFilterModifier and TypeMethodDescriptionEdgeNGramFilterFactory.create
(TokenStream input) NGramFilterFactory.create
(TokenStream input) -
Uses of TokenFilter in org.apache.lucene.analysis.no
Subclasses of TokenFilter in org.apache.lucene.analysis.noModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesNorwegianLightStemmer
to stem Norwegian words.final class
ATokenFilter
that appliesNorwegianMinimalStemmer
to stem Norwegian words.final class
This filter normalize use of the interchangeable Scandinavian characters æÆäÄöÖøØ and folded variants (ae, oe, aa) by transforming them to åÅæÆøØ. -
Uses of TokenFilter in org.apache.lucene.analysis.pattern
Subclasses of TokenFilter in org.apache.lucene.analysis.patternModifier and TypeClassDescriptionfinal class
CaptureGroup uses Java regexes to emit multiple tokens - one for each capture group in one or more patterns.final class
A TokenFilter which applies a Pattern to each token in the stream, replacing match occurrences with the specified replacement string.class
Set a type attribute to a parameterized value when tokens are matched by any of a several regex patterns. -
Uses of TokenFilter in org.apache.lucene.analysis.payloads
Subclasses of TokenFilter in org.apache.lucene.analysis.payloadsModifier and TypeClassDescriptionfinal class
Characters before the delimiter are the "token", those after are the payload.class
Assigns a payload to a token based on theTypeAttribute
class
Adds theOffsetAttribute.startOffset()
andOffsetAttribute.endOffset()
First 4 bytes are the startclass
Makes theTypeAttribute
a payload. -
Uses of TokenFilter in org.apache.lucene.analysis.phonetic
Subclasses of TokenFilter in org.apache.lucene.analysis.phoneticModifier and TypeClassDescriptionfinal class
TokenFilter for Beider-Morse phonetic encoding.final class
Create tokens for phonetic matches based on Daitch–Mokotoff Soundex.final class
Filter for DoubleMetaphone (supporting secondary codes)final class
Create tokens for phonetic matches. -
Uses of TokenFilter in org.apache.lucene.analysis.pt
Subclasses of TokenFilter in org.apache.lucene.analysis.ptModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesPortugueseLightStemmer
to stem Portuguese words.final class
ATokenFilter
that appliesPortugueseMinimalStemmer
to stem Portuguese words.final class
ATokenFilter
that appliesPortugueseStemmer
to stem Portuguese words. -
Uses of TokenFilter in org.apache.lucene.analysis.reverse
Subclasses of TokenFilter in org.apache.lucene.analysis.reverseModifier and TypeClassDescriptionfinal class
Reverse token string, for example "country" => "yrtnuoc". -
Uses of TokenFilter in org.apache.lucene.analysis.ru
Subclasses of TokenFilter in org.apache.lucene.analysis.ruModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesRussianLightStemmer
to stem Russian words. -
Uses of TokenFilter in org.apache.lucene.analysis.shingle
Subclasses of TokenFilter in org.apache.lucene.analysis.shingleModifier and TypeClassDescriptionfinal class
A FixedShingleFilter constructs shingles (token n-grams) from a token stream.final class
A ShingleFilter constructs shingles (token n-grams) from a token stream. -
Uses of TokenFilter in org.apache.lucene.analysis.sinks
Subclasses of TokenFilter in org.apache.lucene.analysis.sinksModifier and TypeClassDescriptionfinal class
This TokenFilter provides the ability to set aside attribute states that have already been analyzed. -
Uses of TokenFilter in org.apache.lucene.analysis.snowball
Subclasses of TokenFilter in org.apache.lucene.analysis.snowballModifier and TypeClassDescriptionfinal class
A filter that stems words using a Snowball-generated stemmer.Methods in org.apache.lucene.analysis.snowball that return TokenFilter -
Uses of TokenFilter in org.apache.lucene.analysis.sr
Subclasses of TokenFilter in org.apache.lucene.analysis.srModifier and TypeClassDescriptionfinal class
Normalizes Serbian Cyrillic and Latin characters to "bald" Latin.final class
Normalizes Serbian Cyrillic to Latin. -
Uses of TokenFilter in org.apache.lucene.analysis.stempel
Subclasses of TokenFilter in org.apache.lucene.analysis.stempelModifier and TypeClassDescriptionfinal class
Transforms the token stream as per the stemming algorithm. -
Uses of TokenFilter in org.apache.lucene.analysis.sv
Subclasses of TokenFilter in org.apache.lucene.analysis.svModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesSwedishLightStemmer
to stem Swedish words.final class
ATokenFilter
that appliesSwedishMinimalStemmer
to stem Swedish words. -
Uses of TokenFilter in org.apache.lucene.analysis.synonym
Subclasses of TokenFilter in org.apache.lucene.analysis.synonymModifier and TypeClassDescriptionfinal class
Deprecated.UseSynonymGraphFilter
instead, but be sure to also useFlattenGraphFilter
at index time (not at search time) as well.final class
Applies single- or multi-token synonyms from aSynonymMap
to an incomingTokenStream
, producing a fully correct graph output. -
Uses of TokenFilter in org.apache.lucene.analysis.synonym.word2vec
Subclasses of TokenFilter in org.apache.lucene.analysis.synonym.word2vecModifier and TypeClassDescriptionfinal class
Applies single-token synonyms from a Word2Vec trained network to an incomingTokenStream
. -
Uses of TokenFilter in org.apache.lucene.analysis.te
Subclasses of TokenFilter in org.apache.lucene.analysis.teModifier and TypeClassDescriptionfinal class
ATokenFilter
that appliesTeluguNormalizer
to normalize the orthography.final class
ATokenFilter
that appliesTeluguStemmer
to stem Telugu words. -
Uses of TokenFilter in org.apache.lucene.analysis.tr
Subclasses of TokenFilter in org.apache.lucene.analysis.trModifier and TypeClassDescriptionfinal class
Strips all characters after an apostrophe (including the apostrophe itself).final class
Normalizes Turkish token text to lower case. -
Uses of TokenFilter in org.apache.lucene.analysis.util
Subclasses of TokenFilter in org.apache.lucene.analysis.util -
Uses of TokenFilter in org.apache.lucene.monitor
Subclasses of TokenFilter in org.apache.lucene.monitor -
Uses of TokenFilter in org.apache.lucene.search.highlight
Subclasses of TokenFilter in org.apache.lucene.search.highlightModifier and TypeClassDescription(package private) final class
This is a simplified version of org.apache.lucene.analysis.miscellaneous.LimitTokenOffsetFilter to prevent a dependency on analysis-common.jar.final class
This TokenFilter limits the number of tokens while indexing by adding up the current offset. -
Uses of TokenFilter in org.apache.lucene.search.suggest.analyzing
Subclasses of TokenFilter in org.apache.lucene.search.suggest.analyzingModifier and TypeClassDescriptionfinal class
LikeStopFilter
except it will not remove the last token if that token was not followed by some token separator. -
Uses of TokenFilter in org.apache.lucene.search.suggest.document
Subclasses of TokenFilter in org.apache.lucene.search.suggest.documentModifier and TypeClassDescriptionfinal class
AConcatenateGraphFilter
but we can set the payload and provide access to config options.private static final class
TheContextSuggestField.PrefixTokenFilter
wraps aTokenStream
and adds a set prefixes ahead. -
Uses of TokenFilter in org.apache.lucene.search.uhighlight
Subclasses of TokenFilter in org.apache.lucene.search.uhighlightModifier and TypeClassDescriptionprivate static final class
Wraps anAnalyzer
and string text that represents multiple values delimited by a specified character.
SpanishPluralStemFilter
instead.