Zum Hauptinhalt springen

Class: SentenceSplitter

SentenceSplitter is our default text splitter that supports splitting into sentences, paragraphs, or fixed length chunks with overlap.

One of the advantages of SentenceSplitter is that even in the fixed length chunks it will try to keep sentences together.

Constructors

new SentenceSplitter()

new SentenceSplitter(options?): SentenceSplitter

Parameters

options?

options.chunkOverlap?: number

options.chunkSize?: number

options.chunkingTokenizerFn?

options.paragraphSeparator?: string | RegExp

options.splitLongSentences?: boolean

options.tokenizer?: any

options.tokenizerDecoder?: any

Returns

SentenceSplitter

Source

packages/core/src/TextSplitter.ts:75

Properties

chunkOverlap

chunkOverlap: number

Source

packages/core/src/TextSplitter.ts:67


chunkSize

chunkSize: number

Source

packages/core/src/TextSplitter.ts:66


chunkingTokenizerFn()

private chunkingTokenizerFn: (text) => string[]

Parameters

text: string

Returns

string[]

Source

packages/core/src/TextSplitter.ts:72


paragraphSeparator

private paragraphSeparator: string | RegExp

Source

packages/core/src/TextSplitter.ts:71


splitLongSentences

private splitLongSentences: boolean

Source

packages/core/src/TextSplitter.ts:73


tokenizer

private tokenizer: any

Source

packages/core/src/TextSplitter.ts:69


tokenizerDecoder

private tokenizerDecoder: any

Source

packages/core/src/TextSplitter.ts:70

Methods

combineTextSplits()

combineTextSplits(newSentenceSplits, effectiveChunkSize): TextSplit[]

Parameters

newSentenceSplits: SplitRep[]

effectiveChunkSize: number

Returns

TextSplit[]

Source

packages/core/src/TextSplitter.ts:216


getEffectiveChunkSize()

private getEffectiveChunkSize(extraInfoStr?): number

Parameters

extraInfoStr?: string

Returns

number

Source

packages/core/src/TextSplitter.ts:111


getParagraphSplits()

getParagraphSplits(text, effectiveChunkSize?): string[]

Parameters

text: string

effectiveChunkSize?: number

Returns

string[]

Source

packages/core/src/TextSplitter.ts:128


getSentenceSplits()

getSentenceSplits(text, effectiveChunkSize?): string[]

Parameters

text: string

effectiveChunkSize?: number

Returns

string[]

Source

packages/core/src/TextSplitter.ts:158


processSentenceSplits()

private processSentenceSplits(sentenceSplits, effectiveChunkSize): SplitRep[]

Splits sentences into chunks if necessary.

This isn't great behavior because it can split down the middle of a word or in non-English split down the middle of a Unicode codepoint so the splitting is turned off by default. If you need it, please set the splitLongSentences option to true.

Parameters

sentenceSplits: string[]

effectiveChunkSize: number

Returns

SplitRep[]

Source

packages/core/src/TextSplitter.ts:187


splitText()

splitText(text, extraInfoStr?): string[]

Parameters

text: string

extraInfoStr?: string

Returns

string[]

Source

packages/core/src/TextSplitter.ts:310


splitTextWithOverlaps()

splitTextWithOverlaps(text, extraInfoStr?): TextSplit[]

Parameters

text: string

extraInfoStr?: string

Returns

TextSplit[]

Source

packages/core/src/TextSplitter.ts:282