How to split JavaScript strings into sentences, words or graphemes with "Intl.Segmenter" (#tilPost)

I've been reading Axel Rauschmayer's post on the new regular expression flag /v, which explains a way to split emoji strings into graphemes using Intl.Segmenter.

I haven't used this Intl object before. Let's find out what it's about!

Consider you want to split user input into sentences. It looks like a quick split() task... But there's a lot of nuance in this problem.

Here's a naive approach:

'Hello! How are you?'.split(/[.!?]/);
// ['Hello', ' How are you', '']

Using split(), you'll lose the defined separators and include all these spaces everywhere. And because it's relying on hardcoded delimiters it's not language-sensitive.

I don't speak Japanese, but how would you try to split the following string into words or sentences?

// I am a cat. My name is Tanuki.
'吾輩は猫である。名前はたぬき。'

Common string methods won't be helpful here, but the Intl JavaScript API is always good for a surprise!

According to MDN, Intl.Segmenter allows you to split strings into meaningful parts:

Define a locale and granularity (sentence, word or grapheme) and throw any string at it to split strings into segments.

const segmenterDe = new Intl.Segmenter('de', { 
  granularity: 'word'
});
const segmentsDe = segmenterDe.segment('Was geht ab, Freunde?');

Headsup: Firefox doesn't support Intl.Segmenter at the time of writing. On the server-side, it's supported since Node.js 16.

MDN Compat Data (source)

Browser support info for Intl.Segmenter

87	87	87	Nö	Nein	14.1	14.1	14.0	87

Play around with a tl;dr demo below. 🫵

[Interactive component: visit the article to see it...]

But let's look at some Intl.Segmenter details.

Segmenter.segment returns an iterable

You might have noticed the Array.from call in the example above. Segmenter.segment doesn't return an array but an iterable. To access all segments, use array spreading, Array.from or a for-of loop.

const segmenterDe = new Intl.Segmenter('de', {
  granularity: 'sentence'
});
const segmentsDe = segmenterDe.segment('Was geht ab?');

// ----
// Access the segments via array spreading
console.log([...segmentsDe]);
// [
//   { segment: 'Was geht ab?', index: 0, input: 'Was geht ab?' }
// ] 

// ----
// Access the segments via Array.from
console.log(Array.from(segmentsDe));
// [
//   { segment: 'Was geht ab?', index: 0, input: 'Was geht ab?' }
// ] 

// ----
// Access the segments via for...of
for (let segment of segmentsDe) {
  console.log(segment);
}
// { segment: 'Was geht ab?', index: 0, input: 'Was geht ab?' }

Each segment includes the original string value, the character index in the original and the actual segment string.

const segmenterDe = new Intl.Segmenter('de', {
  granularity: 'sentence'
});
const segmentsDe = segmenterDe.segment('Was geht ab?');

console.log(Array.from(segmentsDe, s => s.segment));
// [ 'Was geht ab?' ]

Word granularity comes with an extra isWordLike property

If you split a string into words, all segments include spaces and line breaks. Filter them out using the isWordLike property.

const segmenterDe = new Intl.Segmenter('de', {
  granularity: 'word'
});
const segmentsDe = segmenterDe.segment('Was geht ab?');

console.log([...segmentsDe]);
// [
//   { segment: 'Was', index: 0, input: 'Was geht ab?', isWordLike: true },
//   { segment: ' ', index: 3, input: 'Was geht ab?', isWordLike: false },
//   ...
// ]

console.log([...segmentsDe].filter(s => s.isWordLike));
// [
//   { segment: 'Was', index: 0, input: 'Was geht ab?', isWordLike: true},
//   { segment: 'geht', index: 4, input: 'Was geht ab?', isWordLike: true },
//   { segment: 'ab', index: 9, input: 'Was geht ab?', isWordLike: true }
// ]

Note that filtering by isWordLike removes punctuation such as ., -, or ?.

Use Intl.Segmenter to split emojis

And lastly, here's Axel's example that led me down this rabbit hole. I won't get into Unicode specifics, but if you want to split a string into visual emojis, Intl.Segmenter is a great help, too.

const emojis = '🫣🫵👨‍👨‍👦‍👦';

// ----
// Split by code units
console.log(emojis.split(''));
// ['\uD83E', '\uDEE3', '\uD83E', '\uDEF5', '\uD83D', '\uDE48']

// ----
// Split by code points
console.log([...emojis]);
// ['🫣', '🫵', '👨', '‍', '👨', '‍', '👦', '‍', '👦']

// ----
// Split by graphemes
const segmenter = new Intl.Segmenter('en', {
  granularity: 'grapheme'
});
const segments = segmenter.segment(emojis);

console.log(Array.from(
  segmenter.segment(emojis),
  s => s.segment
));
// ['🫣', '🫵', '👨‍👨‍👦‍👦']

Note that graphemes also include spaces and "normal" characters.

I continue to be amazed by the Intl feature set. There's always new functionality to discover. Intl.Segmenter enables fairly easy string splitting that considers locales and keeps the delimiters. 🎉

It's yet another Intl API to make language-dependent string handling easier! I wonder what I'll discover next!