word.ts extendedWordChars seems to not capture everything in the comment above it

[hchiam]

in

https://github.com/kpdecker/jsdiff/blob/master/src/diff/word.ts#L5-L23

it currently is

const extendedWordChars = 'a-zA-Z0-9_\\u{C0}-\\u{FF}\\u{D8}-\\u{F6}\\u{F8}-\\u{2C6}\\u{2C8}-\\u{2D7}\\u{2DE}-\\u{2FF}\\u{1E00}-\\u{1EFF}';

unless i'm misunderstanding something (e.g. exceptions mentioned in the comment, or how the extendedWordChars variable is actually interpreted by JS syntax) it seems to be missing 0080-00BF when i break down the unicode part of the string and compare it to the comment:

\u{C0}-\u{FF}
\u{D8}-\u{F6}
\u{F8}-\u{2C6}
\u{2C8}-\u{2D7}
\u{2DE}-\u{2FF}
\u{1E00}-\u{1EFF}

extendedWordChars	comment
SEEMS TO BE MISSING 0080-00BF?	// Latin-1 Supplement, 0080–00FF
\u{C0}-\u{FF}	// Latin-1 Supplement, 0080–00FF
SEEMS TO NOT BE EXCLUDED PROPERLY UNLESS THIS WAS INTENTIONAL?	// - U+00D7 × Multiplication sign
\u{D8}-\u{F6}	seems redundant since covered by other range \u{C0}-\u{FF}
SEEMS TO NOT BE EXCLUDED PROPERLY UNLESS THIS WAS INTENTIONAL?	// - U+00F7 ÷ Division sign
\u{F8}-\u{2C6}	covered by other ranges 0080–00FF and 0100–017F and 0180–024F and 0250–02AF and 02B0–02FF
included in other ranges	// Latin Extended-A, 0100–017F
included in other ranges	// Latin Extended-B, 0180–024F
included in other ranges	// IPA Extensions, 0250–02AF
\u{2C8}-\u{2D7}	// Spacing Modifier Letters, 02B0–02FF covered by other ranges and exclusions combined
intentionally excluded	// - U+02C7 ˇ ˇ Caron
intentionally excluded	// - U+02D8 ˘ ˘ Breve
intentionally excluded	// - U+02D9 ˙ ˙ Dot Above
intentionally excluded	// - U+02DA ˚ ˚ Ring Above
intentionally excluded	// - U+02DB ˛ ˛ Ogonek
intentionally excluded	// - U+02DC ˜ ˜ Small Tilde
intentionally excluded	// - U+02DD ˝ ˝ Double Acute Accent
\u{2DE}-\u{2FF}	// Spacing Modifier Letters, 02B0–02FF
\u{1E00}-\u{1EFF}	// Latin Extended Additional, 1E00–1EFF

[hchiam]

sorry, i made a few edits to the table in my first comment

[ExplodingCabbage]

Hmm...

So, if I understand right, you've found what you reckon are two differences between the comment and the actual regex:

1. The actual regex treats × and ÷ as word characters (just like letters), but the comment indicates that they should be exceptions that are not treated as word characters
2. The comment indicates that the entire range 0080–00FF (besides the two exceptions from point 1) should be treated as word characters, but the actual regex only treats a smaller sub-range, namely 00C0-00FF, as word characters. That is, the comment says that 0080-00BF are word characters, and the regex says they aren't.

(That is, I understand these to be the only two differences you've found - it looks like the rest of your table is basically saying that besides these two points the rest of the regex is consistent with the comment? If I'm misunderstanding that, please correct me!)

The regex and comment both pre-date my time maintaining jsdiff, so let me dig in and have a think...

[... some time later...]

Okay, I agree that point 2 is definitely an inconsistency between the regex and the comment. The actual implementation in the regex seems more reasonable to me than the comment, although probably a better implementation would not quite match either. A quick glance at what the characters in the range 0080-00BF are (e.g. at https://en.wikipedia.org/wiki/Latin-1_Supplement#Character_table) shows that most of them should clearly not be considered word characters. The range includes various things like:

• obscure control characters (U+0093 "Set Transmit State", U+0095 "Message Waiting", U+0097 "End of Protected Area")
• whitespace (U+00A0 "Non-breaking space")
• punctuation (¡, ¿)
• mathematical symbols that seem like they definitely shouldn't be construed as part of a word (·, ± ¬)

One character strikes me as a clear exception:

• Control character U+00AD, the "soft hyphen", which marks a preferred place to insert a hyphen into a word if the software laying out the text decides to break the word across two lines. Since the whole point of a soft hyphen is that it appears within a single word, it would seem that we ought to treat it as a word character for the purpose of splitting text into words.

Then there are a whole bunch of murky, ambiguous cases where I'm not really sure what's appropriate:

• currency symbols ("¢", "£"). Is "£5" one word, or a punctuation mark followed by a word?
• the section symbol, "§", generally pronounced "section" when reading text aloud. The main place I've seen this used is in US law, where "section 123" of a statute is conventionally written "§ 123". Here it seems to me that § is acting more like a word than a punctuation mark... although it does appear that Unicode puts it in category "Po", classifying it as punctuation
• the paragraph symbol, "¶", used similarly
• the "copyright" and "registered" symbols, © and ®. Is "Google®" in its entirety a word, or is it the word "Google" followed by a punctuation mark?
• fractions ("¼", "½", "¾"). These feel to me like they're probably word characters? Like, I would say the sentence "My daughter is 1½ years old." has 6 words, "My", "daughter", "is", "1½", "years", and "old". But perhaps others would disagree, I dunno.
• weird standalone diacritics, like U+00B4, the standalone acute accent. (To be clear, this is NOT the modifier character that adds an acute accent onto the preceding letter. Instead it's a character that just produces a standalone, free-floating acute accent, like t´h´i´s. Other diacritic characters in this range are similar.) It's not obvious to me how these would ever be used except in a work writing about the accent itself (e.g. a guide to a language that uses the particular diacritic). I guess it's appropriate to treat them as punctuation.

I also think you are right that point 1 is an inconsistency. I agree with your interpretation that the comment is saying "×" and "÷" are exceptions from the rule that characters in the range 0080-00FF (or 00C0-00FF) should be treated as word characters - i.e. it is saying they alone should not be treated as word characters - yet, as you note, the regex contains no such exception and simply treats those two characters as word characters. (The comment isn't very explicit about this, but I can't see what else the mention of those two characters in the comment could possibly have been intended to mean.) In this case, the rule documented in the comment seems more reasonable than the one in the actual implementation; "×" and "÷" surely should be treated as punctuation.

I'll therefore make the following changes:

• update the comment to say "00C0-00FF" instead of "0080-00FF"
• update both the comment and the regex to say that soft hyphens are word characters
• update the regex to treat "×" and "÷" as non-word characters, per the existing comment