Non-terminated HTML Entities are not recognized properly

[Muthukirthan]

Case1

Input: <p>a&nbspc</p>
Brower result: a c
&nbsp is recognized as   html entity

Jsoup parsed content: <p>a&amp;nbspc</p>
Brower result: a&nbspc
&nbsp is not recognized which shows different result in browser

---

Case2

Input: <p>a&nbsp&shyc</p>
Brower result: a ­c
&nbsp and &shy is recognized as   and ­ respective html entity

Jsoup parsed content: <p>a&nbsp;&amp;shyc</p>
Brower result: a &shyc
&nbsp is recognized (might be due to succeeding & character), but &shy is not recognized as ­. Shows different result in browser

---

Case3

Input: <p>a&shyc&nbsp</p>
Brower result: a­c
&nbsp and &shy is recognized as   and ­ respective html entity

Jsoup parsed content: <p>a&amp;shyc&nbsp;</p>
Brower result: a&shyc
&nbsp is recognized (as the string ends with that entity), but &shy is not recognized as ­. Shows different result in browser

---

On checking few more cases, this issue is seen only for named entities (like  , &, ", and others) where the entity is not ended with semi-colon and followed by letters. Hexa-decimal entities and numeric entities are detected even if they are not ended with semi-colon.

Examples:
Proper detection as expected,
&nbsp,ddd (Expected  ,ddd , and got same results)
&nbsp ddd (Expected   ddd , and got same results)
djdjb&nbsp (Expected djdjb  , and got same results)

Invalid detections (ISSUES):
&nbspdhdj (Expected  dhdj but got &nbspdhdj)
&ampdfgsj (Expected &dfgsj but got &ampdfgsj)

Browsers are able to detect these html entities. Validated in https://mothereff.in/html-entities as well

Parser: Html parser
Escape mode: Same result for both base and extended. nbsp entity is replaced by   in xhtml escape mode but the result is same

I also raised this doubt related to entity: #2206

[jhy]

I guess the spec changed here a bit (https://html.spec.whatwg.org/multipage/parsing.html#named-character-reference-state). jsoup actually used to work like that way back when. Will need to review.

[jhy]

Here's the change from when we used to look for the longest matching prefix: a31ec08

[jhy]

Thanks for the clear report! Fixed.