mb_check_encoding doesn't respect UTF-8 RFC3629

[LLyaudet]

Description

Hello,

RFC3629 says:

   The definition of UTF-8 prohibits encoding character numbers between
   U+D800 and U+DFFF, which are reserved for use with the UTF-16
   encoding form (as surrogate pairs) and do not directly represent
   characters.

But both of mb_check_encoding and preg_match('//u', $input) doesn't detect when such characters occurs.
I found this whilst validating my code here:
Seldaek/jsonlint#91

I had to comment out the fast path relying on mb_check_encoding and preg_match.

Maybe there are related security issues.

Have a nice day, best regards,
Laurent Lyaudet

PHP Version

PHP 8.5.4 (cli) (built: May 25 2026 12:19:37) (NTS)
Copyright (c) The PHP Group
Built by Ubuntu
Zend Engine v4.5.4, Copyright (c) Zend Technologies
    with Zend OPcache v8.5.4, Copyright (c), by Zend Technologies

Operating System

Ubuntu 26.04

[youkidearitai]

The definition of UTF-8 prohibits encoding character numbers between
U+D800 and U+DFFF, which are reserved for use with the UTF-16
encoding form (as surrogate pairs) and do not directly represent
characters.

In this, I think correct behavior of mb_check_encoding and preg_match('//u').
UTF-8 is not allow surrogate pair.

I tried 3v4l. Is there anything wrong?
https://3v4l.org/BlEIF#veol

@alexdowad Do you have any concern? Perhaps I lost anyone.

[LLyaudet]

The definition of UTF-8 prohibits encoding character numbers between
U+D800 and U+DFFF, which are reserved for use with the UTF-16
encoding form (as surrogate pairs) and do not directly represent
characters.

In this, I think correct behavior of mb_check_encoding and preg_match('//u'). UTF-8 is not allow surrogate pair.

You inverted what I said.
Indeed "UTF-8 doesn't allow surrogate pair." as we both said.
But "correct behavior of mb_check_encoding and preg_match('//u')" is not the case,
because both allow surrogate pair.
Please don't fool us.

[youkidearitai]

Please don't fool us.

I'm sorry if you angry.
However, your case is D8 81 in UTF-8, so valid character in Arabic.

~/php86/bin/php -r 'var_dump(chr(216).chr(129), hex2bin("d881"), "\u{00d8}\u{0081}", mb_check_encodi
ng("\u{00d8}\u{0081}", "UTF-8"));'
string(2) "؁"
string(2) "؁"
string(4) "Ø"
bool(true)

I don't think problem. And, unrelated surrogate pair.

[LamentXU123]

@LLyaudet Hi. Could you provide a proof-of-concept script to help reproducing the bug? As far as I can tell. both

<?php
var_dump(mb_check_encoding("\u{d800}\u{dc00}", "UTF-8"));
var_dump(preg_match("//u", "\u{d800}\u{dc00}"));

returns false. You can reproduce this result in: https://3v4l.org/jZTXA#v
Also I actually think the code checks these cases. See: ext/mbstring/mbstring.c:5248

		case 2:
			/* 3-byte character. Check third byte for 0x80. Then check first 2 bytes for
			 * 1110 0000, xx0x xxxx (overlong sequence) or 1110 1101, 1010 xxxx (0xd800-0xdfff) */
			if ((*(++p) & 0xc0) != 0x80 || (c == 0xe0 && (d & 0x20) == 0) || (c == 0xed && d >= 0xa0)) {
				return false;
			}
			break;

[LLyaudet]

Hello @LamentXU123,
Sorry @youkidearitai @LamentXU123, I made a bug in my code.
I got confused when reading the RFC between notation UD800 to UDFFF for unicode code points,
with notation %xD8 %x00:

UTF8-3      = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) /
                 %xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail )

I corrected my code:
Seldaek/jsonlint@728b4eb

[LamentXU123]

No worries! Thank you for reporting.

[youkidearitai]

I'm glad to it was solved.