duktape/doc/utf8-internal-representation.rst

======================================
Using UTF-8 as internal representation
======================================

Some notes on using UTF-8 as internal representation for ECMAScript strings
when surrogate pairs can be combined.

Current representation
======================

Current internal representation is a union of:

* CESU-8: to support full 16-bit codepoint sequences without limitations.
  In particular, individual and unpaired surrogates must work without
  interpretation or conversion.

* UTF-8: to support non-BMP characters, if they are created from C code
  or e.g. using String.fromCharCode(0x12345).

* Extended UTF-8: to support codepoints up to U+FFFFFFFF.  This is now
  only needed by the regexp bytecode, which uses extended UTF-8 as its
  internal representation and needs to represent long offsets as
  codepoints.

C API problem with current representation
=========================================

One concrete problem with this arrangement is that non-BMP strings are
internally represented as CESU-8:

* If source code contains a non-BMP character, the ECMAScript specification
  requires that such a character is decoded into surrogates, from
  https://www.ecma-international.org/ecma-262/5.1/#sec-6:

  - If an actual source text is encoded in a form other than 16-bit code
    units it must be processed as if it was first converted to UTF-16.

* This means that ``x = '\u{12345}'`` and ``x = '\ud808\udf45'`` MUST be
  treated identically.  For example, for both inputs:

  - The string's ``.length`` must be 2.

  - ``x[0]`` must be 0xd808, and ``x[1]`` must be 0xdf45.

  - RegExps must be able to match the individual surrogates, and one must
    be able to e.g. backtrack each surrogate separately.

  - It must be possible to take a substring whose one end is between
    the surrogate codepoints.

* In the current C API such a string will appear CESU-8 encoded because
  that's the internal representation used for surrogate codepoints.

* Applications dealing natively with UTF-8 would often prefer to see UTF-8
  rather than CESU-8, thus avoiding the need to transcode CESU-8 to UTF-8.

The ECMAScript specification doesn't (and cannot) mandate any specific
internal representation, nor does it provide any requirements on how a
C API must represent strings.  The current convention of using CESU-8
for standard ECMAScript strings is thus not really mandatory.  However,
if an alternative representation is used, it MUST behave identically as
far as script code is concerned.

Automatically combining surrogates in internal representation
=============================================================

One alternative to the current internal representation is to:

* Keep the current CESU-8 + UTF-8 + extended UTF-8 as the base representation.

* When conceptual ECMAScript strings contain correctly paired surrogates,
  combine the surrogates into the actual non-BMP codepoint.  The resulting
  codepoint is then valid UTF-8 and not CESU-8.

* When a non-paired surrogate is found, encode it as CESU-8 as before.

* This process must be applied to all inputs, both script code and C code,
  so that a certain conceptual ECMAScript string has a unique duk_hstring
  representation.  (If this is not the case, string comparison using an
  interned string pointer would no longer be valid which leads to a lot of
  complications.)

This would have the upside that:

* Valid Unicode strings in UTF-8 codepoint range (U+0000 to U+10FFFF without
  surrogate range U+D800 to U+DFFF) would appear as valid UTF-8 (not CESU-8)
  in the C API.

* Pushing UTF-8 strings would produce strings that behaved like standard
  ECMAScript strings, i.e. they would conceptually have surrogate pairs in
  place of non-BMP.

And a few downsides:

* All the internal code would need to maintain an "as if" illusion: such
  strings must appear as uninterpreted 16-bit codepoint sequences, and all
  16-bit codepoint sequences must still work without difference as far as
  script code is concerned.  This is not trivial, more on this below.

* One would no longer be able to push an arbitrary byte sequence as a string
  (duk_push_string()) and then read it back as is.  The automatic surrogate
  combination would mean the output might be different, with surrogates
  represented in CESU-8 combined into UTF-8.  This is a loss of current
  functionality which has been useful for some applications; one can e.g.
  push ISO-8859-1 strings as is, and read them back.  Script code will see
  such strings as being somewhat broken, but they have previously passed
  through without modification.

Some internals where the "as if" illusion must be maintained:

* String ``.length`` must count non-BMP codepoints as 2 codepoints to get
  the standard length.

* String.charCodeAt() and all other String functions must use an index scheme
  that references the conceptual 16-bit codepoint sequence index (where each
  non-BMP counts as two indices), and allow reading, substringing, etc, both
  of the surrogate pairs individually.

* There's no longer an easy "char offset to byte offset" internal primitive.
  Currently such a conversion maps an integer to an integer (or error).  For
  non-BMP characters the result would now be a tuple: an integer pointing to
  the start of the codepoint, and a flag indicating whether we want the high
  or the low surrogate.  All places maintaining "current offset" must track
  that additional flag somehow (it could maybe be encoded as the high bit of
  a 32-bit unsigned value?).

* When doing string replacements, code must always check whether the
  replacements created valid surrogate pairs from previously unpaired
  surrogates.  They must be merged, to maintain a unique strnig representation.
  Such surrogates may appear at the edges of replacement strings.

* When combining strings, must check for previously unpaired surrogates at
  string join point.

* RegExp matching must match non-BMP codepoints as two surrogates individually
  as far as patterns are concerned.  It must be possible to capture only one
  of the surrogates, backtrack each surrogate individually, match start offset
  must try both surrogates as starting points, etc.

* RegExp /u mode would work trivially with this internal representation, as
  the codepoints are already combined.