* Use same "proxy rejected" for all Proxy invariant rejects.
* #ifdef guard for proxy policy checks.
* Test coverage for 'preventExtensions' trap.
* Convert many duk_hobject.h macro calls into function calls.
* Add functions to interact with idx_props allocation.
* Compile warning fixes.
Switch to using WTF-8 for duk_hstring string representation. The main
differences to previous extended CESU-8/UTF-8 are: (1) valid surrogate
pairs are automatically combined to UTF-8 on string intern while invalid
surrogate characters are encoded in CESU-8, and (2) ECMAScript code always
sees surrogate pairs for non-BMP characters.
Together, these make it more natural to work with non-BMP strings for both
ECMAScript (which no longer sees extended codepoints as before) and native
code (which now sees valid UTF-8 for non-BMP whenever possible).
Internally the main change is in string interning which now always sanitizes
input strings (but no Symbols) to WTF-8. Also all call sites where the byte
representation of strings are dealt with need fixing. WTF-8 leads to some
challenges because it's no longer possible to e.g. find a substring with a
naive byte compare: surrogate characters may either appear directly (CESU-8)
or baked into a non-BMP UTF-8 byte sequence.
The main places where this needs complex handling include:
* charCodeAt / codePointAt
* Extracting a substring
* String .replace()
* String .startsWith() and .endsWith()
* String .split() and search functions (like .indexOf())
* RegExp matching
* String cache behavior
This commit fixes all the necessary sites with minimal baseline implementations
which are in some cases much slower than the previous CESU-8 ones. Further work
is needed to optimize the WTF-8 variants to perform close to CESU-8.
Isolate all char-offset-to-byte-offset and character access calls
behind helpers to help prepare for a switch to WTF-8 representation.
This change should have no visible effect yet.
* Rename hobject helper functions for property entry access.
* Add helpers for looking up _Formals and _Varmap without inheritance.
* Add duk_xget_owndataprop() + variants for internal property lookups
without inheritance or side effects.
* Date built-in fix for .setTime(), .setYear(), etc on a frozen Date
instance. These should be allowed because they operate on an internal
slot which is not under normal property access control. Previously
the operations were incorrectly rejected with a TypeError.
* Changes to internal property definition and lookup here and there.
Mostly the changes should be no-op because e.g. inheritance only
matters if the internal property might be missing (which is not
usually the case).
* Automatically pin C literals interned into heap strings. Or if the
literal maps to an already interned string, pin it too. Pinning is
implemented using a duk_hstring flag and a one-off refcount bump.
Mark-and-sweep avoids sweeping pinned strings based on the flag.
* Add a lookup cache for quickly mapping a C literal address (which is
assumed stable) into a duk_hstring pointer. Once a mapping has been
formed, it never needs to be invalidated because the duk_hstring is
always pinned if the cache is used. Only heap destruction will free
the pinned duk_hstrings.
* More internal call site conversion for literals.
* Wording trivia.
* Convert memory helpers like duk_memcmp() to macros where possible: for some
reason automatic inlining with -Os doesn't do the right thing and the
footprint impact is over 1kB. The duk_memcmp() helper is an inlined
function because it returns a value and assertions would otherwise lead
to multiple evaluation of arguments.
* Differentiate between C99+ and "unsafe" call sites. For example, use
duk_memcpy() when the call site obeys C99+ restrictions, and
duk_memcpy_unsafe() when either pointer may be NULL when the size is zero.
Same for all helpers.
* Add internal wrappers also to DUK_MEMMOVE(), DUK_MEMSET(), and
DUK_MEMZERO().
* Use the wrappers everywhere for consistency: the zero-size cases will then
always be safe, and if the target is fine with invalid pointers in the zero
size case, the whole check can be omitted easily.
* Remove a few zero size checks as they're no longer necessary.
Use a single pause flags field to implement Resume, StepInto, StepOver,
and StepOut. This opens up possibilities for more Resume options, like
explicit control over whether to pause on caught vs. uncaught error.
Change StepOver, StepInto, and StepOut behavior when current activation
has no line information. Previously the commands were silently ignored
in this tate. The updated behavior is to ignore the line-based pause
trigger but obey the others, e.g. StepInto will pause on function entry,
function exit, and an error thrown past the current function.
* Change duk_bool_to to duk_small_uint_t from duk_small_int_t. This may
cause some sign warnings in calling code.
* Reject attempt to unpack an array-like value whose length is 2G or over;
previously was not checked explicitly, and the length was cast to duk_idx_t
with a sign change and the unpack would then later fail. Now it fails with
a clean RangeError.
* Add wrap check for Node.js Buffer.concat().
* API DUK_TYPE_xxx, DUK_TYPE_MASK_xxx, flag constants etc are now unsigned.
Both duk_hthread and duk_context typedefs resolve to struct duk_hthread
internally. In external API duk_context resolves to struct duk_hthread
which is intentionally left undefined as the struct itself is not
dereferenced. Change internal code to use duk_hthread exclusively which
removes unnecessary and awkward thr <-> ctx casts from internals.
The basic guidelines are:
* Public API uses duk_context in prototype declarations. The intent is to
hide the internal type, and there's already a wide dependency on the
type name.
* All internal code, both declarations and definitions, use duk_hthread
exclusively. This is done even for API functions, i.e. an API function
declared as "void duk_foo(duk_context *ctx);" is then defined as
"void duk_foo(duk_hthread *thr);".
Remove the special ecma-to-ecma call setup code and just use the normal
unprotected call setup code for that instead. Most of the code is the
same; just before calling into the bytecode executor check if the current
executor can be reused, and if so, indicate the situation using a special
return code.
Also remove internal duk_handle_call_protected() and implement all
protected API calls via duk_safe_call(). This reduces footprint and code
duplication further.
Rework call handling to use helpers more to make the call handling code
easier to follow.
Various other minor changer, e.g. DUK_OP_NEW is now DUK_OP_CONSCALL and
bytecode sets up the initial default instance.
* Make value stack and call stack limits configurable via DUK_USE_xxx
options. Also make value stack grow/shrink constants configurable.
* Rewrite value stack grow/shrink check primitives for better hot/cold path
handling.
* Use a proportional spare for grow and shrink sizes so that applications
needing a large value stack have fewer value stack resizes.
* Grow value stack allocation when entering a call or when explicitly requested
via e.g. duk_require_stack().
* Never shrink the value stack when entering a call, so that the unwind path
is guaranteed to have value stack to handle a protected call return. This
guarantee is only needed for protected call but is now applied to all calls
for simplicity.
* Don't perform a value stack shrink check at all in function return anymore.
It would be OK from protected call semantics perspective to do a shrink
attempt without throwing if it fails.
* Perform a value stack shrink check in mark-and-sweep only for now. When
emergency GC is running, shrink to a minimal size respecting current value
stack reserve.
Remove thr->callstack as a monolithic array and replace it with a linked list
of duk_activations. thr->callstack_curr is the current call (or NULL if no
call is in progress), and act->parent chains to a previous call or NULL.
thr->callstack_top is kept because it's needed by some internals at present;
it may be removed in the future.
When the flag is set, there is either no subclass C struct for the
duk_hobject, or there is a subclass C struct but there are no references
needing DECREF/marking in the struct.
This allows DECREF and mark-and-sweep to handle duk_hobjects with less
overhead for the common cases of plain objects and arrays (and some other
less commonly collected structs like duk_hnatfunc).
Also change Duktape.Thread.prototype internal class from Thread to Object:
with the other changes internal code now assumes that if an object's class
is Thread, it has the duk_hthread memory layout which wouldn't be the case
for Duktape.Thread.prototype.
One bottleneck in refzero and mark-and-sweep handling is checking whether an
object has an own or inherited _Finalizer property. This check walked the
prototype chain and did a property lookup for every object. Because a
finalizer is usually not present, the prototype chain would almost always be
walked to completion.
Improve this behavior by:
* Adding a DUK_HOBJECT_FLAG_HAVE_FINALIZER flag. The flag is set when the
object has an own _Finalizer property with a callable value, and cleared
otherwise. The flag is *only* set by duk_set_finalizer(), so any other
means of changing the internal _Finalizer property will leave the flag out
of sync (which causes a finalizer run to be skipped).
* Adding duk_hobject_has_finalizer_fast() which checks for finalizer existence
by walking the prototype chain, but only checking the flag, not the property
table.
* Use the fast finalizer check in refzero and mark-and-sweep.
Out-of sync cases:
* If the flag is set but there is no actual finalizer, the object will go
through finalizer processing when garbage collecting. This is harmless:
the finalizer call will fail and the object will be garbage collected, but
with some potential delay (especially for mark-and-sweep).
* If the flag is cleared but there is an actual finalizer, the finalizer will
be ignored.
Related changes:
* When duk_dump_function() is called, zero DUK_HOBJECT_FLAG_HAVE_FINALIZER on
serialization, so it won't be set when the function is loaded back. If this
is not done, the loaded function will (harmlessly) go through finalizer
processing when garbage collected.
* Update debugger artificial properties to include "have_finalizer" flag.
Other changes:
* A few DUK_UNLIKELY() attributes for prototype sanity limits which are
almost never hit.
Throw notify might be thrown from an empty callstack e.g. if duk_throw()
is called by C code outside of any call. If the throw argument is not an
error instance, the code would attempt to access the topmost activation
even if one didn't exist.
* Replace the two alternative algorithms with a single one which works for
both desktop and low memory cases.
* Basic algorithm is a hash table with size 2^N, hash mask is simply
(size - 1), e.g. if size is 0x100, mask is 0xFF. duk_hstring has a 'next'
pointer (single linked list) for chaining strings mapping to the same
slot.
* Change plain buffers to inherit from Uint8Array. This affects a lot of
small things like Object.prototype.toString() output, enumeration of plain
buffers, etc. It also changes JSON serialization for plain buffers because
the index properties are enumerable as with Uint8Array instances.
* Disable JSON stringify fastpath for plain buffers for now so that the
virtual index properties get serialized correctly.
* Remove ArrayBuffer non-standard virtual properties.
* Remove DataView non-standard virtual properties.
* Move .byteLength, .byteOffset, .BYTES_PER_ELEMENT, and .buffer into
inherited getters as required in ES6. However, the .length property
remains a virtual own property for now (it too is an inherited getter
in ES6).
* Move ArrayBuffer.allocPlain() and ArrayBuffer.plainOf() to
Uint8Array.allocPlain() and Uint8Array.plainOf() to match the
semantics change for plain buffers.
* Fix Node.js buffer .slice() behavior, the returned Node.js buffer
would have ArrayBuffer.isView == 0 which doesn't match the revised
Node.js behavior (Buffers being Uint8Array instances)
* Reject ArrayBuffers with a view offset/length in Node.js Buffer .slice()
rather than accept such ArrayBuffers without actually respecting the
view offset/length.
* Allow a plain buffer or a lightfunc as a constructor "replacement object"
return value.
* Comment on duk_buffer_to_string() safety w.r.t. potentially pushing
Symbol values.
* Went through all duk_push_(l)string() call sites too.
* Minor footprint optimization for pushing empty strings (use more
compact internal helper).
* Strings with 0xFF byte prefix are considered special symbols: they have
typeof "symbol" but still mostly behave as strings (e.g. allow ToString)
so that existing code dealing with internal keys, especially inside
Duktape, can work with fewer changes.
* Strings with 0x80 byte prefix are global symbols, e.g. Symbol.for('foo')
creates the byte representatio: 0x80 "foo"
* Strings with 0x81 byte prefix are unique symbols; the 0x81 byte is followed
by the Symbol description, and an internal string component ensuring
uniqueness is separated by a 0xFF byte (which can never appear anywhere in
an extended UTF-8 string). The unique suffix is up to Duktape internals,
currently two 32-bit counters are used. For example:
0x81 "mySymbol" 0xFF "0-17".
* Well-known symbols use the 0x81 prefix but lack a unique suffix, so their
format is 0x81 <description> 0xFF.
* ES6 distinguishes between an undefined symbol description and an empty
string symbol description. This distinction is not currently visible via
Ecmascript bindings but may be visible in the future. Append an extra
0xFF to the unique suffix when the description is undefined, i.e.
0x81 0xFF <unique suffix> 0xFF.
* Callstack index is required for most debug commands which accept one
* Index comes before other parameters, e.g. Eval has the signature:
REQ 0x1e <index> <code>
* Eval accepts null for the callstack index, indicating an indirect
eval.
These can be used whenever we're 100% certain that the value stack index
exists and the type matches expected type. When these are true, a
duk_hstring, duk_hbuffer, or duk_hobject pointer fetch can be inlined to
small code.
* Plain buffers still inherit from ArrayBuffer.prototype.
* Plain buffers won't object coerce, so Object(plainBuffer) fails.
* All buffer object related methods throw an error; their function bodies
are essentially empty. Note that this includes bindings such as
String.fromBuffer(), ArrayBuffer.allocPlain(), ArrayBuffer.plainOf(),
and so on. In essence, you can index plain buffers in Ecmascript but
the buffer values must be created via the C API.
* Duktape custom bindings like Duktape.dec('hex', 'deadbeef') still work
and produce plain buffers.
Change handling of plain buffers so that they behave like ArrayBuffer
instances to Ecmascript code, with limitations such as not being
extensible and all properties being virtualized. This simplifies
Ecmascript code as plain buffers are just lightweight ArrayBuffers
(similarly to how lightfuncs appear as function objects). There are
a lot of small changes in how the built-in objects and methods, and
the C API deals with plain buffer values.
Also make a few small changes to plain pointer and lightfunc handling
to improve consistency with how plain buffers are now handled.
Sami Vaarala
Bruce Pascoe