duktape/doc/hobject-design.txt

===========
duk_hobject
===========

Overview
========

The ``duk_hobject`` type represents an object with key-value properties,
and is the most important type from an implementation point of view.
It provides objects for various purposes:

* Objects with E5 normal object semantics
* Objects with E5 array object special behavior
* Objects with E5 string object special behavior
* Objects with E5 arguments object special behavior
* Objects with no E5 semantics, for internal use

This document discusses the ``duk_hobject`` object in detail, including:

* Requirements overview
* Features of Ecmascript E5 objects
* Internal data structure and algorithms
* Enumeration guarantees
* Ecmascript property behavior (default and special)
* Design notes, future work

The details of property-related algorithms in E5 are pretty intricate
and are described separately in ``hobject-algorithms.txt``.

The following parts of Ecmascript E5 are useful background:

+-----------+-------------------------------------------------------------+
| Section   | Description                                                 |
+===========+=============================================================+
| 8.6       | Object type, internal properties, property attributes       |
+-----------+-------------------------------------------------------------+
| 8.10      | Property descriptors                                        |
+-----------+-------------------------------------------------------------+
| 8.12      | Default property access methods                             |
+-----------+-------------------------------------------------------------+
| 10.6      | Arguments object special behavior                           |
+-----------+-------------------------------------------------------------+
| 15.4.5.1  | Array object special behavior                               |
+-----------+-------------------------------------------------------------+
| 15.5.5.2  | String object special behavior                              |
+-----------+-------------------------------------------------------------+

See also the following documentation:

* ``hobject-algorithms.txt``: detailed derivation of object algorithms

* ``hobject-enumeration.txt``: more discussion on enumeration

* ``error-objects.txt``: error object properties

* ``function-objects.txt``: function template and instance properties

Requirements overview
=====================

Ecmascript object compatibility requires:

* Properties with a string key and a value that is either a plain data
  value or an accessor (getter/setter)
* Property attributes which control the behavior of individual properties
  (e.g. enumerability and writability)
* Object extensibility flag which controls addition of new properties
* Prototype-based inheritance of properties along a loop-free prototype chain
* Some very basic enumeration guarantees for both mutating and non-mutating
  enumeration
* Object internal properties (at a conceptual level)

Additional practical requirements include:

* Additional enumeration guarantees (e.g. enumeration order matches key
  insertion order); see separate discussion on enumeration
* Minimal memory footprint, especially for objects with few properties
  which dominate common use
* Near constant property lookup performance, even for large objects
* Near constant amortized property insert performance, even for large objects
* Fast read/write access for array entries, in particular avoiding string
  interning whenever possible
* Sparse array support (e.g. ``var x=[]; x[0]=1; x[1000000]=2;``): must be
  compliant, shouldn't allocate megabytes of memory, but does not have to
  be fast
* Support long-lived objects with an arbitrary number of key insertions
  and deletions (implies "compaction" of keys / ordering structure)

There are unavoidable trade-offs involved, the current trade-off preferences
are roughly as follows (most important to least important):

#. Compliance
#. Compactness
#. Performance
#. Low complexity

Compliance is a must-have goal for all object features.  Performance is only
really relevant for common idioms.  Rare cases need to be compliant but not
especially compact or performant: for instance, sparse arrays don't perform
very well but are still compliant.

Object features
===============

Named and internal properties
-----------------------------

An Ecmascript object consists of:

* A set of externally visible *named properties*

* A set of (conceptual) *internal properties*

The externally visible named properties are characterized by:

* A string key

  + 16-bit characters (any 16-bit unsigned integer codepoints may be used)
  + Even array indices are strings, e.g. ``x[0]`` really means ``x["0"]``

* A property value which may be:

  + A *data property*, a plain Ecmascript value
  + An *accessor property*, a setter/getter function pair invoked
    for property accesses

* Property attributes which control property accesses:

  + For data properties:

    - ``[[Configurable]]``
    - ``[[Enumerable]]``
    - ``[[Value]]``
    - ``[[Writable]]``
 
  + For accessor properties:

    - ``[[Configurable]]``
    - ``[[Enumerable]]``
    - ``[[Get]]``
    - ``[[Set]]``

* The ``[[Extensible]]`` internal property determines whether new (own) keys
  can be added to an object.  Many other internal properties exist.

Internal properties are used in E5 to specify required behavior; the
concrete property implementation is implementation specific.  The
current implementation for internal properties is covered in more
detail below.

Property attributes and descriptors
-----------------------------------

Property attributes affect property access algorithms internally.  They are
also externally visible and can be manipulated through built-in methods.
The property attributes are:

* ``[[Configurable]]``
* ``[[Enumerable]]``
* ``[[Value]]``
* ``[[Writable]]``
* ``[[Get]]``
* ``[[Set]]``

New properties added to objects by an assignment are by default data
properties with the following attributes: ``[[Writable]]=true``,
``[[Enumerable]]=true``, ``[[Configurable]]=true``.  This is implicit
in the ``[[Put]]`` algorithm (E5 Section 8.12.5, step 6).  Note that
these defaults differ from the "official default values" (all attributes
``false``) in E5 Section 8.6.1 which are used for e.g.
``[[DefineOwnProperty]]``.

User code can deviate from the defaults for assignments by defining or
modifying properties using ``Object.defineProperty()``.  This is not
very common, so almost all user properties have default attributes.
Built-in objects often have properties with non-default attributes, though.

A *property descriptor* contains zero or more property attributes,
and is used both internally and externally to describe or modify
property attributes.  Property descriptors are used internally in the E5
specification with the following notation:

  { ``[[Value]]``: 42, ``[[Writable]]``: true }

The same property descriptor would be represented as an external Ecmascript
value::

  { "value": 42, "writable": true }

The internal and external property descriptors are converted through the
internal ``FromPropertyDescriptor()`` and ``ToPropertyDescriptor()``
methods.

Property descriptors are classified into several categories based on
what keys they contain:

* Data property descriptor: contains ``[[Value]]`` or ``[[Writable]]``
* Accessor property descriptor: contains ``[[Set]]`` or ``[[Get]]``
* Generic property descriptor: a descriptor which is neither a data nor
  an accessor property descriptor, i.e. does not contain
  ``[[Value]]``, ``[[Writable]]``, ``[[Set]]``, or ``[[Get]]``

Although a property descriptor can technically be both a data property
descriptor and an accessor property descriptor at the same time, such
descriptors are rejected whenever they are encountered.  The
``[[Configurable]]`` and ``[[Enumerable]]`` attributes can be in any
kind of a descriptor.

A property descriptor is *fully populated* if it contains all the keys of
its type, i.e.:

* A fully populated data descriptor contains all of the following:
  ``[[Configurable]]``, ``[[Enumerable]]``, ``[[Value]]``, ``[[Writable]]``
* A fully populated accessor descriptor contains all of the following:
  ``[[Configurable]]``, ``[[Enumerable]]``, ``[[Get]]``, ``[[Set]]``

The property attributes stored in the object for a certain property always
form a fully populated property descriptor.  Any missing values are always
filled in with defaults when a property is first inserted, and although
attribute values can be changed after insertion, the attributes themselves
cannot be removed.

Partially populated property descriptors are used internally in the
specification for describing property modifications.  They can also be
used externally (for similar purposes) through ``Object.defineProperty()``.

Consider the following example, which illustrates how properties and
their attributes can be defined in various ways::

  // array initializer
  var o = [ "one", "two" ];  // also sets "length" to 2

  // add "foo" (writable, enumerable, configurable)
  o.foo = 1;

  // modify attributes after insertion
  Object.defineProperty(o, "foo", {
    writable: false
  });

  // insert an accessor, with [[DefineOwnProperty]] default
  // attributes (non-enumerable, non-configurable)
  Object.defineProperty(o, "bar", {
    "get": function() { return "bar"; },
    "set": function(x) { throw new Error("cannot write bar"); }
  });

The object would have the following internal state (represented as
external property descriptors)::

  "0"       -->  { "configurable": true,
                   "enumerable": true,
                   "writable": true,
                   "value": 'one' }

  "1"       -->  { "configurable": true,
                   "enumerable": true,
                   "writable": true,
                   "value": 'two' }

  // non-default attributes (E5 Section 15.4.5.2)
  "length"  -->  { "configurable": false,
                   "enumerable": false,
                   "writable": true,
                   "value": 2 }

  "foo"     -->  { "configurable": true,
                   "enumerable": true,
                   "writable": false,
                   "value": 1 }

  // [[DefineOwnProperty]] defaults for new properties
  // differ from [[Put]] (defaults to false)
  "bar"     -->  { "configurable": false,
                   "enumerable": false,
                   "get": <function reference>,
                   "set": <function reference> }

Property descriptors can also be read back from user code through
``Object.getOwnPropertyDescriptor()``.  Example using NodeJS / V8::

  var o = {
    foo: 1,
  
    get bar() { return "bar"; },
    set bar(x) { throw new Error("cannot write bar"); }
  };

  // Prints:
  // { value: 1,
  //   writable: true,
  //   enumerable: true,
  //   configurable: true }

  console.log(Object.getOwnPropertyDescriptor(o, "foo"));

  // Prints:
  // { get: [Function: bar],
  //   set: [Function: bar],
  //   enumerable: true,
  //   configurable: true }

  console.log(Object.getOwnPropertyDescriptor(o, "bar"));

The visibility of property attributes to user code poses some implementation
challenges.  Ordinary property access occurs through the ``[[Get]]``,
``[[Put]]``, and ``[[Delete]]`` algorithms, hiding some of the internal
complexity regarding property attributes etc.  However, the built-in method
``Object.defineProperty()`` exposes the internal ``[[DefineOwnProperty]]``
in all its complexity to user code.

The current implementation manages to use fully populated descriptors
internally, and expose partial descriptors only through
``Object.defineProperty()``.

Prototype chain
---------------

Each object has a non-mutable *internal prototype* established at
object creation (``[[Prototype]]`` internal property).  The value of
the internal prototype must be another object or null.  Since the
prototype object may also have an internal prototype (and so on), the
objects form a non-mutable *prototype chain* terminating at an object
whose internal prototype is ``null``.

The prototype chain affects most property access algorithms with the
general principle that if a property is not found in a certain object,
the prototype chain is then searched in ascending order.  To simplify:

* Property read operations return the value found in the first object
  in the prototype chain containing the property.  If an accessor
  property is found, the getter is called.

* Property write operations first check the prototype chain to see
  whether the property exists.  If so, the property may prevent the
  write (if a non-writable data property), cause a setter call (if an
  accessor property), or allow the write.  The write is allowed and
  not captured by a setter, the property is added to the *original target
  object* (instead of an object possibly higher up in the prototype chain).

* Property delete operations do not consult the prototype chain and
  only have an effect on the target object.

The non-mutability of the prototype chain is not very explicit in the
specification (nor does the current implementation assume or take
advantage of its non-mutability).  The requirement is stated in a
footnote in E5 Section 8.6.2:

  NOTE This specification defines no ECMAScript language operators or
  built-in functions that permit a program to modify an object’s
  ``[[Class]]`` or ``[[Prototype]]`` internal properties or to change
  the value of [[Extensible]] from false to true.

The prototype chain is required to be loop-free; it is required to
terminate in a null reference (again, in E5 Section 8.6.2):

  Every ``[[Prototype]]`` chain must have finite length (that is, starting
  from any object, recursively accessing the [[Prototype]] internal
  property must eventually lead to a null value).

The current implementation makes no specific effort to ensure this because
plain Ecmascript code cannot create prototype loops (though C code can
easily do so).  To see why this is the case, assume that the current set of
objects have no prototype loops and a new object is created.  The internal
prototype of the created object is either set to ``null`` or one of the
existing objects during its creation.  Since the prototypes of the existing
objects cannot be updated, the result is a new set of objects without
prototype loops.  Even so, all prototype walking loops in the implementation
contain a sanity limit for prototype chain length to break out should a
loop be somehow created.

Valid array index and length
----------------------------

The E5 specification has very specific definitions for *valid array index*
property names and *valid array length* property values.  Only these trigger
the ``Array`` specific ``[[DefineOwnProperty]]`` behavior in E5 Section
15.4.5.1 steps 3 and 4.

Note that these requirements do not simply specify a minimum length for
supported arrays: they also specify a maximum length for arrays at least
with respect to how ``length`` behaves.  It would be nice to be able to
support larger arrays as a build option, but there are probably no realistic
cases with arrays larger than 4G elements which would still be relevant for
embedding Duktape.

Array index
:::::::::::

Ecmascript E5 Section 15.4 states:

  A property name ``P`` (in the form of a String value) is an *array index*
  if and only if ``ToString(ToUint32(P))`` is equal to ``P`` and
  ``ToUint32(P)`` is not equal to 2^32-1 (0xffffffff).

This implies that the maximum array length is 2**32-1 (0xffffffff) and
the maximum array index is one less, 2**32-2 (0xfffffffe).  All valid
array index and length values can be represented with unsigned 32-bit
values.  Because 0xffffffff is not a valid array index, it is used internally
as a convenient "no array index" marker (``DUK_HSTRING_NO_ARRAY_INDEX``).
For instance, a coercion call can return the marker to indicate that an input
was not a valid array index.

Because Ecmascript object property keys are strings, all array indices
encountered in property access expressions are conceptually first coerced
to a string form using ``ToString()`` and then checked whether they are
valid array indexes (this is unlike array ``length`` values, which can
have any type, see below).  (Obviously, this explicit coercion should be
avoided whenever possible.)

The following table lists the possible coercions:

+---------------+-----------------+---------------------------------------+
| Property name | ``ToString``    | Valid array index?                    |
+===============+=================+=======================================+
| ``undefined`` | ``"undefined"`` | no                                    |
+---------------+-----------------+---------------------------------------+
| ``null``      | ``"null"``      | no                                    |
+---------------+-----------------+---------------------------------------+
| ``false``     | ``"false"``     | no                                    |
+---------------+-----------------+---------------------------------------+
| ``true``      | ``"true"``      | no                                    |
+---------------+-----------------+---------------------------------------+
| a number      | various         | yes, if a whole number in the range   |
|               |                 | [0,2**32-2]                           |
+---------------+-----------------+---------------------------------------+
| a string      | same            | yes, if a "canonical" representation  |
|               |                 | for a whole number in the range       |
|               |                 | [0,2**32-2] (``"2"`` is valid, while  |
|               |                 | ``"0.2e1"`` is not)                   |
+---------------+-----------------+---------------------------------------+
| an object     | various         | depends on coerced string value       |
+---------------+-----------------+---------------------------------------+

Note that for instance ``"0.2e1"`` which numerically represents 2 is not
a valid array index: ``ToString(ToUint32("0.2e1"))`` produces ``"2"``, but
this is not equal to the original string ``"0.2e1"`` (which is required).

Array length
::::::::::::

The requirements for a valid *array length* are a bit different, because
array length is assigned as an arbitrary property value and is not therefore
automatically coerced to a string first.

The requirements for a valid array length are implicit in E5 Section 15.4.5.1,
steps 3.c to 3.d:

* Step 3.c: Let ``newLen`` be ``ToUint32(Desc.[[Value]])``.
* Step 3.d: If ``newLen`` is not equal to ``ToNumber(Desc.[[Value]])``, throw
  a ``RangeError`` exception

The requirements boils down to (for input value ``X``):

* ``ToUint32(X)`` == ``ToNumber(X)``

The requirements are seemingly similar to the array index requirements, but
in fact allow a wider set of values, such as:

* ``true`` represents array length ``1``, but is not a valid array index
* ``"0.2e1"`` represents array length ``2``, but is not a valid array index
* ``0xffffffff`` represents array length 2**32-1, but is not a valid array index

A potential ``length`` value ``X`` is treated as follows (see E5 Sections
9.3 and 9.6 for definitions of the coercions ``ToNumber`` and ``ToUint32``):

+----------------+--------------+--------------+--------------------------+
| Property value | ``ToNumber`` | ``ToUint32`` | Valid array length?      |
+================+==============+==============+==========================+
| ``undefined``  | ``NaN``      | ``+0``       | no                       |
+----------------+--------------+--------------+--------------------------+
| ``null``       | ``+0``       | ``+0``       | yes, length ``0``        |
+----------------+--------------+--------------+--------------------------+
| ``false``      | ``+0``       | ``+0``       | yes, length ``0``        |
+----------------+--------------+--------------+--------------------------+
| ``true``       | ``1``        | ``1``        | yes, length ``1``        |
+----------------+--------------+--------------+--------------------------+
| a number       | various      | various      | yes, if whole number in  |
|                |              |              | the range [0,2**32-1]    |
+----------------+--------------+--------------+--------------------------+
| a string       | various      | various      | yes, if representation of|
|                |              |              | a whole number in the    |
|                |              |              | range [0,2**32-1] (does  |
|                |              |              | not need to be canonical,|
|                |              |              | e.g. ``"2"``, ``"2.0"``, |
|                |              |              | ``"0.2e1"`` are all      |
|                |              |              | acceptable               |
+----------------+--------------+--------------+--------------------------+
| an object      | various      | various      | depends on coerced       |
|                |              |              | values                   |
+----------------+--------------+--------------+--------------------------+

As an example of using a non-number as Array length::

  duk> var a = [ 'foo', 'bar', 'quux' ]; a
  = foo,bar,quux
  duk> a.length = true;  // same as a.length = 1
  = true
  duk> a
  = foo
  duk> a.length
  = 1

Enumeration requirements
------------------------

Enumeration requirements are discussed in a separate section below,
together with the current implementation for enumerating object keys.

.. raw:: LaTeX

   \newpage

Structure overview
==================

The memory layout of an ``duk_hobject`` is illustrated below::

  duk_hobject                       property allocation
  (fixed allocation)                (dynamic allocation)
  
  +------------------------+        +---------------------------+
  | duk_heaphdr    (flags) |  .---->| entry part keys           |
  +========================+  |     | (e_size x duk_hstring *)  |
  | duk_u8 *p -------------+--'     +---------------------------+
  | duk_u32 e_size         |     .->| entry part values         |
  | duk_u32 e_used         |     :  | (e_size x duk_propvalue)  |
  | duk_u32 a_size         |     :  +---------------------------+
  | duk_u32 h_size         |     +->| entry part flags          |
  | duk_hobject *prototype |     :  | (e_size x duk_u8)         |
  +------------------------+     :  +---------------------------+
  : duk_hcompiledfunction  :     +->| array part values         |
  : duk_hnativefunction    :     :  | (a_size x duk_tval)       |
  : duk_hthread            :     :  +---------------------------+
  :                        :     +->| hash part indices         |
  : (extended structures)  :     :  | (h_size x duk_u32)        |
  +------------------------+     :  +---------------------------+
                                 :
  'p' is NULL if no property     `- these pointers are computed
  allocation exists                 on-the-fly using e_size and
                                    a_size


The heap header structure ``duk_heaphdr`` contains:

* flags with both heap level flags (``DUK_HEAPHDR_FLAG_*`` in
  ``duk_heaphdr.h``) and object specific flags (``DUK_HOBJECT_FLAG_*``
  in ``duk_hobject.h``)
* heap allocated list linkage
* reference counter field

The object specific part of ``duk_hobject`` contains:

* property allocation: A data structure for storing properties
* internal prototype field for fast prototype chain walking;
  other internal properties are stored in the property allocation
* ``duk_hcompiledfunction``, ``duk_hnativefunction``, and ``duk_hthread``
  object sub-types have an extended structure with more fields

The property allocation part is a single memory allocation containing all
the object properties, both external and internal.  It is subdivided
internally into the following parts:

* *Entry part* stores ordered key-value properties with arbitrary
  property attributes (flags), and supports accessor properties
  (getter/setter properties), i.e., full E5 semantics
* *Array part* (optional) stores plain values with default property
  attributes (writable, enumerable, configurable) for valid array indices
  (``"0"``, ``"1"``, ..., ``"4294967294"``); does not support accessor
  properties
* *Hash part* (optional) provides accelerated key lookups for the
  entry part, mapping a key into an entry part index

Internal properties are stored in the entry part, and are only distinguished
from normal properties in that their keys are invalid UTF-8 sequences which
cannot be generated (and thus not accessed) from Ecmascript code.  Internal
properties should never be enumerable or visible in other ways.  See separate
discussion of internal properties later in the document.

The ``duk_hobject`` allocation is fixed and its address never changes after
initialization.  The property allocation part is allocated on demand, and
its address may change when the object is resized; this currently always
happens because there is no in-place resizing.  The resizing process is
described in a separate section below.  The property allocation part can also
be missing, the ``p`` pointer is NULL in this case.

To avoid storing multiple pointers/offsets pointing to the individual sections
of the property allocation, the different parts are reached with run-time
pointer computations.  There are a lot of convenience accessor macros in
``duk_hobject.h`` to access the various parts and elements within the parts.
**Always** use these macros to manipulate object properties so that changing
the layout is contained to a small section of code.

Notes:

* For a newly allocated object with no properties, there is no property
  allocation, and the ``p`` pointer is ``NULL``.  It may also become
  ``NULL`` later if all object properties are deleted and the object is
  then compacted.

* The array part is assumed to be comprehensive, i.e. if the array part
  exists, all valid array index keys must reside in the array part.  If
  this invariant would need to be violated, the array part is abandoned
  and its entries moved into the entry part.

* The array part entries are assumed to have default property attributes
  (writable, configurable, enumerable).  If this invariant would need to
  be violated, the array part is also abandoned.

* The array part is also abandoned if the array part would become too
  sparse, i.e. it would take too much memory compared to the number of
  entries actually present.  This behavior is not compliance related.

* A certain key can be present at most once (in either the entry or array
  part).  This invariant must be enforced when adding new keys into the
  object.  Other implementation code can simply assume it.

* The default attributes for new properties depend on how they are inserted:

  + For ordinary assignment, the defaults are defined in the ``[[Put]]``
    algorithm (E5 Section 8.12.5, step 6): writable, enumerable, configurable.
    Note that these differ from the official default values defined in
    E5 Section 8.6.2.

  + For ``[[DefineOwnProperty]]`` the defaults are defined in E5 Section
    8.12.9 step 4, which refers to the official "default attribute values"
    in E5 Section 8.6.2: non-writable, non-enumerable, non-configurable.

Entry and hash part
===================

The entry part contains ordered key-value pairs, and supports full Ecmascript
E5 semantics: property values can be plain *data properties* or *accessor
properties*, and can have any property attributes, stored in property flags.
Internal properties, identified with a special key prefix, can also be stored.

The hash part is optional, and allows faster lookups into the entry part.
It is only used for objects with at least ``DUK_HOBJECT_E_USE_HASH_LIMIT``
properties in the entry part.

.. raw:: LaTeX

   \newpage

Layout
------

The entry part consists of three separate arrays arranged sequentially:
keys, values, flags.  An optional hash part may exist to speed up key
lookups::

    +---------+                  
    | key 0   |   entry part keys
    | NULL    |   (duk_hstring *)
 .->| key 2   |
 |  | / / / / |
 |  | / / / / |
 |  +---------+
 |  | value 0 |   entry part values
 |  | / / / / |   (duk_propvalue)
 +->| value 2 |
 |  | / / / / |
 |  | / / / / |
 |  +---------+
 |  | flags 0 |   entry part flags
 |  | / / / / |   (duk_u8)
 +->| flags 2 |
 |  | / / / / |
 |  | / / / / |   / / / /   Denotes uninitialized data
 |  +---------+             which is not reachable from
 |  :         :             a GC perspective
 |  : array   :
 |  : part    :
 |  :         :
 |  +---------+
 |  | UNUSED  |   hash part
 `--| 2       |   (duk_u32)
    | DELETED |
    | UNUSED  |   UNUSED  = DUK_HOBJECT_HASHIDX_UNUSED
    | 0       |           = 0xffffffffU
    | UNUSED  |
    | UNUSED  |   DELETED = DUK_HOBJECT_HASHIDX_DELETED
    +---------+           = 0xfffffffeU
 
                  DELETED entries don't terminate hash
                  probe sequences, UNUSED entries do.
 
    Here, e_size = 5, e_used = 3, h_size = 7.

.. FIXME for some unknown reason the illustration breaks with pandoc

Each array in the entry part contains ``e_size`` allocated entries.
The entries at indices [0,\ ``e_used``\ [ are currently in use, and any
entries above that are uninitialized (garbage) data, and not reachable
from a GC perspective.

New keys are always appended to the current ``e_used`` position (with the
entry part resized if it is already full).  Existing entries are deleted
by marking a key as ``NULL``; they are not reused for new properties to
avoid disrupting the key enumeration order (which should match insertion
order).  NULL entries are removed (compacted) whenever the property
allocation is resized.  If a key entry is ``NULL``, the corresponding value
and flag fields MUST NOT be interpreted, and are not considered reachable
from a GC perspective.  Thus, the property value must be decref'd when the
key is set to NULL.  If a key entry is non-\ ``NULL``, it is considered
reachable and must be incref'd on insertion.

Flags are represented by an ``duk_u8`` field, with flags defined in
``duk_hobject.h``.  The current flags are:

* ``DUK_PROPDESC_FLAG_WRITABLE``
* ``DUK_PROPDESC_FLAG_ENUMERABLE``
* ``DUK_PROPDESC_FLAG_CONFIGURABLE``
* ``DUK_PROPDESC_FLAG_ACCESSOR``

The value field is a union of (1) a plain value, and (2) an accessor value
which contains ``get`` and ``set`` function pointers.  The interpretation
of the union depends on the ``DUK_PROPDESC_FLAG_ACCESSOR``; if set, the
value is treated as the accessor part of the union, otherwise it is treated
as the value part of the union.  This interpretation must be done everywhere
where the value is accessed, otherwise garbage values will be read.
In particular, the reference count and garbage collection code must always
interpret the union correctly based on the current entry flags.

The hash part (if it exists) maps an ``duk_hstring`` key ``K`` to an index
``I`` of the current entries part, or indicates that the key does not exist
in the object.  The hash index structure has no bearing on garbage
collection; in particular, the index references from the hash part to the
entry part are not considered counted references.

If the hash part exists, it is always kept up-to-date with the entry part
so that both structures always contain the same keys.  Deleted entries
in the hash data structure are explicitly marked DELETED.  Such entries
don't terminate hash probe sequences but act otherwise as UNUSED entries;
see more detailed discussion below.  DELETED entries are eliminated
(converted to UNUSED) when the property allocation is resized, improving
hash part performance.

Notes:

* This layout of three separate arrays has been chosen so that a linear key
  scan is efficient, e.g. works nicely with cache lines and prefetches,
  which is important because "small" objects don't have a hash part at all.
  Linear scan is more space efficient and often also faster than a
  hash lookup, which does one or more random accesses to the hash part when
  going through the probe sequence.

* Objects in dynamic languages often don't guarantee a key enumeration
  order, which allows objects to be implemented with easy and efficient
  "pure" hash tables.  Although Ecmascript E5 does not require a particular
  key ordering for enumeration, a practical implementation must provide
  some ordering guarantees to be compatible with existing code base.  Such
  guarantees include enumerating keys in their insertion order; see the
  section on enumeration for details.  This has a big impact on the viable
  data structure alternatives; the current entry and hash part model is a
  relatively simple approach to satisfy the practical requirements.

Hash part details
-----------------

The hash part maps a key ``K`` to an index ``I`` of the entry part or
indicates that ``K`` does not exist.  The hash part uses a `closed hash
table`__, i.e. the hash table has a fixed size and a certain key has
multiple possible locations in a *probe sequence*.  The current probe
sequence uses a variant of *double hashing*.

__ http://en.wikipedia.org/wiki/Hash_table#Open_addressing

.. note:: The current hash algorithm does not perform especially well,
          and it is future work to make it work better especially with
          high load factors.

The hash part is an array of ``h_size`` ``duk_u32`` values.  Each value
is either an index to the entry part, or one of two markers:

* ``UNUSED``: entry is currently unused
* ``DELETED``: entry has been deleted

Hash table size (``h_size``) is selected relative to the maximum number
of inserted elements ``N`` (equal to ``e_size`` in practice) in two steps:

#. A temporary value ``T`` is selected relative to the number of entries,
   as ``c * N`` where ``c`` is currently about 1.2.

#. ``T`` is rounded upwards to the closest prime from a pre-generated
   list of primes with an approximately fixed prime-to-prime ratio.

  + The list of primes generated by ``genhashsizes.py``, and is encoded
    in a bit packed format, decoded on the fly.  See ``genhashsizes.py``
    for details.
  + The fact that the hash table size is a prime simplifies probe sequence
    handling: it is easy to select probe steps which are guaranteed to
    cover all entries of the hash table.
  + The ratio between successive primes is currently about 1.15.
    As a result, the hash table size is about 1.2-1.4 times larger than
    the maximum number of properties in the entry part.  This implies a
    maximum hash table load factor of about 72-83%.
  + The current minimum prime used is 17.

The probe sequence for a certain key is guaranteed to walk through every
hash table entry, and is generated as follows:

#. The initial hash index is computed directly from the string hash,
   modulo hash table size as: ``I = string_hash % h_size``.

#. The probe step is then selected from a pre-generated table of 32
   probe steps as: ``S = probe_steps[string_hash % 32]``.

   + The probe steps are is guaranteed to be non-zero and relatively prime
     to all precomputed hash table size primes.  See ``genhashsizes.py``.
   + Currently the precomputed steps are small primes which are not present
     in the precomputed hash size primes list.  Technically they don't need
     to be primes (or small), as long as they are relatively prime to all
     possible hash table sizes, i.e. ``gcd(S, h_size)=1``, to guarantee that
     the probe sequence walks through all entries of the hash.

#. The probe sequence is: ``(X + i*S) % h_size`` where i=0,1,...h_size-1.

When looking up an element from the hash table, we walk through the probe
sequence looking at the hash table entries.  If a UNUSED entry is found, the
probe sequence is terminated, and we determine that the entry cannot be in
the hash (and thus, not in the entry part).  If a DELETED entry is found,
we continue with the probe sequence.  This is necessary to handle deletions
correctly.

When inserting an element to the hash table, we must first ensure it does
not already exist.  The probe sequence must be verified up to the first
UNUSED entry (but not beyond).  The element is then inserted to the first
UNUSED *or* DELETED entry.  DELETED entries can thus be reused, but they
can never be marked UNUSED, otherwise probe sequences would be "broken".

When an element is deleted, it is first located by following the probe
sequence, and if found, is then replaced with a DELETED marker.

If the hash part is full, the probe sequence eventually comes back to the
initial entry and is thus in an infinite loop.  An explicit loop check
would be an unnecessary cost: it suffices to ensure there is at least one
UNUSED entry in the hash part.  As the probe sequence is guaranteed to
cover every hash entry, it will eventually hit the UNUSED entry and
terminate.

DELETED entries don't terminate hash probe sequences.  If they did, existing
hash chains could be broken as a side effect of deletions.  Since the hash
must contain at least one UNUSED entry, DELETED entries must be "purged" from
time to time: if all entries were either occupied or marked DELETED, probe
sequences would never terminate.  Currently DELETED entries are only removed
during property allocation resizing, which always rehashes all entries,
purging any DELETED entries as a side effect.  The handling of key insertion
and deletion in the entry part actually guarantees that a rehashing occurs
before the hash part fills up with DELETED entries as follows.

Because all new entries are appended to the existing entry part key array
(deleted entry part keys are marked ``NULL`` but not reused until a resize
happens), the hash part contains exactly ``e_used`` used and DELETED entries
combined, and exactly ``h_size - e_used`` UNUSED entries.  As long as the hash
part is larger than the entry part (``h_size > e_size``) the hash is thus
guaranteed to contain at least one UNUSED entry.  When an insertion is
attempted to a full entry part (``e_used = e_size``), a property allocation
resize is triggered which also resizes and rehashes the hash part, purging
any ``DELETED`` entries.

.. raw:: LaTeX

   \newpage

Array part
==========

Layout
------

The array part simply contains a sequence of tagged values::

  values
  (duk_tval)

  +---------+
  | value 0 |  Represents the array:
  | UNUSED  |    { "0": (value 0), "2": (value 2) }
  | value 2 |
  | UNUSED  |  UNUSED = duk_tval 'undefined unused' value
  | UNUSED  |           (DUK_TVAL_IS_UNDEFINED_UNUSED(tv))
  +---------+

  Here, a_size = 5.

The array part stores all properties whose string key is a *valid
array index*, a canonical string representation of a whole number in
the range [0,0xfffffffe] (discussed in more detail above).  The array
part is *comprehensive*, which means that if an object has an array part,
any string key which is a valid array index *must* reside in the array
part (it can never be in the entry part).  (Because 0xffffffff is not a
valid array index, it is used internally as a convenient "no array index"
marker, ``DUK_HSTRING_NO_ARRAY_INDEX``.)

The array part does not store any property attribute flags: all entries
are implicity assumed to be data properties with the default ``[[Put]]``
property attributes: writable, enumerable, configurable.  This assumption is
true for almost all real world code.  If a property insertion or modification
were to violate this assumption, the entire array part needs to be abandoned
and moved to the entry part to maintain E5 semantics.

All array entries are always reachable from a GC perspective, up to
the allocated size, ``a_size``.  Unused values are marked with the special
"undefined unused" value, set using the
``DUK_TVAL_SET_UNDEFINED_UNUSED`` macro.  Any other entries, including
"undefined actual" values, set using the ``DUK_TVAL_SET_UNDEFINED_ACTUAL``
macro, are considered to be in use, and their corresponding key is
considered to exist in the object, and they are thus visible to enumeration.
In the illustration above, values at indices "0" and "2" are considered used,
so an attempt to enumerate the array part would result in ``["0", "2"]``, also
e.g. ``2 in obj`` would be true while ``3 in obj`` would be false.

Notes:

* The array part is an optimized structure for reading and writing
  array indexed properties efficiently.  It can be used for *any* object,
  not just the Ecmascript ``Array`` object, and Ecmascript ``Array``
  special behaviors are unrelated to the array part's existence.

* A non-\ ``Array`` object with an array part does not get the ``Array``
  related special behaviors (like automatic interaction between array
  indexed elements and the ``length`` property).

* An ``Array`` object may be created without an array part, or may have its
  array part abandoned.  The ``Array`` special behaviors must keep on
  working even if the ``Array`` object has no array part.

* The ``Array`` ``length`` property is stored as an ordinary property in
  the entries part, and has no relation with array part size (``a_size``).

Abandoning the array part
-------------------------

The array entries are assumed to be data properties with default
attributes (writable, configurable, enumerable).  This has the
following implications:

* When a new property with an array index outside the currently allocated
  array part is being added (e.g. as part of a property write), we must
  either:

  #. extend the array allocation to cover the new entry; or
  #. abandon the entire array part, moving all array part entries to the
     entry part.

  The first option may not be viable if the array were to become very
  sparse (e.g. when executing: ``var a = []; a[1000000000] = 1``).

* When a property in the array part would become an accessor property
  (getter/setter) or would need to have incompatible attributes, the entire
  array part must be abandoned.

  Note that the property cannot be stored in the entry part while keeping
  the array part, because the array part is assumed to be comprehensive.

When an array part is abandoned, its entries are all moved into the entries
part as ordinary key-value properties with string keys.  If an array part
is abandoned for a certain object it is currently never reinstated.  The
current implementation performs array abandonment only as part of a property
allocation resize; the need to abandon the array thus triggers a resize with
the side effect of doing key compaction, rehashing, etc.

The enumeration ordering of keys is preserved for existing array index
keys by adding them first to the resized entry part, before non-array-index
keys.  However, key ordering behavior for new array indexed entries after
the resize (which abandons the array part) differs from an object with an
array part: array index keys are appended to the entry part as ordinary keys.
This implementation specific behavior is illustrated below::

  // two example arrays
  var a = [1,2,3]; a.foo = "bar";
  var b = [1,2,3]; b.foo = "bar";

  // force 'a' to abandon array part
  a[1000000] = 4;  // array part abandoned
  a.length = 3;    // array part not reinstated

  // arrays 'a' and 'b' have the same enumeration
  // ordering at this point: [ "0", "1", "2", "foo" ].

  a[4] = 5;
  a[3] = 4;
  b[4] = 5;
  b[3] = 4;

  // enumeration ordering differs here:
  print(Object.keys(a));   // -> 0,1,2,foo,4,3
  print(Object.keys(b));   // -> 0,1,2,3,4,foo

Note that Ecmascript implementation behavior differs greatly when it comes
to "sparse arrays".  For instance, the above example has varying results
with existing Ecmascript implementations::

  // Rhino (Rhino 1.7 release 3 2012 02 13)
  // (behavior matches example above)
  [...]
  0,1,2,foo,4,3
  0,1,2,3,4,foo

  // smjs (JavaScript-C 1.8.5+ 2011-04-16)
  // (enumerates based purely on insertion order
  // for both objects)
  [...]
  0,1,2,foo,4,3
  0,1,2,foo,4,3

  // V8 (nodejs v0.4.12)
  // (enumerates array indices before other keys
  // for both objects)
  print = console.log;
  [...]
  [ '0', '1', '2', '3', '4', 'foo' ]
  [ '0', '1', '2', '3', '4', 'foo' ]

Fast array access
-----------------

The reason why a separate array part exists is to:

* Store normal array structures compactly: normal arrays are dense and
  have default properties
* Provide relatively fast access to array elements: avoid entry or hash
  part lookup
* Avoid string interning of array index keys for numeric indices

Ecmascript array indices are always strings, so conceptually arrays
map string indices of the form "0", "1", etc to arbitrary values.
Non-string keys for property accesses are coerced to strings at run time.
For instance::

  var a = [1,2];
  print(a[0]);    // 0 coerced to "0" before access
  print(a["0"]);  // equivalent access

  var i = 1;
  print(a[i]);    // equivalent to a["1"]

If the property access key happens to be a number which is also a valid
array index (whole number in the range [0,2**32-1[) and the target array
happens to have an array part, we can avoid the string interning and
look up the entry directly.  The same applies to assignments to array
index properties.

This is not trivial to implement in practice because of the prototype
chain, the details of property access algorithms etc.  Currently the
"fast path" behavior applies to a very narrow set of circumstances.
See the following functions in ``duk_hobject_props.c``:

* ``duk_hobject_get_value_u32()``
* ``duk_hobject_get_value_tval()``
* ``duk_hobject_has_property_u32()``
* ``duk_hobject_has_property_tval()``

There is currently no fast path for array writes, which means the key is
temporarily interned for the duration of the array write.  The array write
fast path is a bit tricky: if the element does not already exist, a property
higher up in the prototype chain may block or capture the write, and currently
the prototype chain lookup is only possible with a string key.  See future
work.

Resizing the property allocation
================================

The property allocation resizing algorithm handles: growing and shrinking
of the entry, array, and hash parts; abandoning the array part, key compaction
(elimination of ``NULL`` keys); and rehashing (elimination of ``DELETED``
entries).  Only one resizing algorithm is used; all parts of the property
allocation are always processed during resizing.  Multiple resizing
algorithms would be useful (e.g. to just resize one part, perhaps in-place),
but would increase code size.  The current resize algorithm is
``realloc_props()`` in ``duk_hobject_props.c``.

The property allocation is currently resized e.g. when:

* The entry part runs out during insertion of a new property.

* The array part needs to be extended during insertion of a new
  property.

* The array part needs to be abandoned due to:

  + a property insert which would result in a too sparse array part;
  + a property insert incompatible with the array part assumptions; or
  + a property modification incompatible with the array part assumptions.

* The object is compacted, i.e. its active entry and array part properies
  are counted, and an optimal (small) new size is allocated.

The resizing algorithm:

* Allocates a new memory area for properties (in-place resizing is not
  supported).  This may trigger a garbage colleciton, and may fail.

* If array abandoning is requested, existing array properties are first
  moved into the beginning of the new entry part to keep the enumeration
  ordering identical to that before abandonding (array indices are normally
  enumerated before other entry keys).  The array abandoning process is a
  bit tricky because it requires string interning which may trigger garbage
  collection and may also fail.  Any temporary values must thus be reachable
  and correctly referenced counted for every intern call.

* Existing entry part properties are moved into the new entry part.  Any
  ``NULL`` keys are skipped, so that the entry part keys are "compacted".

* If the new allocation has a hash part, the new entry part keys are
  hashed into the new hash part.  Note that an existing hash part (of
  the current allocation) is irrelevant and is ignored here; in any case,
  the new hash part contains no ``DELETED`` entries.

If the array part is not abandoned, reference counts for the object as a
whole remain constant: the reachable keys and values are exactly the same.
If the array part is abandoned, the newly interned array index string keys
(e.g. ``"0"``) will be newly reachable and need to be incref'd.

Some complications:

* The tricky reachability issues related to array abandoning are handled by
  using the current thread's value stack as a place to store temporaries;
  the value stack has an existing process for cleanup if an error occurs.
  This is not the whole story, though; see code for details.

* The allocation calls required during resizing (for the new memory area,
  string interning, and value stack resizing) may cause a garbage collection.
  The garbage collection may attempt to resize any object as part of an
  "emergency GC" compaction.  This needs to be prevented for the current
  object (or in general, for any object being concurrently resized).

  The current solution is to use the ``heap->mark_and_sweep_base_flags``
  mechanism to prevent finalizers from running (= prevents attempts to add,
  remove, or modify properties in the middle of a resize) and to prevent
  object compaction (so that a certain object won't be resized when it is
  already being resized).

Enumeration
===========

Enumeration poses a lot of problems for implementing the Ecmascript
object/array semantics efficiently.

Below, the relevant parts of the specification are first discussed
(and quoted for easy reference), followed by some useful additional
requirements and features.  Some implementation pitfalls are then
discussed.  Finally, the current enumeration mechanism is discussed
in some detail.

The current implementation can be found in ``duk_hobject_enum.c``.

Ecmascript specification requirements
-------------------------------------

E5 Section 12.6.4: "The for-in statement" contains the main requirements
for enumeration in the E5 specification:

* The mechanics and order of enumerating the properties [...] is
  not specified.

* Properties of the object being enumerated may be deleted during
  enumeration. If a property that has not yet been visited during
  enumeration is deleted, then it will not be visited.

* If new properties are added to the object being enumerated during
  enumeration, the newly added properties are not guaranteed to be
  visited in the active enumeration.

* Enumerating the properties of an object includes enumerating
  properties of its prototype, and the prototype of the prototype,
  and so on, recursively; but a property of a prototype is not
  enumerated if it is "shadowed" because some previous object in
  the prototype chain has a property with the same name.

E5 Section 15.2.3.7: "Object.defineProperties ( O, Properties )"
requires that when multiple properties are defined with
``Object.defineProperties()``, the order should be kept:

* If an implementation defines a specific order of enumeration
  for the for-in statement, that same enumeration order must be
  used to order the list elements in step 3 of this algorithm.

E5 Section 15.2.3.14: "Object.keys ( O )" requires that the "for-in"
enumeration order should also be used for ``Object.keys()``:

* If an implementation defines a specific order of enumeration
  for the for-in statement, that same enumeration order must be
  used in step 5 of this algorithm.

Practical enumeration requirements
----------------------------------

The E5 requirements for enumeration are rather loose; for instance, there
is no requirement that object keys are enumerated in their insertion order
or even that array indexes are enumerated in an ascending order.  However,
real world code sometimes makes such assumptions.

For instance, it is a common idiom to assume that the following works::

  var a = [ "foo", "bar", "quux" ];
  for (var i in a) {
    print(i, a[i]);
  }

Many programmers expect this to print::

  0 foo
  1 bar
  2 quux

where it might just as well, while being fully E5 compliant, print::

  2 quux
  0 foo
  1 bar

Similarly, much existing code assumes that properties are enumerated
in the order they were inserted.  See, for instance:

* http://code.google.com/p/chromium/issues/detail?id=2605

* http://ejohn.org/blog/javascript-in-chrome/

  "However, specification is quite different from implementation.
  All modern implementations of ECMAScript iterate through object
  properties in the order in which they were defined. Because of
  this the Chrome team has deemed this to be a bug and will be fixing it."

* ``hobject-enumeration.txt`` for practical testing results with
  actual implementations.

We impose the following additional requirements for compatibility:

* Non-array-index keys should be enumerated in their insertion order.

* The keys for ``Array`` elements should be enumerated in an
  ascending order, and before non-array-index keys.

  + This is currently provided for all objects with an array part.
    Ecmascript ``Array`` instances should thus always have an array
    part (at least when they are created).
  + If an object has an array part which is abandoned, e.g. because
    the array becomes too sparse, the enumeration ordering reverts
    to enumerating entries in insertion order (regardless of whether
    the property is a valid array index or not).

* All keys of a certain object should be enumerated (including both
  array index and non-array-index keys) before proceeding to the
  prototype.  Keys already enumerated must not be repeated during
  enumeration even if they occur again in the prototype chain.

* If an entry is deleted during enumeration before it has appeared
  in the enumeration sequence, it must not turn up later in the
  enumeration.

* A certain key must never appear twice in the enumeration sequence,
  despite any mutation.

* A key which was present during the "initialization" of the enumeration
  (before the first key was enumerated) must not be omitted from the
  enumeration sequence, if they are not deleted during enumeration
  (before they have appeared in the enumeration sequence).

Note the following *non-requirements*:

* New entries added during enumeration are not required to show up
  during the enumeration in progress.

  + The current implementation will *never* enumerate such keys.
    This is not desirable as such, but is a side effect of the
    (simplistic) implementation strategy.

  + The same behavior seems to apply to smjs, Rhino, and V8 at the
    time of writing.

Implementation issues
---------------------

It is tempting to implement enumeration with sort of enumerator state
which maintains some iteration pointers or indices to the target object
and steps through object properties and the (immutable) prototype chain
on request.  However, this approach has many practical difficulties:

* Object mutation may cause the internal structure of the target object
  (or any object in its prototype chain) to change.

  + This poses a problem for any approach based on maintaining an index
    to the array/entry part, as an index may be invalidated by internal 
    data structure maintenance such as compaction of keys.

  + This problem can be avoided if the object is "frozen" for enumeration,
    but this requires awkward book-keeping, which must work even if errors
    are thrown, threads yield (and perhaps never resume, or are garbage
    collected) etc.

* Any keys may be deleted during enumeration.

  + This poses a problem for any approach based on maintaining a key
    based state, e.g. "current key".  The key in question may be
    deleted; how can one then find the next key in the sequence?

Current enumeration mechanism
-----------------------------

The current enumeration approach is based on creating an internal
enumeration object (enumerator) when enumeration is initialized, before
any keys are needed.  The entire list of enumerated keys is generated
during initialization and stored in the enumerator.  This avoids any
issues with mutation, because no user code runs while the enumerator
is being initialized.  This step is memory intensive; in particular,
all enumerated array index keys are interned.

The enumerator is then used to generate a sequence of keys on demand.
When a new key is requested, the enumerator advances to the next key
in its internal sequence.  The key is then checked to ensure it has
not been deleted during enumeration; if it has, we skip and try the
next key in the sequence.

Note that the key may be deleted and still found in an ancestor (and
should still be enumerated)::

  function F() {};
  F.prototype = { "foo": "inherited" };

  var a = new F();
  a.bar = "skip";
  a.foo = "own";

  // enumeration order: "bar", "foo"
  for (var i in a) {
    delete a.foo;  // only affects 'a', not F.prototype
    print(i, a[i]);
  }

This is expected to print::

  bar skip
  foo inherited

However, behavior seems to differ across implementations: V8 and Smjs
work as above, while Rhino does not enumerate ``"foo"``.  Rhino *will*
enumerate ``foo`` if the for-loop is executed twice.  There are other
corner cases in enumeration too, see test cases.

Suppose the enumeration target is::

  x = [ "foo", "bar", "quux" ];
  x.foo = "bar";

After initialization, the internal enumerator object would contain
the following::

  e = {
    // internal control properties first
    "_target": (target object),
    "_next": (numeric index),

    // followed by enumeration keys
    // (as properties, not array entries)
    "0": true,
    "1": true,
    "2": true,
    "foo": true
  }

The enumerator object takes advantage of two features:

#. Keys inserted into an object maintain their order in the entry part
   (even during resizes).  Thus, we can insert keys into the enumerator
   and trust that their order is maintained.  The entry part is always
   gap-free, i.e. there are no NULL keys in the sequence.

#. Inserting enumerated keys as properties instead of array entries
   allows duplicate keys to be handled correctly.  Duplicate keys
   may occur when the prototype chain is walked.  The first occurrence
   is recorded in its correct position, and any later occurrences are
   ignored.

The ``_next`` internal property is a numeric index which indicates
where to find the next potential key.  It is an index to the *entry
part* of the enumerator, and it's initialized to the value 2 because
positions 0 and 1 are taken by ``_target`` and ``_next`` properties,
respectively.  Entry index 2 in the example above is the "0" key.  This
needs to be managed carefully as the indexing model depends on the entry
part having a very specific, unchanged form.

Note that the enumerator is not mutated after its creation, so this
entry part index approach is reliable.  It is reliable even if the
enumerator is resized, as long as properties are not deleted; that
would cause problems in a resize, when the entry part was compacted.

Notes
-----

The current implementation has some nice qualities:

* It is very simple and robust, and avoids any issues with mutation
  (except that keys added during mutation are never enumerated, which
  is not nice but a common feature in other implementations, too).

* It has small code space.

* It has minimal impact on anything else, e.g. it requires no co-operation
  from the object, such as avoiding key compaction until enumeration is
  over.

However, it has many drawbacks:

* It has a relatively large memory footprint for the enumerator.  Because
  the keys are stored as key-value properties (not as array entries), each
  enumerated key takes about 13 bytes on a typical 32-bit architecture
  (4 bytes for key, 8 bytes for value, 1 byte for flags).

  + This footprint could be reduced somewhat by using the property-based
    approach to generate the enumeration sequence (eliminating duplicate
    keys etc), and then converting that to an array; array entries typically
    take 8 bytes.  But this would temporarily increase memory footprint
    even more.

* Numeric key indices of an array part or the virtual numeric key indices
  of a ``String`` object are interned and are reachable simultaneously
  during enumeration.

* Execution of program code stops while the enumerator is initially
  created.  This probably has little impact in most cases, but it might
  be an issue if a very large object is being enumerated (consider for
  instance enumerating a very large array).

As a concrete illustration of some of the drawbacks, consider the
following::

  var a = [... large array of 1 million elements ...];
  for (var i in a) {
    print(i, a[i]);
  }

The enumerator created in this example would contain a million interned
keys for "0", "1", and so on.  *All* of these keys would remain reachable
for the entire duration of the enumeration.  The following code would
perform *much* better (and would be more portable, as it makes no
assumptions on enumeration order)::

  var a = [... large array of 1 million elements ...];
  var n = a.length;
  for (var i = 0; i < n; i++) {
    print(i, a[i]);
  }

This problem could be alleviated a bit by deleting any already-enumerated
keys from the enumerator as enumeration proceeds.  Care would then have
to be taken to avoid any possibility of a resize of the object to avoid
mixing up the key order (due to key compaction).  This might not be easy
to arrange, if GC is allowed to opportunistically compact objects
(at least in "emergency GC" mode).  A much easier approach would be to
replace enumeration entry keys with a fixed string (like the empty string)
instead of deleting them.  This would still free the string resources but
keep the object property key list intact.  However, it would lead to
duplicate keys in the entry part, which needs careful consideration to work
correctly.

Internal properties
===================

Duktape implements E5 internal properties in differing ways, depending
on the property in question:

* concretely stored internal properties
* ``duk_hobject`` header flags
* ``duk_hobject`` structure fields (only internal prototype currently)
* implicit behaviors in specification algorithms based on e.g.
  object flags, type, or class

The current approach for storing internal properties which are never
visible to program code and never overlap with externally visible named
properties is simple: since legitimate keys all encode into valid UTF-8
sequences (to be fully exact, valid CESU-8 sequences) in memory, internal
properties are prefixed with an invalid UTF-8 sequence which user code
cannot generate and thus cannot access.  The current prefix is a single
``0xff`` byte.  The prefix is denoted with an underscore in this document;
e.g. ``_map`` would be represented as the byte sequence: ``0xff`` ``'m'``
``'a'`` ``'p'`` in memory.

To avoid complications:

* Internal properties MUST NOT be enumerable

* Internal properties MUST NOT be visible in any other way either, e.g.
  through ``Object.getOwnPropertyNames()`` which outputs also
  non-enumerable properties

* User Ecmascript code should not be given references to internal strings,
  i.e. strings other than valid UTF-8/CESU-8 encodings

The current C bindings *do allow* internal properties to be accessed as C
code cannot be easily restrained in any case.  However, it's not recommended
to access internal properties from C code as the internal properties may
change between versions.

Internal property names use a bit of shorthand and often the same internal
key is reused in many contexts.  This is simply to save a bit of memory with
minimal impact on readability.

The following table summarizes the internal properties specified in E5,
and how they are mapped to the ``duk_hobject`` implementation.  The
double brackets are omitted from the specification property names
(e.g. ``[[Class]]`` is listed as "Class").

+-------------------+------------------------------------------------------+
| Property          | Implementation                                       |
+===================+======================================================+
| Prototype         | ``duk_hobject`` struct ``prototype`` field.          |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| Class             | ``duk_hobject`` flags field, encoded as a number.    |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| Extensible        | ``duk_hobject`` flag ``DUK_HOBJECT_FLAG_EXTENSIBLE``.|
|                   |                                                      |
+-------------------+------------------------------------------------------+
| Get               | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| GetOwnProperty    | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| GetProperty       | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| Put               | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| CanPut            | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| HasProperty       | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| Delete            | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| DefaultValue      | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| DefineOwnProperty | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| PrimitiveValue    | Internal property ``_value``.                        |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| Construct         | Not stored, implicit in algorithms.  ``duk_hobject`` |
|                   | flag ``DUK_HOBJECT_FLAG_CONSTRUCTABLE`` indicates    |
|                   | whether the object is a constructor, i.e.            |
|                   | conceptually implements the internal                 |
|                   | ``[[Construct]]`` function.  Note that all callable  |
|                   | objects are not constructable.                       |
+-------------------+------------------------------------------------------+
| Call              | Not stored, implicit in algorithms.  ``duk_hobject`` |
|                   | macro ``DUK_HOBJECT_IS_CALLABLE`` determines whether |
|                   | the object is callable, i.e. conceptually implements |
|                   | the internal ``[[Call]]`` function.  The check is    |
|                   | made using (other) object type flags, there is no    |
|                   | dedicated "callable" flag.                           |
+-------------------+------------------------------------------------------+
| HasInstance       | Not stored, implicit in algorithms.                  |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| Scope             | Internal properties ``_lexenv`` and ``_varenv``.     |
|                   | (Unlike E5, global and eval code are also compiled   |
|                   | into functions, hence two scope fields are needed.)  |
+-------------------+------------------------------------------------------+
| FormalParameters  | Internal property ``_formals``.                      |
|                   |                                                      |
+-------------------+------------------------------------------------------+
| Code              | An Ecmascript function (``duk_hcompiledfunction``)   |
|                   | has a pointer to compiled bytecode and associated    |
|                   | data (such as constants), see                        |
|                   | ``duk_hcompiledfunction.h``.                         |
|                   | A C function (``duk_hnativefunction``) has a pointer |
|                   | to a C function and some related control data, see   |
|                   | ``duk_hnativefunction.h``.                           |
+-------------------+------------------------------------------------------+
| TargetFunction    | ``duk_hobject`` flag ``DUK_HOBJECT_FLAG_BOUND`` is   |
|                   | set, and the internal property ``_target`` is set    |
|                   | to the target function.                              |
+-------------------+------------------------------------------------------+
| BoundThis         | ``duk_hobject`` flag ``DUK_HOBJECT_FLAG_BOUND`` is   |
|                   | set and the internal property ``_this`` is set       |
|                   | to the ``this`` binding.                             |
+-------------------+------------------------------------------------------+
| BoundArguments    | ``duk_hobject`` flag ``DUK_HOBJECT_FLAG_BOUND`` is   |
|                   | set and the internal property ``_args`` is set       |
|                   | to a list of bound arguments.                        |
+-------------------+------------------------------------------------------+
| Match             | Not stored, implicit in algorithms.  Object type     |
|                   | (class number is DUK_HOBJECT_CLASS_REGEXP) determines|
|                   | whether ``[[Match]]`` is conceptually supported.     |
|                   | The compiled regexp and its flags are stored as the  |
|                   | ``_bytecode`` internal property, whose value is an   |
|                   | internal string.                                     |
+-------------------+------------------------------------------------------+
| ParameterMap      | Internal property ``_map``.                          |
|                   |                                                      |
+-------------------+------------------------------------------------------+


Special behavior and virtual properties
=======================================

Terminology
-----------

The E5 specification defines default property access algorithms like
``[[GetProperty]]`` and ``[[DefineOwnProperty]]`` in E5 Section 8.12.
Some objects have behavior which differs from default behavior; we call
these *special properties* (or properties with *special behavior*), as
opposed to *normal properties* (or properties with *normal behavior* or
*default behavior*).

Conceptually each object has a number of algorithms "stored" in its
internal properties (E5 Section 8.6.2), including all the property access
algorithms.  The current implementation of property access is completely
different: there are fixed algorithms for property access, which change
their behavior based on object type and flags.  The special behaviors are
thus "inlined" into a single algorithm.

From a purely implementation viewpoint some properties are stored in a data
structure as concrete key-value pairs, while others are computed
on-the-fly.  The former are called *concrete properties* and the latter
*virtual properties*.  Whether a property is concrete or virtual should
have no externally visible impact with respect to compliance. 
Note that these two concepts ("being special" and "being virtual") are
indepedent: a special property can be implemented with a concrete property
storing its value, and a normal property can be implemented as a virtual
property.

Special behaviors in E5 specification
-------------------------------------

Special behaviors are discussed at least in the following places in the E5
specification (page numbers refer to page numbers on the page contents, not
the "PDF page number").

Section 8.6.2, pages 32-33 summarizes special behavior and refers to:

* Array objects: ``[[DefineOwnProperty]]``, E5 Section 15.4.5.1

* String objects: ``[[GetOwnProperty]]``, E5 Section 15.5.5.2

* Arguments objects: ``[[Get]]``, ``[[GetOwnProperty]]``,
  ``[DefineOwnProperty]]``, ``[[Delete]]``, E5 Section 10.6
  (the special behavior of a non-strict arguments object is pretty
  intricate and is discussed separately in ``arguments-object.txt``)

* Function objects: ``[[Get]]``, E5 Section 15.3

Special behavior for ``[[Get]]``:

* The ``arguments`` object: E5 Section 10.5

  + If ``arguments.caller`` has a value, which is a strict function object,
    the ``[[Get]]`` operation fails after standard lookup is complete.
  + Note that the special behavior occurs at the level of ``[[Get]]`` and
    is *not* visible through property descriptors, e.g. through
    ``[[GetProperty]]`` or ``[[GetOwnProperty]]``.
  + Special behavior only applies to non-strict arguments objects.

* The ``Function`` object: E5 Section 15.3.5.4

  + Same special behavior for ``caller`` property as for ``arguments``
    object.

Special behavior for ``[[GetOwnProperty]]``:

* ``String`` object array-index properties: E5 Section 15.5.5.2

  + Covers properties which are valid array indexes as specified in E5
    Section 15.4, i.e. P for which ``ToString(ToUint32(P)) == P`` and
    ``ToUint32(P) != 0xffffffff``.

  + Ecmascript E5.1 extended behavior to all number-like properties,
    and thus allows strings longer than 4G characters.

* ``Array`` ``length`` property: E5 Section 15.4.5

  + May be implemented as a concrete property or as a virtual property.
    Currently implemented as a concrete property.

* The ``arguments`` object: E5 Section 10.5

  + The ``[[Value]]`` of a property descriptor may be overridden for
    "magically bound" properties (some numeric indices).
  + Special behavior only applies to non-strict arguments objects.

Special behavior for ``[[DefineOwnProperty]]``:

* ``Array`` ``length`` property: E5 Section 15.4.5.1

  + Has side effects on array elements (deleting elements above newly
    written length).

* ``Array`` index properties: E5 Section 15.4.5.1

  + Has the side effect of automatically updating array ``length``.

* The ``arguments`` object: E5 Section 10.6

  + Automatic interaction with "magically bound" variables (some
    numeric indices).  May also remove magic binding.
  + Special behavior only applies to non-strict arguments objects.

Special behavior for ``[[Delete]]``:

* The ``arguments`` object: E5 Section 10.6

  + Automatic interaction with "magically" bound variables (some
    numeric indices), may remove magic binding.
  + Special behavior only applies to non-strict arguments objects.

When implementing special or virtual properties, property attributes must
be respected normally.  Special or virtual properties may have specific
initial attributes, but these are not fixed and may be changed later by
user code.  The *only* properties which are "truly fixed" are:

* Non-configurable, non-writable data properties
* Non-configurable accessor properties

In particular, a data property which is non-configurable but writable
*can* be changed to non-writable (see E5 Section, step 10).  The property
cannot be changed back to writable after that.  This has the practical
implication that only "truly fixed" properties can be easily implemented
as stateless virtual properties.

Summary of special properties
-----------------------------

The following table summarizes special properties defined in the E5
specification, along with their (initial) property attributes in the
columns W(ritable), E(numerable), and C(onfigurable): 
``y`` means "true", ``n`` means "false", ``a`` means "any":

+------------+------------+---+---+---+-----------------------------------+
| Object     | Property   | W | E | C | Notes                             |
+============+============+===+===+===+===================================+
| ``Array``  | ``length`` | y | n | n | Write may affect array elements   |
| instance   |            |   |   |   | (indices above new length are     |
|            |            |   |   |   | deleted)                          |
+------------+------------+---+---+---+-----------------------------------+
| ``Array``  | array      | a | a | a | Write may affect array            |
| instance   | indices    |   |   |   | ``length`` (if new index is above |
|            |            |   |   |   | existing length)                  |
+------------+------------+---+---+---+-----------------------------------+
| ``String`` | ``length`` | n | n | n | No special behavior as such, but  |
| instance   |            |   |   |   | easy to implement as a virtual    |
|            |            |   |   |   | property because not writable     |
|            |            |   |   |   | or configurable.                  |
+------------+------------+---+---+---+-----------------------------------+
| ``String`` | array      | n | y | n | No special behavior as such, but  |
| instance   | indices    |   |   |   | maps individual characters to     |
|            | inside     |   |   |   | indicates; affects enumeration.   |
|            | string     |   |   |   |                                   |
|            | length     |   |   |   |                                   |
+------------+------------+---+---+---+-----------------------------------+
| plain      | ``length`` | n | n | n | See notes below.                  |
| string     |            |   |   |   |                                   |
| value      |            |   |   |   |                                   |
+------------+------------+---+---+---+-----------------------------------+
| plain      | array      | n | y | n | See notes below.                  |
| string     | indices    |   |   |   |                                   |
| value      | inside     |   |   |   |                                   |
|            | string     |   |   |   |                                   |
|            | length     |   |   |   |                                   |
+------------+------------+---+---+---+-----------------------------------+
| Arguments  | some       | y | y | y | Some numeric indices of an        |
| object,    | numeric    |   |   |   | arguments object "magically bind" |
| non-strict | indices    |   |   |   | to formal arguments.  Only affects|
|            |            |   |   |   | a non-strict arguments object.    |
+------------+------------+---+---+---+-----------------------------------+
| Arguments  | ``caller`` | a | a | a | If *value* of ``caller`` property |
| object,    |            |   |   |   | is a strict function, ``[[Get]]`` |
| non-strict |            |   |   |   | fails (but ``[[GetOwnProperty]]`` |
|            |            |   |   |   | does not!).  Only affects a       |
|            |            |   |   |   | non-strict arguments object.      |
+------------+------------+---+---+---+-----------------------------------+

Notes:

* The special properties for ``String`` instances (which are objects) also
  apply in practice to plain strings, because properties of plain strings
  can also be accessed (the string is automatically promoted to a temporary
  object; the implementation handles this without an actual temporary object
  being created).

* The ``caller`` property of a non-strict arguments object is curious: it has
  special behavior but no such property is established for non-strict argument
  objects.  (This is why its property attributes are listed as "any" above.)

* The only special properties which are easy to implement as fully virtual,
  stateless properties are the ``String`` instance ``length`` and
  array index properties, because they are non-configurable and non-writable.
  They are enumerable, though, which must be taken into account in enumeration.

* The array ``length`` property has an initial value which is a valid array
  length (32-bit unsigned integer).  The special behavior of the property
  ensures that whatever values are assigned to it, they are either rejected
  or coerced into a valid array length (32-bit unsigned integer).

Implementation of special properties
------------------------------------

The following table summarizes the implementation of special properties at
the moment.

+------------+------------+-----------------------------------------------+
| Object     | Property   | Description                                   |
+============+============+===============================================+
| ``Array``  | ``length`` | Stored as a concrete property.                |
| instance   |            | ``DUK_HOBJECT_FLAG_SPECIAL_ARRAY`` enables    |
|            |            | special behavior in:                          |
|            |            | ``duk_hobject_put_value()``,                  |
|            |            | ``duk_hobject_object_define_property()``.     |
+------------+------------+-----------------------------------------------+
| ``Array``  | array      | Stored as conrete properties in array part    |
| instance   | indices    | or entry part (if array part abandoned).      |
|            |            | ``DUK_HOBJECT_FLAG_SPECIAL_ARRAY`` enables    |
|            |            | special behavior in:                          |
|            |            | ``duk_hobject_put_value()``,                  |
|            |            | ``duk_hobject_object_define_property()``.     |
+------------+------------+-----------------------------------------------+
| ``String`` | ``length`` | Virtual property computed from the string     |
| instance   |            | length of the internal ``_value`` property.   |
|            |            | ``DUK_HOBJECT_FLAG_SPECIAL_STRINGOBJ`` enables|
|            |            | special behavior in:                          |
|            |            | ``get_own_property_desc()``.                  |
+------------+------------+-----------------------------------------------+
| ``String`` | array      | Virtual properties computed by looking up     |
| instance   | indices    | characters of the internal ``_value``         |
|            | inside     | property.                                     |
|            | string     | ``DUK_HOBJECT_FLAG_SPECIAL_STRINGOBJ`` enables|
|            | length     | special behavior in:                          |
|            |            | ``get_own_property_desc()``.                  |
+------------+------------+-----------------------------------------------+
| plain      | ``length`` | Special handling in property access code,     |
| string     |            | return string character length without        |
| value      |            | promoting the plain string value to a         |
|            |            | temporary ``String`` instance.                |
+------------+------------+-----------------------------------------------+
| plain      | array      | Special handling in property access code,     |
| string     | indices    | return individual character value without     |
| value      | inside     | promoting the plain string value to a         |
|            | string     | temporary ``String`` instance.                |
|            | length     |                                               |
+------------+------------+-----------------------------------------------+
| Arguments  | some       | Stored as concrete property values.           |
| object,    | numeric    | ``DUK_HOBJECT_FLAG_SPECIAL_ARGUMENTS`` enables|
| non-strict | indices    | special behavior in:                          |
|            |            | ``get_own_property_desc()``,                  |
|            |            | ``duk_hobject_get_value()``,                  |
|            |            | ``duk_hobject_put_value()``,                  |
|            |            | ``duk_hobject_delete_property()``,            |
|            |            | ``duk_hobject_object_define_property()``.     |
+------------+------------+-----------------------------------------------+
| Arguments  | ``caller`` | Stored as a concrete property.                |
| object,    |            | ``DUK_HOBJECT_FLAG_SPECIAL_ARGUMENTS`` enables|
| non-strict |            | special behavior in:                          |
|            |            | ``duk_hobject_get_value()``.                  |
|            |            | This special behavior only affects            |
|            |            | ``[[Get]]``, it is not visible through e.g.   |
|            |            | property descriptors or                       |
|            |            | ``[[GetOwnProperty]]``.                       |
+------------+------------+-----------------------------------------------+

Notes:

* The only virtual properties are ``String`` object ``length`` and array
  index properties.  These are easy to implement as virtual properties
  because they are non-configurable and non-writable.  However, they *are*
  enumerable which affects enumeration handling.

* If array ``length`` becomes non-writable, the special behavior ensures
  no elements above the specified length can ever be inserted.  The array
  part could thus be compacted without risk of it being extended afterwards.

Notes on array length and array indices
---------------------------------------

The special array semantics only apply to valid array indices.  Nothing
prevents user code from writing to numeric array indices higher than the
maximum valid array index, but such writes will get no special behavior::

  var x = [];
  x["4294967294"] = 1;    // has array semantics, updates length
  print(x.length);        // length is 4294967295

  x["9999999999"] = 2;    // no array semantics
  print(x.length);        // length is still 4294967295

  print(x[4294967294]);   // coerced to string -> value "1"
  print(x[9999999999]);   // like above -> value "2"

  print(Object.keys(x));  // -> ["4294967294", "999999999"]

  x.length = 0;           // deletes valid array indices
  print(x[4294967294]);   // undefined
  print(x[9999999999]);   // value still "2", not auto-deleted

Note that assigning an ``undefined`` value extends the array length, as
it is a valid value::

  var x = [];
  print(x.length);        // -> 0
  x[10] = undefined;
  print(x.length);        // -> 11

  print(Object.keys(x));  // -> [ "10" ]

Internal objects
================

The following internal objects are currently used:

* Function templates which are "instantiated" into concrete closures
* A declarative environment record
* An object environment record
* Function formals name list
* Function variable map

Internal objects don't always need Ecmascript properties like:

* Enumeration order
* Property attributes
* Prototype chain

The current implementation does not take advantage of these: internal
objects are handled just like Ecmascript objects.

Function instances
==================

The creation of function instances is described in E5 Section 13.2.
Each function instance (each closure created from a function
expression or declaration) has the following properties:

* ``length``

* ``prototype``: points to a fresh object which has a ``constructor``
  property pointing back to the function

* ``caller``: thrower (strict functions only)

* ``arguments``: thrower (strict functions only)

There is considerable variance in practical implementations:

* smjs::

    // the "name" property is non-standard; "arguments" and "caller" are
    // present for a non-strict function

    js> f = function foo() {}
    (function () {})
    js> Object.getOwnPropertyNames(f)
    ["prototype", "length", "name", "arguments", "caller"]

    // for strict mode, the same properties are present.

    js> f = function foo() { "use strict"; }
    (function foo() {"use strict";})
    js> Object.getOwnPropertyNames(f);
    ["prototype", "length", "name", "arguments", "caller"]

    // the "name" property contains the function expression name

    js> f.name
    "foo"

    // "name" is non-writable, non-configurable (and non-enumerable)
    // -> works as a reliable "internal" property too

    js> Object.getOwnPropertyDescriptor(f, 'name')
    ({configurable:false, enumerable:false, value:"foo", writable:false})

* nodejs (v8)::

    // "name" is non-standard; "arguments" and "caller" are present
    // for even a non-strict function

    > f = function foo() {}
    [Function: foo]
    > Object.getOwnPropertyNames(f)
    [ 'length',
      'caller',
      'arguments',
      'name',
      'prototype' ]
    > f.name
    'foo'

    // strict mode is the same

    > f = function foo() { "use strict"; }
    [Function: foo]
    > Object.getOwnPropertyNames(f)
    [ 'name',
      'length',
      'arguments',
      'prototype',
      'caller' ]

    // 'name' is writable but not configurable/enumerable

    > f.name
    'foo'
    > Object.getOwnPropertyDescriptor(f, 'name')
    { value: 'foo',
      writable: true,
      enumerable: false,
      configurable: false }

* rhino::

    // "name" is non-standard, "arity" is non-standard, "arguments"
    // is present (but "caller" is not)

    js> f = function foo() {}
    [...]
    js> Object.getOwnPropertyNames(f)
    arguments,prototype,name,arity,length

    // name is non-writable, non-enumerable, non-configurable

    js> pd = Object.getOwnPropertyDescriptor(f, 'name')
    [object Object]
    js> pd.writable
    false
    js> pd.enumerable
    false
    js> pd.configurable
    false

    // strict mode functions are similar

See ``function-objects.txt`` for more discussion.

Built-in functions
------------------

The properties of built-in functions are a special case, because
they are not created with the algorithm in E5 Section 13.2;
instead, their properties are described explicitly in E5 Section 15.

There is considerable variance between implementations on what
properties built-in functions get.

Design notes and future work
============================

ES6 Proxy objects or Lua-like metatables
----------------------------------------

It would be nice to have a Lua metatable like mechanism for creating
custom object behavior extensions and full object virtualization,
see http://www.lua.org/pil/13.html for a description of Lua metatables.
There is a similar mechanism in the Ecmascript 6 draft called "Proxy
object":

* https://people.mozilla.org/~jorendorff/es6-draft.html#sec-proxy-object-internal-methods-and-internal-slots

The ES6 Proxy object is of course a natural target for implementation,
but it's not clear what the underlying mechanism should be: should the
Proxy object mechanism be the internal mechanism, or should there be some
other implementation specific mechanism underneath which is used to provide
the Proxy object implementation but also provide non-standard additional
features?

Nice-to-have features:

* Sufficient for creating arbitrary "host objects"

* Sufficient for providing array-like access to byte buffers

* Allow "full virtualization" of E5 semantics

Some notes:

* Should interact reasonably with the E5 object model, e.g. property
  descriptors.

* Should metatable behavior only affect non-existent properties (as in
  Lua)?  To apply it to all properties, simply use an empty table.

* May require raw access functions for dealing with the underlying
  properties.

* What's the best level for capturing operations?

  a. Concrete, exposed operations like getprop, putprop, hasprop, delprop,
     Object.getOwnPropertyDescriptor(), Object.defineProperty(), etc?

  b. Specification functions like ``[[GetOwnProperty]]``,
     ``[[DefineProperty]]`` etc?

* In addition, should cover:

  + Enumeration
  + Getting object ``[[Class]]``
  + Garbage collection => finalizer

ES6 features
------------

There are many ES6 features which may need changes to the basic object model.

For instance, there are keyed collections:

* https://people.mozilla.org/~jorendorff/es6-draft.html#sec-keyed-collection

The ``Map`` object provides an arbitrary collection of key/value pairs,
where keys and values can be arbitrary Ecmascript objects.  This is very
useful compared to the standard Ecmascript object whose keys can only be
strings.

How to implement this?  One could extend the basic model to provide enough
functionality to implement ``Map`` values.  One could also implement them
entirely separate from the basic object model.  The trouble is, the ``Map``
keys and values must be GC reachable so they have to reside either in basic
objects or there needs to be additional native structure for them.

Array-like access to underlying byte buffers
--------------------------------------------

There are various proposals for typed access to an underlying buffer.
For instance:

* http://www.khronos.org/registry/typedarray/specs/latest/
* http://nodejs.org/docs/v0.4.7/api/buffers.html

There is no implementation for any kind of typed arrays which behave
like normal arrays now.  There probably should be, as such an interface
provides a good match for low level memory manipulation in parsing,
serializing, etc.

Notes:

* The interface model provided does not need to implement any of the full
  APIs above, but must provide the core "virtual property" behavior
  somehow.  Setup wrappers etc can be added as ordinary code.

* The underlying should be bindable to either a "buffer" object (which can
  either be fixed or reallocatable, and its allocation may move in memory
  from time to time) or a "raw" pointer (and length) for general C code
  interfacing.

  + Allowing a caller to specify a length is nice to provide some memory
    protection guarantees.  The desired behavior could either be silent
    errors or error throwing (RangeError?) for out-of-bound accesses.

* Could be implemented as virtual properties as for ``String`` objects,
  but writable.

  + Keys should probably be non-enumerable for practical reasons.

* Could also be implemented using "metatables", with some performance hit.

WindowProxy
-----------

The HTML5 WindowProxy object seems to require behavior outside E5.

See: http://www.w3.org/TR/html5/browsers.html#windowproxy.

Script origin for security checks
---------------------------------

HTML5 web storage, as an example, uses 'script origin' to do required
security checks on property access.

Possible implementation e.g. through: a metatable-like mechanism for
capturing accesses, and some way to introspect the current caller or
call chain to check script origin.

It would be nice to be able to determine an origin for every function,
mapped to an Ecmascript object with a bunch of propeties like URI,
load time.

Alternatives to current entry part
----------------------------------

Are there better alternatives to the current entry part model?

The pointer computations have a run-time performance cost and also a
code footprint cost.  Is it worth it?  This depends on the number of
objects.  For instance, if there are 1000 objects and a pointer to the
array part and hash part are added to ``duk_hobject``, this would have
a footprint of 8000 extra bytes on a 32-bit platform.

Hash algorithm notes
--------------------

Some hash algorithm goals:

* Minimal memory allocation
* High load factor (minimizes memory use)
* Small code space

Closed hashing (open addressing) provides fixed allocation, but requires a
"probe sequence" to deal with hash collisions.  Options for dealing with
collisions include:

* http://en.wikipedia.org/wiki/Linear_probing
* http://en.wikipedia.org/wiki/Quadratic_probing
* http://en.wikipedia.org/wiki/Double_hashing
* http://en.wikipedia.org/wiki/Cuckoo_hashing

Notes on current solution:

* Linear probing is more cache efficient but requires a lower load (= higher
  allocated size relative to used size) to avoid 'clustering' issues.
  Current approach prefers compact object size over cache efficiency.

* Slightly better probe would be: ``probe_steps[(hash >> 16) % 32]``.
  This could possibly correlate less with the initial hash value, but
  requires an extra shift operation.  Most likely not worth it.

* The current algorithm is not very good, as it requires the load factor
  to be relatively low (around 70-80%) to be efficient.  Much better
  results are possible.  This is definite future work.

Property key ordering
---------------------

Because of the practical enumeration requirements, a data structure for
object keys must both maintain keys in their insertion order, and also
support some form of indexed access (e.g. a hash table) to work efficiently
with large objects.

One such approach is to use an "ordered hash table" which maintains
entries both in the hash structure and a linked list for order.

The current approach is to maintain a simple, ordered entry part, and
then provide an optional hash table on top of that.  This has some
nice properties:

* The entry part maintains key ordering trivially.

* The hash part is optional, which minimizes object size.  For small
  objects a linear scan of keys is also relatively efficient because
  the keys are adjacent in memory (being in the own "sub array").

* The hash part is non-critical.  For instance, it could be dumped in an
  emergency out of memory situation without losing any information.

* It would be relatively straightforward to support multiple hash
  algorithms for e.g. different object sizes.  In particular, it might
  be useful to have a variant for medium size objects, where entry part
  indices could be 8-bit or 16-bit values (instead of 32-bit).

What are the practical alternatives?

Separate object type for internal objects
-----------------------------------------

There used to be a separate object type for internal objects.

The differences to standard objects include: no need to support
array part, no property attributes, no accessor properties,
no need to preserve property ordering, no special behaviors, etc.

However, the extra cost of having another object data structure
does not seem worth it.  The effects are:

* Code size is increased by several kilobytes.
* Internal objects data size decreases slightly (no need to track
  property attributes, for instance).
* Internal object property lookup is slightly more performant.

Currently it seems to make more sense to use the same object
abstraction but to provide "short cut" raw property accessors
for faster / simpler accessing, if necessary.

Array part "freezing" vs abandoning
-----------------------------------

The current policy is that the array part is either actively used and contains
all array indexed properties, or it is abandoned entirely and all entries
moved to the entry part.  An alternative design would allow some array entries
to be in the array part and others in the entry part.

For instance, the array part could be "frozen" and entries above the
allocated size could reside in the entry part.  This is an easy and
relatively straightforward policy too:

* If the "frozen" flag is set, never resize the array part (except perhaps
  to abandon it entirely).

* If an array index key is not found from the array part, and the "frozen"
  flag is *not* set, no need to look at the entry part (this would be the
  common case).  If the "frozen" flag is set, need to look at the entry
  part, too.

This policy has the nice side effect that there is no need to necessarily
support array abandoning at all.  However, without array abandoning the
frozen array part could be left arbitrarily sparse after deletions.

Array size management improvements
----------------------------------

* When to do compaction checks (e.g. property deletion)?

* Periodic actual density checks (relative to e_size + a_size to keep
  average cost limited)?

* Triggered density checks?

More fast paths for array indexing
----------------------------------

It would be nice to have more fast paths (intern free accesses) for the
array part.  In particular, it would be very nice to have a fast path for
``[[Put]]``, at least for the very common case of a plain Array value
inheriting from ``Array.prototype`` with no array indexes entries anywhere
in the prototype chain.

How to do the prototype chain check required by ``[[Put]]`` efficiently?
In other hands, how to the check *without interning the index* that:

* The property does not exist in any ancestor.

* Or if it exists in an ancestor, the write is not prevented by a 
  non-writable property or "captured" by an accessor property.

Some approaches:

* Rework the property lookup to do lazy interning in general.

* Add an object flag ``DUK_HOBJECT_FLAG_NO_ARRIDX_PROPS``.

  + If set, the flag guarantees that the object has no concrete or virtual
    array indexed properties.

  + The prototype walking check can then simply check that no object in the
    prototype chain has this flag cleared.  The flag must be cleared for e.g.
    ``String`` objects which have virtual array indexed properties.

  + Unfortunately the ``Array`` prototype is itself an array, see E5 Section
    15.4.4.  However, it normally has no array elements, so it could have the
    flag set initially, and if someone set an array index to the prototype
    (which does not really make sense) the flag would be cleared, wrecking
    performance.

  + Whenever adding a new property (by whichever means) and the key is an
    array index, the flag must be cleared for the target object.

Note that the fast path really only needs to work for ``Array`` instances
in practice.  Their prototype chain is::

      null
       ^
       |
  Object prototype (E5 Section 15.2.4)
       ^
       |
  Array prototype (E5 Section 15.4.4)
       ^
       |
  Array instance (see E5 Section 15.4.2)

The special flag based approach would work if the object and array prototypes
have no array indexed keys.

Existing Ecmascript implementations do not seem to implement the
official ``[[Put]]`` algorithm accurately, especially for checking the
ancestor properties for protection.  For instance, NodeJS / V8::

  > var p = Array.prototype;
  > Object.defineProperty(p, '0', { 'value': 'inherited', 'writable': false });
  []
  > var a = [];
  > a[0] = 'own';  // should fail
  'own'
  > console.log(a[0]);  // but does not
  own

NodeJS / V8 doesn't seem to respect ``[[Put]]`` for normal objects either::

  > var p = Object.prototype;
  > Object.defineProperty(p, 'foo', { 'value': 'inherited', 'writable': false });
  {}
  > var o = {};
  > o.foo = 'own';  // should fail
  'own'
  > console.log(o.foo);  // but does not
  own

Rhino respects the algorithm in both cases::

  Rhino 1.7 release 3 2012 02 13
  js> var p = Array.prototype;
  js> Object.defineProperty(p, '0', { 'value': 'inherited', 'writable': false });
  inherited
  js> var a = [];
  js> a[0] = 'own';  // should fail
  own
  js> print(a[0]);
  inherited

and::

  Rhino 1.7 release 3 2012 02 13
  js> var p = Object.prototype;
  js> Object.defineProperty(p, 'foo', { 'value': 'inherited', 'writable': false });
  [object Object]
  js> var o = {};
  js> o.foo = 'own';  // should fail
  own
  js> print(o.foo);
  inherited

Similarly for Smjs::

  js> var p = Array.prototype;
  js> Object.defineProperty(p, '0', { 'value': 'inherited', 'writable': false });
  ["inherited"]
  js> var a = [];
  js> a[0] = 'own';  // should fail
  "own"
  js> print(a[0]);
  inherited

and::

  js> var p = Object.prototype;
  js> Object.defineProperty(p, 'foo', { 'value': 'inherited', 'writable': false });
  ({})
  js> var o = {};
  js> o.foo = 'own';
  "own"
  js> print(o.foo);
  inherited

Delete keys from enumerator state during enumeration
----------------------------------------------------

It would be nice to delete keys from enumerator state once the keys have
been enumerated.  This would reduce memory pressure as quickly as it is
currently possible.  Care must be taken to avoid a resize and key compaction
while enumeration is running.

A more practical alternative is to replace the already enumerated keys with
a fixed string (e.g. the empty string).  This would allow the enumerated keys
to be freed as the enumeration progresses.  However, this would create duplicate
keys in the entry part, something which is not currently allowed and would thus
need careful consideration.

Enumeration ordering vs. mutation
---------------------------------

Enumeration vs. mutation causes a lot of implementation headache, as no
internal reorganizations (hash part resizing, array part reallocation or
dropping, compactions) can cause enumeration to fail. There are at least
three basic approaches to manage these:

#. Freeze the object when one or more enumerations are in progress.
   However, to do this, enumerations must be tracked which complicates
   execution.

#. Design the data structures to work with enumeration (especially mutation
   during enumeration) so that the object does not need to know that
   enumerations are in progress.  This is difficult.

#. Create a snapshot of keys to-be-enumerated when the enumerator is
   created.  This wastes memory but is guaranteed to work - although newly
   added keys will not show up in the enumeration (which is compliant
   behavior).

The current approach is to create a snapshot of keys and then re-check them
to see whether they've been deleted during enumeration.

More modular virtual/special properties
---------------------------------------

More modular (but still compact) way of implementing virtual and special
properties?

Test cases
----------

Black box and white box test cases.