duktape/doc/testcases.rst

=========
Testcases
=========

There are two main testcase sets for Duktape:

* Ecmascript testcases (``tests/ecmascript``) for testing Ecmascript
  compliance, "real world" behavior, and Duktape specific behavior.

* API testcases (``tests/api``) for testing the Duktape specific C API.

Testcases are written using an "expected string" approach: a testcase file
describes the expected output using a custom markup (described below) and also
contains the Ecmascript or C code that is intended to produce that output.
A test runner compares actual output to expected; known issue files are used
to document "known bad" outputs.

This document describes the testcase formats and current test tools.

Ecmascript testcase format
==========================

Testcases are plain Ecmascript (``.js``) files with custom markup inside
comments for providing test metadata, expected output, and include files.
A testcase is "prepared" before execution using ``util/runtest.py``:

* Testcase metadata and expected string are parsed from inside custom markup.

* A minified prologue is injected to provide a global ``Test`` object.
  The prologue also harmonizes the execution environment so that e.g.
  ``print()`` and ``console.log()`` are available so that the prepared
  test can be executed using Duktape, V8, etc.

* Include files are located, minified, and included into the prepared test.
  All utilities included must work in both strict and non-strict contexts
  because testcases may be either strict or non-strict programs.

* A ``"use strict";`` declaration is prepended (even before the prologue)
  if test metadata indicates it is needed.  This is needed when the testcase
  is exercising strict program code behavior.

The prologue and include files are minified to one-liners so that they don't
offset the line numbers of the testcase.  This is important for tests that
exercise traceback line numbers for example.

Include files are specified using the following syntax::

  /*@include util-buffer.js@*/

Testcase metadata is provided as JSON or YAML inside a comment block.  If
multiple blocks are present they are merged, with the last occurrence of a
key overwriting previous occurrences::

  /*---
  {
      "custom": true
  }
  ---*/

  // or

  /*---
  custom: true
  ---*/

The metadata keys change over time; current keys are described below.
Metadata is optional.

Finally, the expected output is specified using the following syntax::

  /*===
  hello world
  ===*/

  print('hello world');

There's also a single-line shorthand::

  print('hello world');  //>hello world

Full testcase example::

  /*
   *  Example test.
   */

  /*@include util-foo.js@*/

  /*---
  # Optional metadata is encoded in JSON or YAML.
  slow: false
  ---*/

  /*===
  hello world
  ===*/

  if (1) {
      print("hello world");   /* automatic newline */
  } else {
      print("not quite");
  }

  /*===
  second test
  ===*/

  /* there can be multiple "expected" blocks (but only one metadata block) */
  print("second test");

  /* Shorthand can also be used. */
  print("shorthand");  //>shorthand

Ecmascript testcase metadata keys
=================================

Metadata keys are added and removed as necessary so this list may be
out-of-date; see ``util/runtest.py`` for current keys.  All keys are
optional:

+----------------------+------------------------------------------------------+
| Key                  | Description                                          |
+======================+======================================================+
| comment              | Optional string to comment on the testcase briefly.  |
+----------------------+------------------------------------------------------+
| slow                 | If true, test is (very) slow and increased time      |
|                      | limits may be necessary to avoid test timeouts.      |
+----------------------+------------------------------------------------------+
| skip                 | If true, test is skipped without causing a test      |
|                      | failure.  Useful for unfinished tests and tests      |
|                      | that need to be executed manually.                   |
+----------------------+------------------------------------------------------+
| custom               | If true, some implementation dependent behavior      |
|                      | is expected and comparison to other Ecmascript       |
|                      | engines is not relevant.  The behavior may either    |
|                      | be entirely Duktape specific (e.g. relying on JX     |
|                      | format) or specific behavior not required by the     |
|                      | Ecmascript specification (e.g. additional enumeration|
|                      | guarantees).                                         |
+----------------------+------------------------------------------------------+
| nonstandard          | If true, expected behavior is not standards          |
|                      | compliant but matches "real world" expectations.     |
+----------------------+------------------------------------------------------+
| endianness           | If set, indicates that the testcase requires a       |
|                      | specific endianness, needed for e.g. some TypedArray |
|                      | testcases.  Values: ``little``, ``big``, ``mixed``.  |
+----------------------+------------------------------------------------------+
| use_strict           | Testcase is a strict mode program.  When preparing   |
|                      | the test, prepend a ``"use strict";`` declaration as |
|                      | very first statement of the test, before the test    |
|                      | prologue.                                            |
+----------------------+------------------------------------------------------+
| intended_uncaught    | Testcase intentionally fails by throwing an uncaught |
|                      | error (which may even be a SyntaxError).  This is    |
|                      | needed to test some program level behavior.          |
+----------------------+------------------------------------------------------+

Ecmascript testcase known issues
================================

Sometimes testcases fail due to known bugs or environment specific differences
such as endianness.  Known issue files describe the "known bad" testcase
output and describes the reason for the failure.  This allows a failing test
to be flagged as a "known issue" rather than a failure.

Known issue files have a YAML metadata block, followed by ``---``, followed by
the "known bad" verbatim testcase output::

  summary: wurld is printed instead of world
  ---
  hello wurld

The "known bad" output can also be provided as an MD5 hash which is useful if
the full output is very large and uninteresting::

  summary: wurld is printed instead of world
  md5: 49a9895803ec23a6b41dd346c32203b7

Each known issue file describes a single known failure for a specific testcase.
A certain testcase may have several known issue files, for different Duktape
versions, different config options, different environments, etc.  The current
naming convention is just a numbered sequence based on the testcase name::

  # For test-dev-hello-world.js:
  test-dev-hello-world-1.txt
  test-dev-hello-world-2.txt
  test-dev-hello-world-3.txt
  ...

Ecmascript testcase best practices
==================================

Indentation
-----------

Indent with 4 spaces, no tabs.

Verifying exception type
------------------------

Since Ecmascript doesn't require specific error messages for errors
thrown, the messages should not be inspected or printed out in test
cases.  Ecmascript does require specific error types though (such as
``TypeError``.  These can be verified by printing the ``name``
property of an error object.

For instance::

  try {
      null.foo = 1;
  } catch (e) {
      print(e.name);
  }

prints::

  TypeError

When an error is not supposed to occur in a successful test run, the
exception message can (and should) be printed, as it makes it easier
to resolve a failing testcase.  This can be done most easily as::

  try {
      null.foo = 1;
  } catch (e) {
      print(e.stack || e);
  }

This is portable and prints a stack trace when available.

Printing tracebacks, pointers, etc
----------------------------------

While it should be generally avoided, in some testcases it's necessary to
print out tracebacks, JX-serialize pointers, etc.  When doing so:

* Replace filenames and line numbers in tracebacks with e.g. ``FILE:LINE``.
  Otherwise the test output will include temporary file names and it won't
  be possible to describe a stable expected output.

* Replace pointers with e.g. ``PTR``.  Pointer format is platform dependent
  and can include ``0x12345678``, ``0x123456789abcdef``, and ``12345678``.

There are utility includes to perform these replacements.

API testcase format
===================

Testcase files are C files with a ``test()`` function.  The test function
gets as its argument an already initialized ``duk_context *`` and print out
text to ``stdout``.  The testcase can assume ``duktape.h`` and common headers
like ``stdio.h`` have been included.  There are also some predefined macros
(like ``TEST_SAFE_CALL()`` and ``TEST_PCALL()``) to minimize duplication in
testcase code.

Expected output and metadata is defined as for Ecmascript testcases.  However,
the expected output shorthand syntax (``//>output``) cannot be used because
it's not portable C89.

Example::

  /*===
  Hello world from Ecmascript!
  Hello world from C!
  ===*/

  void test(duk_context *ctx) {
      duk_push_string("print('Hello world from Ecmascript!');");
      duk_eval(ctx);
      printf("Hello world from C!\n");
  }

API testcase known issues
=========================

As for Ecmascript testcases, known issues are documented using known issue
files providing the "known bad" output.  The format is the same as for
Ecmascript tests.

Test tools
==========

* ``util/runtest.py``: prepares and executes a single testcase, and prints
  out a readable result summary.  Optionally writes JSON test result file,
  prepared testcase, and various other outputs to specified files.  The tool
  can also be used to just prepare a test.  The runtest.py tool can be used
  both manually and as part of running a test suite.

* ``util/prep_test.py``: earlier version of ``runtest.py``, used by
  runtests.js and likely be to deprecated.

* ``runtests/runtests.js``: original Node.js based test runner which is
  likely to be rewritten as a Python program.

* ``testrunner/``: distributed test runner for Github commit/pull webhook
  tests.

Future work
===========

* Put testcases in a directory hierarchy instead (``test/stmt/trycatch.js``),
  perhaps scales better (at the expense of adding hassle to e.g. grepping).