mirror of https://github.com/svaarala/duktape.git
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
303 lines
11 KiB
303 lines
11 KiB
=========
|
|
Testcases
|
|
=========
|
|
|
|
There are two main testcase sets for Duktape:
|
|
|
|
* ECMAScript testcases (``tests/ecmascript``) for testing ECMAScript
|
|
compliance, "real world" behavior, and Duktape specific behavior.
|
|
|
|
* API testcases (``tests/api``) for testing the Duktape specific C API.
|
|
|
|
Testcases are written using an "expected string" approach: a testcase file
|
|
describes the expected output using a custom markup (described below) and also
|
|
contains the ECMAScript or C code that is intended to produce that output.
|
|
A test runner compares actual output to expected; known issue files are used
|
|
to document "known bad" outputs.
|
|
|
|
This document describes the testcase formats and current test tools.
|
|
|
|
ECMAScript testcase format
|
|
==========================
|
|
|
|
Testcases are plain ECMAScript (``.js``) files with custom markup inside
|
|
comments for providing test metadata, expected output, and include files.
|
|
A testcase is "prepared" before execution using ``util/runtest.py``:
|
|
|
|
* Testcase metadata and expected string are parsed from inside custom markup.
|
|
|
|
* A minified prologue is injected to provide a global ``Test`` object.
|
|
The prologue also harmonizes the execution environment so that e.g.
|
|
``print()`` and ``console.log()`` are available so that the prepared
|
|
test can be executed using Duktape, V8, etc.
|
|
|
|
* Include files are located, minified, and included into the prepared test.
|
|
All utilities included must work in both strict and non-strict contexts
|
|
because testcases may be either strict or non-strict programs.
|
|
|
|
* A ``"use strict";`` declaration is prepended (even before the prologue)
|
|
if test metadata indicates it is needed. This is needed when the testcase
|
|
is exercising strict program code behavior.
|
|
|
|
The prologue and include files are minified to one-liners so that they don't
|
|
offset the line numbers of the testcase. This is important for tests that
|
|
exercise traceback line numbers for example.
|
|
|
|
Include files are specified using the following syntax::
|
|
|
|
/*@include util-buffer.js@*/
|
|
|
|
Testcase metadata is provided as JSON or YAML inside a comment block. If
|
|
multiple blocks are present they are merged, with the last occurrence of a
|
|
key overwriting previous occurrences::
|
|
|
|
/*---
|
|
{
|
|
"custom": true
|
|
}
|
|
---*/
|
|
|
|
// or
|
|
|
|
/*---
|
|
custom: true
|
|
---*/
|
|
|
|
The metadata keys change over time; current keys are described below.
|
|
Metadata is optional.
|
|
|
|
Finally, the expected output is specified using the following syntax::
|
|
|
|
/*===
|
|
hello world
|
|
===*/
|
|
|
|
print('hello world');
|
|
|
|
There's also a single-line shorthand::
|
|
|
|
print('hello world'); //>hello world
|
|
|
|
Full testcase example::
|
|
|
|
/*
|
|
* Example test.
|
|
*/
|
|
|
|
/*@include util-foo.js@*/
|
|
|
|
/*---
|
|
# Optional metadata is encoded in JSON or YAML.
|
|
slow: false
|
|
---*/
|
|
|
|
/*===
|
|
hello world
|
|
===*/
|
|
|
|
if (1) {
|
|
print("hello world"); /* automatic newline */
|
|
} else {
|
|
print("not quite");
|
|
}
|
|
|
|
/*===
|
|
second test
|
|
===*/
|
|
|
|
/* there can be multiple "expected" blocks (but only one metadata block) */
|
|
print("second test");
|
|
|
|
/* Shorthand can also be used. */
|
|
print("shorthand"); //>shorthand
|
|
|
|
ECMAScript testcase metadata keys
|
|
=================================
|
|
|
|
Metadata keys are added and removed as necessary so this list may be
|
|
out-of-date; see ``util/runtest.py`` for current keys. All keys are
|
|
optional:
|
|
|
|
+----------------------+------------------------------------------------------+
|
|
| Key | Description |
|
|
+======================+======================================================+
|
|
| comment | Optional string to comment on the testcase briefly. |
|
|
+----------------------+------------------------------------------------------+
|
|
| slow | If true, test is (very) slow and increased time |
|
|
| | limits may be necessary to avoid test timeouts. |
|
|
+----------------------+------------------------------------------------------+
|
|
| skip | If true, test is skipped without causing a test |
|
|
| | failure. Useful for unfinished tests and tests |
|
|
| | that need to be executed manually. |
|
|
+----------------------+------------------------------------------------------+
|
|
| custom | If true, some implementation dependent behavior |
|
|
| | is expected and comparison to other ECMAScript |
|
|
| | engines is not relevant. The behavior may either |
|
|
| | be entirely Duktape specific (e.g. relying on JX |
|
|
| | format) or specific behavior not required by the |
|
|
| | ECMAScript specification (e.g. additional enumeration|
|
|
| | guarantees). |
|
|
+----------------------+------------------------------------------------------+
|
|
| nonstandard | If true, expected behavior is not standards |
|
|
| | compliant but matches "real world" expectations. |
|
|
+----------------------+------------------------------------------------------+
|
|
| endianness | If set, indicates that the testcase requires a |
|
|
| | specific endianness, needed for e.g. some TypedArray |
|
|
| | testcases. Values: ``little``, ``big``, ``mixed``. |
|
|
+----------------------+------------------------------------------------------+
|
|
| use_strict | Testcase is a strict mode program. When preparing |
|
|
| | the test, prepend a ``"use strict";`` declaration as |
|
|
| | very first statement of the test, before the test |
|
|
| | prologue. |
|
|
+----------------------+------------------------------------------------------+
|
|
| intended_uncaught | Testcase intentionally fails by throwing an uncaught |
|
|
| | error (which may even be a SyntaxError). This is |
|
|
| | needed to test some program level behavior. |
|
|
+----------------------+------------------------------------------------------+
|
|
|
|
ECMAScript testcase known issues
|
|
================================
|
|
|
|
Sometimes testcases fail due to known bugs or environment specific differences
|
|
such as endianness. Known issue files describe the "known bad" testcase
|
|
output and describes the reason for the failure. This allows a failing test
|
|
to be flagged as a "known issue" rather than a failure.
|
|
|
|
Known issue files have a YAML metadata block, followed by ``---``, followed by
|
|
the "known bad" verbatim testcase output::
|
|
|
|
summary: wurld is printed instead of world
|
|
---
|
|
hello wurld
|
|
|
|
The "known bad" output can also be provided as an MD5 hash which is useful if
|
|
the full output is very large and uninteresting::
|
|
|
|
summary: wurld is printed instead of world
|
|
md5: 49a9895803ec23a6b41dd346c32203b7
|
|
|
|
Each known issue file describes a single known failure for a specific testcase.
|
|
A certain testcase may have several known issue files, for different Duktape
|
|
versions, different config options, different environments, etc. The current
|
|
naming convention is just a numbered sequence based on the testcase name::
|
|
|
|
# For test-dev-hello-world.js:
|
|
test-dev-hello-world-1.txt
|
|
test-dev-hello-world-2.txt
|
|
test-dev-hello-world-3.txt
|
|
...
|
|
|
|
ECMAScript testcase best practices
|
|
==================================
|
|
|
|
Indentation
|
|
-----------
|
|
|
|
Indent with 4 spaces, no tabs.
|
|
|
|
Verifying exception type
|
|
------------------------
|
|
|
|
Since ECMAScript doesn't require specific error messages for errors
|
|
thrown, the messages should not be inspected or printed out in test
|
|
cases. ECMAScript does require specific error types though (such as
|
|
``TypeError``. These can be verified by printing the ``name``
|
|
property of an error object.
|
|
|
|
For instance::
|
|
|
|
try {
|
|
null.foo = 1;
|
|
} catch (e) {
|
|
print(e.name);
|
|
}
|
|
|
|
prints::
|
|
|
|
TypeError
|
|
|
|
When an error is not supposed to occur in a successful test run, the
|
|
exception message can (and should) be printed, as it makes it easier
|
|
to resolve a failing testcase. This can be done most easily as::
|
|
|
|
try {
|
|
null.foo = 1;
|
|
} catch (e) {
|
|
print(e.stack || e);
|
|
}
|
|
|
|
This is portable and prints a stack trace when available.
|
|
|
|
Printing tracebacks, pointers, etc
|
|
----------------------------------
|
|
|
|
While it should be generally avoided, in some testcases it's necessary to
|
|
print out tracebacks, JX-serialize pointers, etc. When doing so:
|
|
|
|
* Replace filenames and line numbers in tracebacks with e.g. ``FILE:LINE``.
|
|
Otherwise the test output will include temporary file names and it won't
|
|
be possible to describe a stable expected output.
|
|
|
|
* Replace pointers with e.g. ``PTR``. Pointer format is platform dependent
|
|
and can include ``0x12345678``, ``0x123456789abcdef``, and ``12345678``.
|
|
|
|
There are utility includes to perform these replacements.
|
|
|
|
API testcase format
|
|
===================
|
|
|
|
Testcase files are C files with a ``test()`` function. The test function
|
|
gets as its argument an already initialized ``duk_context *`` and print out
|
|
text to ``stdout``. The testcase can assume ``duktape.h`` and common headers
|
|
like ``stdio.h`` have been included. There are also some predefined macros
|
|
(like ``TEST_SAFE_CALL()`` and ``TEST_PCALL()``) to minimize duplication in
|
|
testcase code.
|
|
|
|
Expected output and metadata is defined as for ECMAScript testcases. However,
|
|
the expected output shorthand syntax (``//>output``) cannot be used because
|
|
it's not portable C89.
|
|
|
|
Example::
|
|
|
|
/*===
|
|
Hello world from ECMAScript!
|
|
Hello world from C!
|
|
===*/
|
|
|
|
void test(duk_context *ctx) {
|
|
duk_push_string("print('Hello world from ECMAScript!');");
|
|
duk_eval(ctx);
|
|
printf("Hello world from C!\n");
|
|
}
|
|
|
|
API testcase known issues
|
|
=========================
|
|
|
|
As for ECMAScript testcases, known issues are documented using known issue
|
|
files providing the "known bad" output. The format is the same as for
|
|
ECMAScript tests.
|
|
|
|
Test tools
|
|
==========
|
|
|
|
* ``util/runtest.py``: prepares and executes a single testcase, and prints
|
|
out a readable result summary. Optionally writes JSON test result file,
|
|
prepared testcase, and various other outputs to specified files. The tool
|
|
can also be used to just prepare a test. The runtest.py tool can be used
|
|
both manually and as part of running a test suite.
|
|
|
|
* ``util/prep_test.py``: earlier version of ``runtest.py``, used by
|
|
runtests.js and likely be to deprecated.
|
|
|
|
* ``runtests/runtests.js``: original Node.js based test runner which is
|
|
likely to be rewritten as a Python program.
|
|
|
|
* ``testrunner/``: distributed test runner jobs for GitHub commit/pull webhook
|
|
tests. The testrunner client/server code is in its own repo
|
|
https://github.com/svaarala/duktape-testrunner.
|
|
|
|
Future work
|
|
===========
|
|
|
|
* Put testcases in a directory hierarchy instead (``test/stmt/trycatch.js``),
|
|
perhaps scales better (at the expense of adding hassle to e.g. grepping).
|
|
|