Documentation changes for tools reorg

config/README.rst

@@ -25,15 +25,15 @@ Usage
 
 To create an autodetect duk_config.h header (compatible with Duktape 1.2.x)::
 
-    $ python config/genconfig.py --metadata config --output /tmp/duk_config.h \
-          autodetect-header
+    $ python tools/genconfig.py --metadata config --output /tmp/duk_config.h \
+          duk-config-header
 
 To create a barebones duk_config.h header for a specific platform (easier to
 edit manually)::
 
-    $ python config/genconfig.py --metadata config --output /tmp/duk_config.h \
+    $ python tools/genconfig.py --metadata config --output /tmp/duk_config.h \
           --platform linux --compiler gcc --architecture x64 \
-          barebones-header
+          duk-config-header
 
 There are further commands to e.g. autogenerate config option documentation;
 see ``genconfig.py`` for details.

dist-files/README.rst

@@ -67,9 +67,15 @@ This distributable contains:
 
 * ``src-separate/``: main Duktape library in multiple files format.
 
-* ``config/``: genconfig utility for creating duk_config.h configuration
+* ``src-input/``: raw input source files used for a config-and-prepare which
+  recreates the combined/separate prepared sources with specific options.
+
+* ``tools/``: various Python tools, such as prepare_sources.py for doing a
+  config-and-prepare and genconfig.py for creating duk_config.h configuration
   files, see: http://wiki.duktape.org/Configuring.html.
 
+* ``config/``: configuration metadata for genconfig.py.
+
 * ``examples/``: further examples for using Duktape.  Although Duktape
   itself is widely portable, some of the examples are Linux only.
   For instance the ``eventloop`` example illustrates how ``setTimeout()``

doc/duk-config.rst

@@ -94,7 +94,7 @@ Using genconfig
 Overview of genconfig
 ---------------------
 
-Genconfig (``config/genconfig.py``) is a helper script which provides
+Genconfig (``tools/genconfig.py``) is a helper script which provides
 several commands related to config handling:
 
 * Generate a ``duk_config.h`` for a specified platform, compiler, and
@@ -121,16 +121,16 @@ To generate an autodetect header suitable for directly supported platforms
     # The --metadata option can point to a metadata directory or a tar.gz
     # file with packed metadata (included in end user distributable).
 
-    $ cd duktape-1.4.0
-    $ python config/genconfig.py \
+    $ cd duktape-2.0.0
+    $ python tools/genconfig.py \
         --metadata config/genconfig_metadata.tar.gz \
         --output /tmp/duk_config.h \
         duk-config-header
 
     # The same command using unpacked metadata present in Duktape source repo.
 
-    $ cd duktape
-    $ python config/genconfig.py \
+    $ cd duktape-2.0.0
+    $ python tools/genconfig.py \
         --metadata config/ \
         --output /tmp/duk_config.h \
         duk-config-header
@@ -145,7 +145,7 @@ through a file or inline.
 
 If you're building Duktape as a DLL, you should use the ``--dll`` option::
 
-    $ python config/genconfig.py \
+    $ python tools/genconfig.py \
         --metadata config/ \
         --dll \
         --output /tmp/duk_config.h \
@@ -167,7 +167,7 @@ Generating a barebones duk_config.h
 To generate a barebones header you need to specify a platform, compiler, and
 architecture for genconfig::
 
-    $ python config/genconfig.py \
+    $ python tools/genconfig.py \
         --metadata config/ \
         --platform linux \
         --compiler gcc \
@@ -229,7 +229,7 @@ file, and a few options are then tweaked using the C compiler format.  An
 autodetect header is then generated::
 
     $ cd duktape
-    $ python config/genconfig.py \
+    $ python tools/genconfig.py \
         --metadata config/ \
         --option-file low_memory.yaml \
         -DDUK_USE_TRACEBACK_DEPTH=100 \
@@ -254,7 +254,7 @@ This file, another override file, and a few inline YAML forced options
 could be used as follows to generate a barebones header::
 
     $ cd duktape
-    $ python config/genconfig.py \
+    $ python tools/genconfig.py \
         --metadata config/ \
         --platform linux \
         --compiler gcc \
@@ -264,7 +264,7 @@ could be used as follows to generate a barebones header::
         --option-yaml 'DUK_USE_JX: false' \
         --option-yaml 'DUK_USE_JC: false' \
         --output /tmp/duk_config.h \
-        barebones-header
+        duk-config-header
 
 For inline YAML, multiple forced options can be given either by using a YAML
 value with multiple keys, or by using multiple options::
@@ -335,7 +335,7 @@ sanity checks (if enabled).
 
 For example, to generate a barebones header with two fixup headers::
 
-    $ python config/genconfig.py \
+    $ python tools/genconfig.py \
         --metadata config/ \
         --platform linux \
         --compiler gcc \
@@ -343,7 +343,7 @@ For example, to generate a barebones header with two fixup headers::
         --fixup-file my_env_strings.h \
         --fixup-file my_no_json_fastpath.h \
         --output /tmp/duk_config.h \
-        barebones-header
+        duk-config-header
 
 The ``my_env_strings.h`` fixup header could be::
 
@@ -476,7 +476,7 @@ header::
     # my_custom.h is applied after generated header; functionally similar
     # to Duktape 1.2.x duk_custom.h
 
-    $ python config/ genconfig.py [...] --fixup-file my_custom.h [...]
+    $ python tools/genconfig.py [...] --fixup-file my_custom.h [...]
 
 A genconfig-generated barebones header also has the following line near the end
 for detecting where to add override defines; this is easy to detect reliably::

doc/low-memory.rst

@@ -344,12 +344,12 @@ The following may be appropriate when even less memory is available
     is more memory efficient: it creates a writable (empty) global object
     which inherits from the ROM global object.
 
-  - Rerun ``make_dist.py`` with ``--rom-support`` to create a distributable
-    with support for ROM builtins.  ROM builtin support is not enabled by
-    default because it increases the size of ``duktape.c`` considerably.
-    Add the option ``--rom-auto-lightfunc`` to convert built-in function
-    properties into lightfuncs to reduce ROM footprint.  (See
-    ``util/example_rombuild.sh`` for some very simple examples.)
+  - Rerun ``prepare_sources.py`` with ``--rom-support`` to create prepared
+    sources with support for ROM builtins.  ROM builtin support is not
+    enabled by default because it increases the size of ``duktape.c``
+    considerably.  Add the option ``--rom-auto-lightfunc`` to convert
+    built-in function properties into lightfuncs to reduce ROM footprint.
+    (See ``util/example_rombuild.sh`` for some very simple examples.)
 
   - Moving built-ins into ROM makes them read-only which has some side
     effects.  Some side effects are technical compliance issues while
@@ -377,8 +377,8 @@ The following may be appropriate when even less memory is available
 
     + ``src/builtins.yaml``: documents some more format details
 
-    + ``util/example_rombuild.sh``: illustrates how to run ``make_dist.py``
-      with user builtins
+    + ``util/example_rombuild.sh``: illustrates how to run
+      ``prepare_sources.py`` with user builtins
 
 * Consider using lightfuncs for representing function properties of ROM
   built-ins.

tools/README.rst

@@ -0,0 +1,63 @@
+=============
+Duktape tools
+=============
+
+This directory contains various Duktape Python tools which are included in
+the end user distributable.
+
+The main use case is config-and-prepare which simultaneously creates
+a ``duk_config.h`` configuration file and prepares source files for
+compilation.  These two operations are combined because:
+
+1. Configuration options may affect source file preparation.
+
+2. Some features involve metaprogramming.  For example, when using ROM
+   built-ins YAML metadata is used for automatic code generation for the
+   ROM object initializers.
+
+There are also other miscellaneous tools, e.g. utilities to scan potential
+fixed strings from source files, dump bytecode, and resolving file/line
+from combined source.
+
+Because this tooling is part of the end user distributable, the tooling must
+be as portable as possible.  For example:
+
+* Avoid Unix path separators.
+
+* Avoid depending on Python executable name, use ``sys.executable`` instead
+  to launch Python commands.
+
+The tooling has been written for Python 2.x and there's no support for
+Python 3.x at present.
+
+TODO
+====
+
+- Go through all .py files once more and figure out what to move into tools
+
+- Go through all moves, and git grep for "call sites"
+
+- Add dist/config/genconfig.py copy for convenience?
+
+STEPS
+=====
+
+- Move all scripts that should be in the end user distributable into 'tools/'.
+
+- Change all references for the scripts for their new location so that existing
+  dist and build works.  No other changes at this point.
+
+- Split make_dist.py into two parts:
+
+  1. Revised dist part which first copies files into the end user distributable
+     and then runs the prepare step *in the distributable* for the default
+     configuration.
+
+  2. Separate config-and-prepare utility which does most of the current dist
+     processing.  This will take the --rom-auto-lightfunc and other stuff now.
+     It also interfaces with genconfig.
+
+- Update documentation: new locations (e.g. in genconfig docs, READMEs etc),
+  new config process, ROM built-in stuff, etc.
+
+- At this point the initial rework is complete.

util/README.rst

@@ -0,0 +1,6 @@
+=======================
+Miscellaneous utilities
+=======================
+
+This directory contains miscellaneous Python, Shell, and other utilities
+which are not part of the end user distributable.