Browse Source

Dist/prepare tooling improvements

* Rename prepare_sources.py to configure.py; the operation is similar to
  autoconf ./configure so the association is useful.

* Rename make_dist.py to dist.py.

* Generate only one source set (combined or separate) for one run of
  configure.py.

* Change dist.py to run configure.py three times to generate the default
  source sets.

* Use an autodeleted temporary directory for preparing sources, which
  removes some of the manual temporary file handling.
pull/933/head
Sami Vaarala 8 years ago
parent
commit
80c99a9a9f
  1. 2
      Makefile
  2. 8
      README.md
  3. 4
      RELEASES.rst
  4. 2
      appveyor.yml
  5. 23
      dist-files/README.rst
  6. 11
      doc/duk-config.rst
  7. 15
      doc/low-memory.rst
  8. 2
      doc/release-checklist.rst
  9. 37
      doc/release-notes-v2-0.rst
  10. 9
      examples/cmdline/duk_cmdline_ajduk.c
  11. 5
      extras/alloc-pool/Makefile
  12. 12
      testrunner/client-simple-node/run_commit_test.py
  13. 38
      tools/README.rst
  14. 275
      tools/configure.py
  15. 14
      tools/genbuiltins.py
  16. 34
      tools/genconfig.py
  17. 20
      tools/prepare_unicode_data.py
  18. 127
      util/dist.py
  19. 8
      util/example_rombuild.sh
  20. 6
      util/example_user_builtins1.yaml

2
Makefile

@ -1067,7 +1067,7 @@ cloc: dist cloc-1.60.pl
# XXX: make prints a harmless warning related to the sub-make.
dist:
@make codepolicycheck
$(PYTHON) util/make_dist.py --create-spdx
$(PYTHON) util/dist.py --create-spdx
.PHONY: dist-src
dist-src: dist

8
README.md

@ -64,7 +64,7 @@ Automatically generated bleeding edge snapshots from master are available at
[duktape.org/snapshots](http://duktape.org/snapshots).
You can also clone this repository, make modifications, and build a source
distributable on Linux, OSX, and Windows using `python util/make_dist.py`.
distributable on Linux, OSX, and Windows using `python util/dist.py`.
Getting started: modifying and rebuilding the distributable
-----------------------------------------------------------
@ -74,17 +74,17 @@ distributable in Linux, OSX, or Windows:
# Linux; can often install from packages or using 'pip'
$ sudo apt-get install python python-yaml
$ python util/make_dist.py
$ python util/dist.py
# OSX
# Install Python 2.7.x
$ pip install PyYAML
$ python util/make_dist.py
$ python util/dist.py
# Windows
; Install Python 2.7.x from python.org, and add it to PATH
> pip install PyYAML
> python util\make_dist.py
> python util\dist.py
The source distributable directory will be in `dist/`.

4
RELEASES.rst

@ -1761,8 +1761,8 @@ Planned
* Incompatible change: genconfig.py has been relocated to tools/genconfig.py
in the end user distributable (GH-929)
* Incompatible change: make_dist.py no longer supports ROM built-ins, use
tools/prepare_sources.py instead (GH-929)
* Incompatible change: util/dist.py no longer supports ROM built-ins, use
tools/configure.py instead (GH-929)
* Include raw input sources and a prepare-and-config tool in the distributable,
which allow user code to regenerate a config file and source code files for

2
appveyor.yml

@ -25,7 +25,7 @@ build_script:
# Make dist.
- cmd: cd C:\projects\duktape
- cmd: python util\make_dist.py
- cmd: python util\dist.py
# --- Visual Studio 2015 ---

23
dist-files/README.rst

@ -58,23 +58,26 @@ To build an example command line tool, use the following::
This distributable contains:
* ``src/``: main Duktape library in a "single source file" format (duktape.c,
* Pre-configured Duktape header and source files using the Duktape default
configuration:
- ``src/``: main Duktape library in a "single source file" format (duktape.c,
duktape.h, and duk_config.h).
* ``src-noline/``: contains a variant of ``src/duktape.c`` with no ``#line``
directives which is preferable for some users. See discussion in
https://github.com/svaarala/duktape/pull/363.
* ``src-noline/``: contains a variant of ``src/duktape.c`` with no ``#line``
directives which is preferable for some users. See discussion in
https://github.com/svaarala/duktape/pull/363.
* ``src-separate/``: main Duktape library in multiple files format.
* ``src-separate/``: main Duktape library in multiple files format.
* ``src-input/``: raw input source files used for a config-and-prepare which
* ``src-input/``: raw input source files used by ``configure.py`` which
recreates the combined/separate prepared sources with specific options.
* ``tools/``: various Python tools, such as prepare_sources.py for doing a
config-and-prepare and genconfig.py for creating duk_config.h configuration
files, see: http://wiki.duktape.org/Configuring.html.
* ``tools/``: various Python tools, such as ``configure.py`` for preparing
a ``duk_config.h`` header and Duktape source files for compilation, see
http://wiki.duktape.org/Configuring.html.
* ``config/``: configuration metadata for genconfig.py.
* ``config/``: configuration metadata for ``configure.py``.
* ``examples/``: further examples for using Duktape. Although Duktape
itself is widely portable, some of the examples are Linux only.

11
doc/duk-config.rst

@ -118,17 +118,6 @@ Generating an autodetect duk_config.h
To generate an autodetect header suitable for directly supported platforms
(matches Duktape 1.2 platform support)::
# The --metadata option can point to a metadata directory or a tar.gz
# file with packed metadata (included in end user distributable).
$ cd duktape-2.0.0
$ python tools/genconfig.py \
--metadata config/genconfig_metadata.tar.gz \
--output /tmp/duk_config.h \
duk-config-header
# The same command using unpacked metadata present in Duktape source repo.
$ cd duktape-2.0.0
$ python tools/genconfig.py \
--metadata config/ \

15
doc/low-memory.rst

@ -238,7 +238,7 @@ Suggested options
There are example files in the distributable for Unicode data limited
to 8-bit codepoints.
- Provide the stripped files to ``prepare_sources.py`` to reduce Unicode
- Provide the stripped files to ``configure.py`` to reduce Unicode
table size.
- Possible footprint savings are about 2-3kB.
@ -355,7 +355,7 @@ The following may be appropriate when even less memory is available
is more memory efficient: it creates a writable (empty) global object
which inherits from the ROM global object.
- Rerun ``prepare_sources.py`` with ``--rom-support`` to create prepared
- Rerun ``configure.py`` with ``--rom-support`` to create prepared
sources with support for ROM builtins. ROM builtin support is not
enabled by default because it increases the size of ``duktape.c``
considerably. Add the option ``--rom-auto-lightfunc`` to convert
@ -388,8 +388,8 @@ The following may be appropriate when even less memory is available
+ ``src/builtins.yaml``: documents some more format details
+ ``util/example_rombuild.sh``: illustrates how to run
``prepare_sources.py`` with user builtins
+ ``util/example_rombuild.sh``: illustrates how to run ``configure.py``
with user builtins
* Consider using lightfuncs for representing function properties of ROM
built-ins.
@ -448,10 +448,11 @@ Note that:
is counterproductive because the external pointer takes more room than the
character data.
The Duktape built-in strings are available from build metadata:
The Duktape built-in strings are available from prepared source metadata:
* ``dist/duk_build_meta.json``, the ``builtin_strings_base64`` contains
the byte exact strings used, encoded with base-64.
* For example, ``dist/src/duk_source_meta.json``, the
``builtin_strings_base64`` contains the byte exact strings used, encoded
with base-64.
Strings used by application C and Ecmascript code can be extracted with
various methods. The Duktape main repo contains an example script for

2
doc/release-checklist.rst

@ -279,7 +279,7 @@ Checklist for ordinary releases
- Trivial compile test for combined source
- Trivial compile test for separate sources (important because
it's easy to forget to add files in make_dist.sh)
it's easy to forget to add files in util/dist.py)
* Store binaries to duktape-releases repo

37
doc/release-notes-v2-0.rst

@ -75,15 +75,15 @@ There are some tooling changes in this release:
* The distributable now includes raw sources (``src/`` in Duktape main repo)
in ``src-input/`` and some tooling in ``tools/``.
* The tooling includes a new ``tools/prepare_sources.py`` tool which creates
* The tooling includes a new ``tools/configure.py`` tool which creates
a ``duk_config.h`` and matching prepared sources simultaneously. This
allows use of ROM built-ins from the distributable (previously required a
manual ``make_dist.py --rom-support ...`` command.
manual ``dist.py --rom-support ...`` command.
* The ``make_dist.py`` utility in Duktape main repo no longer supports
``--rom-support``, ``--rom-auto-lightfunc``, and ``--user-builtin-metadata``
options. Use the ``tools/prepare_sources.py`` tool instead, which supports
these options.
* The ``make_dist.py`` utility in Duktape main repo has been renamed to
``dist.py`` and no longer supports ``--rom-support``,
``--rom-auto-lightfunc``, and ``--user-builtin-metadata`` options. Use
the ``tools/configure.py`` tool instead, which supports these options.
* The distributable still includes sources prepared using default configuration
(``src/``, ``src-noline/``, and ``src-separate``) and some configuration
@ -102,7 +102,30 @@ To upgrade:
``tools/genconfig.py``.
* If you're using ROM built-ins via ``make_dist.py``, change your build to
use ``tools/prepare_sources.py`` instead.
use ``tools/configure.py`` instead.
Dist package file changes
-------------------------
* Configuration metadata is now in unpacked form on ``dist/config`` to match
the Duktape master repo and make config files more convenient to patch.
The ``dist/tools/genconfig.py`` tool no longer accepts a tar.gz metadata
argument.
* The pre-built ``duk_config.h`` examples have been removed as somewhat
useless. Use ``dist/tools/configure.py`` (or ``dist/tools/genconfig.py)``
to generate ``duk_config.h`` files.
* ``dist/duk_build_meta.json`` has been renamed to ``dist/duk_dist_meta.json``
for clarity. It no longer contains string data scanned from source files.
This metadata is now in source directories, e.g.
``dist/src/duk_source_meta.json`` as the string set potentially depends
on options used to prepare sources.
* Source metadata, e.g. ``dist/src/metadata.json``, has been renamed to
``dist/src/duk_source_meta.json`` for clarity. The metadata contains
Duktape version information, strings scanned from source files, and for
combined (amalgamated) sources the line number metadata.
Buffer behavior changes
-----------------------

9
examples/cmdline/duk_cmdline_ajduk.c

@ -507,10 +507,11 @@ void ajsheap_extstr_free_1(const void *ptr) {
* is gathered during application compile time and baked into the application
* binary.
*
* Duktape built-in strings are available from duk_build_meta.json, see
* tools/duk_meta_to_strarray.py. There may also be a lot of application
* specific strings, e.g. those used by application specific APIs. These
* must be gathered through some other means, see e.g. tools/scan_strings.py.
* Duktape built-in strings are available from duk_source_meta.json in a
* prepared source directory, see tools/duk_meta_to_strarray.py. There
* may also be a lot of application specific strings, e.g. those used by
* application specific APIs. These must be gathered through some other
* means, see e.g. tools/scan_strings.py.
*/
static const char *strdata_duk_builtin_strings[] = {

5
extras/alloc-pool/Makefile

@ -16,11 +16,10 @@ test:
.PHONY: ptrcomptest
ptrcomptest:
tar -x -v -z -f ../../config/genconfig_metadata.tar.gz examples/low_memory.yaml
python ../../tools/genconfig.py \
--metadata ../../config/genconfig_metadata.tar.gz \
--metadata ../../config \
--output ./duk_config.h \
--option-file examples/low_memory.yaml \
--option-file ../../config/examples/low_memory.yaml \
--option-file ptrcomp.yaml \
--fixup-file ptrcomp_fixup.h \
duk-config-header

12
testrunner/client-simple-node/run_commit_test.py

@ -315,10 +315,10 @@ def context_helper_minsize_fltoetc(archopt, strip):
execute([ 'rm', '-rf', os.path.join(cwd, 'dist', 'src-separate') ])
cmd = [
'python2', os.path.join(cwd, 'dist', 'tools', 'prepare_sources.py'),
'python2', os.path.join(cwd, 'dist', 'tools', 'configure.py'),
'--source-directory', os.path.join(cwd, 'dist', 'src-input'),
'--output-directory', os.path.join(cwd, 'dist'),
'--config-metadata', os.path.join(cwd, 'dist', 'config', 'genconfig_metadata.tar.gz'),
'--config-metadata', os.path.join(cwd, 'dist', 'config'),
'--option-file', os.path.join(cwd, 'config', 'examples', 'low_memory.yaml')
]
if strip:
@ -556,7 +556,7 @@ def context_linux_x86_dist_genconfig():
os.chdir(os.path.join(cwd, 'dist'))
execute([
'python2', os.path.join(cwd, 'dist', 'tools', 'genconfig.py'),
'--metadata', os.path.join(cwd, 'dist', 'config', 'genconfig_metadata.tar.gz'),
'--metadata', os.path.join(cwd, 'dist', 'config'),
'--output', os.path.join(cwd, 'dist', 'src', 'duk_config.h'), # overwrite default duk_config.h
'-DDUK_USE_FASTINT', '-UDUK_USE_JX', '-UDUK_USE_JC',
'duk-config-header'
@ -604,7 +604,7 @@ def context_linux_x64_error_variants():
os.chdir(os.path.join(cwd, 'dist'))
execute([
'python2', os.path.join(cwd, 'dist', 'tools', 'genconfig.py'),
'--metadata', os.path.join(cwd, 'dist', 'config', 'genconfig_metadata.tar.gz'),
'--metadata', os.path.join(cwd, 'dist', 'config'),
'--output', os.path.join(cwd, 'dist', 'src', 'duk_config.h') # overwrite default duk_config.h
] + params['genconfig_opts'] + [
'duk-config-header'
@ -657,10 +657,10 @@ def context_helper_hello_ram(archopt):
execute([ 'rm', '-rf', os.path.join(cwd, 'dist', 'src-separate') ])
cmd = [
'python2', os.path.join(cwd, 'dist', 'tools', 'prepare_sources.py'),
'python2', os.path.join(cwd, 'dist', 'tools', 'configure.py'),
'--source-directory', os.path.join(cwd, 'dist', 'src-input'),
'--output-directory', os.path.join(cwd, 'dist'),
'--config-metadata', os.path.join(cwd, 'dist', 'config', 'genconfig_metadata.tar.gz'),
'--config-metadata', os.path.join(cwd, 'dist', 'config'),
'--rom-support'
] + genconfig_opts
print(repr(cmd))

38
tools/README.rst

@ -5,8 +5,8 @@ Duktape tools
This directory contains various Duktape Python tools which are included in
the end user distributable.
The main use case is config-and-prepare which simultaneously creates
a ``duk_config.h`` configuration file and prepares source files for
The main tool is ``configure.py`` which simultaneously creates a
``duk_config.h`` configuration file and prepares source files for
compilation. These two operations are combined because:
1. Configuration options may affect source file preparation.
@ -27,37 +27,5 @@ be as portable as possible. For example:
* Avoid depending on Python executable name, use ``sys.executable`` instead
to launch Python commands.
The tooling has been written for Python 2.x and there's no support for
The tooling has been written for Python 2.x. There's no support for
Python 3.x at present.
TODO
====
- Go through all .py files once more and figure out what to move into tools
- Go through all moves, and git grep for "call sites"
- Add dist/config/genconfig.py copy for convenience?
STEPS
=====
- Move all scripts that should be in the end user distributable into 'tools/'.
- Change all references for the scripts for their new location so that existing
dist and build works. No other changes at this point.
- Split make_dist.py into two parts:
1. Revised dist part which first copies files into the end user distributable
and then runs the prepare step *in the distributable* for the default
configuration.
2. Separate config-and-prepare utility which does most of the current dist
processing. This will take the --rom-auto-lightfunc and other stuff now.
It also interfaces with genconfig.
- Update documentation: new locations (e.g. in genconfig docs, READMEs etc),
new config process, ROM built-in stuff, etc.
- At this point the initial rework is complete.

275
tools/prepare_sources.py → tools/configure.py

@ -1,7 +1,16 @@
#!/usr/bin/env python2
#
# Config-and-prepare: create a duk_config.h and combined/separate sources
# for configuration options specified on the command line.
# Prepare a duk_config.h and combined/separate sources for compilation,
# given user supplied config options, built-in metadata, Unicode tables, etc.
#
# This is intended to be the main tool application build scripts would use
# before their build step, so convenient, versions, Python compatibility,
# etc all matter.
#
# When obsoleting options, leave the option definitions behind (with
# help=optparse.SUPPRESS_HELP) and give useful suggestions when obsolete
# options are used. This makes it easier for users to fix their build
# scripts.
#
import os
@ -13,7 +22,9 @@ import optparse
import tarfile
import json
import yaml
import tempfile
import subprocess
import atexit
import genconfig
@ -134,7 +145,7 @@ def get_duk_version(apiheader_filename):
# Python module check and friendly errors
def check_python_modules():
# make_dist.py doesn't need yaml but other dist utils will; check for it and
# dist.py doesn't need yaml but other dist utils will; check for it and
# warn if it is missing.
failed = False
@ -163,7 +174,12 @@ check_python_modules()
# Option parsing
def main():
parser = optparse.OptionParser()
parser = optparse.OptionParser(
usage='Usage: %prog [options]',
description='Prepare Duktape source files and a duk_config.h configuration header for compilation. ' + \
'Source files can be combined (amalgamated) or kept separate. ' + \
'See http://wiki.duktape.org/Configuring.html for examples.'
)
# Forced options from multiple sources are gathered into a shared list
# so that the override order remains the same as on the command line.
@ -202,41 +218,46 @@ def main():
line = line[:-1]
fixup_header_lines.append(line)
# Options for config-and-prepare tool itself.
# Options for configure.py tool itself.
parser.add_option('--source-directory', dest='source_directory', default=None, help='Directory with raw input sources (src-input/)')
parser.add_option('--output-directory', dest='output_directory', default=None, help='Directory for output files, must already exist')
parser.add_option('--duk-build-meta', dest='duk_build_meta', default=None, help='duk_build_meta.json for git commit info etc')
parser.add_option('--git-commit', dest='git_commit', default=None, help='Force git commit hash')
parser.add_option('--git-describe', dest='git_describe', default=None, help='Force git describe')
parser.add_option('--git-branch', dest='git_branch', default=None, help='Force git branch name')
parser.add_option('--duk-dist-meta', dest='duk_dist_meta', default=None, help='duk_dist_meta.json to read git commit etc info from')
# Options for combining sources.
parser.add_option('--separate-sources', dest='separate_sources', action='store_true', default=False, help='Output separate sources instead of combined source (default is combined)')
parser.add_option('--line-directives', dest='line_directives', action='store_true', default=False, help='Output #line directives in combined source (default is false)')
# Options forwarded to genbuiltins.py.
parser.add_option('--rom-support', dest='rom_support', action='store_true', help='Add support for ROM strings/objects (increases duktape.c size considerably)')
parser.add_option('--rom-auto-lightfunc', dest='rom_auto_lightfunc', action='store_true', default=False, help='Convert ROM built-in function properties into lightfuncs automatically whenever possible')
parser.add_option('--user-builtin-metadata', dest='user_builtin_metadata', action='append', default=[], help='User strings and objects to add, YAML format (can be repeated for multiple overrides)')
# Options forwarded to genconfig.py.
genconfig.add_genconfig_optparse_options(parser)
parser.add_option('--user-builtin-metadata', dest='user_builtin_metadata', metavar='FILENAME', action='append', default=[], help='User strings and objects to add, YAML format (can be repeated for multiple overrides)')
# Options for Unicode.
parser.add_option('--unicode-data', dest='unicode_data', default=None, help='Provide custom UnicodeData.txt')
parser.add_option('--special-casing', dest='special_casing', default=None, help='Provide custom SpecialCasing.txt')
# Options forwarded to genconfig.py.
genconfig.add_genconfig_optparse_options(parser)
(opts, args) = parser.parse_args()
assert(opts.source_directory)
srcdir = opts.source_directory
assert(opts.output_directory)
outdir = opts.output_directory
assert(opts.config_metadata)
# Figure out directories, git info, etc
entry_pwd = os.getcwd()
duk_build_meta = None
if opts.duk_build_meta is not None:
with open(opts.duk_build_meta, 'rb') as f:
duk_build_meta = json.loads(f.read())
duk_dist_meta = None
if opts.duk_dist_meta is not None:
with open(opts.duk_dist_meta, 'rb') as f:
duk_dist_meta = json.loads(f.read())
duk_version, duk_major, duk_minor, duk_patch, duk_version_formatted = \
get_duk_version(os.path.join(srcdir, 'duk_api_public.h.in'))
@ -245,12 +266,10 @@ def main():
git_branch = None
git_describe = None
if duk_build_meta is not None:
git_commit = duk_build_meta['git_commit']
git_branch = duk_build_meta['git_branch']
git_describe = duk_build_meta['git_describe']
else:
print('No --duk-build-meta, git commit information determined automatically')
if duk_dist_meta is not None:
git_commit = duk_dist_meta['git_commit']
git_branch = duk_dist_meta['git_branch']
git_describe = duk_dist_meta['git_describe']
if opts.git_commit is not None:
git_commit = opts.git_commit
@ -260,10 +279,13 @@ def main():
git_branch = opts.git_branch
if git_commit is None:
print('Git commit not specified, autodetect from current directory')
git_commit = exec_get_stdout([ 'git', 'rev-parse', 'HEAD' ], default='external').strip()
if git_describe is None:
print('Git describe not specified, autodetect from current directory')
git_describe = exec_get_stdout([ 'git', 'describe', '--always', '--dirty' ], default='external').strip()
if git_branch is None:
print('Git branch not specified, autodetect from current directory')
git_branch = exec_get_stdout([ 'git', 'rev-parse', '--abbrev-ref', 'HEAD' ], default='external').strip()
git_commit = str(git_commit)
@ -283,15 +305,14 @@ def main():
else:
special_casing = opts.special_casing
print('Config-and-prepare for Duktape version %s, commit %s, describe %s, branch %s' % \
print('Configuring Duktape version %s, commit %s, describe %s, branch %s' % \
(duk_version_formatted, git_commit, git_describe, git_branch))
# For now, create the src/, src-noline/, and src-separate/ structure into the
# output directory. Later on the output directory should get the specific
# variant output directly.
mkdir(os.path.join(outdir, 'src'))
mkdir(os.path.join(outdir, 'src-noline'))
mkdir(os.path.join(outdir, 'src-separate'))
# Temporary directory.
tempdir = tempfile.mkdtemp(prefix='tmp-duk-prepare-')
atexit.register(shutil.rmtree, tempdir)
mkdir(os.path.join(tempdir, 'src'))
#print('Using temporary directory %r' % tempdir)
# Separate sources are mostly copied as is at present.
copy_files([
@ -413,15 +434,13 @@ def main():
'duk_strings.h',
'duk_replacements.c',
'duk_replacements.h'
], srcdir, os.path.join(outdir, 'src-separate'))
], srcdir, os.path.join(tempdir, 'src'))
# Build temp versions of LICENSE.txt and AUTHORS.rst for embedding into
# autogenerated C/H files.
# XXX: use a proper temp directory
copy_and_cquote('LICENSE.txt', os.path.join(outdir, 'LICENSE.txt.tmp'))
copy_and_cquote('AUTHORS.rst', os.path.join(outdir, 'AUTHORS.rst.tmp'))
copy_and_cquote('LICENSE.txt', os.path.join(tempdir, 'LICENSE.txt.tmp'))
copy_and_cquote('AUTHORS.rst', os.path.join(tempdir, 'AUTHORS.rst.tmp'))
# Create a duk_config.h.
# XXX: might be easier to invoke genconfig directly, but there are a few
@ -465,7 +484,7 @@ def main():
cmd = [
sys.executable, os.path.join('tools', 'genconfig.py'),
'--output', os.path.join(outdir, 'duk_config.h.tmp'),
'--output', os.path.join(tempdir, 'duk_config.h.tmp'),
'--git-commit', git_commit, '--git-describe', git_describe, '--git-branch', git_branch
]
cmd += forward_genconfig_options()
@ -475,9 +494,7 @@ def main():
#print(repr(cmd))
exec_print_stdout(cmd)
copy_file(os.path.join(outdir, 'duk_config.h.tmp'), os.path.join(outdir, 'src', 'duk_config.h'))
copy_file(os.path.join(outdir, 'duk_config.h.tmp'), os.path.join(outdir, 'src-noline', 'duk_config.h'))
copy_file(os.path.join(outdir, 'duk_config.h.tmp'), os.path.join(outdir, 'src-separate', 'duk_config.h'))
copy_file(os.path.join(tempdir, 'duk_config.h.tmp'), os.path.join(outdir, 'duk_config.h'))
# Build duktape.h from parts, with some git-related replacements.
# The only difference between single and separate file duktape.h
@ -485,10 +502,10 @@ def main():
#
# Newline after 'i \':
# http://stackoverflow.com/questions/25631989/sed-insert-line-command-osx
copy_and_replace(os.path.join(srcdir, 'duktape.h.in'), os.path.join(outdir, 'src', 'duktape.h'), {
copy_and_replace(os.path.join(srcdir, 'duktape.h.in'), os.path.join(tempdir, 'duktape.h'), {
'@DUK_SINGLE_FILE@': '#define DUK_SINGLE_FILE',
'@LICENSE_TXT@': read_file(os.path.join(outdir, 'LICENSE.txt.tmp'), strip_last_nl=True),
'@AUTHORS_RST@': read_file(os.path.join(outdir, 'AUTHORS.rst.tmp'), strip_last_nl=True),
'@LICENSE_TXT@': read_file(os.path.join(tempdir, 'LICENSE.txt.tmp'), strip_last_nl=True),
'@AUTHORS_RST@': read_file(os.path.join(tempdir, 'AUTHORS.rst.tmp'), strip_last_nl=True),
'@DUK_API_PUBLIC_H@': read_file(os.path.join(srcdir, 'duk_api_public.h.in'), strip_last_nl=True),
'@DUK_DBLUNION_H@': read_file(os.path.join(srcdir, 'duk_dblunion.h.in'), strip_last_nl=True),
'@DUK_VERSION_FORMATTED@': duk_version_formatted,
@ -499,11 +516,14 @@ def main():
'@GIT_BRANCH@': git_branch,
'@GIT_BRANCH_CSTRING@': git_branch_cstring
})
# keep the line so line numbers match between the two variant headers
copy_and_replace(os.path.join(outdir, 'src', 'duktape.h'), os.path.join(outdir, 'src-separate', 'duktape.h'), {
'#define DUK_SINGLE_FILE': '#undef DUK_SINGLE_FILE'
})
copy_file(os.path.join(outdir, 'src', 'duktape.h'), os.path.join(outdir, 'src-noline', 'duktape.h'))
if opts.separate_sources:
# keep the line so line numbers match between the two variant headers
copy_and_replace(os.path.join(tempdir, 'duktape.h'), os.path.join(outdir, 'duktape.h'), {
'#define DUK_SINGLE_FILE': '#undef DUK_SINGLE_FILE'
})
else:
copy_file(os.path.join(tempdir, 'duktape.h'), os.path.join(outdir, 'duktape.h'))
# Autogenerated strings and built-in files
#
@ -518,10 +538,10 @@ def main():
+ glob.glob(os.path.join(srcdir, '*.h')) \
+ glob.glob(os.path.join(srcdir, '*.h.in'))
)
with open(os.path.join(outdir, 'duk_used_stridx_bidx_defs.json.tmp'), 'wb') as f:
with open(os.path.join(tempdir, 'duk_used_stridx_bidx_defs.json.tmp'), 'wb') as f:
f.write(res)
# XXX: call as direct python? does this need to work outside of prepare_sources.py?
# XXX: call as direct python? does this need to work outside of configure.py?
cmd = [
sys.executable,
os.path.join('tools', 'genbuiltins.py'),
@ -533,12 +553,12 @@ def main():
'--duk-version', str(duk_version)
]
cmd += [
'--used-stridx-metadata=' + os.path.join(outdir, 'duk_used_stridx_bidx_defs.json.tmp'),
'--used-stridx-metadata=' + os.path.join(tempdir, 'duk_used_stridx_bidx_defs.json.tmp'),
'--strings-metadata=' + os.path.join(srcdir, 'strings.yaml'),
'--objects-metadata=' + os.path.join(srcdir, 'builtins.yaml'),
'--out-header=' + os.path.join(outdir, 'src-separate', 'duk_builtins.h'),
'--out-source=' + os.path.join(outdir, 'src-separate', 'duk_builtins.c'),
'--out-metadata-json=' + os.path.join(outdir, 'duk_build_meta.json')
'--out-header=' + os.path.join(tempdir, 'src', 'duk_builtins.h'),
'--out-source=' + os.path.join(tempdir, 'src', 'duk_builtins.c'),
'--out-metadata-json=' + os.path.join(tempdir, 'genbuiltins_metadata.json')
]
cmd.append('--ram-support') # enable by default
if opts.rom_support:
@ -620,8 +640,8 @@ def main():
exec_print_stdout([
sys.executable,
os.path.join('tools', 'prepare_unicode_data.py'),
unicode_data,
os.path.join(outdir, 'src-separate', 'UnicodeData-expanded.tmp')
'--unicode-data', unicode_data,
'--output', os.path.join(tempdir, 'UnicodeData-expanded.tmp')
])
def extract_chars(incl, excl, suffix):
@ -629,14 +649,14 @@ def main():
res = exec_get_stdout([
sys.executable,
os.path.join('tools', 'extract_chars.py'),
'--unicode-data=' + os.path.join(outdir, 'src-separate', 'UnicodeData-expanded.tmp'),
'--unicode-data=' + os.path.join(tempdir, 'UnicodeData-expanded.tmp'),
'--include-categories=' + incl,
'--exclude-categories=' + excl,
'--out-source=' + os.path.join(outdir, 'src-separate', 'duk_unicode_%s.c.tmp' % suffix),
'--out-header=' + os.path.join(outdir, 'src-separate', 'duk_unicode_%s.h.tmp' % suffix),
'--out-source=' + os.path.join(tempdir, 'duk_unicode_%s.c.tmp' % suffix),
'--out-header=' + os.path.join(tempdir, 'duk_unicode_%s.h.tmp' % suffix),
'--table-name=' + 'duk_unicode_%s' % suffix
])
with open(os.path.join(outdir, 'src-separate', suffix + '.txt'), 'wb') as f:
with open(os.path.join(tempdir, suffix + '.txt'), 'wb') as f:
f.write(res)
def extract_caseconv():
@ -645,14 +665,14 @@ def main():
sys.executable,
os.path.join('tools', 'extract_caseconv.py'),
'--command=caseconv_bitpacked',
'--unicode-data=' + os.path.join(outdir, 'src-separate', 'UnicodeData-expanded.tmp'),
'--unicode-data=' + os.path.join(tempdir, 'UnicodeData-expanded.tmp'),
'--special-casing=' + special_casing,
'--out-source=' + os.path.join(outdir, 'src-separate', 'duk_unicode_caseconv.c.tmp'),
'--out-header=' + os.path.join(outdir, 'src-separate', 'duk_unicode_caseconv.h.tmp'),
'--out-source=' + os.path.join(tempdir, 'duk_unicode_caseconv.c.tmp'),
'--out-header=' + os.path.join(tempdir, 'duk_unicode_caseconv.h.tmp'),
'--table-name-lc=duk_unicode_caseconv_lc',
'--table-name-uc=duk_unicode_caseconv_uc'
])
with open(os.path.join(outdir, 'src-separate', 'caseconv.txt'), 'wb') as f:
with open(os.path.join(tempdir, 'caseconv.txt'), 'wb') as f:
f.write(res)
#print('- extract_caseconv canon lookup')
@ -660,13 +680,13 @@ def main():
sys.executable,
os.path.join('tools', 'extract_caseconv.py'),
'--command=re_canon_lookup',
'--unicode-data=' + os.path.join(outdir, 'src-separate', 'UnicodeData-expanded.tmp'),
'--unicode-data=' + os.path.join(tempdir, 'UnicodeData-expanded.tmp'),
'--special-casing=' + special_casing,
'--out-source=' + os.path.join(outdir, 'src-separate', 'duk_unicode_re_canon_lookup.c.tmp'),
'--out-header=' + os.path.join(outdir, 'src-separate', 'duk_unicode_re_canon_lookup.h.tmp'),
'--out-source=' + os.path.join(tempdir, 'duk_unicode_re_canon_lookup.c.tmp'),
'--out-header=' + os.path.join(tempdir, 'duk_unicode_re_canon_lookup.h.tmp'),
'--table-name-re-canon-lookup=duk_unicode_re_canon_lookup'
])
with open(os.path.join(outdir, 'src-separate', 'caseconv_re_canon_lookup.txt'), 'wb') as f:
with open(os.path.join(tempdir, 'caseconv_re_canon_lookup.txt'), 'wb') as f:
f.write(res)
print('Create Unicode tables for codepoint classes')
@ -695,40 +715,28 @@ def main():
# The injection points use a standard C preprocessor #include syntax
# (earlier these were actual includes).
copy_and_replace(os.path.join(outdir, 'src-separate', 'duk_unicode.h'), os.path.join(outdir, 'src-separate', 'duk_unicode.h'), {
'#include "duk_unicode_ids_noa.h"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_ids_noa.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_noabmp.h"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_ids_noabmp.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_m_let_noa.h"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_ids_m_let_noa.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_m_let_noabmp.h"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_ids_m_let_noabmp.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_idp_m_ids_noa.h"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_idp_m_ids_noa.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_idp_m_ids_noabmp.h"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_idp_m_ids_noabmp.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_caseconv.h"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_caseconv.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_re_canon_lookup.h"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_re_canon_lookup.h.tmp'), strip_last_nl=True)
copy_and_replace(os.path.join(tempdir, 'src', 'duk_unicode.h'), os.path.join(tempdir, 'src', 'duk_unicode.h'), {
'#include "duk_unicode_ids_noa.h"': read_file(os.path.join(tempdir, 'duk_unicode_ids_noa.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_noabmp.h"': read_file(os.path.join(tempdir, 'duk_unicode_ids_noabmp.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_m_let_noa.h"': read_file(os.path.join(tempdir, 'duk_unicode_ids_m_let_noa.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_m_let_noabmp.h"': read_file(os.path.join(tempdir, 'duk_unicode_ids_m_let_noabmp.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_idp_m_ids_noa.h"': read_file(os.path.join(tempdir, 'duk_unicode_idp_m_ids_noa.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_idp_m_ids_noabmp.h"': read_file(os.path.join(tempdir, 'duk_unicode_idp_m_ids_noabmp.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_caseconv.h"': read_file(os.path.join(tempdir, 'duk_unicode_caseconv.h.tmp'), strip_last_nl=True),
'#include "duk_unicode_re_canon_lookup.h"': read_file(os.path.join(tempdir, 'duk_unicode_re_canon_lookup.h.tmp'), strip_last_nl=True)
})
copy_and_replace(os.path.join(outdir, 'src-separate', 'duk_unicode_tables.c'), os.path.join(outdir, 'src-separate', 'duk_unicode_tables.c'), {
'#include "duk_unicode_ids_noa.c"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_ids_noa.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_noabmp.c"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_ids_noabmp.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_m_let_noa.c"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_ids_m_let_noa.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_m_let_noabmp.c"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_ids_m_let_noabmp.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_idp_m_ids_noa.c"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_idp_m_ids_noa.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_idp_m_ids_noabmp.c"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_idp_m_ids_noabmp.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_caseconv.c"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_caseconv.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_re_canon_lookup.c"': read_file(os.path.join(outdir, 'src-separate', 'duk_unicode_re_canon_lookup.c.tmp'), strip_last_nl=True)
copy_and_replace(os.path.join(tempdir, 'src', 'duk_unicode_tables.c'), os.path.join(tempdir, 'src', 'duk_unicode_tables.c'), {
'#include "duk_unicode_ids_noa.c"': read_file(os.path.join(tempdir, 'duk_unicode_ids_noa.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_noabmp.c"': read_file(os.path.join(tempdir, 'duk_unicode_ids_noabmp.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_m_let_noa.c"': read_file(os.path.join(tempdir, 'duk_unicode_ids_m_let_noa.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_ids_m_let_noabmp.c"': read_file(os.path.join(tempdir, 'duk_unicode_ids_m_let_noabmp.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_idp_m_ids_noa.c"': read_file(os.path.join(tempdir, 'duk_unicode_idp_m_ids_noa.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_idp_m_ids_noabmp.c"': read_file(os.path.join(tempdir, 'duk_unicode_idp_m_ids_noabmp.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_caseconv.c"': read_file(os.path.join(tempdir, 'duk_unicode_caseconv.c.tmp'), strip_last_nl=True),
'#include "duk_unicode_re_canon_lookup.c"': read_file(os.path.join(tempdir, 'duk_unicode_re_canon_lookup.c.tmp'), strip_last_nl=True)
})
# Clean up some temporary files
delete_matching_files(os.path.join(outdir, 'src-separate'), lambda x: x[-4:] == '.tmp')
delete_matching_files(os.path.join(outdir, 'src-separate'), lambda x: x in [
'ws.txt',
'let.txt', 'let_noa.txt', 'let_noabmp.txt',
'ids.txt', 'ids_noa.txt', 'ids_noabmp.txt',
'ids_m_let.txt', 'ids_m_let_noa.txt', 'ids_m_let_noabmp.txt',
'idp_m_ids.txt', 'idp_m_ids_noa.txt', 'idp_m_ids_noabmp.txt'
])
delete_matching_files(os.path.join(outdir, 'src-separate'), lambda x: x[0:8] == 'caseconv' and x[-4:] == '.txt')
# Create a combined source file, duktape.c, into a separate combined source
# directory. This allows user to just include "duktape.c", "duktape.h", and
# "duk_config.h" into a project and maximizes inlining and size optimization
@ -798,7 +806,7 @@ def main():
for fn in handpick:
files.append(fn)
for fn in sorted(os.listdir(os.path.join(outdir, 'src-separate'))):
for fn in sorted(os.listdir(os.path.join(tempdir, 'src'))):
f_ext = os.path.splitext(fn)[1]
if f_ext not in [ '.c' ]:
continue
@ -806,41 +814,60 @@ def main():
continue
files.append(fn)
res = map(lambda x: os.path.join(outdir, 'src-separate', x), files)
res = map(lambda x: os.path.join(tempdir, 'src', x), files)
#print(repr(files))
#print(repr(res))
return res
with open(os.path.join(outdir, 'prologue.tmp'), 'wb') as f:
f.write(create_source_prologue(os.path.join(outdir, 'LICENSE.txt.tmp'), os.path.join(outdir, 'AUTHORS.rst.tmp')))
if opts.separate_sources:
for fn in os.listdir(os.path.join(tempdir, 'src')):
copy_file(os.path.join(tempdir, 'src', fn), os.path.join(outdir, fn))
else:
with open(os.path.join(tempdir, 'prologue.tmp'), 'wb') as f:
f.write(create_source_prologue(os.path.join(tempdir, 'LICENSE.txt.tmp'), os.path.join(tempdir, 'AUTHORS.rst.tmp')))
exec_print_stdout([
sys.executable,
os.path.join('tools', 'combine_src.py'),
'--include-path', os.path.join(outdir, 'src-separate'),
'--include-exclude', 'duk_config.h', # don't inline
'--include-exclude', 'duktape.h', # don't inline
'--prologue', os.path.join(outdir, 'prologue.tmp'),
'--output-source', os.path.join(outdir, 'src', 'duktape.c'),
'--output-metadata', os.path.join(outdir, 'src', 'metadata.json'),
'--line-directives'
] + select_combined_sources())
cmd = [
sys.executable,
os.path.join('tools', 'combine_src.py'),
'--include-path', os.path.join(tempdir, 'src'),
'--include-exclude', 'duk_config.h', # don't inline
'--include-exclude', 'duktape.h', # don't inline
'--prologue', os.path.join(tempdir, 'prologue.tmp'),
'--output-source', os.path.join(outdir, 'duktape.c'),
'--output-metadata', os.path.join(tempdir, 'combine_src_metadata.json')
]
if opts.line_directives:
cmd += [ '--line-directives' ]
cmd += select_combined_sources()
exec_print_stdout(cmd)
# Merge metadata files.
doc = {
'type': 'duk_source_meta',
'comment': 'Metadata for prepared Duktape sources and configuration',
'git_commit': git_commit,
'git_branch': git_branch,
'git_describe': git_describe,
'duk_version': duk_version,
'duk_version_string': duk_version_formatted
}
with open(os.path.join(tempdir, 'genbuiltins_metadata.json'), 'rb') as f:
tmp = json.loads(f.read())
for k in tmp.keys():
doc[k] = tmp[k]
if opts.separate_sources:
pass
else:
with open(os.path.join(tempdir, 'combine_src_metadata.json'), 'rb') as f:
tmp = json.loads(f.read())
for k in tmp.keys():
doc[k] = tmp[k]
exec_print_stdout([
sys.executable,
os.path.join('tools', 'combine_src.py'),
'--include-path', os.path.join(outdir, 'src-separate'),
'--include-exclude', 'duk_config.h', # don't inline
'--include-exclude', 'duktape.h', # don't inline
'--prologue', os.path.join(outdir, 'prologue.tmp'),
'--output-source', os.path.join(outdir, 'src-noline', 'duktape.c'),
'--output-metadata', os.path.join(outdir, 'src-noline', 'metadata.json')
] + select_combined_sources())
# Clean up remaining temp files
delete_matching_files(outdir, lambda x: x[-4:] == '.tmp')
print('Config-and-prepare finished successfully')
with open(os.path.join(outdir, 'duk_source_meta.json'), 'wb') as f:
f.write(json.dumps(doc, indent=4))
print('Configure finished successfully')
if __name__ == '__main__':
main()

14
tools/genbuiltins.py

@ -2875,7 +2875,7 @@ def main():
rom_emit_object_initializer_types_and_macros(gc_src)
rom_emit_objects(gc_src, rom_meta, rom_bi_str_map)
else:
gc_src.emitLine('#error ROM support not enabled, rerun prepare_sources.py with --rom-support')
gc_src.emitLine('#error ROM support not enabled, rerun configure.py with --rom-support')
gc_src.emitLine('#else /* DUK_USE_ROM_STRINGS */')
emit_ramstr_source_strinit_data(gc_src, ramstr_data)
gc_src.emitLine('#endif /* DUK_USE_ROM_STRINGS */')
@ -2886,7 +2886,7 @@ def main():
gc_src.emitLine('#error DUK_USE_ROM_OBJECTS requires DUK_USE_ROM_STRINGS')
gc_src.emitLine('#endif')
else:
gc_src.emitLine('#error ROM support not enabled, rerun prepare_sources.py with --rom-support')
gc_src.emitLine('#error ROM support not enabled, rerun configure.py with --rom-support')
gc_src.emitLine('#else /* DUK_USE_ROM_OBJECTS */')
if opts.ram_support:
emit_ramobj_source_nativefunc_array(gc_src, ram_native_funcs) # endian independent
@ -2900,7 +2900,7 @@ def main():
gc_src.emitLine('#error invalid endianness defines')
gc_src.emitLine('#endif')
else:
gc_src.emitLine('#error RAM support not enabled, rerun prepare_sources.py with --ram-support')
gc_src.emitLine('#error RAM support not enabled, rerun configure.py with --ram-support')
gc_src.emitLine('#endif /* DUK_USE_ROM_OBJECTS */')
gc_hdr = dukutil.GenerateC()
@ -2913,13 +2913,13 @@ def main():
emit_header_stridx_defines(gc_hdr, rom_meta)
rom_emit_strings_header(gc_hdr, rom_meta)
else:
gc_hdr.emitLine('#error ROM support not enabled, rerun prepare_sources.py with --rom-support')
gc_hdr.emitLine('#error ROM support not enabled, rerun configure.py with --rom-support')
gc_hdr.emitLine('#else /* DUK_USE_ROM_STRINGS */')
if opts.ram_support:
emit_header_stridx_defines(gc_hdr, ram_meta)
emit_ramstr_header_strinit_defines(gc_hdr, ram_meta, ramstr_data, ramstr_maxlen)
else:
gc_hdr.emitLine('#error RAM support not enabled, rerun prepare_sources.py with --ram-support')
gc_hdr.emitLine('#error RAM support not enabled, rerun configure.py with --ram-support')
gc_hdr.emitLine('#endif /* DUK_USE_ROM_STRINGS */')
gc_hdr.emitLine('')
gc_hdr.emitLine('#if defined(DUK_USE_ROM_OBJECTS)')
@ -2936,7 +2936,7 @@ def main():
emit_header_native_function_declarations(gc_hdr, rom_meta)
rom_emit_objects_header(gc_hdr, rom_meta)
else:
gc_hdr.emitLine('#error RAM support not enabled, rerun prepare_sources.py with --ram-support')
gc_hdr.emitLine('#error RAM support not enabled, rerun configure.py with --ram-support')
gc_hdr.emitLine('#else /* DUK_USE_ROM_OBJECTS */')
if opts.ram_support:
emit_header_native_function_declarations(gc_hdr, ram_meta)
@ -2952,7 +2952,7 @@ def main():
gc_hdr.emitLine('#error invalid endianness defines')
gc_hdr.emitLine('#endif')
else:
gc_hdr.emitLine('#error RAM support not enabled, rerun prepare_sources.py with --ram-support')
gc_hdr.emitLine('#error RAM support not enabled, rerun configure.py with --ram-support')
gc_hdr.emitLine('#endif /* DUK_USE_ROM_OBJECTS */')
gc_hdr.emitLine('#endif /* DUK_BUILTINS_H_INCLUDED */')

34
tools/genconfig.py

@ -1437,12 +1437,12 @@ def add_genconfig_optparse_options(parser, direct=False):
fixup_header_lines.append(line)
if direct:
parser.add_option('--metadata', dest='config_metadata', default=None, help='metadata directory or metadata tar.gz file')
parser.add_option('--metadata', dest='config_metadata', default=None, help='metadata directory')
parser.add_option('--output', dest='output', default=None, help='output filename for C header or RST documentation file')
else:
# Different option name when called through prepare_sources.py,
# Different option name when called through configure.py,
# also no --output option.
parser.add_option('--config-metadata', dest='config_metadata', default=None, help='metadata directory or metadata tar.gz file')
parser.add_option('--config-metadata', dest='config_metadata', default=None, help='metadata directory')
parser.add_option('--platform', dest='platform', default=None, help='platform (default is autodetect)')
parser.add_option('--compiler', dest='compiler', default=None, help='compiler (default is autodetect)')
@ -1456,14 +1456,14 @@ def add_genconfig_optparse_options(parser, direct=False):
parser.add_option('--omit-deprecated-config-options', dest='omit_deprecated_config_options', action='store_true', default=False, help='omit deprecated config options from generated headers')
parser.add_option('--omit-unused-config-options', dest='omit_unused_config_options', action='store_true', default=False, help='omit unused config options from generated headers')
parser.add_option('--add-active-defines-macro', dest='add_active_defines_macro', action='store_true', default=False, help='add DUK_ACTIVE_DEFINES macro, for development only')
parser.add_option('--define', type='string', dest='force_options_yaml', action='callback', callback=add_force_option_define, default=force_options_yaml, help='force #define option using a C compiler like syntax, e.g. "--define DUK_USE_DEEP_C_STACK" or "--define DUK_USE_TRACEBACK_DEPTH=10"')
parser.add_option('-D', type='string', dest='force_options_yaml', action='callback', callback=add_force_option_define, default=force_options_yaml, help='synonym for --define, e.g. "-DDUK_USE_DEEP_C_STACK" or "-DDUK_USE_TRACEBACK_DEPTH=10"')
parser.add_option('--undefine', type='string', dest='force_options_yaml', action='callback', callback=add_force_option_undefine, default=force_options_yaml, help='force #undef option using a C compiler like syntax, e.g. "--undefine DUK_USE_DEEP_C_STACK"')
parser.add_option('-U', type='string', dest='force_options_yaml', action='callback', callback=add_force_option_undefine, default=force_options_yaml, help='synonym for --undefine, e.g. "-UDUK_USE_DEEP_C_STACK"')
parser.add_option('--option-yaml', type='string', dest='force_options_yaml', action='callback', callback=add_force_option_yaml, default=force_options_yaml, help='force option(s) using inline YAML (e.g. --option-yaml "DUK_USE_DEEP_C_STACK: true")')
parser.add_option('--option-file', type='string', dest='force_options_yaml', action='callback', callback=add_force_option_file, default=force_options_yaml, help='YAML file(s) providing config option overrides')
parser.add_option('--fixup-file', type='string', dest='fixup_header_lines', action='callback', callback=add_fixup_header_file, default=fixup_header_lines, help='C header snippet file(s) to be appended to generated header, useful for manual option fixups')
parser.add_option('--fixup-line', type='string', dest='fixup_header_lines', action='callback', callback=add_fixup_header_line, default=fixup_header_lines, help='C header fixup line to be appended to generated header (e.g. --fixup-line "#define DUK_USE_FASTINT")')
parser.add_option('--define', type='string', metavar='OPTION', dest='force_options_yaml', action='callback', callback=add_force_option_define, default=force_options_yaml, help='force #define option using a C compiler like syntax, e.g. "--define DUK_USE_DEEP_C_STACK" or "--define DUK_USE_TRACEBACK_DEPTH=10"')
parser.add_option('-D', type='string', metavar='OPTION', dest='force_options_yaml', action='callback', callback=add_force_option_define, default=force_options_yaml, help='synonym for --define, e.g. "-DDUK_USE_DEEP_C_STACK" or "-DDUK_USE_TRACEBACK_DEPTH=10"')
parser.add_option('--undefine', type='string', metavar='OPTION', dest='force_options_yaml', action='callback', callback=add_force_option_undefine, default=force_options_yaml, help='force #undef option using a C compiler like syntax, e.g. "--undefine DUK_USE_DEEP_C_STACK"')
parser.add_option('-U', type='string', metavar='OPTION', dest='force_options_yaml', action='callback', callback=add_force_option_undefine, default=force_options_yaml, help='synonym for --undefine, e.g. "-UDUK_USE_DEEP_C_STACK"')
parser.add_option('--option-yaml', type='string', metavar='YAML', dest='force_options_yaml', action='callback', callback=add_force_option_yaml, default=force_options_yaml, help='force option(s) using inline YAML (e.g. --option-yaml "DUK_USE_DEEP_C_STACK: true")')
parser.add_option('--option-file', type='string', metavar='FILENAME', dest='force_options_yaml', action='callback', callback=add_force_option_file, default=force_options_yaml, help='YAML file(s) providing config option overrides')
parser.add_option('--fixup-file', type='string', metavar='FILENAME', dest='fixup_header_lines', action='callback', callback=add_fixup_header_file, default=fixup_header_lines, help='C header snippet file(s) to be appended to generated header, useful for manual option fixups')
parser.add_option('--fixup-line', type='string', metavar='LINE', dest='fixup_header_lines', action='callback', callback=add_fixup_header_line, default=fixup_header_lines, help='C header fixup line to be appended to generated header (e.g. --fixup-line "#define DUK_USE_FASTINT")')
parser.add_option('--sanity-warning', dest='sanity_strict', action='store_false', default=True, help='emit a warning instead of #error for option sanity check issues')
parser.add_option('--use-cpp-warning', dest='use_cpp_warning', action='store_true', default=False, help='emit a (non-portable) #warning when appropriate')
@ -1493,21 +1493,13 @@ def parse_options():
def genconfig(opts, args):
meta_dir = opts.config_metadata
if opts.config_metadata is None:
if os.path.isfile(os.path.join('.', 'genconfig_metadata.tar.gz')):
opts.config_metadata = 'genconfig_metadata.tar.gz'
elif os.path.isdir(os.path.join('.', 'config-options')):
if os.path.isdir(os.path.join('.', 'config-options')):
opts.config_metadata = '.'
if opts.config_metadata is not None and os.path.isdir(opts.config_metadata):
meta_dir = opts.config_metadata
metadata_src_text = 'Using metadata directory: %r' % meta_dir
elif opts.config_metadata is not None and os.path.isfile(opts.config_metadata) and tarfile.is_tarfile(opts.config_metadata):
meta_dir = get_auto_delete_tempdir()
tar = tarfile.open(name=opts.config_metadata, mode='r:*')
tar.extractall(path=meta_dir)
metadata_src_text = 'Using metadata tar file %r, unpacked to directory: %r' % (opts.config_metadata, meta_dir)
else:
raise Exception('metadata source must be a directory or a tar.gz file')
raise Exception('metadata argument must be a directory (tar.gz no longer supported)')
scan_helper_snippets(os.path.join(meta_dir, 'helper-snippets'))
scan_use_defs(os.path.join(meta_dir, 'config-options'))

20
tools/prepare_unicode_data.py

@ -6,10 +6,18 @@
import os
import sys
import optparse
def main():
f_in = open(sys.argv[1], 'rb')
f_out = open(sys.argv[2], 'wb')
parser = optparse.OptionParser()
parser.add_option('--unicode-data', dest='unicode_data')
parser.add_option('--output', dest='output')
(opts, args) = parser.parse_args()
assert(opts.unicode_data is not None)
assert(opts.output is not None)
f_in = open(opts.unicode_data, 'rb')
f_out = open(opts.output, 'wb')
while True:
line = f_in.readline()
if line == '' or line == '\n':
@ -23,9 +31,13 @@ def main():
cp1 = long(parts[0], 16)
cp2 = long(parts2[0], 16)
suffix = ';'.join(parts[1:])
for i in xrange(cp1, cp2 + 1): # inclusive
tmp = parts[1:]
tmp[0] = '-""-'
suffix = ';'.join(tmp)
f_out.write(line)
for i in xrange(cp1 + 1, cp2):
f_out.write('%04X;%s' % (i, suffix))
f_out.write(line2)
else:
f_out.write(line)

127
util/make_dist.py → util/dist.py

@ -7,6 +7,7 @@
import os
import sys
import re
import json
import shutil
import glob
import optparse
@ -136,9 +137,9 @@ def create_dist_directories(dist):
shutil.rmtree(dist)
mkdir(dist)
mkdir(os.path.join(dist, 'src-input'))
#mkdir(os.path.join(dist, 'src-separate')) # created by prepare_sources.py
#mkdir(os.path.join(dist, 'src'))
#mkdir(os.path.join(dist, 'src-noline'))
mkdir(os.path.join(dist, 'src-separate'))
mkdir(os.path.join(dist, 'src'))
mkdir(os.path.join(dist, 'src-noline'))
mkdir(os.path.join(dist, 'tools'))
mkdir(os.path.join(dist, 'config'))
mkdir(os.path.join(dist, 'extras'))
@ -192,9 +193,9 @@ def parse_options():
parser.add_option('--git-commit', dest='git_commit', default=None, help='Force git commit hash')
parser.add_option('--git-describe', dest='git_describe', default=None, help='Force git describe')
parser.add_option('--git-branch', dest='git_branch', default=None, help='Force git branch name')
parser.add_option('--rom-support', dest='rom_support', action='store_true', help='Deprecated, use prepare_sources.py instead')
parser.add_option('--rom-auto-lightfunc', dest='rom_auto_lightfunc', action='store_true', default=False, help='Deprecated, use prepare_sources.py instead')
parser.add_option('--user-builtin-metadata', dest='user_builtin_metadata', action='append', default=[], help='Deprecated, use prepare_sources.py instead')
parser.add_option('--rom-support', dest='rom_support', action='store_true', help=optparse.SUPPRESS_HELP)
parser.add_option('--rom-auto-lightfunc', dest='rom_auto_lightfunc', action='store_true', default=False, help=optparse.SUPPRESS_HELP)
parser.add_option('--user-builtin-metadata', dest='user_builtin_metadata', action='append', default=[], help=optparse.SUPPRESS_HELP)
(opts, args) = parser.parse_args()
return opts, args
@ -202,7 +203,7 @@ def parse_options():
# Python module check and friendly errors.
def check_python_modules(opts):
# make_dist.py doesn't need yaml but other dist utils will; check for it and
# dist.py doesn't need yaml but other dist utils will; check for it and
# warn if it is missing.
failed = False
@ -244,7 +245,7 @@ def main():
# Obsolete options check.
if opts.rom_support or opts.rom_auto_lightfunc or len(opts.user_builtin_metadata) > 0:
raise Exception('obsolete ROM support argument(s), use tools/prepare_sources.py instead')
raise Exception('obsolete ROM support argument(s), use tools/configure.py instead')
# Figure out directories, git info, Duktape version, etc.
@ -411,8 +412,7 @@ def main():
'UnicodeData-8bit.txt',
], 'src', os.path.join(dist, 'src-input'))
os.chdir(os.path.join(entry_pwd, 'config'))
create_targz(os.path.join(dist, 'config', 'genconfig_metadata.tar.gz'), [
for fn in [
'tags.yaml',
'platforms.yaml',
'architectures.yaml',
@ -426,11 +426,15 @@ def main():
'header-snippets',
'other-defines',
'examples'
])
os.chdir(entry_pwd)
]:
# Copy directories in their entirety
if os.path.isfile(os.path.join('config', fn)):
shutil.copyfile(os.path.join('config', fn), os.path.join(dist, 'config', fn))
else:
shutil.copytree(os.path.join('config', fn), os.path.join(dist, 'config', fn))
copy_files([
'prepare_sources.py',
'configure.py',
'combine_src.py',
'create_spdx_license.py',
'duk_meta_to_strarray.py',
@ -722,73 +726,48 @@ def main():
'--opcodes', os.path.join('debugger', 'duk_opcodes.yaml')
])
# Build some example duk_config.h headers. This may go away later.
# Add a build metadata file.
print('Create duk_config.h headers')
doc = {
'type': 'duk_dist_meta',
'comment': 'Metadata for Duktape distributable',
'git_commit': git_commit,
'git_branch': git_branch,
'git_describe': git_describe,
'duk_version': duk_version,
'duk_version_string': duk_version_formatted
}
with open(os.path.join(dist, 'duk_dist_meta.json'), 'wb') as f:
f.write(json.dumps(doc, indent=4))
exec_print_stdout([
sys.executable, os.path.join('tools', 'genconfig.py'), '--metadata', 'config',
'--output', os.path.join(dist, 'config', 'duk_config.h-modular-static'),
'--git-commit', git_commit, '--git-describe', git_describe, '--git-branch', git_branch,
'--omit-removed-config-options', '--omit-unused-config-options',
'--emit-legacy-feature-check', '--emit-config-sanity-check',
'duk-config-header'
])
# Build prepared sources (src/, src-noline/, src-separate/) with default
# config. This is done using tools and metadata in the dist directory.
exec_print_stdout([
sys.executable, os.path.join('tools', 'genconfig.py'), '--metadata', 'config',
'--output', os.path.join(dist, 'config', 'duk_config.h-modular-dll'),
'--git-commit', git_commit, '--git-describe', git_describe, '--git-branch', git_branch,
'--omit-removed-config-options', '--omit-unused-config-options',
'--emit-legacy-feature-check', '--emit-config-sanity-check',
'--dll',
'duk-config-header'
])
print('Create prepared sources for default configuration')
def genconfig_barebones(platform, architecture, compiler):
exec_print_stdout([
sys.executable, os.path.join('tools', 'genconfig.py'), '--metadata', 'config',
'--output', os.path.join(dist, 'config', 'duk_config.h-%s-%s-%s' % (platform, architecture, compiler)),
def prep_default_sources(dirname, extraopts):
cmd = [
sys.executable, os.path.join(dist, 'tools', 'configure.py'),
'--source-directory', os.path.join(dist, 'src-input'),
'--output-directory', os.path.join(dist, dirname),
'--config-metadata', os.path.join(dist, 'config'),
'--git-commit', git_commit, '--git-describe', git_describe, '--git-branch', git_branch,
'--platform', platform, '--architecture', architecture, '--compiler', compiler,
'--omit-removed-config-options', '--omit-unused-config-options',
'--emit-legacy-feature-check', '--emit-config-sanity-check',
'duk-config-header'
])
#genconfig_barebones('linux', 'x86', 'gcc')
#genconfig_barebones('linux', 'x64', 'gcc')
#genconfig_barebones('linux', 'x86', 'clang')
#genconfig_barebones('linux', 'x64', 'clang')
#genconfig_barebones('windows', 'x86', 'msvc')
#genconfig_barebones('windows', 'x64', 'msvc')
#genconfig_barebones('apple', 'x86', 'gcc')
#genconfig_barebones('apple', 'x64', 'gcc')
#genconfig_barebones('apple', 'x86', 'clang')
#genconfig_barebones('apple', 'x64', 'clang')
# Build prepared sources (src/, src-noline/, src-separate/) with default
# config. This is done using tools and metadata in the dist directory.
print('Config-and-prepare sources for default configuration')
cmd = [
sys.executable, os.path.join(dist, 'tools', 'prepare_sources.py'),
'--source-directory', os.path.join(dist, 'src-input'),
'--output-directory', dist,
'--config-metadata', os.path.join(dist, 'config', 'genconfig_metadata.tar.gz'),
'--git-commit', git_commit, '--git-describe', git_describe, '--git-branch', git_branch,
'--omit-removed-config-options', '--omit-unused-config-options',
'--emit-config-sanity-check', '--support-feature-options'
]
if opts.rom_support:
cmd.append('--rom-support')
if opts.rom_auto_lightfunc:
cmd.append('--rom-auto-lightfunc')
for i in opts.user_builtin_metadata:
cmd.append('--user-builtin-metadata')
cmd.append(i)
exec_print_stdout(cmd)
'--emit-config-sanity-check', '--support-feature-options'
]
cmd += extraopts
if opts.rom_support:
cmd.append('--rom-support')
if opts.rom_auto_lightfunc:
cmd.append('--rom-auto-lightfunc')
for i in opts.user_builtin_metadata:
cmd.append('--user-builtin-metadata')
cmd.append(i)
exec_print_stdout(cmd)
prep_default_sources('src', [ '--line-directives' ])
prep_default_sources('src-noline', [])
prep_default_sources('src-separate', [ '--separate-sources' ])
# Clean up remaining temp files.

8
util/example_rombuild.sh

@ -12,14 +12,14 @@ make clean dist
# metadata can be provided through one or more YAML files (which are applied
# in sequence). Duktape configuration can be given at the same time.
rm -rf dist/src dist/src-noline dist/src-separate
$PYTHON dist/tools/prepare_sources.py \
$PYTHON dist/tools/configure.py \
--source-directory dist/src-input \
--output-directory dist \
--rom-support \
--rom-auto-lightfunc \
--user-builtin-metadata util/example_user_builtins1.yaml \
--user-builtin-metadata util/example_user_builtins2.yaml \
--config-metadata dist/config/genconfig_metadata.tar.gz \
--config-metadata dist/config \
-DDUK_USE_ROM_STRINGS \
-DDUK_USE_ROM_OBJECTS \
-DDUK_USE_ROM_GLOBAL_INHERIT \
@ -34,14 +34,14 @@ make duk dukd # XXX: currently fails to start, DUK_CMDLINE_LOGGING_SUPPORT, DUK
# --support-feature-options by moving the options into a genconfig
# YAML config file.
rm -rf dist/src dist/src-noline dist/src-separate
$PYTHON dist/tools/prepare_sources.py \
$PYTHON dist/tools/configure.py \
--source-directory dist/src-input \
--output-directory dist \
--rom-support \
--rom-auto-lightfunc \
--user-builtin-metadata util/example_user_builtins1.yaml \
--user-builtin-metadata util/example_user_builtins2.yaml \
--config-metadata dist/config/genconfig_metadata.tar.gz \
--config-metadata dist/config \
--support-feature-options \
-DDUK_USE_ROM_STRINGS \
-DDUK_USE_ROM_OBJECTS \

6
util/example_user_builtins1.yaml

@ -8,9 +8,9 @@
#
# See examples below for details on how to use these.
#
# Note that genbuiltins.py and prepare_sources.py accept multiple user
# built-in YAML files, so that you can manage your custom strings and
# objects in individual YAML files for modularity.
# Note that genbuiltins.py and configure.py accept multiple user built-in
# YAML files, so that you can manage your custom strings and objects in
# individual YAML files for modularity.
#
# When using pointer compression, all ROM strings and objects need a number
# from the ROM pointer compression range (e.g. [0xf800,0xffff]). By default

Loading…
Cancel
Save