======= Modules ======= Introduction ============ This document discusses the barebone CommonJS-based module framework built into Duktape: * Ecmascript modules are defined using the CommonJS module format: - http://wiki.commonjs.org/wiki/Modules/1.1.1 * The user must provide a *module search function* which locates a module corresponding to a resolved module ID, and can register module symbols directly and/or return module source code as a string. To remain portable and unaware of file systems and such, Duktape does not provide a default module search function. * C modules, static or DLL-based, can be implemented on top of the module search function by user code. There is no built-in C module support in the main Duktape library to avoid portability issues for exotic platforms. However, there is a recommended convention which works on most platforms and allows both static and DLL loading, see ``c-module-convention.rst``. Using modules from Ecmascript code (require) ============================================ Duktape provides a global ``require()`` function which allows a module to be loaded based on a module identifier:: var lib = require('package/lib') The module identifier may contain forward slashes to provide some hierarchy. There are both absolute and relative identifiers:: // absolute identifier, resolves to 'a/b' everywhere var mod = require('a/b'); // relative identifier, resolves to 'package/c' inside package/lib var mod = require('./c'); // relative identifier, resolves to 'foo/xyz' inside foo/bar/quux var mod = require('../xyz'); Relative paths are resolved starting from the absolute root of the current module, i.e. the module calling ``require()``, see separate section on the resolution algorithm. A module with a certain resolved identifier is loaded only once per a global environment: ``Duktape.modLoaded`` keeps track of modules which are fully or partially loaded, mapping a resolved identifier to the module's ``exports`` table. If a required module has already been (fully or partially) loaded, the same ``exports`` value is returned without further action. If a module has not yet been registered to ``Duktape.modLoaded``, Duktape calls ``Duktape.modSearch()``, a module search function which must be provided by the user (there is no default):: Duktape.modSearch = function (id, require, exports, module) { // ... }; The identifier given to the ``modSearch()`` function is a fully resolved, absolute identifier. If the search function cannot locate a module based on its identifier, it is expected to throw an error. If a module is found, the search function can register symbols directly to 'exports' (this is used to implement C modules), and can also return a string to be used as the module Ecmascript source code:: /* An actual implementation would usually scan and return module sources * e.g. from the filesystem or a compressed source pack. This example * illustrates the return value options. */ Duktape.modSearch = function (id, require, exports, module) { if (id === 'foo') { /* 'foo' is a native module, register symbols to 'exports' in a * platform specific way. */ return; // undefined: no Ecmascript source } if (id === 'bar') { /* 'bar' is a pure Ecmascript module. */ return 'exports.hello = function () { print("Hello world from bar!"); };'; } if (id === 'quux') { /* 'quux' is a mixed C/Ecmascript module. C code provides the * exports.rawFunc() binding while Ecmascript code implements * a safe variant on top of that. */ return 'exports.func = function () {\n' + ' try { exports.rawFunc(); } catch (e) { print(e); }\n' + '};\n'; } /* If a module is not found, an error must be thrown. */ throw new Error('cannot find module: ' + id); }; If the search function returns without throwing an error, the exports value is registered to ``Duktape.modLoaded`` before evaluating module source code possibly returned by the search function. This is important so that circular requires are properly supported (the current barebones mechanism does not support circular references for C modules, though). The user ``Duktape.modSearch()`` function encapsulates functionality such as module search paths and file I/O. This is a nice division of labor as it encapsulates practically all of the platform dependent parts of module loading, while keeping the user callback unaware of almost every other aspect of module loading. Using stub module search functions is also easy, which is good for testing. Ecmascript modules follow the CommonJS format, e.g. ``package/lib`` could be represented by the source file:: exports.add = function (a, b) { return a + b; }; CommonJS instructs that modules should be evaluated with certain bindings in force. Duktape currently implements the CommonJS requirements by simply wrapping the module code inside some footer/header code:: (function (require, exports, module) { /* module code here */ }) So the example module would become:: (function (require, exports, module) { exports.add = function (a, b) { return a + b; }; // return value is ignored }) When evaluated, the expression results in a function object (denoted ``F``) which is then called (more or less) like:: var exports = {}; F.call(exports, /* exports also used as 'this' binding */ require, /* require method */ exports, /* exports */ { id: 'package/lib' }); /* module */ A few notes: * The return value of this call is ignored. * The first argument is a new function object whose underlying native function is the same as the global ``require()`` function. This fresh function is needed to facilitate resolution of relative module identifiers: relative identifers are resolved relative to the current module. The resolved absolute identifier of the current module is tracked in ``require.id``. Native code can then pick up the resolution path from the current function object. * The third argument provides the module with its own, resolved identifier. The value in ``module.id`` is guaranteed to be in absolute form, and resolve to the module itself if required from any other module. Duktape doesn't currently support ``module.exports`` like NodeJS, as it is not required by CommonJS. CommonJS module identifier resolution ===================================== CommonJS specifies that identifier terms must be "camelCase": * http://wiki.commonjs.org/wiki/Modules/1.1#Module_Identifiers Some interpret this to mean that e.g. a dash character is not allowed. Such an interpretation seems counterproductive because e.g. filenames often contain dashes, underscores, etc. Duktape allows terms to contain any characters (including non-ASCII and white space) except that: * A term must not begin with a period (``.``) to simplify resolution. Such terms are rejected. * A term cannot contain a forward slash, which (of course) gets interpreted as a separator. * A term cannot contain a U+0000 character. Such terms are currently not rejected. Instead, they terminate the resolution as if the requested identifier had ended. If user code wishes to impose further limits, the module search function can check a resolved identifier and throw an error if it is not of a desirable form. Logger names and tracebacks =========================== Logger name defaulting uses the calling function's ``fileName`` property. The ``fileName`` of the internal module wrapper function is set to the resolved module identifier to make the logger default name come out right. Tracebacks show both ``name`` and ``fileName`` of the internal wrapper function. The ``name`` property is currently not set, so the wrapper function appears anonymous. It could also be set to the module name. module.exports ============== NodeJS allows the default ``exports`` value to be changed by the module being loaded; it can even be replaced e.g. by a function (it's normally an object value). To change the value, the module must assign to ``module.exports`` which initially has the same value as ``exports``: * http://timnew.github.io/blog/2012/04/20/exports_vs_module_exports_in_node_js/ Duktape doesn't currently support assignment to ``module.exports``. C modules and DLLs ================== Recommended convention ---------------------- ``c-module-convention.rst`` describes a recommended convention for defining an init function for a C module. The convention allows a C module to be initialized manually when using static linking, or as part of loading the module from a DLL. The recommendation is in no way mandatory and you can easily write a module loader with your own conventions (see below). However, modules following the recommended convention will be easier to share between projects. Implementing a C module / DLL loader ------------------------------------ The user provided module search function can be used to implement DLL support. Simply load the DLL based on the module identifier, and call some kind of init function in the DLL to register module symbols into the 'exports' table given to the module loader. Mixed C/Ecmascript modules are also possible by first registering symbols provided by C code into the 'exports' table, and then returning the Ecmascript part of the module. The Ecmascript part can access the symbols provided by C code through the shared 'exports' table. Limitations: * Because the module is not yet registered into ``Duktape.modLoaded`` when the module search function executes, circular requires are not handled correctly for C modules. * There is no automatic mechanism to know when a DLL can be unloaded from memory. Tracking the reachability of the exports table of the module (e.g. through a finalizer) is **not** enough because other modules can copy references to individual exported values. Background ========== Module frameworks ----------------- Ecmascript has not traditionally had a module mechanism. In browser environments a web page can load multiple script files in a specific order, each of them introducing more global symbols. This is not very elegant because the order of loading must be correct in case any code runs during loading. Several module mechanisms have since been created for the browser environment to make writing modular Ecmascript easier. Similar needs also exist in non-browser environments and several mechanisms have been defined. References summarizing several module frameworks: * http://addyosmani.com/writing-modular-js/ * http://wiki.commonjs.org/wiki/Modules Module loading APIs or "formats": * Asynchronous Module Definition (AMD) API: - https://github.com/amdjs/amdjs-api/wiki/AMD * CommonJS: - http://wiki.commonjs.org/wiki/Modules/1.1.1 - https://github.com/joyent/node/blob/master/lib/module.js - https://github.com/commonjs/commonjs/tree/master/tests/modules - http://requirejs.org/docs/commonjs.html - http://dailyjs.com/2010/10/18/modules/ * NodeJS, more or less CommonJS: - http://nodejs.org/docs/v0.11.13/api/modules.html * ES6: - https://people.mozilla.org/~jorendorff/es6-draft.html#sec-modules AMD is optimized for the web client side, and requires callback based asynchronous module loading. This model is not very convenient for server side programming, or fully fledged application programming which is more natural with Duktape. CommonJS module format is a server side module mechanism which seems most appropriate to be the default Duktape mechanism. Some NodeJS tests ================= This section illustrates some NodeJS module loader features, as it's nice to align with NodeJS behavior when possible. Assignments ----------- Test module:: // test.js var foo = 123; // not visible outside bar = 234; // assigned to global object this.quux = 345; // exported from module exports.baz = 456; // exported from module Test code:: > var t = require('./test'); undefined > console.log(JSON.stringify(t)); {"quux":345,"baz":456} undefined > console.log(bar); 234 Future work =========== module.exports -------------- Could add support to ``module.exports``. Ability to load modules from C code ----------------------------------- For instance, implement something like:: // Pushes the 'exports' table of 'foo/bar' module to the stack. duk_require_module(ctx, "foo/bar"); This is not a high priority thing as one can simply:: duk_eval_string(ctx, "require('foo/bar')"); Eval invokes the compiler which is not ideal, but modules are usually imported during initialization so this should rarely matter. Better C module support ----------------------- Several ideas to improve the C module support: * Allow C modules to participate in circular requires. Module search and C module init need to be separated for this to be possible, so that the ``Duktape.modLoaded`` registration can be done in-between. * Provide a default DLL loading helper for at least POSIX and Windows. Module unloading support ------------------------ Currently modules cannot be unloaded: once loaded, they're registered to ``Duktape.modLoaded`` permanently, which keeps the exported object permanently reachable (unless removed manually). Adding a finalizer to the exports table is not a solution: another module might hold a reference to a specific symbol within the module but not the exports table itself, e.g.:: var helloFunc = require('hello').func; Collecting a module exports table and executing some unload code is not trivial. Just removing an unused exports object probably requires weak reference support. Isolating a module from the global object ----------------------------------------- Currently ``this`` is bound to ``exports`` so writes through ``this`` do not pollute globals. Variable and function declarations also currently go to the module wrapper function and do not pollute globals. However, plain assignments do write to globals, and reads not matching identifiers declared in scope are read from globals:: fooBar = 123; // if 'fooBar' not in scope, write to global print(barFoo); // if 'barFoo' not in scope, read from global Lua-like module loader ---------------------- The lowest level module mechanism could also be similar to what Lua does. A module would be cached as in CommonJS so that it would only be loaded once per global context. Modules could be loaded with a user callback which takes a module ID and returns the loaded module object (same as the ``exports``) value to be registered into the module cache. The upside of this approach is flexibility: most of the CommonJS module mechanism can be implemented on top of this. One downside is that the module loading mechanism would not be a common one and most users would need to implement or borrow a standard module loader. Another downside is that a Lua-like mechanism doesn't deal with circular module loading while the CommonJS one does (to some extent).