typedoc internally parsed output not working for me

[wildcat63]

Thanks!

I was just getting started documenting a typescript project, and I was very pleased to see that I could use Sphinx with your extension, since I've used Sphinx before for python projects, and I find it very useful and mature.

Before going through my problem and fix, I want to say thanks for the work that all of the contributors have done on this.

My issue

The doclets provided by the typedoc.py post-typedoc parser weren't working for me.

You will note that I'm using typedoc 0.13.0. Unfortunately I was unable to check typedoc 0.11.1 on my project because it complained about react .t.ds files, which was probably due to either a problem with changes to those files or an overly strict tsconfig.json.

The problems I ran into fell into two categories:

• the comment and description field resulted in doclets being seen as relevant that shouldn't have
• the path information in meta was causing an exception in doclet_full_path in the PathVisitor call

Without really having a good idea of how you guys are parsing output, I've come up with modifications that make it work for me, but what I've done may simply be a hack, so feel free to ignore it and find a more appropriate solution.

My test functions

To evaluate what was going on for jsdoc and typedoc, I compared the output from the following functions:

in javascript

/**
 * Adds two numbers together
 *
 * @param {number} first The First Number
 * @param {number} second The Second Number
 * @returns {number}
 */
function add(first, second) {
    return first + second;
}

and

in typescript

/**
 * Adds two numbers together
 *
 */
export const add = (first: number, second: number): number => {
    return first + second;
}

The desired output from typedoc should, more or less, match the jsdoc output, which is the following:

{
    'comment':
    '/**\n * Adds two numbers together\n *\n * @param {number} first The First Number\n * @param {number} second The Second Number\n * @returns {number}\n */',
    'meta': {
        'range': [147, 205],
        'filename': 'simple.js',
        'lineno': 8,
        'columnno': 0,
        'path': '/Users/xxx/development/current/yyy/src',
        'code': {
            'id': 'astnode100000002',
            'name': 'add',
            'type': 'FunctionDeclaration',
            'paramnames': ['first', 'second']
        }
    },
    'description':
    'Adds two numbers together',
    'params': [
        {
            'type': {
                'names': ['number']
            },
            'description': 'The First Number',
            'name': 'first'
        },
        {
            'type': {
            'names': ['number']
            },
            'description': 'The Second Number',
            'name': 'second'
            }
    ],
    'returns': [{
        'type': {
            'names': ['number']
        }
    }],
    'name': 'add',
    'longname': 'add',
    'kind': 'function',
    'scope': 'global'
}

The first challenge - comment and description field values

I first saw an issue occurring in the function gather_doclets when collecting relevant doclets. Currently, the test is

if d.get('comment') and not d.get('undocumented')

All of the doclets I was seeing had a comment key equal to the string <empty> which enabled every doclet to be considered relevant.

The source for <empty> is in typedoc.py and occurs in the following call:

        return self.make_doclet(
            kind=kind,
            access=access,
            comment=node.get('comment', {}).get('text', '<empty>'),
            meta=self.make_meta(node),
            name=node.get('name'),
            longname=self.make_longname(node),
            memberof=memberof,
            description=self.make_description(comment)
        )

In my typedoc node, there was no "text" key, but there was a "shortText" key.

{
    'id': 1139,
    'name': 'add',
    'kind': 4096,
    'kindString': 'Call signature',
    'flags': {},
    'comment': {
        'shortText': 'Adds two numbers together'
    },
    'parameters': [{
        'id': 1140,
        'name': 'first',
        'kind': 32768,
        'kindString': 'Parameter',
        'flags': {},
        'type': {
            'type': 'intrinsic',
            'name': 'number'
        },
        '__parentId': 1139
    },
    {
        'id': 1141,
        'name': 'second',
        'kind': 32768,
        'kindString': 'Parameter',
        'flags': {},
        'type': {
            'type': 'intrinsic',
            'name': 'number'
        },
        '__parentId': 1139
    }],
    'type': {
        'type': 'intrinsic',
        'name': 'number'
    },
    '__parentId':
    1138,
    'sources': [{
        'fileName': 'simple.ts',
        'line': 5,
        'character': 16
    }]
}

For me, changing "text" to "shortText" works, if only for this small test. In addition, the jsdoc description was the short one and the comment the long one, so I made the change to be:

        return self.make_doclet(
            kind=kind,
            access=access,
            comment=self.make_description(comment),
            meta=self.make_meta(node),
            name=node.get('name'),
            longname=self.make_longname(node),
            memberof=memberof,
            description=node.get('comment', {}).get('shortText', ''),
        )

I've done three things: (a) referenced "shortText" rather than "text"; (b) made the default the empty string rather than "", and (c) I've swapped comment and description (I'm not sure this matters, and I'm not sure if it's right, either).

I have no idea if this change is meaningful for anything other than the small test case I ran, too.

The second challenge - path data in the doclet

The jsdoc doclet provides a "meta" key with (relevant) values like the following:

{
    'meta': {
        'filename': 'simple.js',
        'path': '/Users/xxx/development/current/yyy/src',
    }
}

The doclet coming back from typedoc.py, however, had the following "meta" value:

{
    'meta': {
        'path': './',
        'filename': 'simple.ts',
}

I didn't see enough information provided to the TypeDoc class to recreate the jsdoc version of "meta" without passing along abs_source_paths, but since the end result of doclet_full_path appeared to be relatively simple, I made a quick hack to get the result differently if the language is typescript.

It's quite possible that the simplistic approach below removes some of the significant versatility of this package.

I start by passing the language as a variable to doclet_full_path since parsing the doclet will be different depending on the language. The two calls now (lines 46 and 67) add that in:

doclet_full_path(d, root_for_relative_paths, app.config.js_language)  # line 46
segments = doclet_full_path(d, root_for_relative_paths, app.config.js_language, longname_field='memberof') # line 67

The function signature of doclet_full_path becomes:

def doclet_full_path(d, base_dir, jslanguage, longname_field='longname'):

and the original test is wrapped in a language check, with a (very simple and not heavily tested) version for typescript:

    if jslanguage != 'typescript':
        meta = d['meta']
        rel = relpath(meta['path'], base_dir)
        rel = '/'.join(rel.split(sep))
        if not rel.startswith(('../', './')) and rel not in ('..', '.'):
            # It just starts right out with the name of a folder in the cwd.
            rooted_rel = './%s' % rel
        else:
            rooted_rel = rel

        # Building up a string and then parsing it back down again is probably
        # not the fastest approach, but it means knowledge of path format is in
        # one place: the parser.
        path = '%s/%s.%s' % (rooted_rel,
                             splitext(meta['filename'])[0],
                             d[longname_field])
    else:
        item = d[longname_field].split(':')[-1]
        path = '%s%s' % (d['meta']['path'], item)

Conclusion

With these changes, sphinx-js with typedoc 0.13.0 works for the very limited amount of documentation that I've tried. I looked at typedoc to see if they had made changes to their json serializer, but I didn't see any, which makes me wonder if all of this is due to user error in the setup of documentation in my simple code.

One way or another, though, it is working for me, and I thank you guys for the efforts you've made.

[erikrose]

Thanks! I've put this in my review queue. Though perhaps @WimYedema will get to it first.

[erikrose]

The '<empty>'/shortText fix sounds good. If it passes the existing tests (and a new one for itself), I'm happy to merge it.

For the rest, we should probably wait until #120 lands. This will give us a well-defined format to spit info into, rather than having to guess what JSDoc means by "comment" and "description", for instance. You'll be able to squirrel away in the IR whatever information you need to get the right result from doclet_full_path() rather than having to thread the language into it explicitly. Most likely, doclet_full_path will become a method on some object in the IR.

[erikrose]

How are things now? I've rewritten (on master) most of what was giving you problems.