[generator] Add generator --with-javadoc-xml=FILE support.

Context: dotnet#687 (comment)
Commit 69e1b80 added `tools/java-source-utils`, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:
# API-30 `sources` package contains both `Object.java` and `Object.annotated.java`;
# Skip the `.annotated.java` files
$ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \
| grep -v '\.annotated\.' > sources.txt
$ java -jar java-source-utils.jar -v \
--source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
--output-javadoc android-javadoc.xml \
@sources.txt
What can we *do* with the generated `android-javadoc.xml`?
`android-javadoc.xml` contains parameter names, and thus can be used
with `class-parse --docspath`; see commit 806082f.
What we *really* want to do is make the Javadoc information *useful*
to consumers of the binding assembly. This means that we want a
C# XML Documentation file for the binding assembly.
The most straightforward way to get a C# XML Documentation File is to
emit [C# XML Documentation Comments][0] into the binding source code!
Add a new `generator --with-javadoc-xml=FILE` option. When specified,
`FILE` will be treated as an XML file containing the output from
`java-source-utils.jar --output-javadoc` (69e1b80), and all
`<javadoc/>` elements within the XML file will be associated with C#
types and members to emit, based on the `//@jni-signature` and
`//@name` attributes, as appropriate.
When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)
To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to `Java.Interop.Tools.JavaSource.dll`,
in the new `Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser`
type, which parses the Javadoc content and translates to XML.
In addition to transforming the Javadoc comments into C# XML
documentation comments, we *also* want to provide "upstream
information" in the form of:
1. A URL to the corresponding online Javadoc HTML documentation.
2. A copyright notice disclaimer.
Allow provision of this information by updating
`java-source-utils.jar` to support the new options:
--doc-copyright FILE Copyright information for Javadoc. Should be in
mdoc(5) XML, to be held within <remarks/>.
Stored in //javadoc-metadata/copyright.
--doc-url-prefix URL Base URL for links to documentation.
Stored in //javadoc-metadata/link/@Prefix.
--doc-url-style STYLE STYLE of URLs to generate for member links.
Stored in //javadoc-metadata/link/@Style.
Supported styles include:
- developer.android.com/reference@2020-Nov
The new `/api/javadoc-metadata/link@prefix` and
`/api/javadoc-metadata/link@style` XML attributes, stored within
`java-source-utils.jar --output-javadoc` XML output, allow
construction of a URL to the Java member.
For example, given:
java -jar java-source-utils.jar \
--doc-url-prefix https://developer.android.com/reference \
--doc-url-style developer.android.com/reference@2020-Nov
Then `generator` can emit the C# documentation comment for
`java.lang.Object.equals(Object)`:
/// <format type="text/html">
/// <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
/// </format>
The copyright notice disclaimer is supported by
`java-source-utils.jar --doc-copyright FILE`; the contents of `FILE`
are inserted into the `/api/javadoc-metadata/copyright` element, and
will be copied into the output of every C# XML documentation block.
Example output is at:
* <https://gist.github.com/jonpryor/004f01f4cd5ff32299ff590ba7a2fe0e>
Unfortunately, converting Javadoc to C# XML Documentation Comments is
not a "zero cost" operation. Add a new
`generator --doc-comment-verbosity=STYLE` option to control how
"complete" the generated documentation comments are:
--doc-comment-verbosity=STYLE
STYLE of C# documentation comments to emit.
Defaults to `full`. STYLE may be:
* `intellisense`: emit <summary>, <param>,
<returns>, <exception>.
* `full`: plus <remarks>, <seealso>, ...
Using `--doc-comment-verbosity=full` will *most* impact build times.
TODO:
* `SourceJavadocToXmldocParser` doesn't support many constructs.
[0]: https://docs.microsoft.com/en-us/dotnet/csharp/codedoc

Author: jonpryor

src/Java.Interop.Tools.JavaSource/Java.Interop.Tools.JavaSource/IronyExtensions.cs

@@ -0,0 +1,28 @@
+usingSystem;
+usingSystem.Collections.Generic;
+usingSystem.IO;
+usingSystem.Linq;
+usingSystem.Text;
+usingIrony.Ast;
+usingIrony.Parsing;
+
+namespaceJava.Interop.Tools.JavaSource{
+
+staticclassIronyExtensions{
+
+publicstaticvoidMakePlusRule(thisNonTerminalstar,Grammargrammar,BnfTermdelimiter)
+{
+star.Rule=grammar.MakePlusRule(star,delimiter);
+}
+
+publicstaticvoidMakeStarRule(thisNonTerminalstar,Grammargrammar,BnfTermdelimiter,BnfTermof)
+{
+star.Rule=grammar.MakeStarRule(star,delimiter,of);
+}
+
+publicstaticvoidMakeStarRule(thisNonTerminalstar,Grammargrammar,BnfTermof)
+{
+star.Rule=grammar.MakeStarRule(star,of);
+}
+}
+}

src/Java.Interop.Tools.JavaSource/Java.Interop.Tools.JavaSource/JavadocInfo.cs

@@ -0,0 +1,31 @@
+usingSystem;
+usingSystem.Collections.Generic;
+usingSystem.Collections.ObjectModel;
+usingSystem.IO;
+usingSystem.Linq;
+usingSystem.Xml.Linq;
+
+usingIrony.Ast;
+usingIrony.Parsing;
+
+namespaceJava.Interop.Tools.JavaSource{
+
+sealedclassJavadocInfo{
+publicreadonlyICollection<XNode>Exceptions=newCollection<XNode>();
+publicreadonlyICollection<XNode>Extra=newCollection<XNode>();
+publicreadonlyICollection<XNode>Remarks=newCollection<XNode>();
+publicreadonlyICollection<XNode>Parameters=newCollection<XNode>();
+publicreadonlyICollection<XNode>Returns=newCollection<XNode>();
+
+publicoverridestringToString()
+{
+returnnewXElement("Javadoc",
+newXElement(nameof(Parameters),Parameters),
+newXElement(nameof(Remarks),Remarks),
+newXElement(nameof(Returns),Returns),
+newXElement(nameof(Exceptions),Exceptions),
+newXElement(nameof(Extra),Extra))
+.ToString();
+}
+}
+}

src/Java.Interop.Tools.JavaSource/Java.Interop.Tools.JavaSource/SourceJavadocToXmldocGrammar.BlockTagsBnfTerms.cs

@@ -0,0 +1,239 @@
+usingSystem;
+usingSystem.Collections.Generic;
+usingSystem.IO;
+usingSystem.Linq;
+usingSystem.Xml.Linq;
+
+usingIrony.Ast;
+usingIrony.Parsing;
+
+namespaceJava.Interop.Tools.JavaSource{
+
+publicpartialclassSourceJavadocToXmldocGrammar{
+
+// https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javadoc.html#javadoctags
+publicclassBlockTagsBnfTerms{
+
+internalBlockTagsBnfTerms()
+{
+}
+
+internalvoidCreateRules(SourceJavadocToXmldocGrammargrammar)
+{
+AllBlockTerms.Rule=AuthorDeclaration
+|ApiSinceDeclaration
+|DeprecatedDeclaration
+|DeprecatedSinceDeclaration
+|ExceptionDeclaration
+|ParamDeclaration
+|ReturnDeclaration
+|SeeDeclaration
+|SerialDataDeclaration
+|SerialFieldDeclaration
+|SinceDeclaration
+|ThrowsDeclaration
+|UnknownTagDeclaration
+|VersionDeclaration
+;
+BlockValue.Rule=grammar.HtmlTerms.ParsedCharacterData
+|grammar.HtmlTerms.InlineDeclaration
+;
+BlockValues.MakePlusRule(grammar,BlockValue);
+
+AuthorDeclaration.Rule="@author"+BlockValues;
+AuthorDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.AuthorTag))
+return;
+// Ignore; not sure how best to convert to Xmldoc
+FinishParse(context,parseNode);
+};
+
+ApiSinceDeclaration.Rule="@apiSince"+BlockValues;
+ApiSinceDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.SinceTag)){
+return;
+}
+varp=newXElement("para","Added in API level ",AstNodeToXmlContent(parseNode.ChildNodes[1]),".");
+FinishParse(context,parseNode).Remarks.Add(p);
+parseNode.AstNode=p;
+};
+
+DeprecatedDeclaration.Rule="@deprecated"+BlockValues;
+DeprecatedDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.DeprecatedTag)){
+return;
+}
+varp=newXElement("para","This member is deprecated. ",AstNodeToXmlContent(parseNode.ChildNodes[1]));
+FinishParse(context,parseNode).Remarks.Add(p);
+parseNode.AstNode=p;
+};
+
+DeprecatedSinceDeclaration.Rule="@deprecatedSince"+BlockValues;
+DeprecatedSinceDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.DeprecatedTag)){
+return;
+}
+varp=newXElement("para","This member was deprecated in API level ",AstNodeToXmlContent(parseNode.ChildNodes[1]),".");
+FinishParse(context,parseNode).Remarks.Add(p);
+parseNode.AstNode=p;
+};
+
+varnonSpaceTerm=newRegexBasedTerminal("[^ ]","[^ ]+"){
+AstConfig=newAstNodeConfig{
+NodeCreator=(context,parseNode)=>parseNode.AstNode=parseNode.Token.Value,
+},
+};
+
+ExceptionDeclaration.Rule="@exception"+nonSpaceTerm+BlockValues;
+ExceptionDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.ExceptionTag)){
+return;
+}
+// TODO: convert `nonSpaceTerm` into a proper CREF
+vare=newXElement("exception",
+newXAttribute("cref",string.Join("",AstNodeToXmlContent(parseNode.ChildNodes[1]))),
+AstNodeToXmlContent(parseNode.ChildNodes[2]));
+FinishParse(context,parseNode).Exceptions.Add(e);
+parseNode.AstNode=e;
+};
+
+ParamDeclaration.Rule="@param"+nonSpaceTerm+BlockValues;
+ParamDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.ParamTag)){
+return;
+}
+varp=newXElement("param",
+newXAttribute("name",string.Join("",AstNodeToXmlContent(parseNode.ChildNodes[1]))),
+AstNodeToXmlContent(parseNode.ChildNodes[2]));
+FinishParse(context,parseNode).Parameters.Add(p);
+parseNode.AstNode=p;
+};
+
+ReturnDeclaration.Rule="@return"+BlockValues;
+ReturnDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.ReturnTag)){
+return;
+}
+varr=newXElement("returns",
+AstNodeToXmlContent(parseNode.ChildNodes[1]));
+FinishParse(context,parseNode).Returns.Add(r);
+parseNode.AstNode=r;
+};
+
+SeeDeclaration.Rule="@see"+BlockValues;
+SeeDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.SeeTag)){
+return;
+}
+// TODO: @see supports multiple forms; see: https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javadoc.html#see
+vare=newXElement("seealso",
+newXAttribute("cref",string.Join("",AstNodeToXmlContent(parseNode.ChildNodes[1]))));
+FinishParse(context,parseNode).Extra.Add(e);
+parseNode.AstNode=e;
+};
+
+SinceDeclaration.Rule="@since"+BlockValues;
+SinceDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.SinceTag)){
+return;
+}
+varp=newXElement("para","Added in ",AstNodeToXmlContent(parseNode.ChildNodes[1]),".");
+FinishParse(context,parseNode).Remarks.Add(p);
+parseNode.AstNode=p;
+};
+
+ThrowsDeclaration.Rule="@throws"+nonSpaceTerm+BlockValues;
+ThrowsDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.ExceptionTag)){
+return;
+}
+// TODO: convert `nonSpaceTerm` into a proper CREF
+vare=newXElement("exception",
+newXAttribute("cref",string.Join("",AstNodeToXmlContent(parseNode.ChildNodes[1]))),
+AstNodeToXmlContent(parseNode.ChildNodes[2]));
+FinishParse(context,parseNode).Exceptions.Add(e);
+parseNode.AstNode=e;
+};
+
+// Ignore serialization informatino
+SerialDeclaration.Rule="@serial"+BlockValues;
+SerialDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.SerialTag)){
+return;
+}
+FinishParse(context,parseNode);
+};
+
+SerialDataDeclaration.Rule="@serialData"+BlockValues;
+SerialDataDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.SerialTag)){
+return;
+}
+FinishParse(context,parseNode);
+};
+
+SerialFieldDeclaration.Rule="@serialField"+BlockValues;
+SerialFieldDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.SerialTag)){
+return;
+}
+FinishParse(context,parseNode);
+};
+
+varunknownTagTerminal=newRegexBasedTerminal("@[unknown]",@"@\S+"){
+Priority=TerminalPriority.Low,
+};
+unknownTagTerminal.AstConfig.NodeCreator=(context,parseNode)=>
+parseNode.AstNode=parseNode.Token.Value.ToString();
+
+
+UnknownTagDeclaration.Rule=unknownTagTerminal+BlockValues;
+UnknownTagDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.Remarks)){
+return;
+}
+Console.WriteLine($"# Unsupported @block-tag value: {parseNode.ChildNodes[0].AstNode}");
+FinishParse(context,parseNode);
+};
+
+// Ignore Version
+VersionDeclaration.Rule="@version"+BlockValues;
+VersionDeclaration.AstConfig.NodeCreator=(context,parseNode)=>{
+if(!grammar.ShouldImport(ImportJavadoc.VersionTag)){
+return;
+}
+FinishParse(context,parseNode);
+};
+}
+
+publicreadonlyNonTerminalAllBlockTerms=newNonTerminal(nameof(AllBlockTerms),ConcatChildNodes);
+
+publicreadonlyTerminalCdata=newCharacterDataTerminal("#CDATA",preserveLeadingWhitespace:true);
+/*
+ public readonly Terminal Cdata = new RegexBasedTerminal (nameof (BlockValue), "[^<]*") {
+ AstConfig = new AstNodeConfig {
+ NodeCreator = (context, parseNode) => parseNode.AstNode = parseNode.Token.Value.ToString (),
+ },
+ };
+ */
+
+publicreadonlyNonTerminalBlockValue=newNonTerminal(nameof(BlockValue),ConcatChildNodes);
+publicreadonlyNonTerminalBlockValues=newNonTerminal(nameof(BlockValues),ConcatChildNodes);
+publicreadonlyNonTerminalAuthorDeclaration=newNonTerminal(nameof(AuthorDeclaration));
+publicreadonlyNonTerminalApiSinceDeclaration=newNonTerminal(nameof(ApiSinceDeclaration));
+publicreadonlyNonTerminalDeprecatedDeclaration=newNonTerminal(nameof(DeprecatedDeclaration));
+publicreadonlyNonTerminalDeprecatedSinceDeclaration=newNonTerminal(nameof(DeprecatedSinceDeclaration));
+publicreadonlyNonTerminalExceptionDeclaration=newNonTerminal(nameof(ExceptionDeclaration));
+publicreadonlyNonTerminalParamDeclaration=newNonTerminal(nameof(ParamDeclaration));
+publicreadonlyNonTerminalReturnDeclaration=newNonTerminal(nameof(ReturnDeclaration));
+publicreadonlyNonTerminalSeeDeclaration=newNonTerminal(nameof(SeeDeclaration));
+publicreadonlyNonTerminalSerialDeclaration=newNonTerminal(nameof(SerialDeclaration));
+publicreadonlyNonTerminalSerialDataDeclaration=newNonTerminal(nameof(SerialDataDeclaration));
+publicreadonlyNonTerminalSerialFieldDeclaration=newNonTerminal(nameof(SerialFieldDeclaration));
+publicreadonlyNonTerminalSinceDeclaration=newNonTerminal(nameof(SinceDeclaration));
+publicreadonlyNonTerminalThrowsDeclaration=newNonTerminal(nameof(ThrowsDeclaration));
+publicreadonlyNonTerminalUnknownTagDeclaration=newNonTerminal(nameof(UnknownTagDeclaration));
+publicreadonlyNonTerminalVersionDeclaration=newNonTerminal(nameof(VersionDeclaration));
+}
+}
+}