4.6

ANTLR version 4.6 is a major release with many features and bug fixes.

Summary of new features, improvements, fixes

• NEW TARGETS: C++, Swift, Go:
  • Peter Boyer (Go target)
  • Mike Lischke (C++ completed target)
  • Dan McLaughlin (C++ initial target)
  • David Sisson (C++ initial target and test)
  • Janyou (Swift target)
  • Ewan Mellor, Hanzhou Shi (Swift target merging)
• Major performance improvement discovered by @sharwell. We were seeing a number of reports of nonterminating parses for very large expressions with grammars that used left recursive rule specifications (like e : e '*' e | INT ;). Sam noticed that the parsing engine detected an ambiguity between two choices and resolved it properly but that both would lead to a valid parse. It turns out that the first choice is what led to really slow parsing but the second choice is much faster and still yields a valid parse. Most targets have added that improvement. Here is the discussion of the issue and solution and the primary patch for java.
• Error handling improved in cases related to LL(1) decisions for optional and closure subrules. Matching sets of tokens were also sometimes considered resynchronization (error) tokens. Single token deletions that occurred at the start of expressions were sometimes not included in the parse tree as error nodes.
  • Add sync() calls back in for LL(1) decisions. (code-gen, error-handling)
  • Matched tokens being marked as error (comp:runtime, error-handling, trees-contexts, type:bug)
  • Single-token deletion error node not in parse tree (error-handling, trees-contexts, type:bug)
• The tool itself generates better error messages:
  • Crash with NullPointerException in SymbolChecks.checkForModeConflicts with empty lexer (comp:tool, grammars, type:bug)
  • Have ANTLR4 prevent conflict with user rule names by behind-the-scenes renaming of its own variables (status:cannot-reproduce)
  • Add compile-time errors/warnings for use of incompatible lexer commands (comp:tool, error-handling)
  • character set issues (comp:tool, error-handling, type:improvement)
  • Duplicated and incompatible commands (comp:tool, error-handling, type:improvement)
• We fixed a number of issues with the build system
  • Tidy poms (comp:build, type:improvement)
  • divide and conquer Travis CI (comp:build, type:improvement)
  • Version information omitted from MANIFEST.MF in antlr-4.5.1-complete.jar (comp:build, type:bug)
  • antrl4--sources.jar does not contain sources for runtime (comp:build, type:bug)
  • OSGi module name does not reflect Maven module name (comp:build, type:improvement)
  • Running a limited subset of runtime-testsuite (comp:testing, type:improvement)
• Improvements to the ANTLR4 mvn plug-in:
  • Grammar dependency management for Maven plugin (comp:build, comp:maven-plugin, type:improvement)
  • antlr4-maven-plugin: code generated in wrong encoding (comp:maven-plugin, type:improvement)
• The ANTLR unit test mechanism was completely rewritten but it should not affect anyone other than the development team
  • Refactor runtime test generation mechanism (comp:testing, type:improvement)
• We assume java 1.7 not 1.6 now.
  • Drop support for 1.6, move pom.xml etc. to JDK 1.7 (cross-platform-issue, type:improvement)
• Miscellaneous
  • make sure BSD 3-clause license prefix is consistent and on all appropriate files (type:improvement)
  • Allow paths as tokenVocab option (comp:build, comp:tool, type:improvement)
  • add TokenStreamRewriter to Python2 runtime (target:python2, type:improvement)
  • Add an iterative version of the ParseTreeWalker (comp:runtime, target:java, trees-contexts, type:improvement)
  • indicate token types as well as numbers in -tokens dump (target:java, type:improvement)
  • Binary parsing (comp:doc, type:improvement, type:question)

Issues fixed

• Python 2 Target's ErrorStrategy Creates Problematic Tokens (target:python2, type:bug)
• Prefix rules that also recurse are not detected (grammars, type:bug)
• mvn compile failing. Could not resolve dependencies (comp:build, comp:doc, type:bug)
• Repeated epsilon transitions in ATN from left-recursive grammar (atn-analysis, type:bug)
• Add sync() calls back in for LL(1) decisions. (code-gen, error-handling)
• Array2DHashSet.clear does not reset threshold (target:java, type:bug)
• ANTLR 4.5.3 & java.lang.StackOverflowError (comp:tool, type:bug)
• Matched tokens being marked as error (comp:runtime, error-handling, trees-contexts, type:bug)
• Single-token deletion error node not in parse tree (error-handling, trees-contexts, type:bug)
• $e.v yields incorrect value 0 in "e returns [int v] : '1' {$v = 1;} | '(' e ')' {$v = $e.v;} ;" (actions, grammars, type:bug)
• Strange code in JavaScript - PredictionContext.js line 191 (target:javascript, type:bug)
• A bug of the antlr4 python3 runtime 4.5.3 (target:python3, type:bug)
• Empty alternative in rule causes syntax error in *Parser.py for Python2/3 targets (target:python2, target:python3, type:bug)
• getText method with arguments not always works correctly (target:java, type:bug)
• maven plugin dependency tracking ignores imported files (comp:build, comp:maven-plugin, comp:tool)
• NullPointerException on invalid attribute reference (comp:tool, type:bug)
• getLookaheadParseTrees() should reset or create new error handler each alt but doesn't (comp:runtime, trees-contexts, type:bug)
• getAllPossibleParseTrees() doesn't work when ambig input reaches eof. (comp:runtime, trees-contexts, type:bug)
• Changing '+' to '*' crashes antlr4 (atn-analysis, parsers, type:bug)
• Crash with NullPointerException in SymbolChecks.checkForModeConflicts with empty lexer (comp:tool, grammars, type:bug)
• LB() can try to scan to i < 0 (comp:runtime, type:bug)
• Incorrect childs generation with array of element label (grammars, status:invalid)
• TokenStreamRewriter for Python (target:python2)
• getTokenTypeMap should test all token type values (comp:runtime, target:java, type:bug)
• Version information omitted from MANIFEST.MF in antlr-4.5.1-complete.jar (comp:build, type:bug)
• antrl4--sources.jar does not contain sources for runtime (comp:build, type:bug)
• template error: attribute parser isn't defined (target:javascript, type:bug)
• Add compile-time errors/warnings for use of incompatible lexer commands (comp:tool, error-handling)