[PL/SQL] basic example as described by the documentation reports a TypeScript type error

[doberkofler]

When trying to use the PlSql in a basic example as described by the documentation TypeScript reports a type error when using the gramma specific PlSqlLexer and PlSqlParser.

This is how the grammar is generated:

# clean
rm -rf grammar
mkdir grammar

# download gramma
curl --output grammar/PlSqlLexer.g4 https://raw.githubusercontent.com/antlr/grammars-v4/master/sql/plsql/PlSqlLexer.g4
curl --output grammar/PlSqlParser.g4 https://raw.githubusercontent.com/antlr/grammars-v4/master/sql/plsql/PlSqlParser.g4

### convert gramma for Typescript
antlr4 -Dlanguage=TypeScript grammar/PlSqlLexer.g4
antlr4 -Dlanguage=TypeScript grammar/PlSqlParser.g4

This is the example code:

import antlr4 from 'antlr4';
import PlSqlLexer from './grammar/PlSqlLexer';
import PlSqlParser from './grammar/PlSqlParser';

const input = 'select * from dual';
const chars = new antlr4.CharStream(input);
const lexer = new PlSqlLexer(chars);
const tokens = new antlr4.CommonTokenStream(lexer); /* type error 1 */
const parser = new PlSqlParser(tokens);
const tree = parser.sql_script();
console.log(tree.toStringTree(null, parser)); /* type error 2 */

type error 1:

Argument of type 'PlSqlLexer' is not assignable to parameter of type 'Lexer'.
Type 'PlSqlLexer' is missing the following properties from type 'Lexer': _input, _interp, text, line, and 18 more.

type error 2:

Argument of type 'PlSqlParser' is not assignable to parameter of type 'Parser'.
Type 'PlSqlParser' is missing the following properties from type 'Parser': _input, _ctx, _interp, _errHandler, and 25 more.

[bkiers]

When you look into the grammars, you will see that both the lexer- and parser grammars have defined a superClass in their options block. That means you should also be using the TS base lexer- and parser classes: https://github.com/antlr/grammars-v4/tree/master/sql/plsql/TypeScript

[doberkofler]

@bkiers Thank you for your feedback. I agree that it should work, but unfortunately it does not seem to work when following the documentation.
By looking at the source, I now discovered that the imports are from a package antlr4ts and not antlr4 as suggested by the antlr4 TypeScript documentation.
There is an antlr4ts package on npm but it does not seem to be maintained and is not what the official documentation describes.

[doberkofler]

When using the antlr4ts package I get different type errors:

import antlr4ts from 'antlr4ts';
import PlSqlLexer from './grammar/PlSqlLexer';
import PlSqlParser from './grammar/PlSqlParser';

const input = 'select * from dual';
const inputStream = antlr4ts.CharStreams.fromString(input);
const lexer = new PlSqlLexer(inputStream);
const tokenStream = new antlr4ts.CommonTokenStream(lexer);
const parser = new PlSqlParser(tokenStream);
const tree = parser.sql_script();
console.log(tree.toStringTree(null, parser));

Reports:

[kaby76]

All grammars in grammars-v4 have a "desc.xml". This file describes what ports work for a grammar, the entry point (if it cannot be found by analyzing the grammar), how to test the grammar, etc. This is analogous to the pom.xml, but for all targets, not just the Java target.

The desc.xml only lists CSharp and Java.

grammars-v4/sql/plsql/desc.xml

Line 4 in eba5c44

<targets>CSharp;Java</targets>

The plsql/ grammar doesn't work for TypeScript. There may be code for a port, but that never means that the port actually "works". A script, sets up the initial values for the desc.xml, which I did earlier in 2023.

I did a "grep grammars-v4 -R -i -e antlr4ts" a few weeks ago to note that plsql/ still uses the old Tunnelvision Antlr fork for TS. #3621.

I will do the port now.

[kaby76]

The JS port is atrociously slow--times out at 300 seconds for "long-running/" examples. BTW, I added these timeouts so the builds don't hang the system, chew up money and resources, and force the Github Action to be killed. That's part of the reason why a port isn't listed as "working" in the desc.xml.

The TS port is a wrapper around the JS port. Here's how the JS port worked on a few examples in "long-running/".

$ bash run.sh long-running/
aggregate01.sql        cast_multiset07.sql    datetime02.sql         order_by07.sql         query_factoring07.sql
08/07-07:10:58 ~/issues/g4-3647/sql/plsql/Generated-JavaScript
$ bash run.sh long-running/*.sql
JavaScript 0 long-running/aggregate01.sql success 107.64
JavaScript 1 long-running/cast_multiset07.sql success 123.18
JavaScript 2 long-running/datetime02.sql success 102.439
JavaScript 3 long-running/order_by07.sql success 133.974
JavaScript 4 long-running/query_factoring07.sql success 178.287
Total Time: 645.549
08/07-07:22:04 ~/issues/g4-3647/sql/plsql/Generated-JavaScript

I'll redo the tests so we parse puny, "Hello World!" examples for these ports. At a later point, I'll have to go through and fix the grammar so that it can parse examples bigger than the "Hello World!" equivalent for plsql.

[kaby76]

Here's the planned "Hello World" examples for the JS port.

08/07-07:33:02 ~/issues/g4-3647/sql/plsql/Generated-JavaScript-hw
$ bash run.sh hw-examples/*.sql
JavaScript 0 hw-examples/alter_operator.sql success 0.302
JavaScript 1 hw-examples/alter_outline.sql success 0.049
JavaScript 2 hw-examples/drop_operator.sql success 0.018
JavaScript 3 hw-examples/keywordasidentifier02.sql success 14.553
JavaScript 4 hw-examples/lexer01.sql success 2.49
JavaScript 5 hw-examples/lexer02.sql success 0.751
JavaScript 6 hw-examples/lexer03.sql success 7.46
JavaScript 7 hw-examples/lexer04.sql success 13.322
JavaScript 8 hw-examples/lexer05.sql success 1.335
JavaScript 9 hw-examples/simple11.sql success 0.397
JavaScript 10 hw-examples/truncate_table.sql success 0.011
Total Time: 40.704
08/07-07:34:03 ~/issues/g4-3647/sql/plsql/Generated-JavaScript-hw

[doberkofler]

Do I understand you correctly:

1. Officially he PlSql grammar currently only supports C# and Java?
2. antlr4 itself would natively support TypeScript for the PlSql grammar has not yet been ported?
3. The PlSql grammar should work with the antlr4ts library?

I have also already tried using the antlr4ts library but still get type errors. Would you have a working example on how to use the PlSql grammar with the antlr4ts library?

Is there any documentation explaining all of this that I missed?

[kaby76]

All grammars--such as this grammar, sql/plsql/--that are split and have target-specific code have to be in "target agnostic format". That's described here. I'm hoping that the proposed "string template actions" PR is adopted, though.

[doberkofler]

Unfortunately I just started having a look at this project and do not understand the implementation details.
I basically just wanted to have a first look at the PL/SQL parser to evaluate if it could be used as the starting point for some major automated code refactoring in a large PLZ/SQL code base. TypeScript is one of the language I currently use the most and thats what I wanted to start with.

Could you please just confirm that I understood you correctly:

1. Officially he PlSql grammar currently only supports C# and Java?
2. antlr4 itself would natively support TypeScript for the PlSql grammar has not yet been ported?
3. The PlSql grammar should work with the antlr4ts library but they also do no longer?

[kaby76]

1. Officially he PlSql grammar currently only supports C# and Java?

It currently only supports CSharp (the "official" name of the C# port) and Java. But, I have a PR that fixes the JavaScript and TypeScript ports. #3650

2. antlr4 itself would natively support TypeScript for the PlSql grammar has not yet been ported?

Antlr version 4.12.0 supports TypeScript, but at it stands at this second, it does not work for TypeScript, nor JavaScript. The PR I'm finishing up fixes this. I have to bump up the "" in the desc.xml to note to the testing scripts and to people that at least 4.12.0 is required. We test using 4.13.0 for grammars-v4.

3. The PlSql grammar should work with the antlr4ts library but they also do no longer?

It probably does still "work", but "work" is a relative term. The generated parser is really slow, and I doubt that it would be any faster than the "official" Antlr Typescript port. Also, we just don't support the TunnelVision port for TypeScript. We don't have time to support all the different targets out there. I can only work so much, and have time only for the "official" targets.

The plan is to work on optimizing the grammar on a few of the more egregious problems in a day or two.

[doberkofler]

I understand. Thank you for all the good work.