Change sourceSpan for Comment nodes to cover the whole comment
instead of just the opening token.
The primary motivation for this is the interaction between ESLint and
`@angular-eslint`. ESLint can detect unused `eslint-disable` directives
in comments and automatically remove them when running with `--fix`.
This is based on ranges computed from AST spans, and as a result
does not work inside Angular templates - right now all comments
claim to be 4 characters long so only the opening `<!--` is removed.
PR Close#50855
Fixes that using braces in the block parameters would result in incorrect tokens being produced. Currently we don't have any blocks that allow object literal parameters, but it may come up in the future.
PR Close#51143
⚠️Disclaimer⚠️ this PR implements syntax that is still in an open RFC. It will be adjusted once the RFC is closed.
These changes implement the `BlockGroup` and `Block` AST nodes that will then be used to generate instructions based on the new syntax. A `BlockGroup` is a container for `Block` instances. The first block of a block is always implicit and required while any subsequent blocks are optional.
PR Close#50953
⚠️Disclaimer⚠️ this PR implements syntax that is still in an open RFC. It will be adjusted once the RFC is closed.
These changes extend the lexer to recognize the concepts of a block group (`{#foo paramA; paramB}{/foo}`) and a block (`{:foo paramA; paramB;}`) which will be useful later on for the control flow and defer proposals. Block groups can be used anywhere and require a closing tag while block can only be used inside of a block.
The idea is that in the next PRs the markup AST will be expanded to have some more specialized node like `ConditionalBlock` or `DeferBlock` which will then be turned into instructions.
PR Close#50895
Allows for self-closing tags to be used for non-native tag names, e.g. `<foo [input]="bar"></foo>` can now be written as `<foo [input]="bar"/>`. Native tag names still have to have closing tags.
Fixes#39525.
PR Close#48535
.substr() is deprecated so we replace it with functions which work similarily but aren't deprecated
Signed-off-by: Tobias Speicher <rootcommander@gmail.com>
PR Close#45397
After updating to a more recent version of rollup, rollup started to
complain because the `TreeParseResult` class is being re-exported
twice in the `index.ts -> public-api.ts -> compiler.ts` entry-point.
Rollup threw errors like:
```
Error: "ParseTreeResult" cannot be exported from
<..>/ml_parser/parser.mjs as it is a re-export that references itself.
```
It seems like Rollup ideally would not throw here, similar to TypeScript
which detects that these exports are the same and just dedupes them, but
it's low-effort fixing this for now and actually is a good opportunity
to make the public API a little more easy understand (when looking at
the `compiler.ts` file).
PR Close#43431
Adds support for TypeScript 4.4. High-level overview of the changes made in this PR:
* Bumps the various packages to `typescript@4.4.2` and `tslib@2.3.0`.
* The `useUnknownInCatchVariables` compiler option has been disabled so that we don't have to cast error objects explicitly everywhere.
* TS now passes in a third argument to the `__spreadArray` call inside child class constructors. I had to update a couple of places in the runtime and ngcc to be able to pick up the calls correctly.
* TS now generates code like `(0, foo)(arg1, arg2)` for imported function calls. I had to update a few of our tests to account for it. See https://github.com/microsoft/TypeScript/pull/44624.
* Our `ngtsc` test setup calls the private `matchFiles` function from TS. I had to update our usage, because a new parameter was added.
* There was one place where we were setting the readonly `hasTrailingComma` property. I updated the usage to pass in the value when constructing the object instead.
* Some browser types were updated which meant that I had to resolve some trivial type errors.
* The downlevel decorators tranform was running into an issue where the Closure synthetic comments were being emitted twice. I've worked around it by recreating the class declaration node instead of cloning it.
PR Close#43281
These token interfaces will make it easier to reason about tokens in the
parser and in specs.
Previously, it was never clear what items could appear in the `parts`
array of a token given a particular `TokenType`. Now, each token interface
declares a labelled tuple for the parts, which helps to document the token
better.
PR Close#43132
When it was tokenized, text content is split into parts that can include
interpolations and encoded entities tokens.
To make this information available to downstream processing, this commit
adds these tokens to the `Text` AST nodes, with suitable processing.
PR Close#43132
The lexer now splits encoded entity tokens out from text and attribute value tokens.
Previously encoded entities would be decoded and the decoded value would be
included as part of the text token of the surrounding text. Now the entities
have their own tokens. There are two scenarios: text and attribute values.
Previously the contents of `<div>Hello & goodbye</div>` would be a single
TEXT token. Now it will be three tokens:
```
TEXT: "Hello "
ENCODED_ENTITY: "&", "&"
TEXT: " goodbye"
```
Previously the attribute value in `<div title="Hello & goodbye">` would be
a single text token. Now it will be three tokens:
```
ATTR_VALUE_TEXT: "Hello "
ENCODED_ENTITY: "&", "&"
ATTR_VALUE_TEXT: " goodbye"
```
- ENCODED_ENTITY tokens have two parts: "decoded" and "encoded".
- ENCODED_ENTITY tokens are always preceded and followed by either TEXT tokens
or ATTR_VALUE_TEXT tokens, depending upon the context, even if they represent
an empty string.
The HTML parser has been modified to recombine these tokens to allow this
refactoring to have limited effect in this commit. Further refactorings
to use these new tokens will follow in subsequent commits.
PR Close#43132
The lexer now splits interpolation tokens out from text tokens.
Previously the contents of `<div>Hello, {{ name}}<div>` would be a single
text token. Now it will be three tokens:
```
TEXT: "Hello, "
INTERPOLATION: "{{", " name", "}}"
TEXT: ""
```
- INTERPOLATION tokens have three parts, "start marker", "expression"
and "end marker".
- INTERPOLATION tokens are always preceded and followed by TEXT tokens,
even if they represent an empty string.
The HTML parser has been modified to recombine these tokens to allow this
refactoring to have limited effect in this commit. Further refactorings
to use these new tokens will follow in subsequent commits.
PR Close#43132
These token interfaces will make it easier to reason about tokens in the
parser and in specs.
Previously, it was never clear what items could appear in the `parts`
array of a token given a particular `TokenType`. Now, each token interface
declares a labelled tuple for the parts, which helps to document the token
better.
PR Close#42062
Previously, the way templates were tokenized meant that we lost information
about the location of interpolations if the template contained encoded HTML
entities. This meant that the mapping back to the source interpolated strings
could be offset incorrectly.
Also, the source-span assigned to an i18n message did not include leading
whitespace. This confused the output source-mappings so that the first text
nodes of the message stopped at the first non-whitespace character.
This commit makes use of the previous refactorings, where more fine grain
information was provided in text tokens, to enable the parser to identify
the location of the interpolations in the original source more accurately.
Fixes#41034
PR Close#42062
When it was tokenized, text content is split into parts that can include
interpolations and encoded entities tokens.
To make this information available to downstream processing, this commit
adds these tokens to the `Text` AST nodes, with suitable processing.
PR Close#42062
The lexer now splits encoded entity tokens out from text and attribute value tokens.
Previously encoded entities would be decoded and the decoded value would be
included as part of the text token of the surrounding text. Now the entities
have their own tokens. There are two scenarios: text and attribute values.
Previously the contents of `<div>Hello & goodbye</div>` would be a single
TEXT token. Now it will be three tokens:
```
TEXT: "Hello "
ENCODED_ENTITY: "&", "&"
TEXT: " goodbye"
```
Previously the attribute value in `<div title="Hello & goodbye">` would be
a single text token. Now it will be three tokens:
```
ATTR_VALUE_TEXT: "Hello "
ENCODED_ENTITY: "&", "&"
ATTR_VALUE_TEXT: " goodbye"
```
- ENCODED_ENTITY tokens have two parts: "decoded" and "encoded".
- ENCODED_ENTITY tokens are always preceded and followed by either TEXT tokens
or ATTR_VALUE_TEXT tokens, depending upon the context, even if they represent
an empty string.
The HTML parser has been modified to recombine these tokens to allow this
refactoring to have limited effect in this commit. Further refactorings
to use these new tokens will follow in subsequent commits.
PR Close#42062
The lexer now splits interpolation tokens out from attribute value tokens.
Previously the attribute value of `<div attr="Hello, {{ name}}">` would be a single
token. Now it will be three tokens:
```
ATTR_VALUE_TEXT: "Hello, "
ATTR_VALUE_INTERPOLATION: "{{", " name", "}}"
ATTR_VALUE_TEXT: ""
```
- ATTR_VALUE_INTERPOLATION tokens have three parts, "start marker",
"expression" and "end marker".
- ATTR_VALUE_INTERPOLATION tokens are always preceded and followed
by TEXT tokens, even if they represent an empty string.
The HTML parser has been modified to recombine these tokens to allow this
refactoring to have limited effect in this commit. Further refactorings
to use these new tokens will follow in subsequent commits.
PR Close#42062
The lexer now splits interpolation tokens out from text tokens.
Previously the contents of `<div>Hello, {{ name}}<div>` would be a single
text token. Now it will be three tokens:
```
TEXT: "Hello, "
INTERPOLATION: "{{", " name", "}}"
TEXT: ""
```
- INTERPOLATION tokens have three parts, "start marker", "expression"
and "end marker".
- INTERPOLATION tokens are always preceded and followed by TEXT tokens,
even if they represent an empty string.
The HTML parser has been modified to recombine these tokens to allow this
refactoring to have limited effect in this commit. Further refactorings
to use these new tokens will follow in subsequent commits.
PR Close#42062
Angular inserts text either through text nodes (`document.createTextNode`) or using `textContent`, but the drawback of doing so is that HTML entities won't be decoded. In order to work around it, the compiler has some logic that maps the entities to their unicode representation which can safely be inserted. The problem is that our current mapping is arbitrarily limited which means that some entities will be mapped while others will throw an error, even though they're valid.
These changes expand the list to cover all entities that are supported by the HTML spec.
Fixes#41186.
PR Close#42818
In combination with the TS `noImplicitOverride` compatibility changes,
we also want to follow the best-practice of adding `override` to
members which are implemented as part of abstract classes. This
commit fixes all instances which will be flagged as part of the
custom `no-implicit-override-abstract` TSLint rule.
PR Close#42512
Previously the lexer would break out of consuming a text token if it contains
a `<` character. Then if the next characters did not indicate an HTML syntax
item, such as a tag or comment, then it would start a new text token. These
consecutive text tokens are then merged into each other in a post tokenization
step.
In the commit before this, interpolation no longer leaks across text tokens.
The approach given above to handling `<` characters that appear in text is
no longer adequate. This change ensures that the lexer only breaks out of
a text token if the next characters indicate a valid HTML tag, comment,
CDATA etc.
PR Close#42605
When consuming a text token, the lexer tracks whether it is reading characters
from inside an interpolation so that it can identify invalid ICU expressions.
Inside an interpolation there will be no ICU expression so it is safe to
have unmatched `{` characters, but outside an interpolation this is an error.
Previously, if an interpolation was started, by an opening marker (e.g. `{{`)
in a text token but the text came to an end before the closing marker (e.g. `}}`)
then the lexer was not clearing its internal state that tracked that it was
inside an interpolation. When the next text token was being consumed,
the lexer, incorrectly thought it was already within an interpolation.
This resulted in invalid ICU expression errors not being reported.
For example, in the following snippet, the first text block has a prematurely
ended interpolation, and the second text block contains an invalid `{` character.
```
<div>{{</div>
<div>{</div>
```
Previously, the lexer would not have identified this as an error. Now there
will be an EOF error that looks like:
```
TS-995002: Unexpected character "EOF"
(Do you have an unescaped "{" in your template? Use "{{ '{' }}") to escape it.)
```
PR Close#42605
This commit updates the parser logic to continue to try to match an end
tag to an unclosed open tag on the stack. Previously, it would only
push an error to the list and stop looking at unclosed elements.
For example, the invalid HTML of `<li><div></li>`, has an unclosed
element stack of [`li`, `div`] when it encounters the close `li` tag.
We compare against the previously unclosed tag `div` and see that this is
unexpected. Instead of simply giving up here, we continue to move up the
unclosed tags until we find a match (if there is one).
PR Close#42554
The compiler's parsing code has logic to recover from incomplete open
tags (i.e. `<div`) but the recovery logic does not handle when the
incomplete tag is terminated by an EOF. This commit updates the logic to
allow for the EOF character to be interpreted as the end of the tag open
so that the parser can continue processing. It will then fail to find
the end tag and recover by marking the open tag as incomplete.
Part of https://github.com/angular/vscode-ng-language-service/issues/1140
PR Close#41054
If the template parse option `leadingTriviaChars` is configured to
consider whitespace as trivia, any trailing whitespace of an element
would be considered as leading trivia of the subsequent element, such
that its `start` span would start _after_ the whitespace. This means
that the start span cannot be used to mark the end of the current
element, as its trailing whitespace would then be included in its span.
Instead, the full start of the subsequent element should be used.
To harden the tests that for the Ivy parser, the test utility `parseR3`
has been adjusted to use the same configuration for `leadingTriviaChars`
as would be the case in its production counterpart `parseTemplate`. This
uncovered another bug in offset handling of the interpolation parser,
where the absolute offset was computed from the start source span
(which excludes leading trivia) whereas the interpolation expression
would include the leading trivia. As such, the absolute offset now also
uses the full start span.
Fixes#39148
PR Close#40513
The parser has a list of tag definitions that it uses when parsing the template. Each tag has a
`contentType` which tells the parser what kind of content the tag should contain. The problem is
that the browser has two separate `title` tags (`HTMLTitleElement` and `SVGTitleElement`) and each
of them has to have a different `contentType`, otherwise the parser will throw an error further down
the pipeline.
These changes update the tag definitions so that each tag name can have multiple content types
associated with it and the correct one can be returned based on the element's prefix.
Fixes#31503.
PR Close#40259
Similar to #39609 and #38898, though we currently have the knowledge of where the key for an
attribute appears during parsing, we do not propagate this
information to the output AST. This means that once we produce the
template AST, we have no way of mapping a template position to the key
span alone. The best we can currently do is map back to the
sourceSpan. This presents problems downstream, specifically for the
language service, where we cannot provide correct information about a
position in a template because the AST is not granular enough.
PR Close#39613
This commit ensures that when leading whitespace is skipped by
the tokenizer, the original start location (before skipping) is captured
in the `fullStart` property of the token's source-span.
PR Close#39486
The lexer is able to skip leading trivia in the `start` location of tokens.
This makes the source-span more friendly since things like elements
appear to begin at the start of the opening tag, rather than at the
start of any leading whitespace, which could include newlines.
But some tooling requires the full source-span to be available, such
as when tokenizing a text span into an Angular expression.
This commit simply adds the `fullStart` location to the `ParseSourceSpan`
class, and ensures that places where such spans are cloned, this
property flows through too.
PR Close#39486
Let's say we have a code like
```html
<div<span>123</span>
```
Currently this gets parsed into a tree with the element tag `div<span`.
This has at least two downsides:
- An incorrect diagnostic that `</span>` doesn't close an element is
emitted.
- A consumer of the parse tree using it for editor services is unable to
provide correct completions for the opening `<span>` tag.
This patch attempts to fix both issues by instead parsing the code into
the same tree that would be parsed for `<div></div><span>123</span>`.
In particular, we do this by optimistically scanning an open tag as
usual, but if we do not notice a terminating '>', we mark the tag as
"incomplete". A parser then emits an error for the incomplete tag and
adds a synthetic (recovered) element node to the tree with the
incomplete open tag's name.
What's the downside of this? For one, a breaking change.
<ol>
<li>
The first breaking change is that `<` symbols that are ambiguously text
or opening tags will be parsed as opening tags instead of text in
element bodies. Take the code
```html
<p>a<b</p>
```
Clearly we cannot have the best of both worlds, and this patch chooses
to swap the parsing strategy to support the new feature. Of course, `<`
can still be inserted as text via the `<` entity.
</li>
</ol>
Part of #38596
PR Close#38681
Previously, the `sourceSpan` and `startSourceSpan` were the same
object, which meant that you had the following situation:
```
element = <div>some content</div>
sourceSpan = <div>
startSourceSpan = <div>
endSourceSpan = </div>
```
This made `sourceSpan` redundant and meant that if you
wanted a span for the whole element including its content
and closing tag, it had to be computed.
Now `sourceSpan` is separated from `startSourceSpan`
resulting in:
```
element = <div>some content</div>
sourceSpan = <div>some content</div>
startSourceSpan = <div>
endSourceSpan = </div>
```
PR Close#38581
Previously, the `startSourceSpan` property could be null
but in reality it is always well defined - except for a legacy
case in the old i18n extraction/merging code, where the
typings for source-spans are already being undermined.
Making this property non-null, simplifies code elsewhere
in the project.
PR Close#38581
Previously the lexer was responsible for deciding whether an "inline"
template should also have its line-endings normalized.
Now this decision is made higher up in the call stack to allow more
flexibility in the parser/lexer.
PR Close#38581
The HTML parser gets an element's namespace either from the tag name
(e.g. `<svg:rect>`) or from its parent element `<svg><rect></svg>`) which
breaks down when an element is inside of an SVG `foreignElement`,
because foreign elements allow nodes from a different namespace to be
inserted into an SVG.
These changes add another flag to the tag definitions which tells child
nodes whether to try to inherit their namespaces from their parents.
It also adds a definition for `foreignObject` with the new flag,
allowing elements placed inside it to infer their namespaces instead.
Fixes#37218.
PR Close#38477
