The Karma Saucelabs script for Bazel & Saucelabs relies on some CommonJS
specific features. This commit replaces it with an ESM-compatible
alternative so that it can execute because `nodejs_binary` requires ESM
files now.
PR Close#48573
With the recent ESM changes we also started generating Saucelabs
targets for `//devtools` (as part of an effort to avoid code
duplication). We should skip Saucelabs targets for this package
because we don't intend to run them on Saucelabs and this whole
setup needs some more work (and we shouldn't change unexpectedly).
PR Close#48554
Fixes that we temporarily broke the Bazel npm package artifact as
part of the ESM work. This commit adjusts it and also makes the
artifact subsitutions more maintainable.
PR Close#48521
This is basically a pre-step for combining devmode and prodmode into a
single compilation. We are already achieving this now, and can claim
with confidence that we reduced possible actions by half. This is
especially important now that prodmode is used more often, but rules
potentially still using the devmode ESM sources. We can avoid double
compilations (which existed before the whole ESM migration too!).
We will measure this more when we have more concrete documentation
of the changes & a better planning document.
Changes:
* ts_library will no longer generate devmode `d.ts`. Definitions are
generated as part of prodmode. That way only prodmode can be exposed
via providers.
* applied the same to `ng_module`.
* updates migrations to bundle because *everything* using `ts_library`
is now ESM. This is actually also useful in the future if
schematics rely on e.g. the compiler.
* updates schematics for localize to also bundle. similar reason as
above.
PR Close#48521
* Switches all remaining targets (even if not tested and failing as per
build) away from `ts_devserver` to the canonical `http_server` from
dev-infra.
PR Close#48521
ZoneJS is no longer loaded as an UMD, but instead is included as part
of the browser init entry-point. This means that ZoneJS is bundled and
the ESBuild logic needs to be adjusted for that.
PR Close#48521
* The benchmark macro should also use devmode ESM 2020. No CommonJS
* The benchmark macro should always add `benchpress` as runtime
dependency because it is loaded asynchronously.
* The protractor `nodejs_binary` should use our ESM-interop binary
so that ESM resolution works (e.g. when `await import(benchpress)` from
the driver utilities is invoked).
PR Close#48521
* The Karma Bazel Saucelabs binary needs to use `.mjs` as everything in
the repo w/Bazel is supposed to be ESM.
* The symbol extractor test is updated to no longer use CommonJS
features like `require.resolve`.
PR Close#48521
Since the `defaults.bzl` repo-wide macros are now supporting ESM,
the special spec-bundle logic from `devtools` can be removed.
Also the esbuild configurations need to be updated to account
for the recent dev-infra build-tooling changes. Also properly
now ensures that `aysnc/await` is downleveled for ZoneJS compatibility.
PR Close#48521
* Updates build-tooling to benefit from the latest `spec_bundle`
improvements.
* Updates the ESM extension loader to not attempt adding extensions to
builtin `node:` specifiers. This seems to be disallowed and cannot be
handled gracefully (the attempts are part of a try/catch).
```
Use --sandbox_debug to see verbose messages from the sandbox
Error [ERR_UNKNOWN_BUILTIN_MODULE]: No such built-in module: node:fs.mjs
at new NodeError (node:internal/errors:371:5)
at ESMLoader.builtinStrategy (node:internal/modules/esm/translators:276:11)
at ESMLoader.moduleProvider (node:internal/modules/esm/loader:236:14)
```
PR Close#48521
Since the Bazel setup in this repo will now always use ESM,
the tooling scripts/binaries in AIO need to be switched to ESM
too. Most of the scripts are already ESM, but a few had to be converted.
Note that the Dgeni generation does not use ESM because it's unaffected
and the Dgeni CLI is used. In the future we could also update the Dgeni
setup to ESM but there is no need currently.
PR Close#48521
Even with patched resolution, we should always attempt the next/builtin
NodeJS resolution first. It may find a module if there `node_modules`
relative to the context file. This would be more correct than looking
for a module always at the Bazel `npm` repository `node_modules` folder.
PR Close#48521
ESBuild relies on the linker and we currently set up the ESM loader,
along with accidentally enabling the patched resolution loader. This
didn't cause any problems in sandbox, but outside of sandbox incorrect
ESBuild versions may be discovered because the loader looks at the
top-level `npm/` node modules before looking relative to e.g.
`@bazel/esbuild`
PR Close#48521
* Adjusts tests to no longer rely on CommonJS features. Switches them to
ESM
* Updates test initialization files to not double-initialize Jasmine now
that bootstrap files are loaded after Jasmine. The `jasmine.boot`
setup was hacky from `rules_nodejs` and will break in the future
regardless if we e.g. use `rules_js` with actual unmodified `jasmine`.
PR Close#48521
Protractor does not support ESM, so we need to take all the ESM
output and bundle it into a CommonJS file. This comes at the cost
of not using actual ESM for execution, but the ESBuild bundle follows
strict ESM semantics so we can be sure it's compatible when we have
an ESM-compatible e2e test runner in the future.
PR Close#48521
There are two build targets which never had all its runtime dependencies
properly specified. This wasn't noticed because there were macros in
`defaults.bzl` that automatically included these deps.
In a follow-up we will clean-up this legacy auto-deps feature in
`defaults.bzl`.
PR Close#48521
Introduces a variant of `jasmine_node_test` that works with async/await
in combination with ZoneJS. This is needed for tests relying on
async/await in combinatin with change detection.
Browser tests are already benefting from ESbuild async-await
downleveling. Node tests would ideally not need this, but until
ZoneJS supports native async/await we need to also process
source files to avoid native async/await syntax.
PR Close#48521
The Angular CLI does not yet support schematics running as ESM. For
this reason we switch the schematics BUILD targets to explicitly
use ESM (as an exception in the repo).
PR Close#48521
Since Karma with Bazel does not support ESM natively, we bundle the
tests using ESBuild into a single AMD file. This not only solves the
ESM issue until we can run browser ESM tests natively (also pending
in the components repo - the esbuild generation follows ESM semantics
but since collapsed we don't rely on the real module system).
A benefit of bundling is also faster and more reliable Karma browser
tests since only a single file needs to be loaded- compared to hundreds
of individual files.
PR Close#48521
Update dev-infra's build-tooling since multiple ESM changes have
landed there. e.g. not relying on `require.main === module` for API
bundling. This will allow us to also execute all dev-infra rules
in ESM because we plan on applying our ESM patching to `ts_library`
(which would also affect build-tooling then).
PR Close#48521
We use `bazel/esbuild` in various places (e.g. for app bundling tests).
These tests rely on the Angular Compiler-CLI itself for e.g. linking
or the Terser configuration. Since everything in this repo is now
strict ESM, the ESBuild configs (which are already ESM-supported)
need to import from `//packages/compiler-cli`. We also need to be
able to leverage our existing ESM Bazel loader for this though as
otherwise resolution would fail.
Long-term we can remove this if everything in the compiler-cli
would use `.mjs` extensions and the import paths would also specify
an explicit extension. See: https://nodejs.org/api/esm.html#mandatory-file-extensions
PR Close#48521
The ESM loader when used with patched Bazel module resolution
handled subpaths incorrectly. e.g.
`@bazel/concatjs/tsc_wrapped/internal/index.js` could be incorrectly
resolved to `@bazel/concatjs/index.js`.
This commit fixes the flawed logic. Also we prioritize the ESM attempts
before the original specifier. This is necessary because otherwise
the default Node resolution (using `require.resolve`) may incorrectly
prefer the `index.js` file over a `index.mjs` when a directory is
imported.
PR Close#48521
Since we no properly initialize as part of the spec-entrypoint
the hacky logic starting Jasmine can be removed. The init
file now runs before tests run, but after Jasmine is initialized.
Calling `boot` while Jasmine is already initialized results in
tests executing in a different context and before other
initialization work completes.
PR Close#48521
Since we generate a `.mjs` file as entry-point for jasmine tests,
a couple of issues prevented the transitive dependencies from
bootstrap targets to be brought in (causing resolution errors):
1. The `_files` (previously `_esm2015`) targets are no longer needed,
and they also miss all the information on runfiles.
2. The aspect for computing linker mappings does not respect the
`bootstrap` attribute from the `spec_entrypoint` so we manually
add the extract ESM output targets (this rule works with the aspect
and forwards linker mappings).
PR Close#48521
RxJS currently ships ESM output that cannot be executed directly
in NodeJS. This is because RxJS ships ESM as `.js` files but does
not have a `package.json` which instructs Node to execute these as ESM.
RxJS would either need to use the explicit `.mjs` extension, or add
a `type: module` `package.json` next to the `.js` sources.
We manually patch RxJS to do this, while we wait on the upstream fix
to land. See: #7130.
PR Close#48521
The prodmode compilation pipeline -that we intend to use more heavily
now given it emitting files with the `.mjs` extension- exposes an
additional tsickle closure externs file. This file is empty most
of the time anyway since tsickle is not wired up.
We remove this generation as otherwise convenient `$(location` of
such ts library targets break because there always is more than 1 file.
PR Close#48521
Note: `--require` does not work for ESM. `--import` does not exist
in the current Node versions. Started being available in NodeJS v19.
A custom entry-point script, already supported by dev-infra, simplifies
the whole logic and solves the ESM case.
PR Close#48521
For every `ts_library` target we expose a shorthand that grants
access to the JS files because `DefaultInfo` of a ts library
only exposes the `.d.ts` files.
We rename this away from `es2015` since in practice it's a much
higher target these days. Additionally we no longer use the devmode
output but rather use the prodmode output which has the explicit
`.mjs` output- compatible with ESM.
PR Close#48521
We introduced a loader that supports ESM with Bazel. This loader can
only be enabled as part of our `nodejs_binary` defaults.bzl test. This
works fine, but there might be other binaries/tests e.g. from
`@angular/build-tooling` that should be able to use ESM & might need to
for importing the `compiler-cli`.
We move the logic for installing the patch into a `rules_nodejs` patch.
There is an existing one that is just updated to "enable ESM support".
PR Close#48521
This is necessary for e.g. JSON files from the CLDR data repository.
Otherwise these could not be added to the runfiles of the
binaries/tests.
PR Close#48521
For the ESM interop patches we expect to have two types
of patches:
* Patches for `node_modules`
* Patches for Bazel repositories.
We move the patches in respective folders to make it very clear
where a patch is used/applied to.
PR Close#48521
We modified the macros of `nodejs_binary/test` to have a rule
in between that requests the `.mjs` output. This works fine but
breaks make variable substitution for `templated_args` because
Bazel requires referenced labels to be part of the explicit `data`.
The rule in between breaks this, so we add a new argument that
can be used for such "template"/"args" data dependencies.
This can be removed when everything is ESM and we don't need
the rule in between.
PR Close#48521
The `nodejs_binary` rule already prioritizes the `.mjs` output as
of the recent commits. The `nodejs_test` rule should do the same,
and also set `use_esm = True`
PR Close#48521
The Bazel NodeJS rules will always use the `.js` files as entry-points.
Since we only rely on the `.mjs` output going-forward, we need to teach
`nodejs_binary` and `nodejs_test` to use the `.mjs` extensions if
intended.
Our `defaults.bzl` macros will set `use_esm = True`, but other targets
from e.g. external repositories should keep the original behavior.
PR Close#48521
Replaces the existing ESM loader for dealing with external module
imports. This loader was introduced by Aspect for AIO `.mjs` scripts.
The loader will be used as foundation for a more extensive loader
that also properly handles first-party packages.
Additionally another loader is added, all packed as a single
loader because our current NodeJS version only supports a single
loader per node invocation. So we implement chaining ourselves.
The new loader will attempt rewriting `.js` extensions to `.mjs`,
also it will add `.mjs` if not already done. This is necessary
in the transition phase because we don't/cannot use explicit `.mts`
extensions and also we don't specify extensions in imports yet.
Long-term we would likely use `.mts` and explicit import extensions,
but it's not yet clear how we would sync this into g3 too.
PR Close#48521
This is in prearation for having a proper diff when this loader
is adjusted to support more situations than just simple external
node modules. See next commit.
Also the file is formatted to make the diff less verbose later.
The file was never formatted correctly and we don't lint `.mjs` files.
PR Close#48521
Currently the devmode output for `ng_module` and `ts_library` is
using ES5 CommonJS UMD. To bring it in sync with prodmode and
to start with our long-term migration to full ESM- the devmode
is updated to to ES2020 ES modules too.
This will require more tricks to make devmod work with the bazel
setup and also tests may need to be refactored given them relying
on ES5 CJS features, like for `spyOn` jasmine patching etc.
PR Close#48521
Similar to the Rules NodeJS require patch, we have an ESM import patch
as of the AIO Bazel migration (to support ESM scripts better).
This script uses `--loader`, an experimental NodeJS flag. This is
similar to how `ts-node` uses it. We should disable the warnings
as it results in a lot of unreadable Bazel output and the warnings
are okay to be ignored. Note that we cannot fine-grain disable
the specific warning so all others would be disabled too.
Realistically we haven't seen any in the past and long-term we will
be not relying on patched resolution anyway (looking at `rules_js`).
PR Close#48282
Temporary patch until
https://github.com/bazelbuild/rules_nodejs/pull/3517 is available in
another `rules_nodejs` release.
We can remove this patch, but for now it doesn't hurt. On the external
side the tsickle code path is not hit at all anyway, but we need to
satisfy the TypeScript checker.
PR Close#47018
The dev-infra build tooling is now decoupled from `ng-dev`. This will
make it easier to update `ng-dev` without necessarily needing to upgrade
the whole build system, Bazel etc. This is useful when e.g. new release
tool features have been added and should also be ported to active LTS
branches.
PR Close#46976
The JS size-tracking logic has been moved into the dev-infra repository
and the rule has been updated to work better with Bazel labels. The
Starlark target is updated accordingly in this commit.
PR Close#46802
This script was used back when dev-infra code was part of
the Angular framework repository. The script is now unused
and can be deleted.
PR Close#46798
The jasmine seed generator is only used in a single karma configuration
file. Used by the legacy build and the Saucelabs/ZoneJS Karma jobs.
We should move the separate script code directly into the config to make
it clear that the seed generation is not used elsewhere, and to simplify
the Starlark code.
PR Close#46798
The `app_bundle` rule from the shared dev-infra package is no longer in
the benchmarking folder, but instead is part of the general Bazel rules
exposed by `dev-infra`. This commit accounts for this location change.
PR Close#46642
tsec previously did not use runfiles on Windows even when the flag was enabled.
The latest version now adds an option to force its usage.
PR Close#46447
Patches are required for tsec and rules_webtesting. The fix for
rules_webtesting was merged to that repo
(581b1557e3)
but it's unclear when a release will be cut.
PR Close#46313
The NodeJS Bazel linker does not work well on Windows because there
is no sandboxing and linker processes from different tests will attempt
to modify the same `node_modules`, causing concurrency race conditions
and resulting in flakiness.
PR Close#45872
Speeds up the dev-turnaround by only bundling types when packaging. Currently
bundling occurs for all the `ng_module` targets in devmode.
This has various positive benefits:
* Avoidance of this rather slower operation in development
* Makes APF-built packages also handle types for `ts_library` targets consistently.
* Allows us to ensure APF entry-points have `d.ts` _always_ bundled (working with ESM
module resolution in TypeScript -- currently experimental)
* Allows us to remove the secondary `package.json` files from APF (maybe APF v14? - seems
low-impact). This would clean-up the APF even more and fix resolution issues (like in Vite)
PR Close#45405
As part of the devtools migration, we copied the custom http server/
dev-server from the `angular/components` repo. This server implementation
has now moved to the shared dev-infra code, and we can clean up the
copy in this repository now.
PR Close#45452
As mentioned in previous commits (check them for more details), `@bazel/typescript`
no longer contains `ts_library`-specific code, so we no longer need that dependency.
PR Close#45431
Update `@bazel` packages to the latest 5.x version.
Some of the changes here are modeled after
angular/dev-infra@40c0ac8559.
Co-Authored-By: George Kalpakas <kalpakas.g@gmail.com>
PR Close#45431
Node.js v12 will become EOL on 2022-04-30. As a result, Angular CLI v14 will no longer support Node.js v12.
BREAKING CHANGE:
Support for Node.js v12 has been removed as it will become EOL on 2022-04-30. Please use Node.js v14.15 or later.
PR Close#45286
Switches the Karma web test rule from `@bazel/concatjs` to our
wrapped/extended variant from the shared dev-infra code.
One benefit is that we now get a `_debug` target for web tests where
no browser is being launched and the action is kept alive. Allowing
developers to conveniently connect a browser manually for debugging.
Also works with iBazel for the manually connected browser.
PR Close#45117
Previously with ESBuild 0.14.11, when a file had dynamic requires to
builtin NodeJS modules like `url`, the resolution completed successfully
regardless of `--platform browser`. This seems to be fixed in ESBuild now.
This unveiled some resolution errors with our Saucelabs bundle generation
because the framework code sometimes switches dynamically to `require('url')`
if `window.URL` is not defined. Previously this just didn't matter, but now
the `require('url')` is checked and a module resolution error is reported given
`url` not being available in the browser as a builtin module.
We fix this by marking the module as external. We will not hit this code path
anyway in the browser saucelabs code. Similarly we exclude all platform-server
files from the bundle. This is not strictly needed after the `url` module being
marked as external, but the issue showed that lots of unnecessary code for the
server platform is included. This can be omitted (unfortunately not from the TS
compilation without over-complicating things significantly more; experimented with
that).
PR Close#44830
Updates the postinstall patch for the benchmark macro rule from dev-infra.
We moved the ZoneJS setup to the bundler. This was necessary in order to
switch away from the Go-based (windows-incompatible, m1-incompatible)
concatjs devserver to a rather basic HTTP server (also provided by dev-infra now).
PR Close#44830
Previously devtools used a nested workspace for its bazel configurations. This meant framework dependencies were consumed via npm.
Now devtools is part of the root bazel directory that all other files in this codebase fall under. This allows us to build devtools using local angular packages, removing the need to consume these dependencies with npn. This is useful because we no longer have to update these dependencies with an automated tool like renovate, and our CI tests will always run against the most up to date framework packages.
To make our test output i.e. devmode output more aligned
with what we produce in the NPM packages, or to be more
aligned with what Angular applications will usually consume,
the devmode output is switched from ES5 to ES2015.
Additionally various tsconfigs (outside of Bazel) have been
updated to match with the other parts of the build. The rules
are:
ES2015 for test configurations, ES2020 for actual code that will
end up being shipped (this includes the IDE-only tsconfigs).
PR Close#44505
The `ng_rollup_bundle` rule has been replaced with a new rule called
`app_bundle`. This rule replicates the Angular v13 optimization
pipeline in the CLI, so that we can get better benchmarking results.
Also the rule is much simpler to maintain as it relies on ESbuild.
The old `ng_rollup_bundle` rule did rely on e.g. build-optimizer that no
longer has an effect on v13 Angular packages, so technically size
tests/symbol tests were no longer as correct as they were before. This
commit fixes that.
A couple of different changes and their explanation:
* Language-service will no longer use the benchmark rule for creating
its NPM bundles! It will use plain `rollup_bundle`. ESBuild would have
been nice but the language-service relies on AMD that ESBuild cannot
generate (yet?)
* Service-worker ngsw-worker.js file was generated using the benchmark
bundle rule. This is wrong. We will use a simple ESbuild rule in the
future. The output is more predictable that way, and we can have a
clear use of the benchmark bundle rule..
* A couple of benchmarks in `modules/` had to be updated to use e.g.
`initTableUtils` calls. This is done because with the new rule, all
files except for the entry-point are considered side-effect free. The
utilities for benchmarks relied on side-effects in some
transitively-loaded file (bad practice anyway IMO). We are now
initializing the utilities using a proper init function that is
exported...
PR Close#44490
We are in an inconvenient situation where the ng-dev package might rely
on packages from the Angular framework repository. Given that we install
this package in the framework repository, we need to update some
references through a postinstall.
This commit updates the patches to account for the latest changes in the
dev-infra package/repository.
PR Close#44490
Adds the postinstall script as runfile for the `yarn_install`
repository rule, so that the dependencies are re-fetched when
the script changes.
PR Close#44490
Updates the symbol extractor to support IIFE bundles using
arrow-functions instead of function declarations. This is in preparation
for running symbol extraction tests with the overhauled optimization
pipeline for Angular v13, relying on ESBuild internally.
Also removes rollup-specific code that does not seem to be relevant
anymore / rollup will be replaced anyway.
PR Close#44490
Wires up the integration test rule from the shared dev-infra package,
while also deleting the old integration test rule.
The readme is updated to reflect the changes that are being made
to run with the new integration rule. Overall one major difference is
that we will declare the integration test targets within each test
directory, making those actual bazel packages. This is more idiomatic
within Bazel and also reduces the computation within Skyframe as less
globs need to be evaluated for example.
PR Close#44238
Fixes lint errors in the build-saucelabs legacy script. Likely this only
surfaces now after merge due to some other lint-affecting changes
landing just before the script landed.
PR Close#44311
Previously, when we did not bundle tests, we also had the jasmine
default timeout increased to 15000ms. This commit re-introduces this
increased timeout by also maintaining a custom test init file for the
legacy saucelab jobs. This is desirable as all of the Saucelabs test
code is local in the `tools/legacy-saucelabs` folder.
PR Close#44281
Bundle spec files similar to how it is done within the Angular
Components repo. This should simplify the setup and also speed
up the Saucelab job as only a single spec bundle would need to be
downloaded, compared to having to load hundreds of files through the
Saucelabs tunnel.
Also makes a couple of tests more robust with the emulators/and accounts
for ES2015 test runner changes. The tests should be less reluctant to
such build process changes.
Note for reviewers: Some imports have been simplified here. This work
came from Joey's original WIP for this. It's unclear to me whether this
is still needed, but it sounded like this was necessary for the ESBuild
bundling to work. I have robusted the module resolution plugin though,
so I doubt it's still needed. At the same time though: Not worth
reverting/trying as these changes are nice to have anyway!
Co-Authored-By: Joey Perrott <josephperrott@gmail.com>
Co-Authored-By: Paul Gschwendtner <paulgschwendtner@gmail.com>
PR Close#44281
This commit removes most tests that were designated as only covering View
Engine code. It also removes tag filters from CI and local commands to run
tests.
In a few cases (such as with the packages/compiler tests), this tag was
improperly applied, and certain test cases have been added back running in
Ivy mode.
This commit also empties `@angular/compiler/testing` as it is no longer
necessary (this is safe since compiler packages are not public API). It can
be deleted in the future.
PR Close#43884
The API of the `extract_js_module_output` rule has changed with the
latest build of the shared dev-infra package. This commit adds the
missing attributes to the targets using this rule, avoiding the
CI failures as these attributes are mandatory.
Note: For NPM packages the linker mappings are not relevant, neither
do we want to include sources from the external NPM packages inside
those.
PR Close#44145
Contents of generated tsconfig for tsec_test now depend on whether
Bazel uses symlinked runfiles for nodejs_test. The current
implementation assumes that symlinked runfiles are not available
on Windows.
PR Close#43924
Using the tag "view-engine-only" better describes the expected usage of bazel targets with the test. They can
only be run with view engine.
PR Close#43862
Setting the angular_ivy_enabled environment variable to True will default Bazel builds to use the Ivy
compiler rather than defaulting to ViewEngine.
PR Close#43862
