The dev-infra build tooling is now decoupled from `ng-dev`. This will
make it easier to update `ng-dev` without necessarily needing to upgrade
the whole build system, Bazel etc. This is useful when e.g. new release
tool features have been added and should also be ported to active LTS
branches.
PR Close#46976
As there is little to no usage of RBE by individuals and we don't actively ensure
that the process and permissions are working for individuals the documentation is
best removed from the repo.
PR Close#46515
We need to switch the package builder to ESM or TS, like we did
in the components repo, but for now until we can, the release script
should work by loading the package builder CJS through the dynamic
import expression.
PR Close#46454
If the snapshot publish job is manually being restarted, the tag in the
snapshot repo might already exist and the job will fail. We can just
forcibly re-create the tag (even if it will be at the same revision).
We use force mode in a couple of other command as well, such as `git
push` of the actual tag and snapshot revision/SHA.
PR Close#46366
We recently had a couple of issues with the shallowing of snapshot
git repositories. In an attempt to fix this, parts of the publish
script of the COMP repo have been used, but variables have not been
updated properly.
This commit fixes an unbound variable, avoiding errors when snapshots
are published for a new branch (e.g. when we branch-off).
PR Close#46076
Use the absolute path for the artifacts directory, as well as encapsulate the clone
vs init logic to prevent it from changing directories.
Manually remove the shallow marker so that git does not recognize this as a shallow repo.
PR Close#45979
Rather than using git init to create an empty repo and then shallow fetch the
repo for a specific branch, we shallow clone the repository itself for the branch.
This prevents us from running into the issue of Github denying the push because it
comes from a shallow branch which does not map back to the repository.
PR Close#45975
It looks like because the primary branch for our ephemeral repos doesn't match the build repos
we are pushing to it causes errors around shallow clones. This attempts to prevent this.
PR Close#45974
Removes scripts and documentation for tools which are no longer
actively used and likely do not work anyway at this point.
All of our PR-related tooling has moved into the `yarn ng-dev` command.
The `PR_REVIEW` markdown file is deleted since it was soley about
the legacy review scripts and the PR review workflow also does not
seem like the common/standard workflow Angular team members follow.
PR Close#45868
Uploading payload size is unreliable from within Bazel. This is
because tests might not run for every commit, tests might have
been cached from a pull request RBE-build (causing payload uploading
to be skipped most of the time as every change comes from a PR)
We should disable the uploading as this is a fundamental problem
(good thing to note now) that we can solve with better payload
size tracking that we want to establish as part of dev-infra.
PR Close#45446
For quite some time now, since we started to use Bazel for integration tests, we
relied on some size tracking logic that did not actually fully work under Bazel.
It was thought that all the necessary CI push/PR information is available to the
Bazel test, but that was not the case. This was now fixed with the recent Rules NodeJS
v5 update where I made sure the `env.sh` variables are actually available before we
write them to the temporary file for the Bazel-access.
This now will unveil an issue because payload size goldens would start being based
on their branch name. e.g. the golden key in `13.3.x` should not be `master` but
`13.3.x`. This makes more sense than `master` as key, but makes things more
cumbersome and ideally we would not store the branch name at all (this is a larger
change though -- not worth now since we might refactor this anyway). For now we will
update the size tracking logic to always use `master` as golden key (like it worked
in the past year(s))
With the environment fix we now (again) start uploading payload size results to Firebase.
This did not work by accident either. The uploading logic is reliant on the CircleCI
commit range which is not working/reliable in upstream branches. This commit
removes this reliance on `COMMIT_RANGE` since it's not strictly necessary and
currently breaking renovate PRs. We can re-enable this when we have a solution with
CircleCI, or a workaround/resolution logic provided in e.g. `ng-dev ci determine-commit-range`.
PR Close#45444
.substr() is deprecated so we replace it with functions which work similarily but aren't deprecated
Signed-off-by: Tobias Speicher <rootcommander@gmail.com>
PR Close#45397
CI components test install the angular package from the local
version, but still use the zone.js from npm, so this commit let
components also install zone.js from local too.
PR Close#45277
Several people on the team (myself included) have had issues where the `webdriver-manager` step of the `postinstall` script fails on the first run, but work correctly on the second one.
These changes attempt to simplify the workflow by retrying the script up to 3 times before giving up.
PR Close#45252
To speed up the components CI job (avoiding it being a bottleneck in FW),
we will stop building everything in `src/...`, but rather only build targets
which are needed to run all tests within `src/...`.
This notably should avoid unnecessary NPM package building. Arguably that
part would also be valuable to have, but we already test the `ng_package`
rule in FW itself, plus we run snapshot tests in the components repo as well.
PR Close#44832
Setting the angular_ivy_enabled environment variable to True will default Bazel builds to use the Ivy
compiler rather than defaulting to ViewEngine.
PR Close#43862
Re-enables the building of the `src/dev-app` package in the components
repo when the components-repo-unit-test job runs. We previously disabled
the dev-app because it was not compatible with template typechecking of
ngtsc. This should be fixed now and can re-enable it.
This could be valuable for having some actual real test scenarios of
type checking.
PR Close#43798
The package builder script should respect the `BAZEL` environment
variable for running Bazel. If not set, it can fallback to bazelisk
from the `node_modules`. Respecting this variable allows for users
with a global `bazel` binary. This is desirable in some situations,
like on Windows, where running Bazel inside of the Yarn environment
seems slower than running a global variant. This could appear like that
because projects might use different Bazel versions. In some cases,
developers would want to use a single (already-warmed-up) instance of
Bazel instead of launching different versions using bazelisk.
(e.g. when switching a lot between repos like COMP, FW or CLI..)
In any case, it doesn't hurt providing this flexibility for advanced
use-cases. It's low-effort to maintain and is respected in COMP as well.
PR Close#43431
Sets the executable bit for the build-packages-dist script so that
the script can be invoked directly without having to explicitly
prefix it with the `node` command.
PR Close#43431
Previously, if a commit message contained a carriage return in its
header, the `payload-size.sh` script would fail to upload the payload
size data to Firebase, because the JSON payload would be messed up when
trying to concatenate the commit message headers. See an example
[here][1].
This commit avoids this problem by replacing carriage returns before
concatenating the commit message headers with the JSON payload string.
[1]: https://app.circleci.com/pipelines/github/angular/angular/37437/workflows/d0fa4adf-43bb-464e-a2fd-d87da15226dd
PR Close#43569
The shared dev-infra package from `HEAD` should not be installed
when running the `components-repo-unit-tests` job. This is done
because otherwise changes to the dev-infra package can result in
build setup issues for the CI job (while the goal of the CI job
is to test the framework)
PR Close#42268
Currently the ng-dev release tool always run `bazel clean` before
calling the configured build release function. The clean is necessary
to ensure the release output is actually built; and not restored
from previous builds which could have different bazel workspace
status variables (which provide the NPM package version).
Instead of doing this as part of the release tool, the
actual script running to build the release output should
run the `bazel clean`. The release tool does not intend to
know about details on how the release output is built. This
is necessary because the build setup could vary between version
branches (especially for older ones; such as LTS version branches).
PR Close#42101
Previously, the `angular-in-memory-web-api` package was built in
`dist/packages-dist/misc/angular-in-memory-web-api/`. This was different
from other Angular packages, which were placed directly in
`dist/packages-dist/`. This caused the `create-package-archives.sh`
script to create an invalid `misc.tgz` archive (i.e. treating the
`misc/` subdirectory as a package).
See, for example, the artifacts [here][1].
This commit changes the build scripts to have the
`angular-in-memory-web-api` package built in
`dist/angular-in-memory-web-api-dist/`, similar to how the `zone.js`
package is handled. It also updates the CircleCI config to correctly
publish the `angular-in-memory-web-api` package to CI build artifacts.
[1]: https://circleci.com/gh/angular/angular/951491
PR Close#41429
This commit updates the `build-ivy-npm-packages.js` script to also build
the `angular-in-memory-web-api` package. This could be used if someone
wanted to test the Ivy packages in docs examples or an integration test.
PR Close#41429
This commit makes the build scripts for the various packages (framework,
`@angular/dev-infra-private`, `angular-in-memory-web-api`, `zone.js`)
consistent. This makes it easier to maintain them (e.g. make similar
changes across all build scripts).
PR Close#41429
We now include `misc/angular-in-memory-web-api` in the packages that
are build in the `dist/packages-dist` directory. But this package should
not be published as an artifact.
This commit prevents attempting to publishing the `misc` directory by
first checking whether the directory contains a `package.json` file.
PR Close#41393
In some cases, we want to test the AIO app or docs examples against the locally built `angular-in-memory-web-api` for example to ensure that the changes in a commit do not introduce a breaking changes.
PR Close#41313
Previously, when a payload size check failed, the error message prompted
the user to update the size limits using a CI-specific file path, which
was confusing (esp. for external contributors). See, for example,
https://circleci.com/gh/angular/angular/932733.
This commit improves the error message by printing the file path
relative to the repository root instead.
PR Close#41116
Previously, we could not use `npm_package.pack` for building the Zone.js
package due to a bug in Windows. This bug has been fixed in version
2.3.0 of `bazelbuild/rules_nodejs`.
Now that we have updated `bazelbuild/rules_nodejs` to version 2.3.3
(in #40581) we can replace `npm pack` with `npm_package.pack`.
PR Close#40592
In older versions of `bazelbuild/rules_nodejs`, the `npm_package.pack`
rule does not work on Windows. This has been fixed in
bazelbuild/rules_nodejs#2257, but the fix is not available before
[version 2.3.0][1].
Currently, we use version 2.2.0 (see [WORKSPACE][2]). In order to allow
the `zone-js-builder.js` script to work on Windows, this commit switches
to using `npm pack` directly, intead of relying on `npm_package.pack`.
For reference, the version of `bazelbuild/rules_nodejs` was updated to
2.3.2 in PR #39636, but this was later reverted due to CI flakes
(4e6d69cc85).
[1]: https://github.com/bazelbuild/rules_nodejs/releases/tag/2.3.0
[2]: fc64fa8e1a/WORKSPACE (L12)
PR Close#40557
Previously the pre-check confirmed that the stamping value was normal and expected, however
with the new stamping methodology of marking for publish via flag, the check no longer will
ever see anything but a passing version, since it is stamped as publish.
PR Close#40422
This commit updates the versions of Chrome and Firefox used in tests -
both with Bazel and without (via Puppeteer) - to the latest:
- Chrome v87
- Firefox v84
PR Close#40150
Previously, the instructions and process for updating the version of
Chrome ued in tests assumed that there was always going to be a
ChromeDriver version that corresponded to a Chrome version. For example,
if we wanted to use Chrome v87.0.4272.x, we assumed that there was going
to be ChromeDriver v87.0.4272.x. It turns out that this is not always
the case.
This commit updates the instructions and process for updating the Chrome
version to ensure a valid version of ChromeDriver will be used as well.
PR Close#40150
The scripts that build the Angular and Zone.js NPM packages rely on
absolute paths in order to work correctly regardless of what the current
working directory is when the scripts are invoked. However, the
`npm pack` command executed in the `zone-js-builder.js` script outputs
the generated `.tgz` archive in the current working directory. This
causes the script to fail when invoked for any working directory other
than the project root directory.
This commit fixes this by ensuring the `npm pack` command is run with
the project root directory as the working directory. This allows the
build scripts to run correctly regardless of the working directory they
are invoked from.
PR Close#39455
Zone.js 0.11.0 release an empty bundle, and now the npm_package tests all target
bazel rule `npm_package`, but not `npm_package.pack`, and these two rules may
generate different results, for example, Zone.js 0.11.0's issue is `package.json`
define files array which make the bundle only include the files in the files array.
So this PR install the zone.js package from the archive generated from `npm_package.pack` rule.
PR Close#38649
Bump Chrome to the next stable release (84.0.4147) by following the
instructions in dev-infra/browsers/README.md.
With Chrome 86 about to be released as stable, the current local version
(Chrome 83) is starting to lag behind. It also contains a bug that
blocks Angular unit and integration tests from using Trusted Types.
PR Close#39179
Remove @angular/platform-webworker and @angular/platform-webworker-dynamic
as they were deprecated in v8
BREAKING CHANGE: @angular/platform-webworker and @angular/platform-webworker-dynamic
have been removed as they were deprecated in v8
PR Close#38846
Historically files to be formatted were added to a listing (via matchers)
to be included in formatting. Instead, this change begins efforts to
instead include all files in format enforcement, relying instead on an
opt out methodology.
PR Close#36940
Deprecate the old merge script as it no longer correctly chooses
the patch branch due to relying on numerical sorting order from
git. Git actually provides a lexicographical sorting order. This
that 9.0.x will be chosen rather than 10.0.x as it is sorted based
the 9 vs 1, rather than 9 vs 10.
PR Close#37247
This commit updates the script that checks master and patch branches to ignore features with `dev-infra` scope
while verifying that there are no feature commits in patch branch. It's ok and in fact desirable for dev-infra features to be on the patch branch.
PR Close#37210
Adds a deprecation notice to the old merge-pr script informing the
user the script will be removed in favor of the ng-dev merge tooling.
This currently serves as a warning, and does not fail to perform the
merge.
PR Close#37204
This commit includes a couple minor fixes for the script that compares master and patch branch:
- take only relevant release commit into account while generating the diff
- fix the initial version display (avoid '+' sign from being added)
- removes obsolete parameter that was needed for v9.0.x branch only
PR Close#37150
The local rebase-pr script assumes the existence of specific git
aliases. Instead this script should rely on the full written out
command instead.
PR Close#37050
Previously the setup-rbe.sh script did not allow accounts that did
have domains of angular.io or google.com. Since we add emails from
other domains into everyone@angular.io, we are unable to be certain
in the script that the account is not actually a member of the
required group. This change adds the option to choose to continue
with an email account logged in which we cannot verify by domain.
PR Close#36846
Previously the exec command was used, however the exec command would
exit the original calling script regardless of the whether exit was
called. This caused the release script to always exit after the
pre-check phase.
PR Close#36862
Previously, our process included running the pre-check script before
releasing. With our new publishing process this was dropped. This
change adds in automatically executing this check before publish for
both next and latest
PR Close#36527
Currently the commit message and corresponding version are flipped, which makes it hard to review the changes. This commit updates the script to properly recognize the order of arguments.
PR Close#36749
Updating `REQUIRED_BASE_SHA` for master and patch branches to make sure PRs that we merge are rebased after `commit-message` validation script update (to make sure the `lint` CI job fails in case a PR contains commits with invalid commit messages).
PR Close#36750
Prior to this change we manage a local version of commit message validation
in addition to the commit message validation tool contained in the ng-dev
tooling. By adding the ability to validate a range of commit messages
together, the remaining piece of commit message validation that is in the
local version is replicated.
We use both commands provided by the `ng-dev commit-message` tooling:
- pre-commit-validate: Set to automatically run on an git hook to validate
commits as they are created locally.
- validate-range: Run by CI for every PR, testing that all of the commits
added by the PR are valid when considered together. Ensuring that all
fixups are matched to another commit in the change.
PR Close#36172
A few leftovers from from `@bazel/bazel` to `@bazel/bazelisk` migration
are still there. This commit fixes those, so that the repository no
longer relies on `@bazel/bazel`.
PR Close#36132
* integration tests target definitions in integration/BUILD.bazel updated to use a single dict
* payload tracking for integration tests updated to work under Bazel
* legacy integration_test CI job removed
* integration/run_tests.sh script no longer used in CI so it has been updated for running integration tests locally in the legacy way
PR Close#35985
Adding a script that compares commits in master and patch branches and finds a delta between them. This is useful for release reviews, to make sure all the necessary commits are included into the patch branch and there is no discrepancy.
PR Close#35130
In some cases, we want to test the AIO app or docs examples against the
locally built Angular packages (for example to ensure that the changes
in a commit do not introduce a breaking change). In order to achieve
this, we have the `ng-packages-installer` script that handles updating
a project's `package.json` file to use the locally built Angular
packages (and appropriate versions for their (dev-/peer-)dependencies).
Previously, `ng-packages-installer` would only consider the locally
built Angular packages (from `dist/packages-dist/`). However, given that
Zone.js is now part of the `angular/angular` repo, it makes sense to
also use the locally built Zone.js package (from `dist/zone.js-dist/`).
Otherwise, the tests might fail for commits that update both the Angular
packages (and related docs examples) and the Zone.js package. An example
of such a simultaneous change (that would have broken tests) is #33838.
This commit updates the script to install the locally built Zone.js
package (in addition to the Angular ones). The commit ensures that the
Zone.js package will always be available alongside the Angular packages
(i.e. that the Zone.js package will be built by the same script that
builds the Angular packages and that the `dist/zone.js-dist/` directory
will be cached on CI).
Note: This problem was discovered while enabling docs examples unit
tests in #34374.
PR Close#35858
Creates the scaffolding for an @angular/dev-infra-private package
which will not be published to npm but will be pushed to
https://github.com/angular/dev-infra-private-builds repo for each
commit to master.
The contents of this npm package will then be depended on via
package.json dependency for angular/angular angular/angular-cli and
angular/components.
PR Close#35862
This reverts commit 7d832ae1001b6264bb7124086089e9e69c10c9b6; breaks CI
with error `Concurrent upstream jobs persisted the same file(s) into the workspace:`
PR Close#35857
In some cases, we want to test the AIO app or docs examples against the
locally built Angular packages (for example to ensure that the changes
in a commit do not introduce a breaking change). In order to achieve
this, we have the `ng-packages-installer` script that handles updating
a project's `package.json` file to use the locally built Angular
packages (and appropriate versions for their (dev-/peer-)dependencies).
Previously, `ng-packages-installer` would only consider the locally
built Angular packages (from `dist/packages-dist/`). However, given that
Zone.js is now part of the `angular/angular` repo, it makes sense to
also use the locally built Zone.js package (from `dist/zone.js-dist/`).
Otherwise, the tests might fail for commits that update both the Angular
packages (and related docs examples) and the Zone.js package. An example
of such a simultaneous change (that would have broken tests) is #33838.
This commit updates the script to install the locally built Zone.js
package (in addition to the Angular ones). The commit ensures that the
Zone.js package will always be available alongside the Angular packages
(i.e. that the Zone.js package will be built by the same script that
builds the Angular packages and that the `dist/zone.js-dist/` directory
will be cached on CI).
Note: This problem was discovered while enabling docs examples unit
tests in #34374.
PR Close#35780
This commit moves the build-related scripts
(`build-ivy-npm-packages.js`, `build-packages-dist.js` and
`package-builder.js`) to a dedicated directory to keep the `scripts/`
directory cleaner.
It also moves the logic for building the `zone.js` package to a separate
script, `zone-js-builder.js`, to make it re-usable. A subsequent commit
will use it to build the `zone.js` package when building the Ivy Angular
packages as well.
PR Close#35780
Switches our tslint setup to the standard `tslint.json` linter excludes.
The set of files that need to be linted is specified through a Yarn script.
For IDEs, open files are linted with the closest tslint configuration, if the
tslint IDE extension is set up, and the source file is not excluded.
We cannot use the language service plugin for tslint as we have multiple nested
tsconfig files, and we don't want to add the plugin to each tsconfig. We
could reduce that bloat by just extending from a top-level tsconfig that
defines the language service plugin, but unfortunately the tslint plugin does
not allow the use of tslint configs which are not part of the tsconfig project.
This is problematic since the tslint configuration is at the project root, and we
don't want to copy tslint configurations next to each tsconfig file.
Additionally, linting of `d.ts` files has been re-enabled. This has been
disabled in the past and a TODO has been left. This commit fixes the
lint issues and re-enables linting.
PR Close#35800
This is a follow-up to #35049 with a few minor fixes related to using
the browser provided by `puppeteer` to run tests. Included fixes:
- Make the `webdriver-manager-update.js` really portable. (Previously,
it needed to be run from the directory that contained the
`node_modules/` directory. Now, it can be executed from a subdirectory
and will correctly resolve dependencies.)
- Use the `puppeteer`-based setup in AIO unit and e2e tests to ensure
that the downloaded ChromeDriver version matches the browser version
used in tests.
- Use the `puppeteer`-based setup in the `aio_monitoring_stable` CI job
(as happens with `aio_monitoring_next`).
- Use the [recommended way][1] of getting the browser port when using
`puppeteer` with `lighthouse` and avoid hard-coding the remote
debugging port (to be able to handle multiple instances running
concurrently).
[1]: https://github.com/GoogleChrome/lighthouse/blame/51df179a0/docs/puppeteer.md#L49
PR Close#35381
This means integration tests no longer need to depend on a $CI_CHROMEDRIVER_VERSION_ARG environment variable to specify which chromedriver version to download to match the locally installed chrome. This was bad DX and not having it specified was not reliable as webdriver-manager would not always download the chromedriver version to work with the locally installed chrome.
webdriver-manager update --gecko=false --standalone=false $CI_CHROMEDRIVER_VERSION_ARG is now replaced with node webdriver-manager-update.js in the root package.json, which checks which version of chrome puppeteer has come bundled with & downloads informs webdriver-manager to download the corresponding chrome driver version.
Integration tests now use "webdriver-manager": "file:../../node_modules/webdriver-manager" so they don't have to waste time calling webdriver-manager update in postinstall
"// resolutions": "Ensure a single version of webdriver-manager which comes from root node_modules that has already run webdriver-manager update",
"resolutions": {
"**/webdriver-manager": "file:../../node_modules/webdriver-manager"
}
This should speed up each integration postinstall by a few seconds.
Further, integration test package.json files link puppeteer via file:../../node_modules/puppeteer which is the ideal situation as the puppeteer post-install won't download chrome if it is already downloaded. In CI, since node_modules is cached it should not need to download Chrome either unless the node_modules cache is busted.
NB: each version of puppeteer comes bundles with a specific version of chrome. Root package.json & yarn.lock currently pull down puppeteer 2.1.0 which comes with chrome 80. See https://github.com/puppeteer/puppeteer#q-which-chromium-version-does-puppeteer-use for more info.
Only two references to CI_CHROMEDRIVER_VERSION_ARG left in integration tests at integration/bazel-schematics/test.sh which I'm not entirely sure how to get rid of it
Use a lightweight puppeteer=>chrome version mapping instead of launching chrome and calling browser.version()
Launching puppeteer headless chrome and calling browser.version() was a heavy-handed approach to determine the Chrome version. A small and easy to update mappings file is a better solution and it means that the `yarn install` step does not require chrome shared libs available on the system for its postinstall step
PR Close#35049
Previously we needed the `components-repo-ci` blocklist to disable
tests that were failing during the development of Ivy. Since we fixed
all those failing tests, and we don't want to regress, we can remove the
blocklist logic.
Resolves FW-1807
PR Close#35115
* Added a /tools/saucelabs/sauce-service.sh script that manages the sauce-connect as a service which is used by the karma-saucelabs.js wrapper to start the service.
* Added /tools/saucelabs/README.md that covers the details of SauceLabs karma testing with Bazel.
PR Close#34769
We rename the `material-unit-tests` job to `components-repo-unit-tests`
because the job runs all unit tests found in the Angular Components repository.
This includes the Angular CDK, Angular Material and more. Also the repository has
been renamed from `angular/material2` to `angular/components` in the past.
PR Close#34898
The components repository has a Yarn resolution to ensure that
dgeni-packages uses a specific TypeScript version. This resolution
causes the specified TS version to be considered as candidate for the
`@angular/bazel` peer dependency. Ultimately, Yarn decides to use
the TypeScript version from the resolution for `@angular/bazel`,
and builds will fail due to a version mismatch.
This is because `tsickle` will use the hoisted top-level TS
version (set to `3.7.4` ), while `@angular/bazel` uses the
version from the resolution (at the time of writing: v3.6.4)
PR Close#33717
