for #24829
See https://github.com/axios/axios/issues/1045 -- by default Axios
buffers uploaded files into memory fully, to support redirects. For
large file uploads this means we get OOM errors, especially when sending
to multiple teams. There's a few other optimizations we can put in place
here but in the short term we can fix the buffering issue by setting
`maxRedirects: 0` on the requests.
I tested this by adding an `onUploadProgress` handler to the Axios
request that dumps memory usage, and uploading a 209mb software file to
3 teams. Before the update, the readout ticket up continuously (the
first number is the # of bytes uploaded):
```
1540129 {rss: 161652736, heapTotal: 65880064, heapUsed: 55625552, external: 28411157, arrayBuffers: 24338844}
edit-software.js:177
1554313 {rss: 149254144, heapTotal: 65880064, heapUsed: 52445200, external: 25193635, arrayBuffers: 21121327}
edit-software.js:177
2339833 {rss: 151703552, heapTotal: 66404352, heapUsed: 52269280, external: 12664377, arrayBuffers: 8592064}
...a minute later...
192708641 {rss: 619323392, heapTotal: 95240192, heapUsed: 55320960, external: 618952429, arrayBuffers: 614879965}
edit-software.js:177
201523233 {rss: 634613760, heapTotal: 95240192, heapUsed: 58514992, external: 636581613, arrayBuffers: 632509154}
edit-software.js:177
209326677 {rss: 637399040, heapTotal: 95240192, heapUsed: 56800016, external: 639441633, arrayBuffers: 635369173}
```
so we start at ~161mb, and by the time we're done, we're using 637mb of
RAM. Render's free tier has a 250mb limit on apps.
With `maxRedirects: 0`, we see:
```
2669337 {rss: 151846912, heapTotal: 66404352, heapUsed: 53297400, external: 26446868, arrayBuffers: 22374419}
edit-software.js:177
2279929 {rss: 152641536, heapTotal: 66404352, heapUsed: 53453664, external: 27233300, arrayBuffers: 23160851}
edit-software.js:177
2228585 {rss: 153038848, heapTotal: 66404352, heapUsed: 53537096, external: 27626516, arrayBuffers: 23554067}
...a minute later...
209326677 {rss: 146989056, heapTotal: 92094464, heapUsed: 53802856, external: 14617518, arrayBuffers: 10545071}
edit-software.js:177
209326677 {rss: 153051136, heapTotal: 92094464, heapUsed: 55376336, external: 22447478, arrayBuffers: 18375026}
edit-software.js:177
209326677 {rss: 152129536, heapTotal: 92094464, heapUsed: 51857632, external: 22447478, arrayBuffers: 16540013}
```
showing that we start and finish with around the same amount of RAM
used.
Changes:
- (msp dashboard) Updated axios usage in upload-software and
edit-software to use the postForm method.
- Added form-data as a dependency to make sure the app uses
[email protected]
Changes:
- (msp dashboard) Updated the edit-software and upload-software endpoints to log more
information about errors when requests to upload a software installer
fail.
Changes:
- Updated the software page to display different error messages when
requests to software endpoints fail.
- Updated the edit software endpoint to:
- have three new exits, `softwareAlreadyExistsOnThisTeam`,
`couldNotReadVersion` and `softwareDeletionFailed`
- Create database records for newly undeployed software after it has
been removed from all teams on a Fleet instance.
- Return a `softwareAlreadyExistsOnThisTeam` response if undeployed
software is being transferred to a team it is already deployed to.
- Return a `couldNotReadVersion` response if the Fleet instance cannot
read the version information from an uploaded software installer
- Return a `softwareDeletionFailed` response if an installer cannot be
deleted via an API request (If it is configured to be installed during
the macOS setup experience)
- Updated the upload-software endpoint to return a `couldNotReadVersion`
response if the Fleet instance cannot read the version information from
an uploaded software installer
Changes:
- Updated the edit-software endpoint to:
- log a detailed warning when a request to get a stream of an existing
software installer returns a non-200 repsonse
- Use the modify package API endpoint when a software installer is
replaced.
- include the nested response from the Fleet instance in error messages
- Updated the delete-software endpoint to return a
'softwareDeletionFailed` exit when a specified software installer is
included in the macOS setup experience and cannot be deleted via API
requests.
- Updated the software page to show an error message with a link to the
controls page on the connected Fleet instance if the delete-software
endpoint returns a `softwareDeletionFailed` exit
For: https://github.com/fleetdm/confidential/issues/8473
Changes:
- Updated the cloud error components message
- Updated the upload-software action to log errors about failed requests
to the Fleet API
- Updated the edit-software action to log errors about failed requests
to the Fleet API, and to delete temporary files when requests fail.
- Updated the software page to clear cloud errors when modals are
closed.
Related to #21928
Changes:
- Added a /software page, a page where users can manage
(upload/edit/download/delete) software installers on their Fleet
instance across multiple teams at once.
- ~~Removed the `deploy-bulk-operations-dashboard-on-heroku` GitHub
action (This dashboard will be hosted in Render in the future)~~
Reverted this change to unblock merging this PR, I will remove this file
in a separate PR.
