for #24829
See https://github.com/axios/axios/issues/1045 -- by default Axios
buffers uploaded files into memory fully, to support redirects. For
large file uploads this means we get OOM errors, especially when sending
to multiple teams. There's a few other optimizations we can put in place
here but in the short term we can fix the buffering issue by setting
`maxRedirects: 0` on the requests.
I tested this by adding an `onUploadProgress` handler to the Axios
request that dumps memory usage, and uploading a 209mb software file to
3 teams. Before the update, the readout ticket up continuously (the
first number is the # of bytes uploaded):
```
1540129 {rss: 161652736, heapTotal: 65880064, heapUsed: 55625552, external: 28411157, arrayBuffers: 24338844}
edit-software.js:177
1554313 {rss: 149254144, heapTotal: 65880064, heapUsed: 52445200, external: 25193635, arrayBuffers: 21121327}
edit-software.js:177
2339833 {rss: 151703552, heapTotal: 66404352, heapUsed: 52269280, external: 12664377, arrayBuffers: 8592064}
...a minute later...
192708641 {rss: 619323392, heapTotal: 95240192, heapUsed: 55320960, external: 618952429, arrayBuffers: 614879965}
edit-software.js:177
201523233 {rss: 634613760, heapTotal: 95240192, heapUsed: 58514992, external: 636581613, arrayBuffers: 632509154}
edit-software.js:177
209326677 {rss: 637399040, heapTotal: 95240192, heapUsed: 56800016, external: 639441633, arrayBuffers: 635369173}
```
so we start at ~161mb, and by the time we're done, we're using 637mb of
RAM. Render's free tier has a 250mb limit on apps.
With `maxRedirects: 0`, we see:
```
2669337 {rss: 151846912, heapTotal: 66404352, heapUsed: 53297400, external: 26446868, arrayBuffers: 22374419}
edit-software.js:177
2279929 {rss: 152641536, heapTotal: 66404352, heapUsed: 53453664, external: 27233300, arrayBuffers: 23160851}
edit-software.js:177
2228585 {rss: 153038848, heapTotal: 66404352, heapUsed: 53537096, external: 27626516, arrayBuffers: 23554067}
...a minute later...
209326677 {rss: 146989056, heapTotal: 92094464, heapUsed: 53802856, external: 14617518, arrayBuffers: 10545071}
edit-software.js:177
209326677 {rss: 153051136, heapTotal: 92094464, heapUsed: 55376336, external: 22447478, arrayBuffers: 18375026}
edit-software.js:177
209326677 {rss: 152129536, heapTotal: 92094464, heapUsed: 51857632, external: 22447478, arrayBuffers: 16540013}
```
showing that we start and finish with around the same amount of RAM
used.
Changes:
- (msp dashboard) Updated axios usage in upload-software and
edit-software to use the postForm method.
- Added form-data as a dependency to make sure the app uses
[email protected]
Changes:
- (msp dashboard) Updated the edit-software and upload-software endpoints to log more
information about errors when requests to upload a software installer
fail.
Changes:
- Updated the software page to display different error messages when
requests to software endpoints fail.
- Updated the edit software endpoint to:
- have three new exits, `softwareAlreadyExistsOnThisTeam`,
`couldNotReadVersion` and `softwareDeletionFailed`
- Create database records for newly undeployed software after it has
been removed from all teams on a Fleet instance.
- Return a `softwareAlreadyExistsOnThisTeam` response if undeployed
software is being transferred to a team it is already deployed to.
- Return a `couldNotReadVersion` response if the Fleet instance cannot
read the version information from an uploaded software installer
- Return a `softwareDeletionFailed` response if an installer cannot be
deleted via an API request (If it is configured to be installed during
the macOS setup experience)
- Updated the upload-software endpoint to return a `couldNotReadVersion`
response if the Fleet instance cannot read the version information from
an uploaded software installer
Changes:
- Updated the edit-software endpoint to:
- log a detailed warning when a request to get a stream of an existing
software installer returns a non-200 repsonse
- Use the modify package API endpoint when a software installer is
replaced.
- include the nested response from the Fleet instance in error messages
- Updated the delete-software endpoint to return a
'softwareDeletionFailed` exit when a specified software installer is
included in the macOS setup experience and cannot be deleted via API
requests.
- Updated the software page to show an error message with a link to the
controls page on the connected Fleet instance if the delete-software
endpoint returns a `softwareDeletionFailed` exit
https://github.com/fleetdm/fleet/issues/21928#issuecomment-2436371970
Changes:
- updated the edit-software endpoint to make sure that text values
related to software installers (pre-install query, install script,
post-install script, uninstall script) are updated for all teams when
software is edited, and to delete software on the fleet instance when
the installer is replaced and all teams are removed.
- updated the ace editor component to emit an input event when text is
pasted inside of it.
- Updated the error messages in the upload software modal
- Fixed an issue where the edit software endpoint would return a 400
response when all teams are removed from a software installer when a new
installer package is provided.
For: https://github.com/fleetdm/confidential/issues/8473
Changes:
- Updated the cloud error components message
- Updated the upload-software action to log errors about failed requests
to the Fleet API
- Updated the edit-software action to log errors about failed requests
to the Fleet API, and to delete temporary files when requests fail.
- Updated the software page to clear cloud errors when modals are
closed.
Related to #21928
Changes:
- Added a /software page, a page where users can manage
(upload/edit/download/delete) software installers on their Fleet
instance across multiple teams at once.
- ~~Removed the `deploy-bulk-operations-dashboard-on-heroku` GitHub
action (This dashboard will be hosted in Render in the future)~~
Reverted this change to unblock merging this PR, I will remove this file
in a separate PR.
