Package Updates

Package Updates

This is a guide used by the team for updating package versioning. This was driven by the need to update packages according to security vulnerabilities presented in outdated package versions.

Option 1: Manually Updating the Packages

This was proposed as the most sage way of addressing security vulnerabilities.

Using grep, go through all the files in the repository.

For instance, say lodash is out of date and component governance is yelling at us. Running the following command can show us where lodash is being referenced:

grep -r "lodash" --include="*package.json" --exclude-dir="node_modules" .

This will give us a list of the uses of lodash in package.json and also the versions:

$ grep -r "lodash" --include="*package.json" --exclude-dir="node_modules" .
./common/transport/amqp/package.json:    "lodash.merge": "^4.6.1",
./device/core/package.json:    "lodash": "^4.17.11",
./device/samples/package.json:    "lodash": "^4.17.11"
./e2etests/package.json:    "lodash": "^4.17.11",
./service/package.json:    "lodash": "^4.17.11",

From here you go through on vim (if you choose inifite suffering) and manually update the versions to the latest. I didn't try this but the sage says it's the best option.

for f in $(grep -r "lodash" . --include="*package.json" --exclude-dir="node_modules" -l); do vi $f; done

This is fine if the changes are only necessary in package.json. If the changes required are in package-lock.json (likely because of a dependency carrying an outdated version of a package), then it can be more tricky. Manually editing the package-lock.json files is tedious at best.

Option 2: Wheeeeee

Ok for this one I threw the kitchen sink at the problem. Let's start at the beggining. Things aren't working. So starting on a new master, here what I ran:

$ lerna bootstrap --hoist
...
$ lerna exec npm update
...
$ lerna exec npm audit fix
...

After this tiring process alot of things have been updated. But maybe not everything that's necessary. To finish the process, you can rerun the SDL rules on the branch you've created, and note any remaining Component Governance warnings. For instance, after running this there are still warnings about lodash:

|CVE-2019-10744                          |lodash 4.17.11                          |High            |

To fix this, I will look where lodash is being used, following the grep method above.

grep -r "lodash" --include="*package.json" --exclude-dir="node_modules" .

Using sed to change lines in files

After running lerna exec npm update you might run into the issue that your package.json now has carats in front of the "azure-iot-device" packages. This is bad, since we tightly couple all our packages together by versions. Therefore we need to fix this. Using sed we can accomplish this. For instance, say package.json in device/samples needs to be fixed. In device/samples we could execute:

sed -i 's/\"azure-iot-device\": \"\^/\"azure-iot-device\": \"/g' package.json

But what if we want to do this on every package.json file that has azure-iot-device? Simple, with grep:

grep -rl '"azure-iot-common"' . --exclude-dir="node_modules" --include=package.json  | xargs sed -e 's/\"azure-iot-common\": \"\^/\"azure-iot-common\": \"/g' -i

Resources

Update all the Node dependencies to their latest version What does NPM ERR: extraneous! mean? (SO)