Get best of the week

Road to Hell JavaScript Dependencies

Each JavaScript project begins with good intentions, namely that its creators promise themselves not to use too many NPM packages during its development. But even if the developers make considerable efforts to keep this promise, NPM packages gradually penetrate their projects. File size package.jsongrows over time. And with the package-lock.jsoninstallation of dependencies, a real horror occurs, expressed in additions and deletions of packages, especially noticeable with the next PR ... "Everything is fine," says the team leader. The rest of the team nods in agreement. What else to do? We all enjoy the fact that the JavaScript ecosystem is alive and well. We do not need to reinvent the wheel every time and try to solve the problems that have already been solved by the open source community.





Suppose you are going to make a blog and want to use Gatsby.js. Try adding this site generator based on your project. Now, congratulations. Your project just had 19,000 additional dependencies. This is normal? How complex can a JavaScript dependency tree become? How does a dependency tree turn into hell? Let's figure it out.

What is a JavaScript package?


NPM (Node Package Manager, Node Package Manager) stores the largest package registry in the world. These are JavaScript packages. NPM is more than RubyGems, PyPi, and Maven combined. This conclusion can be made on the basis of the analysis of the data of the Module Counts project , which monitors the number of packages in popular registries.


Data on the number of packages in popular registries

You might think that very large amounts of code are represented on this graph. The way it is. In order to turn a project into an NPM package, this package must have a filepackage.json. Such a package can be sent to the NPM registry.

What is package.json?


Here are the tasks it solves package.json:

  • It lists the packages that your project depends on (this is a list of project dependencies).
  • In it, using the rules of semantic versioning, versions of dependency packages that your project can use are set.
  • It allows you to reproduce the environment necessary for the package to work, and, as a result, simplifies the transfer of the project to other developers.

A file package.jsoncan be thought of as a file READMEpumped with steroids. Here you can describe the dependencies of your package, here you can write scripts that are executed during the assembly and testing of the project. The same file contains information about the project version specified by its developer and a description of the project. We are particularly interested in the possibility package.jsonof specifying project dependencies.

Perhaps the fact that the project dependencies are indicated in this file looks somewhat alarming. Imagine that there is a package that depends on another package, and this other package depends on another package. Such a chain of dependencies can be arbitrarily long. For this reason, installing the only package, Gatsby.js, means equipping the project with 19,000 additional dependencies.

Dependency types in package.json


In order to better understand how project dependency lists grow over time, let's talk about the different types of dependencies that a project may have. Namely, the package.jsonfollowing sections can be found that describe various dependencies:

  • dependencies - these are ordinary dependencies, the functionality of which is used in the project, and which are accessed from its code.
  • devDependencies- these are development dependencies. For example, the prettier library used to format code.
  • peerDependencies - if dependencies are written to this section, the package developer thereby informs the one who will install it that he will need a specific version of the package specified in this section.
  • optionalDependencies - they list optional dependencies, such, the inability to install which will not violate the installation process of the package.
  • bundledDependencies β€” , . , NPM, , .

package-lock.json


We all know that the file package-lock.json, in the course of work on the project, constantly undergoes changes. Something is removed from it, something is added to it. This is especially noticeable when viewing PR containing an updated version of this file. We often take it for granted. A file is package-lock.jsonautomatically generated every time a file package.jsonor folder changes node_modules. This allows you to maintain the contents of the dependency tree exactly as it was when you installed the project dependencies. This allows, when installing the project, to reproduce the dependency tree. This solves the problem of having different versions of the same package from different developers.

Consider a project whose dependencies include React. The corresponding entry is available at package.json. If you look at the filepackage-lock.json of this project, then you can see something like the following:

    "react": {
      "version": "16.13.0",
      "resolved": "https://registry.npmjs.org/react/-/react-16.13.0.tgz",
      "integrity": "sha512-TSavZz2iSLkq5/oiE7gnFzmURKZMltmi193rm5HEoUDAXpzT9Kzw6oNZnGoai/4+fUnm7FqS5dwgUL34TujcWQ==",
      "requires": {
        "loose-envify": "^1.1.0",
        "object-assign": "^4.1.1",
        "prop-types": "^15.6.2"
      }
    }

A file package-lock.jsonis a large list of project dependencies. Here are the dependency versions, paths (URIs) to the modules, hashes used to verify the integrity of the module and the packages needed by this module. If you read this file, you can find records of all the packages that React needs. This is where the real hell of dependencies lies. Everything that the project needs is described here.

Understanding Gatsby.js dependencies


How is it that, having established only one dependency, we add as many as 19,000 dependencies to the project? It's all about dependency dependencies. That is why we have what we have:

$ npm install --save gatsby

...

+ gatsby@2.19.28
added 1 package from 1 contributor, removed 9 packages, updated 10 packages and audited 19001 packages in 40.382s

If you look in package.json, there you can find only one dependency. But if you look at it package-lock.json, it turns out that in front of us is an almost 14-kilobyte monster. A more detailed answer about what all those lines of code that fall into mean package-lock.json, can be found in the file package.jsonin the Gatsby.js repository . There are a lot of direct dependencies, namely, according to npm calculations , 132. If each of these dependencies has at least one more dependency, then the total number of project dependencies will double - and it will have 264 dependencies. Of course, in the real world this is not so. Each direct project dependency has more than 1 inherent dependency. As a result, the list of project dependencies is very long.

For example, we’ll take an interest in how many times the lodash library is used as a dependency for other packages :

$ npm ls lodash
example-js-package@1.0.0
└─┬ gatsby@2.19.28
  β”œβ”€β”¬ @babel/core@7.8.6
  β”‚ β”œβ”€β”¬ @babel/generator@7.8.6
  β”‚ β”‚ └── lodash@4.17.15  deduped
  β”‚ β”œβ”€β”¬ @babel/types@7.8.6
  β”‚ β”‚ └── lodash@4.17.15  deduped
  β”‚ └── lodash@4.17.15  deduped
  β”œβ”€β”¬ @babel/traverse@7.8.6
  β”‚ └── lodash@4.17.15  deduped
  β”œβ”€β”¬ @typescript-eslint/parser@2.22.0
  β”‚ └─┬ @typescript-eslint/typescript-estree@2.22.0
  β”‚   └── lodash@4.17.15  deduped
  β”œβ”€β”¬ babel-preset-gatsby@0.2.29
  β”‚ └─┬ @babel/preset-env@7.8.6
  β”‚   β”œβ”€β”¬ @babel/plugin-transform-block-scoping@7.8.3
  β”‚   β”‚ └── lodash@4.17.15  deduped
  β”‚   β”œβ”€β”¬ @babel/plugin-transform-classes@7.8.6
  β”‚   β”‚ └─┬ @babel/helper-define-map@7.8.3
  β”‚   β”‚   └── lodash@4.17.15  deduped
  β”‚   β”œβ”€β”¬ @babel/plugin-transform-modules-amd@7.8.3
  β”‚   β”‚ └─┬ @babel/helper-module-transforms@7.8.6
  β”‚   β”‚   └── lodash@4.17.15  deduped
  β”‚   └─┬ @babel/plugin-transform-sticky-regex@7.8.3
  β”‚     └─┬ @babel/helper-regex@7.8.3
  β”‚       └── lodash@4.17.15  deduped
  ...

Fortunately, most of these dependencies are represented by the same version of lodash. And with this approach, there node_moduleswill be only one lodash library folder. True, this is usually not quite the case. Sometimes different packages need different versions of the same package. That is why a lot of jokes appeared about the huge size of the folder node_modules. In our case, however, everything is not so bad:

$ du -sh node_modules
200M    node_modules

200 megabytes is not so bad. I saw how the size of this folder easily reaches 700 MB. If you are interested in learning which modules take up the most space, you can run the following command:

$ du -sh ./node_modules/* | sort -nr | grep '\dM.*'
 17M    ./node_modules/rxjs
8.4M    ./node_modules/@types
7.4M    ./node_modules/core-js
6.8M    ./node_modules/@babel
5.4M    ./node_modules/gatsby
5.2M    ./node_modules/eslint
4.8M    ./node_modules/lodash
3.6M    ./node_modules/graphql-compose
3.6M    ./node_modules/@typescript-eslint
3.5M    ./node_modules/webpack
3.4M    ./node_modules/moment
3.3M    ./node_modules/webpack-dev-server
3.2M    ./node_modules/caniuse-lite
3.1M    ./node_modules/graphql
...

Yes, rxjs is an insidious package.

Here is a simple command that helps reduce the size of the folder node_modulesand simplify its structure:

$ npm dedup
moved 1 package and audited 18701 packages in 4.622s

51 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

During deduplication, npm tries to simplify the structure of the dependency tree by finding the dependencies used by other dependencies and moving them so that they can be shared. This applies to our lodash example. Many packages are used lodash @4.17.15, as a result, to ensure their operability, it is enough to install this version of the library only once. Of course, this is the situation that we get into from the very beginning, only by establishing dependencies. If in the process of working on a project package.jsonadd new dependencies, it is recommended sometimes to remember the team npm dedup. If you use the yarn package manager, there a similar command looks like yarn dedupe. But, in fact, there is no need for it, since dependency optimization is performed automatically when the command is executed yarn install.

Dependency Visualization


Interested in a graphical representation of the dependencies of your project? If so, you can create such a presentation using special tools. Let's consider some of them.

The following is a dependency visualization result obtained using npm.anvaka.com/ .


Dependency visualization using npm.anvaka.com.

Here you can see the dependencies of the Gatsby.js project package dependencies. The result is similar to a huge web. The Gatsby.js project has so many dependencies that this β€œweb” almost β€œhung” my browser. Now , if interested, a link to this diagram. It can be presented in 3D-form.

Here is a visualization made using npm.broofa.com .


A fragment of a dependency visualization made using npm.broofa.com

This is similar to a flowchart. She, for Gatsby.js, turns out to be very complicated. You can take a look at it here . Circuit elements can be colored based on estimates from npms.io . You can upload your own file to the sitepackage.json.

The Package Phobia toolallows you to find out how much space it needs before installing a package.


Package Information Received Using Package Phobia

Here you can find out about the size of the published package, and how much disk space it will take after installation.

Bottom line: with great power comes great responsibility


In the end, I want to say that JavaScript and NPM are great tools. The good thing is that modern developers have the opportunity to use a huge set of dependencies. Running the command npm installin order to save yourself from writing a couple of lines of code is so easy that sometimes we forget about the consequences.

Now that you have read up to this point, you should have a more complete understanding of the features of the npm-project dependency tree structure. If you add a library to the project that is very large, or if you just explore the dependencies of your project, you can always take advantage of what we have discussed here and analyze the dependencies.

Dear readers! Do you want to use as few dependencies as possible in your npm projects?

More articles:


All Articles