issues with build (reproducability, architecture)

[hohwille]

First Problem: build is not reproducable
node.js and gulp cause a lot of headaches and irreproducible builds.

People that come new into OASP check out the code and something that used to work then simply does not work anymore. The same code-base is leading to different build results for different moments in time. From an software-configuration-point of view this is more or less a catastrophy.
This is really an extremely painful situation that we ran into many many times now and that we should get rid off entirely.

Second Problem: organisation of artifact repositories
IMHO the entire architecture of how node_modules folders are organized and module-versioning is happening is a complete design flaw:

• node_modules are just dumped directly into the current build directory without reuse for other projects
• node_modules are chained recursively causing big problems on windows platforms with long file paths and the silly windows limitations
• node_modules folders are not versioned

Maybe you can already say that maven is sick such that in order to start a small build for the first run you have to download half the internet to your local repo. However maven is putting each artifact version once onto your disk while node.js I have the same artifact version N*M times on the disc.

Is there any chance that this architecture will change in the future? Are we in discussion with the makers?
I see a lot of pain-points in this area.

[hohwille]

For the second problem the makers already had the same thoughts and ideas as it seems:
npm/npm#3697

We should probably only upgrade and things get better...

[hohwille]

Current build error for latest codebase:

Γö£ΓöÇΓöÇ [email protected] ([email protected])
ΓööΓöÇΓöÇ [email protected] ([email protected], [email protected], [email protected], ws@
0.7.2)
[INFO]
[INFO] --- exec-maven-plugin:1.3.1:exec (gulp) @ oasp4js-sample ---
events.js:141
throw er; // Unhandled 'error' event
^
Error: EISDIR: illegal operation on a directory, read
at Error (native)
at Object.fs.readSync (fs.js:603:19)
at Object.fs.readSync (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-sample
s\oasp4j-sample-server\src\main\client\node_modules\wiredep\node_modules\bower-c
onfig\node_modules\graceful-fs\polyfills.js:218:23)
at Object.fs.readFileSync (fs.js:438:24)
at C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples\oasp4j-sample-serv
er\src\main\client\node_modules\gulp-usemin\lib\blocksBuilder.js:64:24
at Array.forEach (native)
at getFiles (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples\oasp4j-s
ample-server\src\main\client\node_modules\gulp-usemin\lib\blocksBuilder.js:61:17
)
at module.exports (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples\oa
sp4j-sample-server\src\main\client\node_modules\gulp-usemin\lib\blocksBuilder.js
:89:20)
at Transform._transform (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samp
les\oasp4j-sample-server\src\main\client\node_modules\gulp-usemin\index.js:16:22
)
at Transform._read (C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples\o
asp4j-sample-server\src\main\client\node_modules\gulp-usemin\node_modules\throug
h2\node_modules\readable-stream\lib_stream_transform.js:184:10)
[13:31:38] Using gulpfile C:\Projekte\oasp\workspaces\main\oasp4j\oasp4j-samples
\oasp4j-sample-server\src\main\client\gulpfile.js

[dumbNickname]

Good point. Node modules flattening was optimized in newer releases of node.js, so it would be nice to upgrade here (why not directly to versions above 4.0 - after merge with socket.io). However flattening algorithm seems to be useful only in case when all the dependencies from all libraries are somehow aligned (same versions of dependencies used where possible). If not, the problem might still occur depending on Windows environment setup or the base path where the developer cloned the repository.

Even after applying the upgrade, we need to be aware that Window paths are limited and can cause problems. I would recommend to experiment with all known workarounds such as using symbolic links or switching to UNICODE paths (http://stackoverflow.com/questions/1880321/why-does-the-260-character-path-length-limit-exist-in-windows). When I faced this issue in the past I managed to overcome it with symbolic links approach, but maybe it would be nice to check UNICODE path possibilities like mentioned here:
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

"The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters). To specify an extended-length path, use the "?" prefix. For example, "?\D:\very long path".
Note The maximum path of 32,767 characters is approximate, because the "?" prefix may be expanded to a longer string by the system at run time, and this expansion applies to the total length."

For sure topic needs some further investigation and a stable solution.

Cheers,
Bartek

[maybeec]

Regarding problem1:

I fixed the latest build issues in PR #79. I tried to fix the dynamic dependencies as far as possible.
I thing due to the current restrictions, we should try to fix all dependencies as far as possible including transitive ones.

To not start with overwhelming initial effort, I would rely on the currently fixed versions as far as we do not observe dynamic build issues in future. However, if there are build issues in future, we should identify the causing (transitive) component and fix its version in addition if not already done before.

Regarding problem2:
node.js update 👍

[hohwille]

See:
oasp/oasp4j-ide#74

[hohwille]

I actually updated node to 5.0.0 and npm to 3.4.1.
However, after spending a lot of time to remove all my node_modules folders and calling npm install again I got the same crappy structure in node_modules. This all totally sucks. Why does the JS world have to reinvent the entire universe but doing so many mistakes that have already been solved in other ecosystems like java nicely after a lot of discussion, experience and sweat?

I went trough npm/npm#3697 again and read the entire thread. They only discussed about a nice solution but never implemented it. The whole issue is not fixed but there is a stupid workaround for a windows problem. The entire point is not only that the Windows filesystem sucks (what is for sure) but also that the entire node_modules structure sucks and is a total waste of disk space and directory nodes. It seems that we have to live with this crap!

[hohwille]

I also found this one:
http://stackoverflow.com/questions/29786887/how-can-i-make-multiple-projects-share-node-modules-directory

I tested it and ... it does not work. I created a fresh node_modules folder using npm install for a project. Then I move this folder up to my top-level project folder. Calling npm install again simply ignored that top-level node_modules folder and loaded all the modules again from the web taking ages dumping my disk full of redundant stuff.
Each node_modules folder of a sub-module of a project has more than thousand nested node_modules folders:

$ find . -name "node_modules" | wc -l
1792

And I have about 70 such sub-modules on my entire disc. A total count is over 50.000 node_modules on my entire disc. And I thought I just started with this modernized Angular and JS based stuff. Where will this end?

[hohwille]

So what I did now is spend another lot of time to delete millions of node_modules folders. Next I created one node_modules folder directly in each major project. For each submodule of such project, I created a symlink pointing on the main node_modules folder of the project.
Then I can build my project with all sub-modules and at least only have one physical node_modules folder with thousand of nested node_modules folders for the entire project instead of having those again 10 or more times.
This is still paintful but a lot better. To make it easier to create the symlinks in windows I recommend this tool:
http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html#download

[hohwille]

So symlink is also not working properly:

npm WARN skippingAction Module is inside a symlinked module: not running add boo
[email protected] node_modules\boom
npm WARN skippingAction Module is inside a symlinked module: not running add cry
[email protected] node_modules\cryptiles
npm WARN skippingAction Module is inside a symlinked module: not running move ht
[email protected] node_modules\html-entities
npm WARN skippingAction Module is inside a symlinked module: not running add htt
[email protected] node_modules\http-signature
npm WARN skippingAction Module is inside a symlinked module: not running add is-
[email protected] node_modules\is-property
npm WARN skippingAction Module is inside a symlinked module: not running add gen
[email protected] node_modules\generate-object-property
npm WARN skippingAction Module is inside a symlinked module: not running add iss
[email protected] node_modules\isstream
npm WARN skippingAction Module is inside a symlinked module: not running move ja
[email protected] node_modules\jasminewd
npm WARN skippingAction Module is inside a symlinked module: not running add jas
[email protected] node_modules\jasminewd2
npm WARN skippingAction Module is inside a symlinked module: not running move js
[email protected] node_modules\json-stringify-safe
npm WARN skippingAction Module is inside a symlinked module: not running add jso
[email protected] node_modules\jsonpointer
npm WARN skippingAction Module is inside a symlinked module: not running add is-
[email protected] node_modules\is-my-json-valid
npm WARN skippingAction Module is inside a symlinked module: not running add har
[email protected] node_modules\har-validator
npm WARN skippingAction Module is inside a symlinked module: not running update
[email protected] node_modules\karma-coverage\node_modules\istanbul
npm WARN skippingAction Module is inside a symlinked module: not running remove
[email protected] node_modules\karma-coverage\node_modules\istanbul\node_modules\files
et\node_modules\glob
npm WARN skippingAction Module is inside a symlinked module: not running update
[email protected] node_modules\karma-coverage\node_modules\minimatch
npm WARN skippingAction Module is inside a symlinked module: not running update
[email protected] node_modules\karma-coverage\node_modules\source-map
npm WARN skippingAction Module is inside a symlinked module: not running remove
[email protected] node_modules\karma-phantomjs-launcher\node_modules\lodash
npm WARN skippingAction Module is inside a symlinked module: not running remove
[email protected] node_modules\karma\node_modules\connect\node_modules\body-parser\no
de_modules\raw-body\node_modules\unpipe
npm WARN skippingAction Module is inside a symlinked module: not running remove
[email protected] node_modules\karma\node_modules\connect
npm WARN skippingAction Module is inside a symlinked module: not running remove
[email protected] node_modules\karma\node_modules\glob
npm WARN skippingAction Module is inside a symlinked module: not running update
[email protected] node_modules\karma\node_modules\graceful-fs
npm WARN skippingAction Module is inside a symlinked module: not running remove
[email protected] node_modules\karma\node_modules\http-proxy
...

npm is so much insane that it gets inconsistent within the same project.
I am giving up with npm. I always thought that maven has some drawbacks and would need some improvements but after working with npm and node I have to say that maven is such a great tool...

[hohwille]

See also npm/npm#8681

[tbialecki]

For deleting node_modules directory i recommend rimraf tool. It can be
installed with npm install -g rimraf
23 lis 2015 23:40 "Jörg Hohwiller" [email protected] napisał(a):

See also npm/npm#8681 npm/npm#8681

—
Reply to this email directly or view it on GitHub
#73 (comment).

[hohwille]

http://www.jongleberry.com/why-i-hate-npm.html

[hohwille]

To avoid this extreme redundancies and waste of diskspace I also tried to convert my package.json to a list of global installs like:

npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]
npm install -g [email protected]

Even after having that installed globally a local install again downloads this to the local node_modules folder. I really have to stop thinking about this stupid tool and just accept that it is causing a world-wild waste of time, disk space and so forth.
Sorry, for this harsh flame war on npm but I am really annoyed...

[hohwille]

Shall we close this issue as wontfix?

[hohwille]

Each time I call npm install I get a different error:

>npm install
npm ERR! tar.unpack error reading C:\Users\hohwille\AppData\Local\Temp\npm-9620-
1185affe\registry.npmjs.org\object-assign\-\object-assign-4.0.1.tgz
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Projekte\\software\\nodejs\\node.exe" "C:\\Users\\hohwille\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v5.0.0
npm ERR! npm  v3.4.1
npm ERR! code EBADF
npm ERR! errno -4083
npm ERR! syscall fstat

npm ERR! EBADF: bad file descriptor, fstat
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Projekte\workspaces\main\app\src\main\client\npm-debug.log

>npm install
npm WARN install Couldn't install optional dependency: Unsupported
npm WARN install Couldn't install optional dependency: Unsupported
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Projekte\\software\\nodejs\\node.exe" "C:\\Users\\hohwille\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v5.0.0
npm ERR! npm  v3.4.1
npm ERR! code ECONNRESET
npm ERR! errno ECONNRESET
npm ERR! syscall read

npm ERR! network read ECONNRESET
npm ERR! network This is most likely not a problem with npm itself
npm ERR! network and is related to network connectivity.
npm ERR! network In most cases you are behind a proxy or have bad network settin
gs.
npm ERR! network
npm ERR! network If you are behind a proxy, please make sure that the
npm ERR! network 'proxy' config is set properly.  See: 'npm help config'

Maybe I am too stupid or too smart for this tool - who knows. I still would expect a build tool to be reproducible.

[hohwille]

Also nice:

npm WARN EPACKAGEJSON [email protected] No repository field.
npm WARN EPACKAGEJSON [email protected] No license field.
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Projekte\\software\\nodejs\\node.exe" "C:\\Users\\hohwille\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "install"
npm ERR! node v5.0.0
npm ERR! npm  v3.4.1

npm ERR! Callback called more than once.
npm ERR!
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

[hohwille]

nodejs/node-v0.x-archive#6960

[maybeec]

current state:

• npm now supports flat file structure. Therefore, the file issue is gone.
• shrinkjs enables a complete reproducable build regarding a fixed snapshot of dependencies.

I will close this issue for now. If there are further issues. Please consider stating a new issue if needed.