Better behavior when can't fetch node release index

[rotu]

What version?

0.41.1

Which command?

No response

What happened?

The site http://nodejs.org was briefly unreachable, causing proto to fail:

$ node
Error: plugin::call_func::failed

  × Failed to make HTTP request for https://nodejs.org/download/release/index.json.

I have the version set to node = "lts". The behavior in this case should probably be to either fail gracefully (as proto does when offline completely) or to print an error message suggesting to set the PROTO_OFFLINE environment variable.

Trace logs?

No response

Operating system?

Windows

Architecture?

x64

[davidwinter]

This might be releated, or something to handle differently - please say and I'll happily create a new issue.

I'm trying to install npm on a Windows 10 machine with: proto install npm 10.8.2 --force --log verbose

It's failing with the following output:

[DEBUG 2024-10-15 18:01:51.932] proto   Running proto v0.41.5  bin="C:\\Windows\\system32\\config\\systemprofile\\.proto\\bin\\proto.exe" args=["install", "npm", "10.8.2", "--force", "--log", "verbose"] pid=7832
[TRACE 18:01:51.934] starbase::app run_with_session:run_startup  Running startup phase 
[DEBUG 18:01:51.939] proto_core::proto run_with_session:run_startup:detect_proto_env  Creating proto environment, detecting store  store="C:\\Windows\\system32\\config\\systemprofile\\.proto" home="C:\\Windows\\system32\\config\\systemprofile"
[TRACE 18:01:51.939] starbase::app run_with_session:run_analyze  Running analyze phase 
[DEBUG 18:01:51.939] proto::systems run_with_session:run_analyze:load_proto_configs  Loading configuration in upwards mode  working_dir="C:\\actions-runner\\_work\\wessel\\wessel"
[DEBUG 18:01:51.939] proto_core::proto_config run_with_session:run_analyze:load_proto_configs:load_all:load_config  Loading .prototools  file="C:\\actions-runner\\_work\\wessel\\wessel\\.prototools"
[TRACE 18:01:51.940] starbase_utils::fs run_with_session:run_analyze:load_proto_configs:load_all:load_config:read_file  Reading file  file="C:\\actions-runner\\_work\\wessel\\wessel\\.prototools"
[TRACE 18:01:51.940] schematic::config::loader run_with_session:run_analyze:load_proto_configs:load_all:load_config:load_partial_config  Loading partial configuration  config="ProtoConfig"
[TRACE 18:01:51.941] schematic::config::loader run_with_session:run_analyze:load_proto_configs:load_all:load_config:load_partial_config:parse_into_layers  Creating layer from source  config="ProtoConfig" source="<code>"
[TRACE 18:01:51.941] schematic::config::loader run_with_session:run_analyze:load_proto_configs:load_all:load_config:load_partial_config:merge_layers  Merging partial layers into a final result  config="ProtoConfig"
[DEBUG 18:01:51.942] proto_core::proto_config run_with_session:run_analyze:load_proto_configs:load_all:load_config  Loading .prototools  file="C:\\Windows\\system32\\config\\systemprofile\\.proto\\.prototools"
[TRACE 18:01:51.942] starbase_utils::fs run_with_session:run_analyze:load_proto_configs:load_all:load_config:open_file  Opening file  file="C:\\Windows\\system32\\config\\systemprofile\\.proto\\.prototools"
[TRACE 18:01:52.005] starbase_utils::fs_lock run_with_session:run_analyze:load_proto_configs:load_all:load_config:lock_file_shared  Locking file  file="C:\\Windows\\system32\\config\\systemprofile\\.proto\\.prototools"
[TRACE 18:01:52.007] starbase_utils::fs_lock run_with_session:run_analyze:load_proto_configs:load_all:load_config:lock_file_shared  Unlocking file  file="C:\\Windows\\system32\\config\\systemprofile\\.proto\\.prototools"
[TRACE 18:01:52.008] schematic::config::loader run_with_session:run_analyze:load_proto_configs:load_all:load_config:load_partial_config  Loading partial configuration  config="ProtoConfig"
[TRACE 18:01:52.008] schematic::config::loader run_with_session:run_analyze:load_proto_configs:load_all:load_config:load_partial_config:parse_into_layers  Creating layer from source  config="ProtoConfig" source="<code>"
[TRACE 18:01:52.008] schematic::config::loader run_with_session:run_analyze:load_proto_configs:load_all:load_config:load_partial_config:merge_layers  Merging partial layers into a final result  config="ProtoConfig"
[DEBUG 18:01:52.008] proto_core::proto_config run_with_session:run_analyze:load_proto_configs  Merging loaded configs without global 
[DEBUG 18:01:52.011] proto_core::proto_config run_with_session:run_analyze:load_proto_configs  Merged 1 configs 
[TRACE 18:01:52.011] starbase::app run_with_session:run_execute  Running execute phase 
[DEBUG 18:01:52.012] proto::commands::install install:install_one  Loading tool  id="npm"
[DEBUG 18:01:52.013] proto_core::tool_loader install:install_one:locate_tool  Finding a configured plugin  tool="npm"
[DEBUG 18:01:52.013] proto_core::proto_config install:install_one:locate_tool  Merging loaded configs with global 
[DEBUG 18:01:52.013] proto_core::proto_config install:install_one:locate_tool  Merged 2 configs 
[DEBUG 18:01:52.013] proto_core::tool_loader install:install_one:locate_tool  Using a built-in plugin  plugin="https://github.com/moonrepo/tools/releases/download/node_depman_tool-v0.13.1/node_depman_tool.wasm"
[TRACE 18:01:52.014] warpgate::loader install:install_one:load_tool  Creating plugin loader  cache_dir="C:\\Windows\\system32\\config\\systemprofile\\.proto\\plugins"
[TRACE 18:01:52.014] warpgate::loader install:install_one:load_tool:load_plugin  Loading plugin npm  id="npm" locator="https://github.com/moonrepo/tools/releases/download/node_depman_tool-v0.13.1/node_depman_tool.wasm"
[TRACE 18:01:52.014] warpgate::loader install:install_one:load_tool:load_plugin:load_plugin_from_url:download_plugin:is_plugin_cached  Plugin not cached, downloading  id="npm"
[TRACE 18:01:52.017] starbase_utils::net install:install_one:load_tool:load_plugin:load_plugin_from_url:download_plugin:is_offline_with_options  Checking for an internet connection  timeout=750
[TRACE 18:01:52.056] starbase_utils::net::offline   Resolving 1.1.1.1:53 
[TRACE 18:01:52.111] starbase_utils::net install:install_one:load_tool:load_plugin:load_plugin_from_url:download_plugin:is_offline_with_options  Online! 
[TRACE 18:01:52.112] starbase_utils::fs check_for_new_version:read_file  Reading file  file="C:\\Windows\\system32\\config\\systemprofile\\.proto\\temp\\.last-version-check"
[TRACE 18:01:52.112] warpgate::loader install:install_one:load_tool:load_plugin:load_plugin_from_url:download_plugin  Downloading plugin from URL  id="npm" from="https://github.com/moonrepo/tools/releases/download/node_depman_tool-v0.13.1/node_depman_tool.wasm" to="C:\\Windows\\system32\\config\\systemprofile\\.proto\\plugins\\npm-b9341305d212d01f814c24338dadc08498000fcc4a6b6e8e1e7b27fc5cd7478c.wasm"
[DEBUG 18:01:52.126] warpgate::client install:install_one:load_tool:load_plugin:load_plugin_from_url:download_plugin  Creating HTTP client 
[DEBUG 18:01:52.139] warpgate::client install:install_one:load_tool:load_plugin:load_plugin_from_url:download_plugin  Created HTTP client 
[TRACE 18:01:52.142] starbase_utils::net install:install_one:load_tool:load_plugin:load_plugin_from_url:download_plugin:download_from_url  Downloading file from remote URL to local file  source_url="https://github.com/moonrepo/tools/releases/download/node_depman_tool-v0.13.1/node_depman_tool.wasm" dest_file="C:\\Windows\\system32\\config\\systemprofile\\.proto\\temp\\npm-b9341305d212d01f814c24338dadc08498000fcc4a6b6e8e1e7b27fc5cd7478c.wasm"
[TRACE 18:01:52.920] starbase_utils::fs install:install_one:load_tool:load_plugin:load_plugin_from_url:download_plugin:download_from_url:create_file  Creating file  file="C:\\Windows\\system32\\config\\systemprofile\\.proto\\temp\\npm-b9341305d212d01f814c24338dadc08498000fcc4a6b6e8e1e7b27fc5cd7478c.wasm"
[TRACE 18:01:53.079] starbase_utils::fs install:install_one:load_tool:load_plugin:load_plugin_from_url:download_plugin:move_or_unpack_download:rename  Renaming file  from="C:\\Windows\\system32\\config\\systemprofile\\.proto\\temp\\npm-b9341305d212d01f814c24338dadc08498000fcc4a6b6e8e1e7b27fc5cd7478c.wasm" to="C:\\Windows\\system32\\config\\systemprofile\\.proto\\plugins\\npm-b9341305d212d01f814c24338dadc08498000fcc4a6b6e8e1e7b27fc5cd7478c.wasm"
[DEBUG 18:01:53.084] proto_core::tool_loader install:install_one:load_tool  Loading WASM plugin  source="C:\\Windows\\system32\\config\\systemprofile\\.proto\\plugins\\npm-b9341305d212d01f814c24338dadc08498000fcc4a6b6e8e1e7b27fc5cd7478c.wasm"
[TRACE 18:01:53.084] warpgate::plugin install:install_one:load_tool:inject_default_manifest_config  Storing plugin identifier  id="npm"
[TRACE 18:01:53.085] warpgate::plugin install:install_one:load_tool:inject_default_manifest_config  Storing host environment  id="npm" env={"arch":"x64","libc":"unknown","os":"windows","home_dir":{"path":"/userhome/","virtual_prefix":"/userhome","real_prefix":"C:\\Windows\\system32\\config\\systemprofile"}}
[DEBUG 18:01:53.085] proto_core::tool install:install_one:load_tool  Creating tool npm and instantiating plugin 
[TRACE 18:01:53.085] warpgate::plugin install:install_one:load_tool:new_plugin  Creating plugin container  id="npm"
[TRACE 18:01:53.238] warpgate::plugin install:install_one:load_tool:new_plugin  Created plugin container  id="npm" plugin="1b9f5acb-6f2f-4927-bb86-d5a07e7bc206"
Installing npm
[DEBUG 18:01:53.238] proto_core::tool install:install_one:load_tool  Created tool npm and its WASM runtime 
[TRACE 18:01:53.239] warpgate::plugin install:install_one:load_tool:register_tool:cache_func_with  Calling guest function register_tool  id="npm" plugin="1b9f5acb-6f2f-4927-bb86-d5a07e7bc206" input={"id":"npm"}
[TRACE 18:01:53.243] warpgate::plugin install:install_one:load_tool:register_tool:cache_func_with  Called guest function register_tool  id="npm" plugin="1b9f5acb-6f2f-4927-bb86-d5a07e7bc206" output={"config_schema":{"name":"NodeDepmanPluginConfig","ty":{"type":"Struct","fields":{"dist-url":{"schema":{"ty":{"type":"String"}},"hidden":false,"nullable":false,"optional":true,"read_only":false,"write_only":false},"shared-globals-dir":{"schema":{"ty":{"type":"Boolean"}},"hidden":false,"nullable":false,"optional":true,"read_only":false,"write_only":false}},"partial":false}},"default_version":"bundled","inventory":{},"name":"npm","plugin_version":"0.13.1","type":"DependencyManager"}
[DEBUG 18:01:53.245] proto_core::tool_manifest install:install_one:load_tool:register_tool:create_inventory:load_tool_manifest  Loading manifest.json  file="C:\\Windows\\system32\\config\\systemprofile\\.proto\\tools\\npm\\manifest.json"
[DEBUG 18:01:53.247] proto_core::flow::resolve install:install_one:do_install:resolve_version  Resolving a semantic version or alias  tool="npm" initial_version="10.8.2"
[DEBUG 18:01:53.247] proto_core::flow::resolve install:install_one:do_install:resolve_version:load_version_resolver  Loading available versions  tool="npm"
[TRACE 18:01:53.248] warpgate::plugin install:install_one:do_install:resolve_version:load_version_resolver:cache_func_with  Calling guest function load_versions  id="npm" plugin="1b9f5acb-6f2f-4927-bb86-d5a07e7bc206" input={"initial":"10.8.2"}
[TRACE 18:01:53.260] warpgate::host install:install_one:do_install:resolve_version:load_version_resolver:cache_func_with:host_func_send_request  Calling host function send_request  plugin="1b9f5acb-6f2f-4927-bb86-d5a07e7bc206" input={"url":"https://registry.npmjs.org/npm/"}
[TRACE 18:01:53.381] starbase::app run_with_session:run_shutdown  Running shutdown phase (because another phase failed) 
[TRACE 18:01:53.381] starbase::app run_with_session:run_shutdown  Setting exit code  code=1
Error: plugin::call_func::failed
  × Failed to make HTTP request for [https://registry.npmjs.org/npm/.](https://registry.npmjs.org/npm/)

I've tried accessing the URL directly on the Windows box via Edge, and it loads fine, eventually, as it seems to be quite a slow response. Could this be timeout related? I'm only experiencing it on Windows currently.

I also tried deleting the .proto directory to ensure a fresh attempt at installation, but still no joy. I'm at a loss. I was having the trouble with installing node earlier, but that seems to have passed. Which makes me think it might be a timeout related issue?

[milesj]

I'm not sure what the http libraries are doing under the hood, but my assumption also would be timeout related. Let me see what the current timeout is and if I can adjust it.

[davidwinter]

I had a quick look, and from what I could tell, unless you specify a timeout, the default is to not set a timeout...

This could likely be a quirk with running Windows within a virtual machine. Continuing to investigate. It's just strange that some tools install (like Node etc), and when accessing the URL directly within the VM using Edge, it loads fine, so not appearing like an SSL issue or the like, but just specifically with Rust executables not being able to connect... 🤔

[davidwinter]

Turns out this was a weird DNS issue on a Windows guest VM. I explicitly set Windows to use CloudFlare DNS and suddenly these connection issues stopped. Weird because accessing the URLs directly in the browser was fine within the VM. Annoying that the underling connection issue didn't mention DNS at all... but guessing that is down to reqwest?

[milesj]

In v0.42, we now cache requests for 12 hours. This should help avoid these hiccups.

[rotu]

In v0.42, we now cache requests for 12 hours. This should help avoid these hiccups.

That helps. Would it be simple to log a warning instead of fail outright? That would also help in the case of python where my desired version is "latest" but I want to use the latest locally installed version if indygreg hasn't updated the prebuilt yet.

[milesj]

The data from these requests are required, so we can't simply log a warning and proceed.

[rotu]

The data from these requests are required, so we can't simply log a warning and proceed.

Sure that’s the case when the tool doesn’t exist. in my case, it was doing an auto-install checking if there’s a newer lts. If there were no internet connection, it would skip the check, no?

[milesj]

It would require a bit of a rework to short-circuit out of the version resolve flow, especially only for an alias.