[BUG] build.sh cannot set CUDAToolkit_INCLUDE_DIR From Bare Metal: https://docs.rapids.ai/api/cuspatial/stable/developer_guide/build/

[wblangdon]

I cannot run ./build.sh libcuspatial cuspatial tests

cuspatial-branch-24.12/build.sh fails to find cuda_runtime.h and cublas_v2.h
Terminates with message "Configuring incomplete, errors occurred!" and status 1

CUDA 12.6 (Driver Version: 560.35.03) installed by others (I have no root access).
cuda_runtime.h and cublas_v2.h are both in /opt/cuda/cuda-12.6/targets/x86_64-linux/include

I am using tcsh

setenv CUDAToolkit_INCLUDE_DIR is ignored by ./build.sh

With both tcsh and bash I cannot find a syntax which allows me to
pass CUDAToolkit_INCLUDE_DIR as an argument of ./build.sh

I can set global environment variable EXTRA_CMAKE_ARGS before running ./build.sh
if I set EXTRA_CMAKE_ARGS -DCUDAToolkit_INCLUDE_DIR=/opt/cuda/cuda-12.6/targets/x86_64-linux/include
./build.sh does run cmake and gets as far as "-- PROJ_LIBRARIES = proj"
and then cmake fails with
"CMake Error at build/_deps/proj-src/cmake/CMakeLists.txt:110 (file):
file RELATIVE_PATH called with incorrect number of arguments"

cmake version 3.26.5

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Wed_Aug_14_10:10:22_PDT_2024
Cuda compilation tools, release 12.6, V12.6.68
Build cuda_12.6.r12.6/compiler.34714021_0

$PATH /opt/cuda/cuda-12.6/bin/:....
$LD_LIBRARY_PATH /opt/cuda/cuda-12.6/lib64:....

To Reproduce
download https://github.com/rapidsai/cuspatial/archive/refs/heads/branch-24.12.zip
unzip branch-24.12.zip
mv cuspatial-branch-24.12.zip cuspatial
cd cuspatial
check $PATH and $LD_LIBRARY_PATH

./build.sh libcuspatial cuspatial tests <== Fails
-- Unable to find cuda_runtime.h in "/opt/cuda/cuda-12.6/include" for CUDAToolkit_INCLUDE_DIR.
-- Unable to find cublas_v2.h in either "" or "/opt/math_libs/include"

Try to set CUDAToolkit_INCLUDE_DIR
Most successful is:
setenv EXTRA_CMAKE_ARGS "-DCUDAToolkit_INCLUDE_DIR=/opt/cuda/cuda-12.6/targets/x86_64-linux/include"
./build.sh libcuspatial cuspatial tests <== Fails
-- Configuring proj library:
-- ENABLE_IPO = OFF
-- PROJ_CORE_TARGET_OUTPUT_NAME = proj
-- BUILD_SHARED_LIBS = ON
-- PROJ_LIBRARIES = proj
CMake Error at build/_deps/proj-src/cmake/CMakeLists.txt:110 (file):
file RELATIVE_PATH called with incorrect number of arguments

Expected behavior
I expected ./build.sh to compile libcuspatial cuspatial and cuspatial's tests

Screenshots
./build.sh generates many lines of output. I have copy+pasted what
seems to be the most relevant ones above. I can send more if it would help.

Desktop (please complete the following information):

• OS: Linux, Rocky 9
• Version 24.12

Smartphone (please complete the following information):
n/a

Additional context
n/a

[wblangdon]

Some progress:-) I have now got ./build.sh to run ok as far as
"-- Installing: ...../cuspatial/lib64/cmake/cuspatial/cuspatial-dependencies.cmake"
I.e. tests not run and cuspatial Python package not built

In summary to do this I used:

setenv PATH /opt/cuda/cuda-12.6/bin/:"$PATH"
setenv LD_LIBRARY_PATH /opt/cuda/cuda-12.6/lib64:"$LD_LIBRARY_PATH"

unzip cuspatial-branch-24.12.zip
mv cuspatial-branch-24.12 cuspatial
setenv CUSPATIAL_HOME pwd/cuspatial
setenv EXTRA_CMAKE_ARGS "-DCMAKE_INSTALL_PREFIX=~/assugi/cuproj/cuspatial -DCUDAToolkit_INCLUDE_DIR=/opt/cuda/cuda-12.6/targets/x86_64-linux/include"
setenv PARALLEL_LEVEL "-j1"
cd $CUSPATIAL_HOME && chmod +x ./build.sh && ./build.sh libcuspatial cuspatial tests

(With just one job, -j1, building and installing cuspatial took about 3.5 hours.)

[GPUtester]

Hi @wblangdon!

Thanks for submitting this issue - our team has been notified and we'll get back to you as soon as we can!
In the mean time, feel free to add any relevant information to this issue.

[trxcllnt]

@wblangdon You should be able to pass -DCUDAToolkit_ROOT=/opt/cuda/cuda-12.6 to point CMake at your CTK location. Would you mind doing a clean build with this option and posting the results? For example:

./build.sh clean libcuspatial cuspatial tests '--cmake-args="-DCUDAToolkit_ROOT=/opt/cuda/cuda-12.6"'

[wblangdon]

Thank you for your help.
All being well I have uploaded the output of setting -DCUDAToolkit_ROOT=/opt/cuda/cuda-12.6 (via setenv) in script_paul_taylor_1.txt (which failed) and setting it to /opt/cuda/cuda-12.6/targets/x86_64-linux/include in script_paul_taylor_2.txt (which compiles, links and copies, but, even though it compiles the tests, it does not run any tests). Both files were generated by "script" and then I cleaned them up by hand.
script_paul_taylor_2.txt
script_paul_taylor_1.txt

Apologies for not following your command line exactly, will try again, but for some reason it did not fully show up this morning (firefox 84.0).

Best wishes
Bill

[wblangdon]

ps: I have tried copying the command line and it fails saying it cannot find cuda_runtime.h and cublas_v2.h I also tried ./build.sh clean libcuspatial cuspatial tests '--cmake-args="-DCUDAToolkit_ROOT=/opt/cuda/cuda-12.6/targets/x86_64-linux/include"' which also fails. All being well I have uploaded a 3rd script output file, showing both.script_paul_taylor_3b.txt

Best wishes
Bill

[trxcllnt]

@wblangdon Thanks, that's helpful. It appears CMake is searching for <CUDAToolkit_ROOT>/include, which is usually a symlink to the platform-specific include dir. For example, on my machine:

$ stat /usr/local/cuda/include
  File: /usr/local/cuda/include -> targets/x86_64-linux/include
  Size: 28        	Blocks: 0          IO Block: 4096   symbolic link

Does your installation have a symlink /opt/cuda/cuda-12.6/include -> /opt/cuda/cuda-12.6/targets/x86_64-linux/include? If not, could you create it? It's also possible that reconfiguring with both CUDAToolkit_ROOT and CUDAToolkit_INCLUDE_DIR arguments will work:

./build.sh clean libcuspatial cuspatial tests '--cmake-args="-DCUDAToolkit_ROOT=/opt/cuda/cuda-12.6 -DCUDAToolkit_INCLUDE_DIR=/opt/cuda/cuda-12.6/targets/x86_64-linux/include"'

[wblangdon]

Dear Paul,
Thank you for your rapid reply.

Ahha !!
Our /usr/local/cuda/ has a logical link include ->
targets/x86_64-linux/include
Will investigate further!! Below is what I tried following your email

In our /opt/cuda/cuda-12.6 include is a subdirectory
( /opt/cuda/cuda-12.6/include contains various cudnn* but not
cuda_runtime.h or cublas_v2.h )

Interestingly there is a symbolic link lib64 -> targets/x86_64-linux/lib
which contains various lib*.so* and lib*.a

I am afraid I do not have root access.

Some progress with your suggestion for --cmake-args
build.sh gets further and now fails at
"CMake Error at build/_deps/proj-src/cmake/CMakeLists.txt:110 (file):
file RELATIVE_PATH called with incorrect number of arguments"
which I hit before http://www.cs.ucl.ac.uk/staff/W.Langdon/cuspatial.html
my hack to get past this was
-DCMAKE_INSTALL_PREFIX=~/assugi/cuproj/cuspatial

Thanks again
Bill
ps: I think it would be useful to have an actual example of how to use
--cmake-args=
(I got confused by the documented syntax which includes < > and I did not
try wrapping it all in ' ' )

pps:
comment sent by email last night lost all formatting, so I have created this new comment.
Also I am appending the output of script in script_paul_taylor_4.txt
script_paul_taylor_4.txt

[wblangdon]

Quick reply some progress: hope to send full update next week Message ID: ***@***.***

[wblangdon]

Last week our technical support added the missing symbolic link
/opt/cuda/cuda-12.6/include -> targets/x86_64-linux/include
and I reran build.sh from scratch on 4 December (see script_paul_taylor_5.txt)

Even though I used
./build.sh clean libcuspatial cuspatial tests '--cmake-args="-DCMAKE_INSTALL_PREFIX=xxx/assugi/cuproj/cuspatial"'
build.sh failed after some hours with pip messages and
no such option: --config-settings
status 2

Although it seems to have compiled and linked the tests ok,
it still does not seem to have run any tests.
script_paul_taylor_5.txt

[harrism]

build.sh with the tests argument configures and builds the C++ unit tests but does not run them. You can find the C++ unit tests in the build directory in a gtests subdirectory. You can run them from there.

I'll let @trxcllnt look into the pip errors.

[wblangdon]

Many thanks for your help.
To confirm I have run the 54 cuspatial/cpp/build/gtests/ script_gtests_ok.txt
tests ok (for completeness script output attached)

[harrism]

Relevant error:

Usage:   
  /usr/bin/python -m pip install [options] <requirement specifier> [package-index-options] ...
  /usr/bin/python -m pip install [options] -r <requirements file> [package-index-options] ...
  /usr/bin/python -m pip install [options] [-e] <vcs project url> ...
  /usr/bin/python -m pip install [options] [-e] <local project path> ...
  /usr/bin/python -m pip install [options] <archive url/path> ...

no such option: --config-settings

The script output you included doesn't even have any calls that use --config-settings, but my hunch is you need to update to a newer pip.