Basic scorecard test not working

[siyer-123]

Bug Report

What did you do?

I am trying to implement the basic-spec-check test for operator-sdk scorecard on v0.18.2. Please note that for my use case, operator-sdk version v0.18.2 is a requirement; I am unable to use a newer version at this time.

I am testing my scorecard implementation on the following cluster specs:

[root@my-oc-cluster]# operator-sdk version
operator-sdk version: "v0.18.2", commit: "f059b5e17447b0bbcef50846859519340c17ffad", kubernetes version: "v1.18.2", go version: "go1.13.10 linux/amd64"

In order to run scorecard, I created .osdk-scorecard.yaml in the root with the following info:

scorecard:
  output: text
  bundle: deploy/olm-catalog/<my>-orchestrator
  plugins:
    - basic:
        cr-manifest:
          -  "deploy/crds/<my-orchestrator>.com_v1alpha1_installation_cr.yaml"          

Then, in order to run the scorecard test, I installed operator-sdk v0.18.2 and ran the scorecard test. Note this is a fresh cluster, so no CRDs exists on the cluster at the moment.

operator-sdk scorecard -o text --selector=test=basic-check-spec-test

After running the above, however, the following error is returned:

[root@my-oc-cluster]# operator-sdk scorecard -o text --selector=test=basic-check-spec-test
INFO[0000] Using config file: /root/<my-bundle>/.osdk-scorecard.yaml 
ERRO[0061] Plugin `Basic Tests` failed with error (failed to create namespaced resources: failed to get proxyPod: timed out waiting for the condition) 
	Basic Tests                         
	CR: 
	Labels: 
	Errors: 
		plugin error
	Log:
		failed to create namespaced resources: failed to get proxyPod: timed out waiting for the condition:
		Logs: 

I figured maybe this might be happening because I was using the old osdk-scorecard.yaml method to run scorecard, instead of the newer config.yaml method. To attempt all possibilities, I tried the newer config.yaml method as well, but ran into a separate (different) panic error.

I ran the below command:

operator-sdk scorecard --config=tests/scorecard/config.yaml -o text --selector=test=basic-check-spec-test 

and got the following panic error:

[root@my-oc-cluster]# operator-sdk scorecard --config=tests/scorecard/config.yaml -o text --selector=test=basic-check-spec-test 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe0 pc=0xb5bb87]

goroutine 1 [running]:
github.com/spf13/viper.(*Viper).realKey(0x0, 0x223c555, 0x6, 0x6, 0x7fea060fed98)
	pkg/mod/github.com/spf13/[email protected]/viper.go:1166 +0x37
github.com/spf13/viper.(*Viper).Set(0x0, 0x223c555, 0x6, 0x1e87fa0, 0xc0003a1ad0)
	pkg/mod/github.com/spf13/[email protected]/viper.go:1208 +0x72
github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard.initConfig(0x1f0, 0xc0006fba18, 0x40c1c4)
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard/cmd.go:138 +0x2d1
github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard.buildScorecardConfig(0xc0006d52c0)
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard/cmd.go:174 +0x37
github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard.NewCmd.func1(0xc0006d1080, 0xc0003b3600, 0x0, 0x4, 0x0, 0x0)
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/scorecard/cmd.go:62 +0x56
github.com/spf13/cobra.(*Command).execute(0xc0006d1080, 0xc0003b35c0, 0x4, 0x4, 0xc0006d1080, 0xc0003b35c0)
	pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x460
github.com/spf13/cobra.(*Command).ExecuteC(0xc0006bedc0, 0x2603f60, 0xc0006d2c00, 0x0)
	pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
	pkg/mod/github.com/spf13/[email protected]/command.go:887
github.com/operator-framework/operator-sdk/cmd/operator-sdk/cli.RunLegacy(0xc0000ae000, 0x0)
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/cli/legacy.go:49 +0x11b
main.main()
	src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:43 +0x92

As you can see, neither method worked properly.

What did you expect to see?

I expect to see the scorecard basic-spec-check test run successfully.

What did you see instead? Under which circumstances?

Unfortunately, as described above, neither of the two possible methods to invoke the scorecard test worked properly.

Environment

Operator type:

/language go

Kubernetes cluster type:

Openshift

$ operator-sdk version

operator-sdk version: "v0.18.2", commit: "f059b5e17447b0bbcef50846859519340c17ffad", kubernetes version: "v1.18.2", go version: "go1.13.10 linux/amd64"

$ go version (if language is Go)

go version go1.15.14 linux/amd64

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1-5-g76a04fc", GitCommit:"c66c03f3012a10f16eb86fdce6330433adf6c9ee", GitTreeState:"clean", BuildDate:"2021-02-13T03:54:59Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0+bd9e442", GitCommit:"bd9e4421804c212e6ac7ee074050096f08dda543", GitTreeState:"clean", BuildDate:"2021-02-11T23:05:38Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

N/A

Additional context

N/A

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[camilamacedo86]

/lifecycle frozen

[porridge]

I think the "timed out waiting for the condition" part might be a duplicate of #5452