🐛 Fix controller-tools doesn't support single files as input

[yongxiu]

Fix #837

Below command

go run ../vendor/sigs.k8s.io/controller-tools/cmd/controller-gen webhook paths=./apis/cluster_webhook.go output:webhook:artifacts:config=./webhook/

Used to work, but didn't work any more since v0.9.0, which is caused by #663. There is a assumption we want the base part of the path to be either "..." or ".", this is untrue if user specified a specific file, I added a logic to exclude golang file in this case.

Another logic is uniquePkgIDs, as from comment there is not harm that comes from this per se, but it makes testing easier when a known number of modules can be asserted, this is fine however if there are two files coming from same package, second file will be ignored, thus I have to move this logic to test file.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: yongxiu / name: yongxiu (dabf09d)

[k8s-ci-robot]

Welcome @yongxiu!

It looks like this is your first PR to kubernetes-sigs/controller-tools 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/controller-tools has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @yongxiu. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: yongxiu
Once this PR has been reviewed and has the lgtm label, please assign sbueringer for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rmohr]

@yongxiu would you mind adding some context on how you are solving this to the PR description and/or commit?

[yongxiu]

@yongxiu would you mind adding some context on how you are solving this to the PR description and/or commit?

Done

[rmohr]

/ok-to-test

[vincepri]

Why not specify the entire folder even if there is a single file in it?

[vincepri]

paths is a list of directories, not files

[rmohr]

/lgtm

lgtom to me, but requires a project owner to have a look :)

[rmohr]

/remove-lgtm

based on:

paths is a list of directories, not files

waiting for the out come of the discussion.

[yongxiu]

Why not specify the entire folder even if there is a single file in it?

There could be multiple files.

├── apis
│   ├── cluster_webhook.go
│   ├── machine_webhook.go
│   ├── typeBCluster_webhook.go
│   ├── typeBMachine_webhook.go

In above case, I want to generate webhook for cluster_webhook and machine_webhook in output directory A, while typeBCluster_webhook and typeBMachine_webhook to output directory B.

go run ../vendor/sigs.k8s.io/controller-tools/cmd/controller-gen webhook paths=./apis/cluster_webhook.go paths=./apis/machine_webhook.go  output:webhook:artifacts:config=./webhook/normal/
go run ../vendor/sigs.k8s.io/controller-tools/cmd/controller-gen webhook paths=./apis/typeBCluster_webhook.go paths=./apis/typeBMachine_webhook.go output:webhook:artifacts:config=./webhook/special/

paths is a list of directories, not files

While, at least the command line example is using single file as well:

controller-tools/cmd/controller-gen/main.go

Line 139 in 881ffb4

controller-gen object paths=./apis/v1beta1/some_types.go

[rmohr]

@vincepri being able to separate output directories like described above sounds useful and it seemed to work before. Is there a clear idea if that should only be dirs, or is it more an interpretation of the word path that it should be a directory?

[joelanford]

@yongxiu, can you add a test that covers using a single file within a directory? The test should probably cover:

1. Generation does not occur for types in other files in the directory.
2. If the file that we're generating for references identifiers (types, functions, variables, etc.) in this package, but in another file in this package, generation still works and includes those referenced items as necessary.

My hunch is that the reason that the current code doesn't accept individual files is because the Go AST parser operates on package directories, which makes handling my item (2) above trivial.

[yongxiu]

Added webhook integration tests as requested, please take another look, thanks. @joelanford

[joelanford]

Thanks for adding the webhook integration test. That looks good. Can you also add similar tests that cover CRD and RBAC generation? I assume the changes in loader.go apply to all generation not just webhooks, correct?

I'm particularly interested in CRDs with my item (2). If a CRD's types are split between multiple files (say widget_types.go and shared.go) where widget_types.go references types in shared.go that are necessary for the CRD schema generation. What happens when widget_types.go is passed in to the loader?

[yongxiu]

@joelanford, added tests for CRDs and rbac as well

For the case you mentioned, I have this case in the test, it needs to pass in both files, otherwise it will only shows type: object for that field without detail fields, and if we run controller-tool cmd, it will give error:

$ go run cmd/controller-gen/main.go  crd paths=pkg/crd/testdata/multiple_files/file_two.go output:dir=pkg/crd/testdata/
multiple_files/
controller-tools/pkg/crd/testdata/multiple_files/file_two.go:31:7: unknown type TwoResourceSpec
Error: not all generators ran successfully

To clarify, I also tested controller-tools v0.8.0, it give the same error.

[joelanford]

Not suggesting you go and do this before we see what other maintainers think, but I wonder if we could:

• accept a single file as the path
• load the entire package directory that contains the file
• detect types that are defined in the specified file and only generate based on those types

Alternately, another potential option is leaving paths supporting only package directories, but then add other option that lets a caller pass in a specific list of types they want to generate for?

[yongxiu]

IMHO, detecting the types or add one more input of types is over kill. This feature has been broken for 5 releases, which is 1.5 years, and there is only one bug: #837 for this. I don't see much value of adding types, accepting multiple files which is changing back to 0.8.0 behavior is good enough for most of developers.

[yongxiu]

I wonder how can we move this forward? Are we waiting for any other stakeholders?

[yongxiu]

Friendly ping!

[k8s-ci-robot]

@yongxiu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-controller-tools-test-master	dabf09d	link	true	/test pull-controller-tools-test-master

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[airycanon]

@yongxiu @joelanford Hi, Is there any progress on this PR?
My project relies on the single files input, and to use it, I have to keep my Go version at 1.21, as the old version that includes this feature causes a panic when running on Go 1.22.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[rmohr]

@joelanford we would still be interested in that change. Would you have some recommendations on how to move this forward?

[rmohr]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale