[Question] How to test my GitHub API based project

[frank-zsy]

Sorry for interruption, I am not sure it is the right place to ask this but I don't know anywhere better.

I build a project based on the octokit/rest.js to collaborate with my work on GitHub, but I don't know how to test the functions because most functions are based on the GitHub Webhook and GitHub API.

Should I use jest and how can I make sure the procedure is right? Do I need a fake GitHub server or only to mock and hijack the requests?

[gr2m]

No worries, that is a very good question!

When testing my libraries using @octokit/rest I usually do one of two things

1. I use nock to mock requests to https://api.github.com
2. I create a hook that replaces sending a request with a mock

Here is a code example for testing with jest and nock

const Octokit = require('@octokit/rest')
const nock = require('nock')

test('root request', async () => {
  const octokit = new Octokit()
  const scope = nock('https://api.github.com')
    .get('/')
    reply(200, {})

  await octokit.request('/')
  scope.done() // see https://github.com/nock/nock#expectations
})

Here is an example using a hook

const Octokit = require('@octokit/rest')
const nock = require('nock')

test('root request', async () => {
  const octokit = new Octokit()
  octokit.wrap('request', (request, options) {
    expect(options.method).toBe('GET')
    expect(options.url).toBe('https://api.github.com/')
  })
  await octokit.request('/')
})

Does that help?

[frank-zsy]

@gr2m Yes, this is very helpful for me and I will try to add some tests to my project, however this may involve a lot of development work on test framework. So maybe there should be a test framework for GitHub API and WebHook to solve these problems.

[gr2m]

Testing the combination of webhooks and REST/GraphQL requests is indeed a bit complicated. It’s a challenge I’ve seen with Probot apps, a framework for GitHub Apps based on Octokit.

While nock is great to mock requests, it does not include a feature to mock received webhooks (yet). A project that supports both is https://github.com/slackapi/steno, but I did not use that myself yet.

Do you know of a great example for similar libraries? Making testing easy is definitely and important for us, so if you have any insights you could share on what the ideal setup would look like, we’d appreciate it

[frank-zsy]

@gr2m Thanks for the advise, I will look into the projects.
But what I encounter is that the functions I need to test depend on the repo status, which means I need to build all kinds of test cases for every function. So request mock is not quite friendly for me, what I expect is that the test framework may take a repo name as input and fake the whole repo by pulling data from GitHub, then the framework can handle the requests same as GitHub and modify the data in it. So as the framework performs identical as GitHub backend then the tests can be quite easy for the projects. And if the framework fakes a whole repo, then the WebHook won't be a problem anymore. Does this make sense for you?

[gr2m]

Could you write out an example of how that would look like?

What I expect is that the test framework may take a repo name as input and fake the whole repo by pulling data from GitHub, then the framework can handle the requests same as GitHub and modify the data in it.

Would the repo already exist, or would you expect the test framework to create a repository with your settings, then record the fixtures?

[frank-zsy]

@gr2m I assume that the repo already exists, the sample code may like this:

let fakeGitHub = new FakeGitHub("apache/rocketmq"); // fakeGitHub will pull the data from GitHub
fakeGitHub.listen(port).then(...); // fakeGitHub will listen on port and handle request like GitHub API

I am not sure this is the right way to accomplish this, but I think this can be really helpful because data from real repo especially big repo may cover all the cases and the mock data is really hard for this.

[gr2m]

The above code could mock GET and HEAD requests, but I’m not sure how it could mock mutating requests without also changing the apache/rocketmq repository 🤔 The FakeGitHub library would need to implement parts of GitHub itself for that.

What is a scenario that you would like to test with your app / library?

[frank-zsy]

Yes, this is what I mean, like the FakeGitHub implements basic data manipulate interface and inner logic as well, so the request to GitHub will be handled by FakeGitHub instead of real GitHub.

For example, my robot will check pull request status regularly and append a comment or label when the PR has conflict with master branch which will block merging, and remove the comment or label when the author fix it. So how can I test this function because the behavior depends on the PR data on GitHub and I need to check the result after comment or label process.

So if I want to test the function above, I need to mock the data of all cases and check if certain API is called? Maybe with the FakeGitHub, I only need to handle the initial status of the backend and check the final state of it instead of mock the data or check the API calls which is more reasonable for me.

[gr2m]

Thanks, I’ll give it some thought

[gr2m]

I’m working on something unrelated which might end up helping with a more complicated test setup such as yours.

I’m looking into recording responses for all REST API endpoints so that we can use it to test our Octokit libraries across all language implementations. Most of the requests require a setup, e.g. in order to record a response of "Get a single issue", the setup includes

• create a repository
• create an issue

I’ve described how that could work at octokit/fixtures#204

Eventually I hope that integrators would be able to utilize such a setup for their own testing. They could create higher-level setups and even share them with the community so others could build upon them.

Could you describe some of workflows that your project implements to collaborate with my work on GitHub?

[frank-zsy]

@gr2m sorry for the late reply, I've been occupied by other stuff recently.

I am not quite sure I understand what you mean. Does you mean that you will record all the responses from a certain repo for all endpoints, so we can get responses for testing. This makes sense for all GET APIs, but for POST APIs, how does this project behave?

[gr2m]

My idea is that all requests will be recorded, not only the GET requests. I’m fairly sure that it would work for your use cases.

Can you explain in more detail the request flows you need to test for your project?

[frank-zsy]

@gr2m Sorry, actually Collabobot is my first project in TypeScript and I never wrote any tests before, so I am not quite sure how to integrate a test framework. Maybe you can implement as you think and I will fit in.

[gr2m]

I’ll let you know, but it will take a while until I get to work on this

[frank-zsy]

@gr2m thanks, you may take some thoughts on the request chain, some requests may cause request - webhook chain. And if there is anything I can do to help, please let me know.

[gr2m]

will do, thank you!

[yuujay]

This would be an excellent feature that would help us test the GitHub APIs that our apps use with the official testtools that any of the octokit client provides. We are using octokit's ruby client. I did look at octokit/fixtures#204. But unsure if this is being worked on. Can you please confirm if there is an implementation of client mocking that we could make use of?

[gr2m]

Can you please confirm if there is an implementation of client mocking that we could make use of?

Not for all endpoints I'm afraid, https://github.com/octokit/fixtures/ / https://github.com/octokit/fixtures-server is all we have so far.

Would octokit/fixtures#204 suffice for your testing requirements? In my experience, you'll often need a test scenario which involves multiple requests. Do you need to only mock requests or also webhook events?

[yuujay]

Thanks for the response. But, I am afraid https://github.com/octokit/fixtures/ / https://github.com/octokit/fixtures-server would not be sufficient to my needs for the following reason.

The two projects that you have specified looks like js modules that can be loaded into a node project, but our's is a ruby app. Hence I was looking octokit's ruby client and ways to mock it in ruby.

The use-case that I am trying to test and mock is that, we have a ruby app which try to get information on a ruby repository specifically

1. To check if a release is available in the repo
2. If answer is yes, to the question above, then get the contents and metadata of the latest release from repo.

And then use the above information in our app to proceed further. I am trying to mock the above two steps in our testing workflow, to prevent requests being made to the actual GitHub APIs.

Do let me know if there is a way to use https://github.com/octokit/octokit.rb or similiar projects for my workflow. Thanks.

[ChaiBapchya]

Is there a similar support for Python based projects?

[gr2m]

You can run the fixtures server as a single file binary. We upload binaries for Windows/Mac/Linux to each release:
https://github.com/octokit/fixtures-server/releases/tag/v6.0.6

So it does not matter what language you want to use it with. It's just a single-file binary server that you start and test against