Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Update API link crawler #18794

Draft
wants to merge 5 commits into
base: production
Choose a base branch
from
Draft

[Docs] Update API link crawler #18794

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
55998fc
[Docs] Update API link crawler
pedrosousa Dec 17, 2024
a0f2385
Re-add crawler script entry to package.json
pedrosousa Jan 3, 2025
7852688
Update sitemap URL
pedrosousa Jan 3, 2025
410d390
Handle missing response
pedrosousa Jan 3, 2025
221e05d
Revert to original sitemap URL (we're checking loc elements in the sc…
pedrosousa Jan 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 17 additions & 11 deletions bin/crawl-api-links.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,14 @@ async function checkLinks() {
});
const page = await browser.newPage();

const sitemapUrl = "https://developers.cloudflare.com/sitemap.xml";
// skip image requests
await page.setRequestInterception(true);
page.on("request", (request) => {
if (request.resourceType() === "image") request.abort();
else request.continue();
});

const sitemapUrl = "https://developers.cloudflare.com/sitemap-0.xml";
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real XML file containing all the URLs (after one redirect and one loc attribute).

await page.goto(sitemapUrl, { timeout: navigationTimeout });

const sitemapLinks = await page.$$eval("url loc", (elements) =>
Expand Down Expand Up @@ -51,23 +58,22 @@ async function checkLinks() {
}

if (
pageLink.includes("developers.cloudflare.com/api/operations/") ||
pageLink.startsWith("/api/operations/")
pageLink.includes("developers.cloudflare.com/api/resources/") ||
pageLink.startsWith("/api/resources/")
) {
console.log(`Evaluating link: ${pageLink}`);
await page.goto(pageLink, {
const response = await page.goto(pageLink, {
waitUntil: "networkidle0",
timeout: navigationTimeout,
});
visitedLinks.push(pageLink);

const statusCode = await page.evaluate(() => {
return {
url: window.location.href,
};
});
if (statusCode.url === "https://developers.cloudflare.com/api/") {
brokenLinks.push(pageLink);
if (response) {
if (response.status() === 404) {
brokenLinks.push(pageLink);
}
} else {
console.log("WARNING: Didn't receive a response... skipping.");
}
}
}
Expand Down
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
"check:astro": "npm run sync && astro check",
"check:functions": "npx tsc --noEmit -p ./functions/tsconfig.json",
"check:worker": "npx tsc --noEmit -p ./worker/tsconfig.json",
"crawl-api-links": "node bin/crawl-api-links.js",
Copy link
Contributor Author

@pedrosousa pedrosousa Jan 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used in our scheduled action that checks API links. Currently returning an error because this entry in scripts is missing.

"dev": "npx astro dev",
"format": "npm run format:core && npm run format:data",
"format:core": "npx prettier --write \"**/*.{js,jsx,ts,tsx,mjs,css}\"",
Expand Down
Loading