How to Detect and Fix Broken Links at Scale: A Developer's Guide

Finding broken links on a website is one of those tasks that sounds simple until you are dealing with hundreds of pages. Manually clicking through every link is not an option at scale. This guide covers how to detect and fix broken links programmatically using an API, with real code examples in Node.js, Python, and PHP.

Why Broken Links Matter

Broken links (returning 404, 500, or other error codes) hurt your website in two key ways:

1. SEO: Search engines crawl your links. Dead links waste crawl budget and signal poor site maintenance.
2. User experience: Nothing erodes trust faster than clicking a link and landing on a 404 page. Google does not penalize you directly for broken links, but the indirect effects — higher bounce rates, lower crawl efficiency, reduced authority — add up quickly. A site with clean internal and external links ranks and converts better.

What Is a Broken Links API?

A Broken Links API crawls a given URL and returns all links on the page along with their HTTP status codes. Instead of building and maintaining a headless browser crawler yourself, you make a single API call and get back structured data. ToolCenter's Broken Links API does exactly this — it loads the target page in a real browser (handling JavaScript-rendered content), extracts all anchor tags, and checks each link's status code in parallel. Sample response:

{
  "url": "https://example.com",
  "total": 42,
  "broken": 3,
  "links": [
    { "url": "https://example.com/about", "status": 200, "ok": true },
    { "url": "https://example.com/old-page", "status": 404, "ok": false },
    { "url": "https://external-site.com/resource", "status": 500, "ok": false }
  ]
}

Node.js Example

const axios = require('axios');
async function checkBrokenLinks(targetUrl) {
  const response = await axios.post(
    'https://api.toolcenter.dev/v1/broken-links',
    { url: targetUrl },
    { headers: { 'X-API-Key': process.env.TOOLCENTER_API_KEY } }
  );
  const { links, broken } = response.data;
  if (broken === 0) {
    console.log('No broken links found!');
    return;
  }
  console.log('Found ' + broken + ' broken link(s):');
  links
    .filter(link => !link.ok)
    .forEach(link => console.log('  ' + link.status + ' -> ' + link.url));
}
checkBrokenLinks('https://yourdomain.com');

Python Example

import os
import requests
def check_broken_links(url: str):
    response = requests.post(
        "https://api.toolcenter.dev/v1/broken-links",
        json={"url": url},
        headers={"X-API-Key": os.environ["TOOLCENTER_API_KEY"]},
    )
    data = response.json()
    broken = [link for link in data["links"] if not link["ok"]]
    if not broken:
        print("No broken links found!")
        return
    print(f"Found {len(broken)} broken link(s):")
    for link in broken:
        print(f"  {link['status']} -> {link['url']}")
check_broken_links("https://yourdomain.com")

PHP Example

function checkBrokenLinks(string $url): void {
    $ch = curl_init('https://api.toolcenter.dev/v1/broken-links');
    curl_setopt_array($ch, [
        CURLOPT_POST => true,
        CURLOPT_POSTFIELDS => json_encode(['url' => $url]),
        CURLOPT_HTTPHEADER => [
            'Content-Type: application/json',
            'X-API-Key: ' . getenv('TOOLCENTER_API_KEY'),
        ],
        CURLOPT_RETURNTRANSFER => true,
    ]);
    $response = json_decode(curl_exec($ch), true);
    curl_close($ch);
    $broken = array_filter($response['links'], fn($link) => !$link['ok']);
    if (empty($broken)) {
        echo "No broken links found!\n";
        return;
    }
    echo "Found " . count($broken) . " broken link(s):\n";
    foreach ($broken as $link) {
        echo "  {$link['status']} -> {$link['url']}\n";
    }
}
checkBrokenLinks('https://yourdomain.com');

Automating Broken Link Monitoring

The real power comes from running checks on a schedule. Here is a pattern for automated monitoring that alerts you when broken links appear:

const axios = require('axios');
const PAGES_TO_MONITOR = [
  'https://yourdomain.com',
  'https://yourdomain.com/blog',
  'https://yourdomain.com/docs',
];
async function monitorAllPages() {
  const allBroken = [];
  for (const page of PAGES_TO_MONITOR) {
    const response = await axios.post(
      'https://api.toolcenter.dev/v1/broken-links',
      { url: page },
      { headers: { 'X-API-Key': process.env.TOOLCENTER_API_KEY } }
    );
    const broken = response.data.links.filter(l => !l.ok);
    if (broken.length > 0) {
      allBroken.push({ page, broken });
    }
  }
  if (allBroken.length > 0) {
    console.log('Broken links detected:', JSON.stringify(allBroken, null, 2));
  } else {
    console.log('All pages clean!');
  }
}
monitorAllPages();

Common Status Codes to Watch

| Status | Meaning | Action | |--------|---------|--------| | 404 | Page not found | Update or remove the link | | 301/302 | Redirect | Update to final destination URL | | 403 | Forbidden | Check if link should be private | | 500 | Server error | May be temporary; recheck later | | 0 | Connection failed | Check if domain is still active |

Internal vs. External Links

For most use cases, treat internal and external links differently:

• Internal links (same domain): Fix immediately — these are fully under your control.
• External links (other domains): Monitor and remove if persistently broken; these change without warning.

const { links } = response.data;
const yourDomain = 'yourdomain.com';
const brokenInternal = links.filter(l => !l.ok && l.url.includes(yourDomain));
const brokenExternal = links.filter(l => !l.ok && !l.url.includes(yourDomain));
console.log('Internal broken: ' + brokenInternal.length);
console.log('External broken: ' + brokenExternal.length);

CI/CD Integration

Add a broken links check to your deployment pipeline so no release ships with dead links:

# .github/workflows/broken-links.yml
name: Broken Links Check
on:
  push:
    branches: [main]
jobs:
  check-links:
    runs-on: ubuntu-latest
    steps:
      - name: Check broken links
        run: |
          RESULT=$(curl -s -X POST https://api.toolcenter.dev/v1/broken-links 
            -H "X-API-Key: ${{ secrets.TOOLCENTER_API_KEY }}" 
            -H "Content-Type: application/json" 
            -d '{"url": "https://yourdomain.com"}')
          BROKEN=$(echo $RESULT | python3 -c "import sys,json; print(json.load(sys.stdin)['broken'])")
          if [ "$BROKEN" -gt "0" ]; then
            echo "Build failed: $BROKEN broken link(s) found."
            exit 1
          fi
          echo "All links OK."

Conclusion

Broken links are easy to ignore and expensive to let accumulate. With the ToolCenter Broken Links API you can audit any page on demand, automate daily monitoring, integrate checks into CI/CD, and distinguish between internal and external failures. Get started free at toolcenter.dev.