Write a Python Script That Finds Broken Links on Your Site Before Black Friday Traffic

You are a careful Python developer who builds small, polite site checking tools. Before Black Friday sends more visitors to my site, I want a script that crawls my own website and finds broken links, redirect chains, and missing images, so shoppers do not land on error pages. The script must only crawl the domain I give it, respect robots.txt, and go slowly enough not to overload my server. Use only the details I give you. Do not assume my platform, sitemap location, or page count; ask if they matter. If something is missing, mark it [NEEDS INPUT].

My site's start URL and domain: [SITE]
Roughly how many pages it has and whether it has a sitemap.xml: [SIZE]
Platform or hosting, if relevant (Shopify, WordPress, custom): [PLATFORM]
Python version and whether I can install packages: [ENVIRONMENT]
What I want in the report (broken internal links, broken external links, redirects, missing images, slow pages): [REPORT]

Requirements:
1. Write a single Python script using requests and the standard library HTML parser or BeautifulSoup, and explain how to install anything it needs for [ENVIRONMENT].
2. Read robots.txt before crawling, skip disallowed paths, and send a clear user agent string that names the tool.
3. Crawl only pages on [SITE], deduplicate URLs, ignore fragments, and cap the number of pages with a setting.
4. Add a delay between requests and a timeout per request, both configurable, with safe defaults for a small server.
5. Check external links with a HEAD request first and fall back to GET when a server rejects HEAD, without crawling those external sites further.
6. For every problem, record the broken URL, its status code or error, and the page where the link was found, so I can fix it at the source.
7. Flag redirect chains longer than one hop and links that redirect to the homepage, which often hide removed products.
8. Write the results to a CSV file and print a short summary grouped by problem type.
9. Explain how to run it, how to read the CSV, and how to fix the most common causes on [PLATFORM] in plain words.

Deliverables:
1. The complete commented script
2. Install and run instructions
3. Example of the CSV columns and summary output
4. A fix guide for the most common problems
5. A short checklist to rerun before the sale starts