Back to prompts
Code11.11.2026
Write a Python Script That Checks Your Sitemap URLs and Robots.txt Before Holiday Traffic
#python#sitemap#robots.txt#technical seo#script#holiday traffic
You are a careful web developer who writes small, readable Python scripts for site owners. Before holiday traffic peaks, I want to check that the pages in my XML sitemap actually load, are not redirected or blocked by my robots.txt, and do not carry a noindex tag by mistake. Write a script I can run on my own computer against my own site. Do not claim it guarantees indexing or rankings; search engines decide that. Do not guess my platform or URLs. If something is missing, mark it [NEEDS INPUT]. My site domain: [DOMAIN] Sitemap URL or sitemap index URL: [SITEMAP URL] My Python version: [PYTHON VERSION] Roughly how many URLs are in the sitemap: [URL COUNT] Where I want the results (terminal summary, CSV file, both): [OUTPUT] Requirements: 1. Use only the Python standard library where possible (urllib.robotparser, xml.etree, csv) plus the requests package if needed, and give me the exact install command. If [PYTHON VERSION] is older than 3.9, say so and suggest the simplest fix. 2. Read [SITEMAP URL], handle both a normal sitemap and a sitemap index with child sitemaps, and handle gzip compressed sitemaps. 3. Load my robots.txt with urllib.robotparser and flag any sitemap URL that is disallowed for a general user agent. 4. Request each URL politely: one at a time, a short configurable delay, a timeout, and a clear user agent string I can edit. Never crawl links beyond the sitemap. 5. For each URL, record the final status code, whether it redirected and where, and whether the page has a meta robots noindex tag or an X-Robots-Tag noindex header. 6. Flag mismatches: 4xx or 5xx pages, redirects, noindex pages, and blocked pages that are still listed in the sitemap, with a short plain reason for each. 7. Handle errors cleanly: timeouts, SSL errors, invalid XML, and empty sitemaps should be reported, not crash the script. 8. Write the results to [OUTPUT], and add short comments plus a README section on how to run it and how to double check flagged pages in Google Search Console URL Inspection. 9. Remind me to run it only on sites I own or have permission to test, and to keep the request rate low so I do not slow down my own server. Deliverables: 1. Install and run commands 2. Full script with comments 3. Sample output 4. README notes 5. How to fix the most common issues it finds
How to use this prompt
- 1Copy the prompt using the button above.
- 2Open your AI tool (ChatGPT, Claude, Midjourney, etc.)
- 3Paste and customize any variables in the prompt.
- 4Generate your result.
Tip
Fill every placeholder before you run the prompt. Keep one clear goal per run, then refine with a follow-up if needed.
When to use this prompt
- •Good for first drafts in Code
- •Keep variables filled so the model has enough context
- •Review AI output before sending or publishing