The Technical SEO Checklist: robots.txt, Sitemaps, Canonicals and Schema

Technical SEO is unglamorous and disproportionately valuable. It does not involve guessing what search engines want. It involves making sure the pages you have built can be found, read and understood — and then getting out of the way.

Most sites lose traffic to a handful of specific, checkable faults. This is a checklist for finding them.

1. robots.txt — the file most likely to be quietly costing you

robots.txt tells crawlers which paths they may request. It is one small file, it sits at your root, and a single wrong line can remove an entire section of your site from search results.

The failure that causes the most damage is a Disallow: / left behind from a staging environment. It does exactly what it says — blocks everything — and because nothing on the site looks broken, it can persist for months.

Two misunderstandings are worth correcting:

• robots.txt is not a security measure. It is a request, honoured by well-behaved crawlers and ignored by everyone else. Listing /admin/ there tells the whole world where your admin panel is. Protect private paths with authentication.
• Disallow does not reliably remove a page from search results. Blocking a URL prevents crawling, but if other pages link to it, it can still appear as a bare listing. To keep a page out of an index, allow the crawl and use a noindex meta tag — a blocked page cannot be crawled, so the noindex is never seen.

Use Robots.txt Generator to produce a correct file, and read your live one carefully before assuming it is fine.

2. XML sitemaps

A sitemap lists the URLs you want crawled, which is particularly valuable for large sites, new sites with few inbound links, and pages buried deep in a navigation hierarchy.

Rules that keep it useful:

• Include only canonical, indexable URLs that return 200. A sitemap full of redirects, 404s and noindex pages is a quality signal working against you.
• Keep lastmod honest. Stamping every URL with today's date on every build makes the field worthless.
• Split at 50,000 URLs or 50 MB uncompressed, using a sitemap index.
• Reference it from robots.txt with a Sitemap: line, and submit it in Search Console.

Sitemap Generator will build one for you.

3. Canonical tags

The same content is often reachable at several URLs — with and without www, over HTTP and HTTPS, with tracking parameters, with a trailing slash, through pagination or filters. To a search engine these are different URLs with identical content, which splits ranking signals across duplicates.

A canonical tag names the preferred version. Get these right:

• Use absolute URLs, including the protocol.
• Every page should have a canonical, including the one pointing at itself.
• Never point every page at your homepage. This is a common and damaging error — it tells search engines your interior pages are duplicates of the homepage and should not rank at all.
• Keep it consistent with your redirects. If you redirect non-www to www, canonicals must use www too.

Canonical Checker reports what a page is actually declaring, which is sometimes not what the template was supposed to output.

4. Redirects

Redirects preserve the value of links pointing at URLs that have moved. Handled badly, they leak it.

• Use 301 for permanent moves and 302 only when the move genuinely is temporary.
• Avoid chains. A redirecting to B redirecting to C redirecting to D wastes crawl budget and slows users. Point A straight at D.
• Redirect to the equivalent page, not to the homepage. Bulk-redirecting a retired section to the root is treated as a soft 404 and helps nobody.
• Never redirect to a URL that redirects back. Loops are more common than you would think, usually from a trailing-slash rule fighting a protocol rule.

Redirect Checker follows the full chain and shows every hop and status code.

5. Broken links

Broken internal links waste crawl budget and frustrate visitors. Broken outbound links are a slower-burning problem — sites disappear, URLs get restructured, and an article written three years ago gradually fills with dead references.

Broken Link Checker will crawl and report them. Prioritise internal links and links on your highest-traffic pages.

6. Titles and meta descriptions

The title tag remains one of the strongest on-page signals, and the meta description, while not a ranking factor, heavily influences whether anyone clicks.

• Titles: roughly 50–60 characters before truncation. Front-load the distinctive part — if every title starts with your brand name, every result looks the same in a list.
• Descriptions: roughly 150–160 characters. Write them as ad copy, not summaries. The goal is the click.
• Never leave them duplicated across pages. Identical titles on fifty pages tell a search engine those pages are interchangeable.
• Never leave them empty. An empty description means the search engine writes one for you from whatever text it finds first.

Use SERP Preview to see how a result will actually render and where it truncates, and Meta Tag Generator to produce a complete tag set.

7. Structured data

Schema markup describes what a page is in a machine-readable way — an article, a product, a recipe, an FAQ, an organisation. It is what makes rich results possible: star ratings, prices, FAQ dropdowns, breadcrumbs.

Points that matter:

• Use JSON-LD. It is the recommended format and keeps markup separate from your HTML.
• Mark up what is actually visible on the page. Describing content the user cannot see is a guideline violation.
• Fill the recommended properties, not just the required ones. Sparse markup is often ignored.
• Valid markup makes a rich result possible, never guaranteed.

Schema Generator builds the markup and Schema Markup Validator checks it for errors before it ships.

8. Social preview tags

Open Graph and Twitter Card tags control how a link renders when shared. Without them, platforms guess — and usually guess badly, picking an arbitrary image and the first text on the page.

The minimum worth having on every page: og:title, og:description, og:image, og:url, og:type and twitter:card. Use a dedicated image around 1200×630; anything smaller may be cropped or ignored entirely.

OG Preview shows how the link will appear before you publish it.

9. HTTPS, and doing it properly

HTTPS is a baseline expectation. The details still catch people out:

• Check expiry and the full certificate chain. An incomplete chain works in some browsers and fails in others, which makes it hard to diagnose from a single machine. SSL Checker reports both.
• Redirect HTTP to HTTPS once, directly — not through an intermediate hop.
• Eliminate mixed content. One image loaded over HTTP on an HTTPS page removes the security indicator.
• Set up renewal reminders. Expired certificates are almost always a forgotten renewal, and the damage is immediate.

10. Speed and Core Web Vitals

Page experience signals measure loading, interactivity and visual stability. Two practical notes:

• Largest Contentful Paint is usually dominated by one image — typically the hero. Compressing it is frequently the entire fix.
• Cumulative Layout Shift is usually caused by images without width and height attributes, or by fonts and ads that reflow content after load. Reserve the space.

A sensible order to work in

If you only have an afternoon, do it in this sequence — it is roughly descending order of how much damage each fault causes:

1. Read your live robots.txt. Confirm you are not blocking anything you need indexed.
2. Confirm your sitemap exists, is referenced from robots.txt, and contains only live canonical URLs.
3. Spot-check canonicals on your five most important pages.
4. Find and fix redirect chains and loops.
5. Find duplicate or empty titles and descriptions.
6. Check the certificate and chain.
7. Fix broken internal links.
8. Add structured data and social tags to your top templates.

None of this is creative work, and that is the appeal — every item is verifiable, and a site that passes all ten is not leaving obvious traffic behind.

The robots.txt generator, sitemap generator, canonical checker, redirect checker, schema generator and SERP preview are all free on ZeeSharp.