How we turned ScrapingBee's Google API into an AI glossary powerhouse.

One of the biggest challenges for our client was scaling up SEO with data-backed content insights, so they could capitalise on every missing opportunity across the site.

Most teams waste hundreds of hours researching competitors, finding content gaps, and analysing content structures, and still end up with mediocre content.

We built a better way.

AI Agency Lab built a fully automated content engine that took every single glossary page, matched it against competitors, ranked it by performance, and fed it into an SEO framework to re-write every page in minutes.

This saved our client a huge amount of time on research and content creation, and gave them a fully optimised content pipeline in a few hours.

SERP research

4 modules

1. TriggerGlossary URLs pulled
2. ScrapeMain keyword found
3. ScSearch APITop SERP results
4. Headers4–5 H2s per page

Briefs & content

4 modules

1. Deep searchCompetitor analysis
2. SEO briefGuidelines & checklists
3. Re-writeDraft into Google Docs
4. PipelineReady for HITL review

The step by step approach

Reverse engineered SERP data

This simple but powerful automation extracts hundreds of short tail keywords across every glossary page, hunts down the top competitors in Google SERPs, and builds data-backed briefs and checklists, then re-writes the pages within a few minutes.

1. 1

   The trigger

   The automation starts with a trigger from a Google Sheet, which pulls through the URLs to feed into the downstream workflow.

2. 2

   The scrape

   As soon as the automation is triggered, we extract the content from every page, highlight the main keyword, and feed this data into GPT to output the exact term for every page.

3. 3

   Search API

   We used ScrapingBee to take the keyword from every glossary page and extract the top positions as SERP data to feed into our SEO briefs.

4. 4

   ChatGPT

   Based on the data from the top articles, we fed competitor content into ChatGPT with a comprehensive brief on H2 header writing, to spin up 4–5 headers for every page.

Building out SEO briefs & glossary content

This workflow takes you from manually turning data insights into SEO briefs, through to drafting next level re-writes for your copywriter to deploy.

1. 1

   Perplexity

   This part of the automation uses the SERP data and hunts down specific glossary pages with Perplexity’s deep search engine, analysing market share across 2–3 competitors, Wikipedia results, and other data sets.

2. 2

   ChatGPT

   ChatGPT is incredibly good at logic tasks, so we used it to build an SEO brief from custom guidelines, SEO checklists and a range of data sets, ready to hand over to Claude. This covered word count, H2 optimisation, contextual paragraphs, tone guidelines, competitor examples, and best practice for glossary pages.

3. 3

   Claude

   We used the SEO brief from GPT and custom prompts for Claude to re-write each glossary article into a Google Doc. This went through rigorous testing and review, so every page was 80% ready for a copywriter to humanise before upload.

4. 4

   Google Sheets

   Once the automation has finished for every page, all of the articles feed into the content pipeline for review. This keeps a human in the loop (HITL), so every page is fact checked, QA’d for originality, and signed off by a member of the team.

Grab the full install guide with step by step instructions.