Google Scraping AI Agent

Overview

This document outlines the process of building a Google scraping AI agent using n8n, OpenAI API, and Google Sheets. The agent is designed to search Google for specific LinkedIn profile URLs (e.g., CEOs in real estate in Chicago) and populate these URLs into a Google Sheet. The solution involves two main n8n workflows: a "tool" workflow that handles the Google scraping and data parsing, and an "agent" workflow that interacts with the user and calls the "tool" as needed.

Key Concepts

1. Automated Data Scraping for Targeted Information

The core functionality of this system is to automate the extraction of highly specific data (LinkedIn profiles) from Google search results. This is achieved by crafting precise Google search queries that filter results to only include LinkedIn profiles (site:linkedin.com/in).

• "The agent... will go search Google and scrape Google to get those profiles into a sheet like this."

• The agent successfully finds "CEOs in real estate in Chicago" and "Founders in technology in San Francisco" from LinkedIn.

2. Leveraging n8n for Workflow Automation

n8n serves as the central platform for building and orchestrating the entire process. It provides the visual workflow builder, pre-built nodes for integrations (OpenAI, Google Sheets, HTTP requests), and the ability to define custom tools and agents.

• "n8n that is the website that we're building the workflows on for the tool and for the agent."

• "We're building two workflows today the first one is going to be the tool that scrapes Google for the URLs and then the second one is the actual agent that we can interact with."

3. AI Integration (OpenAI) for Intelligent Processing and Interaction

OpenAI's API is crucial for two main purposes:

• Parsing User Queries: An OpenAI node in the "tool" workflow parses natural language requests from the user (e.g., "CEOs in real estate in Chicago") into structured parameters (job title, company, industry, location) for the Google search.
• Agent Intelligence: An OpenAI chat model (GPT-4o recommended) acts as the "brain" of the AI agent, allowing it to understand user requests, decide which tools to use, and formulate appropriate responses.

• "We need to connect an open AI node... can you parse the Json query which is this information over here and then output it as the following parameters separately."

• "We need to give him the brain which is a chat model which we always use open AI chat model."

4. Tool-Based AI Agent Architecture

The system employs a "tools agent" model in n8n. The main AI agent doesn't directly perform the scraping but rather calls a separate workflow (the "grab profiles" tool) designed specifically for that task. This modular approach enhances maintainability and scalability.

• "It's going to be a tools agent because it's going to be accessing the tools that we give it."

• The tool has a

  "description of when to be called: call this tool to get um yeah LinkedIn profiles."

5. Handling Complex Web Data (HTML Parsing)

Google search results are returned as a large chunk of HTML. A key challenge is to extract only the desired LinkedIn URLs from this raw HTML. This is solved using a "code node" in n8n, with the actual parsing logic generated by an AI assistant.

• "This is just a ton of HTML huge chunk of nonsense that no one you know can really interpret."

• "The next node is going to be a code node and this is going to parse the information that we're getting this nasty chunk of HTML and it's going to parse it to only return what we want which are the LinkedIn profile URLs."

6. Utilizing AI Assistants for Code Generation and Troubleshooting

A significant innovation highlighted is the use of an n8n assistant GPT (a custom ChatGPT model) to generate the necessary code for parsing HTML and setting up HTTP request headers. This greatly simplifies development for users without extensive coding knowledge.

• "I came into chat gbt here and on the left hand side you've got explore gbt... I just typed in NN and it brought me to um basically an n8n assistant GPT."

• "I basically asked like can you help me set up header for HTTP request node... and it's really helpful."

• "Can you write code to parse out the LinkedIn information we're looking for here is the here is an example chunk of HTML being returned."

7. Google Sheets as a Database

Google Sheets serves as the final destination for the scraped data, providing a simple, accessible database for the LinkedIn profile URLs.

• "Google Sheets... that's like I said the database where we're storing the URLs into."

• "Now we need to get these URLs into our Google sheet so we're going to do that with a Google Sheets node."

8. Memory and Context for Conversational Agents

The AI agent incorporates "window buffer memory" to retain context from previous interactions, enabling more natural and coherent conversations with the user.

• "This memory is just giving the agent context for what we're talking about so it's not just like a question and answer and then his brain is resetting."

9. Limitations and Potential Solutions (Google Search API)

The current method of directly scraping Google via HTTP requests has limitations, particularly regarding the number of results (typically around 10) and potential issues with CAPTCHAs from Google. The speaker suggests that a dedicated Google Search API (like SerAPI) would be necessary for more extensive scraping or pagination beyond the first page.

• "If you guys are wondering sort of why we're only returning 10 responses and if you wanted to do more how could that work so my understanding is that we would need to be using a a an actual Google search API like Sur API in order to do that."

• "Google's may be providing a capcha to to find out if it's a human or if it's really a robot scraping information."

Core Components and Setup Requirements

• n8n: The primary platform for building workflows.
• OpenAI API Key: Required for message parsing and AI agent intelligence.
• Google Sheets: To store the scraped LinkedIn profile URLs.
• Google Cloud Service Account (for Google Sheets): Needed for n8n to authenticate and write to Google Sheets (requires OAuth consent screen setup).

Workflow Breakdown

Workflow 1: "Grab Profiles" (The Tool)

• When Called by Another Workflow Trigger: Initiates the workflow when the AI agent calls it, passing parameters like job title, company, industry, and location.
• OpenAI Node (Message a Model):
  • System Prompt: Instructs the model to parse the incoming query (JSON format) and extract job title, company, industry, and location as separate parameters.
  • User Message: Provides the actual query data to be parsed.
  • Output: Ensures the parsed content is a string using json.stringify().
• HTTP Request Node:
  • Method: GET (to retrieve information).
  • URL: Simple Google search URL.
  • Parameters:q: Constructs the search query by combining site:linkedin.com/in with the parsed job title, industry, and location.
  • Headers: User-Agent (set to a browser-like string to avoid being blocked by Google). This information was obtained using the n8n assistant GPT.
• Code Node:
  • Parses the raw HTML response from the HTTP request node.
  • Extracts only the LinkedIn profile URLs. The code for this step was generated and refined with the help of the n8n assistant GPT.
• Google Sheets Node (Append Row):
  • Connects to a pre-existing Google Sheet.
  • Appends the extracted LinkedIn URLs to a specified column (e.g., "Prospect LinkedIn URL"). Requires Google Cloud service account setup for authentication.
• Set Node:
  • Sets a field named response with the value done. This signals to the calling AI agent that the tool's task is complete, allowing the agent to respond to the user.

Workflow 2: "AI Agent" (The Conversational Interface)

• Chat Message Trigger: Initiates the workflow when a user sends a message to the agent.
• AI Agent Node (Tools Agent):
  • Prompt: Takes input from the previous node (user's chat message). Can include a system message (e.g., "helpful assistant").
  • Chat Model (Brain): Connected to an OpenAI chat model (GPT-4o).
  • Memory: Configured with "Window Buffer Memory" to maintain conversational context.
  • Tools:
    • Workflow Tool: References the "Grab Profiles" workflow (the tool).
    • Tool Description:

      "Call this tool to get LinkedIn profiles"

      – guides the agent on when to invoke this workflow.
    • Response Field: Set to response to receive the done signal from the tool.

Demo and Validation

The agent was successfully tested with various queries:

• "CEOs in real estate in Chicago"
• "Founders in technology in San Francisco"

The system accurately identified and populated the correct LinkedIn profiles into the Google Sheet, demonstrating its dynamic functionality beyond hard-coded variables. Execution logs in n8n confirm the flow of data and the parsing steps.