Netlify

An Edge Function that handles bots before your Netlify site is served.

Markdown

Netlify Edge Functions run before each page is served. Jazui's asks Jazui for pages for bots and, for everything else, calls context.next() so Netlify serves your site as usual.

Before you start

  • A site on Netlify.
  • Your site verified in Jazui and your token.

Installation

Create the folder netlify/edge-functions/jazui/ and put these two files in it. The first one must be named after the folder, jazui.js, so Netlify treats it as the function and core.js as its module:

Terminal
mi-sitio/
└── netlify/
    └── edge-functions/
        └── jazui/
            ├── jazui.js   ← netlify-edge.js
            └── core.js
netlify/edge-functions/jazui/jazui.jsDownload
// Edge Function de Netlify: copia este archivo y core.js en netlify/edge-functions/ y define JAZUI_TOKEN en las
// variables de entorno del sitio.
import { renderRequest } from './core.js';

export default async (request, context) => {
  const rendered = await renderRequest(request, {
    token: Netlify.env.get('JAZUI_TOKEN'),
    api: Netlify.env.get('JAZUI_API'),
    siteUrl: Netlify.env.get('JAZUI_SITE_URL'),
    waitUntil: context.waitUntil?.bind(context),
  }, context.ip);
  return rendered ?? context.next();
};

export const config = { path: '/*', excludedPath: ['/.netlify/*', '/api/*'] };
netlify/edge-functions/jazui/core.jsDownload
// Núcleo común de las integraciones JavaScript de Jazui (Cloudflare Workers, Netlify, Vercel y Express).
// Solo usa la API Fetch estándar: funciona en Node 18+, Deno y los runtimes de borde.

export const DEFAULT_API = 'https://api.jazui.com';
export const DEFAULT_TIMEOUT_MS = 20000;

// Bots a los que se sirve el HTML renderizado (bots.txt; lo rellena Sync-Integrations.ps1)
export const BOTS = [
  // BOTS:BEGIN
  'googlebot', 'google-inspectiontool', 'storebot-google', 'adsbot-google', 'bingbot', 'bingpreview', 'msnbot',
  'adidxbot', 'yandex', 'baiduspider', 'duckduckbot', 'slurp', 'applebot', 'seznambot', 'qwantbot', 'qwantify',
  'petalbot', 'sogou', 'naver', 'exabot', 'yetibot', '360spider', 'soso', 'gptbot', 'chatgpt', 'oai-searchbot',
  'claudebot', 'claude-user', 'claude-searchbot', 'anthropic-ai', 'perplexity', 'google-cloudvertexbot',
  'bytespider', 'ccbot', 'amazonbot', 'meta-externalagent', 'cohere-ai', 'mistralai-user', 'youbot',
  'duckassistbot', 'yibot', 'qwenbot', 'tongyibot', 'erniebot', 'yiyanbot', 'doubaobot', 'tiktokspider',
  'deepseekbot', 'pangubot', 'manus-user', 'kimi-agent', 'kimi-searchbot', 'kimi-user', 'kimibot', 'moonshotbot',
  'hunyuan', 'chatglm-spider', 'agi agent', 'ai2bot', 'ai2bot-deepresearcheval', 'ai2bot-dolma', 'aihitbot',
  'amazon-kendra', 'amazon-qbusiness', 'amazonbuyforme', 'amzn-searchbot', 'amzn-user', 'bedrockbot', 'novaact',
  'andibot', 'claude-web', 'apifybot', 'apifywebsitecontentcrawler', 'atlassian-bot', 'awario', 'bigsur.ai',
  'brightbot', 'buddybot', 'cartai agentic commerce', 'terracotta', 'cledarabot', 'cloudflare-autorag',
  'cloudflarebrowserrenderingcrawler', 'cohere-training-data-crawler', 'cragcrawler', 'crawlspace',
  'datenbank crawler', 'diffbot', 'diffbot-user', 'anomura', 'echobot bot', 'echoboxbot', 'elevenlabsbot',
  'elevenlabscrawler', 'exasearchbot', 'exauser', 'factset_spyderbot', 'firecrawlagent', 'geisthaus-pagefetcher',
  'cloudvertexbot', 'gemini-deep-research', 'google-agent', 'google-firebase', 'google-notebooklm',
  'googleagent-mariner', 'googleagent-urlcontext', 'googleother', 'googleother-image', 'googleother-video',
  'notebooklm', 'iaskbot', 'iaskspider', 'imagesiftbot', 'img2dataset', 'isscyberriskcrawler', 'kagi-fetcher',
  'keenablebot', 'klaviyoaibot', 'laiondownloader', 'linguee bot', 'linkupbot', 'aiwebindex',
  'meta-externalfetcher', 'meta-webindexer', 'facebookbot', 'azureai-searchbot', 'mistralai-index',
  'mistralai-training', 'mozilla-tabstack', 'nagetbot', 'netestate imprint crawler', 'icc-crawler', 'oai-adsbot',
  'panscient', 'panscient.com', 'shapbot', 'payhawk invoice fetching agent', 'phindbot', 'poggio-citations',
  'poseidon research crawler', 'qualifiedbot', 'wpbot', 'querit-searchbot', 'queritbot', 'quillbot',
  'quillbot.com', 'reflectionbot', 'cotoyogi', 'sbintuitionsbot', 'scrybot', 'sidetrade indexer bot', 'tavilybot',
  'agenttimes', 'thinkbot', 'timpibot', 'twinagent', 'velenpublicwebcrawler', 'wardbot', 'omgili', 'omgilibot',
  'grok bot', 'grokappandroid', 'grokbot', 'addsearchbot', 'agentdatabot', 'aranet-searchbot', 'bixelbot',
  'bravebot', 'channel3bot', 'friendlycrawler', 'henkbot', 'imagespider', 'kangaroo bot', 'kunatocrawler',
  'laion-huggingface-processor', 'linerbot', 'mycentralaiscraperbot', 'terra cotta', 'wrtnbot', 'zanistabot',
  'facebookexternalhit', 'facebot', 'twitterbot', 'linkedinbot', 'slackbot', 'whatsapp', 'telegrambot',
  'discordbot', 'pinterest', 'redditbot', 'skypeuripreview', 'embedly', 'quora link preview', 'showyoubot',
  'outbrain', 'developers.google.com/+/web/snippet', 'vkshare', 'tumblr', 'bitlybot', 'nuzzel', 'flipboard',
  'snapchat', 'viber', 'mattermost', 'bitrix link preview', 'xing-contenttabreceiver', 'ahrefsbot', 'semrushbot',
  'mj12bot', 'majestic-12', 'dotbot', 'rogerbot', 'screaming frog', 'sitebulb', 'serpstatbot', 'seokicks',
  'siteimprove', 'lighthouse', 'google page speed', 'gtmetrix', 'pingdom', 'uptimerobot', 'uptimebot',
  'w3c_validator', 'w3c-checklink', 'archive.org_bot', 'paperlibot', 'dataprovider.com', 'grapeshotcrawler',
  'zoominfobot', 'feedly', 'newsblur', 'inoreader', 'integration-test',
  // BOTS:END
];

// Archivos que nunca se renderizan (extensions.txt)
export const STATIC_EXTENSIONS = [
  // EXTENSIONS:BEGIN
  'js', 'mjs', 'css', 'less', 'map', 'json', 'xml', 'rss', 'txt', 'webmanifest', 'ico', 'png', 'jpg', 'jpeg',
  'gif', 'webp', 'avif', 'svg', 'bmp', 'tif', 'tiff', 'psd', 'ai', 'woff', 'woff2', 'ttf', 'otf', 'eot', 'mp3',
  'm4a', 'wav', 'ogg', 'mp4', 'm4v', 'webm', 'avi', 'mov', 'mpg', 'mpeg', 'wmv', 'flv', 'swf', 'pdf', 'doc',
  'docx', 'xls', 'xlsx', 'ppt', 'pptx', 'csv', 'zip', 'gz', 'rar', '7z', 'iso', 'torrent', 'exe', 'dmg', 'apk',
  'wasm', 'dll', 'dat', 'blat',
  // EXTENSIONS:END
];

const escapeRegex = (text) => text.replace(/[.*+?^${}()|[\]\\/]/g, '\\$&');
// Patrón de la lista incluida; refreshBots lo sustituye por el del catálogo de Jazui (GET /v2/bots)
export const BOT_PATTERN = new RegExp(BOTS.map(escapeRegex).join('|'), 'i');
export const STATIC_PATTERN = new RegExp(`\\.(${STATIC_EXTENSIONS.join('|')})$`, 'i');

let botPattern = BOT_PATTERN;
let botsVersion = 'incluida';
let botsCheckedAt = 0;
let botsLoading = null;
const BOTS_REFRESH_MS = 24 * 60 * 60 * 1000;
const BOTS_RETRY_MS = 60 * 60 * 1000;

/** Patrón de bots en uso: el del catálogo si ya se descargó, si no el de la lista incluida. */
export const currentBotPattern = () => botPattern;

/** Versión de la lista en uso ('incluida' hasta la primera descarga). */
export const currentBotsVersion = () => botsVersion;

/**
 * Descarga la lista de bots del catálogo de Jazui (GET /v2/bots) si toca: al arrancar y cada 24 h; si falla se
 * sigue con la que hay y se reintenta en una hora. Nunca falla ni retrasa la petición: devuelve la promesa para
 * waitUntil (Cloudflare, Vercel, Netlify) o null si no toca. Se desactiva con remoteBots: false.
 */
export function refreshBots({ api = DEFAULT_API, remoteBots = true } = {}) {
  const now = Date.now();
  if (!remoteBots || botsLoading || now - botsCheckedAt < BOTS_REFRESH_MS) return null;
  botsCheckedAt = now;
  botsLoading = fetch(`${(api || DEFAULT_API).replace(/\/+$/, '')}/v2/bots`, { signal: AbortSignal.timeout(5000) })
    .then((response) => (response.ok ? response.json() : null))
    .then((data) => {
      if (data && Array.isArray(data.bots) && data.bots.length > 0) {
        botPattern = new RegExp(data.bots.map(escapeRegex).join('|'), 'i');
        botsVersion = data.version || botsVersion;
      } else {
        botsCheckedAt = now - BOTS_REFRESH_MS + BOTS_RETRY_MS;
      }
    }, () => { botsCheckedAt = now - BOTS_REFRESH_MS + BOTS_RETRY_MS; })
    .finally(() => { botsLoading = null; });
  return botsLoading;
}

// Cabeceras de Jazui que se devuelven al bot
const PASS_HEADERS = ['content-type', 'location', 'etag', 'last-modified', 'x-jazui', 'x-jazui-cache'];

/**
 * Si la petición hay que pasarla por Jazui: GET/HEAD de un bot conocido (o con _escaped_fragment_, el esquema
 * antiguo de Google para las SPA) a una página, no a un archivo.
 */
export function shouldRender({ method, url, userAgent, headers }) {
  if (method !== 'GET' && method !== 'HEAD') return false;
  if (headers && headers.get('x-jazui')) return false; // nunca a sí mismo
  const { pathname, search } = new URL(url);
  if (STATIC_PATTERN.test(pathname)) return false;
  return (!!userAgent && botPattern.test(userAgent)) || search.includes('_escaped_fragment_');
}

/**
 * Pide a Jazui el HTML de la URL. Devuelve la Response para el bot o null si hay que servir la página normal
 * (Jazui no respondió, se pasó del límite, el sitio no está verificado…): el bot nunca ve un error de Jazui.
 * Solo un 403 marcado con X-Jazui-Blocked (regla o bot de IA bloqueado) llega al bot.
 */
export async function fetchRendered({ url, method = 'GET', userAgent, clientIp, ifNoneMatch, token, api = DEFAULT_API,
                                      siteUrl, timeoutMs = DEFAULT_TIMEOUT_MS }) {
  if (!token) return null;
  let target = url;
  if (siteUrl) {
    // Dominio público del sitio cuando el servidor lo ve con otro (proxy, entorno de pruebas)
    const original = new URL(url);
    target = new URL(original.pathname + original.search, siteUrl).toString();
  }
  const query = new URLSearchParams({ url: target, botAgent: userAgent || '' });
  const headers = { Authorization: `Bearer ${token}` };
  if (clientIp) headers['X-Jazui-Client-IP'] = clientIp;
  if (ifNoneMatch) headers['If-None-Match'] = ifNoneMatch;

  let response;
  try {
    response = await fetch(`${(api || DEFAULT_API).replace(/\/+$/, '')}/Render?${query}`, {
      headers,
      redirect: 'manual',
      signal: AbortSignal.timeout(timeoutMs),
    });
  } catch {
    return null;
  }

  const blocked = response.status === 403 && response.headers.get('x-jazui-blocked');
  if (!response.headers.get('x-jazui') && !blocked) {
    response.body?.cancel();
    return null;
  }
  const out = new Headers();
  for (const name of PASS_HEADERS) {
    const value = response.headers.get(name);
    if (value) out.set(name, value);
  }
  if (blocked) out.set('x-jazui-blocked', blocked);
  const empty = method === 'HEAD' || response.status === 304;
  if (empty) response.body?.cancel();
  return new Response(empty ? null : response.body, { status: response.status, headers: out });
}

// User-agents con pinta de bot: si no están en BOTS se avisa a Jazui sin enviarle la página (el mismo patrón que la API)
export const UNKNOWN_BOT_PATTERN = /bot|crawl|spider|scrap|fetch|headless|preview|archiv|index/i;
const MAX_REPORTED = 1000;
const reported = new Set();
let reportedDay = '';

/**
 * Avisa a Jazui (POST /Render/Seen) de un user-agent con pinta de bot que no está en la lista, una vez al día por
 * user-agent y proceso, para que se pueda añadir al catálogo. No gasta renders y nunca falla: devuelve la promesa para
 * waitUntil (Cloudflare, Vercel) o null si no hay que avisar.
 */
export function reportUnknownBot({ method, url, userAgent, token, api = DEFAULT_API, reportUnknownBots = true }) {
  if (!reportUnknownBots || !token || !userAgent || (method !== 'GET' && method !== 'HEAD')) return null;
  if (STATIC_PATTERN.test(new URL(url).pathname)) return null;
  if (!UNKNOWN_BOT_PATTERN.test(userAgent) || botPattern.test(userAgent)) return null;
  const today = new Date().toISOString().slice(0, 10);
  if (reportedDay !== today) {
    reported.clear();
    reportedDay = today;
  }
  if (reported.size >= MAX_REPORTED || reported.has(userAgent)) return null;
  reported.add(userAgent);
  return fetch(`${(api || DEFAULT_API).replace(/\/+$/, '')}/Render/Seen`, {
    method: 'POST',
    headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ userAgent, url }),
    signal: AbortSignal.timeout(5000),
  }).then((response) => { response.body?.cancel(); }, () => {});
}

/**
 * Atajo para los runtimes con Request estándar: la Response de Jazui o null. Con options.waitUntil (Cloudflare,
 * Vercel) la descarga de la lista de bots y el aviso de bots desconocidos siguen aunque la respuesta ya se haya enviado.
 */
export function renderRequest(request, options, clientIp) {
  const refresh = refreshBots(options);
  if (refresh && options.waitUntil) options.waitUntil(refresh);
  const userAgent = request.headers.get('user-agent') || '';
  if (!shouldRender({ method: request.method, url: request.url, userAgent, headers: request.headers })) {
    const report = reportUnknownBot({ ...options, method: request.method, url: request.url, userAgent });
    if (report && options.waitUntil) options.waitUntil(report);
    return null;
  }
  return fetchRendered({
    ...options,
    url: request.url,
    method: request.method,
    userAgent,
    clientIp,
    ifNoneMatch: request.headers.get('if-none-match'),
  });
}

Environment variables

VariableRequiredWhat for
JAZUI_TOKENYesYour account or site token. Store it as a secret, never in the code.
JAZUI_APINoAPI address. Defaults to https://api.jazui.com.
JAZUI_SITE_URLNoYour public domain (for example https://mysite.com) if the platform sees another one, such as a preview address: Jazui renders the page with that domain.

Without JAZUI_TOKEN the integration does nothing and the whole website is served normally.

Terminal
netlify env:set JAZUI_TOKEN "tu-token"
netlify deploy --prod

What it does

The function applies to every path except /.netlify/* and /api/* (set in its config; you can add more exclusions in excludedPath). Netlify provides the bot's IP in context.ip, and the bot list and reports run in the background with waitUntil.

<p>Request your home page as if you were a bot, with the test user agent <code>integration-test</code>:</p><pre><code>curl -I -A "integration-test" https://mysite.com/</code></pre><p>The response must include <code>x-jazui: true</code>, with <code>x-jazui-cache: MISS</code> the first time and <code>HIT</code> afterwards. Without a bot user agent (<code>curl -I https://mysite.com/</code>) you should get your usual website, without <code>x-jazui</code>. In the dashboard, the request shows up in <b>Bot traffic</b> and the page in <b>Cached pages</b>.</p>

Request your home page as if you were a bot, with the test user agent integration-test:

curl -I -A "integration-test" https://mysite.com/

The response must include x-jazui: true, with x-jazui-cache: MISS the first time and HIT afterwards. Without a bot user agent (curl -I https://mysite.com/) you should get your usual website, without x-jazui. In the dashboard, the request shows up in Bot traffic and the page in Cached pages.

If something doesn't work

  • No x-jazui: the deployment log should list the jazui Edge Function. If not, check the folder and the file name.
  • The variable doesn't arrive: set it under Site configuration › Environment variables with the functions scope and redeploy.
  • Preview addresses (*.netlify.app): set JAZUI_SITE_URL to your verified domain.