Ariadne

Overview

Ariadne is a polite web crawler used only to discover new publicly available pages. It is designed to be “friendly” to websites and operators.

Purpose

Ariadne is used only for link discovery. It does not:

• Use website content for anything related to AI
• Train machine learning models or support AI systems
• Profile users or collect personal data
• Reuse or republish your page content

Where possible, Ariadne keeps what it retrieves to the minimum required to discover links; it is not intended for content analysis.

“Friendly” crawling behavior

• One request at a time per domain (no parallel requests to the same site)
• No intentional traffic spikes
• Respects standard HTTP status codes and caching directives
• Respects robots.txt

User-Agent string

Ariadne identifies itself with a clear User-Agent header. You can set the entire User-Agent string to a value you control.

Mozilla/5.0 (compatible; Ariadne/0.1; +http://stc.onl/ariadne) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36

Robots.txt compliance

Ariadne follows the Robots Exclusion Protocol and respects rules for both: User-agent: Ariadne and User-agent: *.

How to disable Ariadne

To block only Ariadne, add the following to your robots.txt:

User-agent: Ariadne
Disallow: /

To block all crawlers (including Ariadne):

User-agent: *
Disallow: /

Ariadne will not crawl pages that are disallowed by either User-agent: Ariadne rules or User-agent: * rules.

Transparency

Ariadne does not attempt to disguise itself. If you see traffic attributed to Ariadne, it will be labeled accordingly via its User-Agent.