Web Toolset Documentation

Generated byAI

This toolset provides tools for making HTTP GET requests to retrieve web content.

Tool settings: host policies New in 0.9.14

Both tools support host policies that control which servers they may be used to access. Configure them per tool under tool_settings::

allowed_hosts: an array of host names that are allowed; when unset (default) all hosts are allowed.
denied_hosts: an array of host names to refuse to connect to; when empty (default) no host is denied.
allow_private_hosts: set to true to allow access to private/local hosts; the default is false.

tool_settings:
  http_get_web_page_as_markdown:
    denied_hosts: ["google.com"]
  http_get_web_page:
    allowed_hosts: ["google.com"]

`http_get_web_page`

Makes an HTTP GET request to a given URL that returns text and returns the content type and content. It strips line indents and coalesces line breaks for HTML, JSON, and XML. Unless asked not to, it removes the head, script, and style elements as well as all class attributes from HTML content. Content size of response must be less than 256K.

Parameters

Parameter	Type	Required	Description
`url`	String	Yes	The URL to GET the web page from.
`user_agent`	String	No	Optional user agent header; default is Firefox for macOS.
`accept`	String	No	Optionally specify an acceptable content type (e.g., `text/markdown`) to ask the server for a specific format. Default is none.
`preserve_source`	Boolean	No	Optional flag to ask to preserve the content from the server without any attempts to condense the text; default is false.

Notes

Strips line indents and coalesces line breaks for HTML, JSON, and XML.
Removes head, script, and style elements from HTML content unless preserve_source is set to true.
Content size must be less than 32K.

`http_get_web_page_as_markdown`

Makes an HTTP GET request to a given web site URL that returns the content as markdown, either because the website returns markdown directly, or by parsing the incoming HTML and converting it to markdown. If the text content from the web site is neither HTML nor markdown, the tool returns a markdown document with the web page content within a code block.

Parameters

Parameter	Type	Required	Description
`url`	String	Yes	The URL to GET the web page from.
`user_agent`	String	No	Optional user agent header; default is Firefox for macOS.

Notes

Returns content as markdown when the website supports it.
Converts HTML to markdown by properly parsing the HTML.
When the resulting content is too big (>32K), the markdown is truncated; the tool returns the URL plus a truncated: true | false property alongside the (possibly truncated) markdown New in 0.9.14.
Content that is neither HTML nor markdown is wrapped in a code block with appropriate content type tagging where possible.