LLM based text -> destination checker

[carlsverre]

I recently switched to using Lychee to check links on https://sqlsync.dev and https://graft.rs. Really enjoying the tool! Works wonderfully well.

I'm curious if this community has thought of a LLM assisted link checker. I've been using ChatGPT's operator to sanity check that not only are links working - but they are also going to a reasonable location based on the link text. The rational is that sometimes when writing a blog post or documentation, you may accidentally use the wrong link reference. Hence, the link still "works" but it logically doesn't make sense.

Lychee could support this via outputting a bit of link context (the surrounding sentence + where the link, perhaps as markdown), the destination url (maybe embedded in markdown), and optionally a snippet of the destination page (perhaps some of the meta header tags). The user could then feed this into their favorite LLM or just read through it to sanity check links.

And if it becomes popular, Lychee could automate the above process via LLM api details. Although the user could probably do the same by just outputting to a file and then using a different tool to send it to their favorite LLM - which is safer in this world of ultra-fast-paced AI dev.

Cheers and thanks again for a great tool!

[mre]

The idea sounds enticing, but I would like to be upfront and say that this is not an area of focus for the tool. If anything, we might be able to support a different "context" output at some point, which is a machine-readable format for each link and the text around it.
Here's how an MVP version could look like:

{
  "links": [
    {
      "link_text": "official documentation",
      "destination_url": "https://example.com/docs",
      "surrounding_context": "For more information about this feature, check the [official documentation](https://example.com/docs) which explains the implementation details."
    },
    {
      "link_text": "troubleshooting guide",
      "destination_url": "https://example.com/help/faq",
      "surrounding_context": "If you encounter errors during installation, refer to our [troubleshooting guide](https://example.com/help/faq) for common solutions."
    }
  ]
}

We could discuss a format like this, but there's no bandwidth to work on this at the moment.
I would say your best bet would be to use the lychee-lib and write a small wrapper around it for your use-case. The hard part would be the link context, which would require changes in lychee-lib itself. At the moment, we don't store any link context, and doing so is not easy -- at least not if done efficiently. In the worst case, we'd go from O(n) parsing to O(n^2) as we'd have to backtrack from where we'd detect a link. If there is an existing library or more efficient algorithm for that, I'm not aware of it.

[carlsverre]

Thanks for the reply! Yea, I totally get that LLM isn't the are of focus for the tool. I think we're completely in alignment that at most this tool could simply assist in outputing the links + context and another tool could take it from there.

Good point regarding the efficiency of calculating link context. Assuming that the existing library can output spans of each discovered link, a reasonable first pass would be to simply expand the span before and after each link by a configurable window. LLMs can handle partial sentences well enough that finding the precise sentence barriers shouldn't be needed.

I don't have explicit time to work on this right now, but I'll keep it in mind for the future since it sounds like you'd be open to another output format. Cheers!