Add LLMs.txt

[vinogradovkonst]

It is not really crucial, but easy to add and will likely make OSE the first ever endowment in the world with a proper interface for AI web agents and LLM crawlers.

[vladh]

Hmm, I'm not sure llms.txt is the best approach. For one thing, no major LLM provider supports it. But I also see some problems with the spec itself:

• It seems like it would be a better idea to put such a file not in the root, but in .well-known, which was made for this purpose
• I wonder why the extension is .txt when the file format is Markdown
• Implementing such a file would require us to duplicate some content from the site, which in the long term will have to be kept in sync

I think a better idea might be to try to generally use minimal HTML and to adhere to semantic markup. This will have the added benefit of improving our website for users in general, particularly by making our website more accessible.

[vinogradovkonst]

There are already many live examples, and we don't need to wait for all major LLM providers for this. Let the OSE be among the first to adopt this, alongside with top tech companies:

• https://docs.anthropic.com/llms-full.txt
• https://docs.perplexity.ai/llms-full.txt
• https://huggingface-projects-docs-llms-txt.hf.space/accelerate/llms.txt
• https://docs.zapier.com/llms-full.txt
• https://mastra.ai/llms.txt

Currently the startup world is crazy about AI web crawlers, and they are using this standard. Let's tell their LLMs about our endowment via llms.txt and llms-full.txt

[vladh]

Would you be able to suggest contents for an llms.txt and/or llms-full.txt file? The files you linked to seem to describe APIs, and this doesn't seem to apply to our website.

[vinogradovkonst]

It seems that now we could have only llms-full.txt with the homepage markdown content, and add llms.txt with index and summaries later, when our website will have many pages and docs.

Let's start with the homepage content, and I will add more factual data about the OSE later.

[vladh]

Sure, we can create an alternate version of our homepage that uses Markdown. This is similar to how RSS feeds create an alternate version of website content using XML. But we would need a generator for this. If we copy-paste the website content into a Markdown file, as soon as we change the HTML, the Markdown will become out of date, and this will give us a bunch of admin work to keep the two in sync.

I personally don't believe in this llms.txt approach, but what I can say is, if you want an alternate version of the homepage that uses Markdown, I can implement the redesigned version of the website in a way that automatically supports this.

[vinogradovkonst]

Let's just use any existing llms.txt generator automatically, for instance: https://github.com/mendableai/create-llmstxt-py

Also, please add OSE to https://github.com/thedaviddias/llms-txt-hub when it is implemented.

[vinogradovkonst]

https://github.com/supermemoryai/markdowner

[vladh]

I really don't think using an LLM generator is a good solution. For one thing, some, like create-llmstxt-py, will summarise our website. Do we really want to risk people getting false information about the Endowment because of a bad AI summary that we then publish on our website? Secondly, the tools just don't seem very good, for example markdowner is unable to find our team members. And anyway, this is a very complex solution to a simple problem.

If this can wait, I would recommend we wait until the new website implementation where we can build in support for Markdown as an alternative format.

If you think this is important, you could, for example, create an llms.txt file however you think is best, and I can publish that.

[vinogradovkonst]

This is not urgent, but important because LLM-based queries generate more and more traffic to websites, we should proactively address this.

It requires providing markdown-focused crawlers with the right content. I suppose such crawlers see our website like at https://r.jina.ai/endowment.dev, and that's not great.

Let's make sure that the next version of our website (EM) can be correctly read by AI in the markdown format.

[vladh]

I'm unassigning myself from this task since I wouldn't do a good job on it because I don't believe in it. I'll focus my efforts on more urgent things, and someone else can implement this after the redesign is implemented if they wish.