ANALYSIS OF A REAL MISTAKE

How We Built an Autonomous AI Article Publication Pipeline

We asked an AI agent to prepare a series of articles. The texts were produced, but a review revealed that one article mentioned NotebookLM without linking to the glossary. Then we discovered another problem: llms.txt had not been updated after the materials were rewritten. These were not minor typos. The publication process had failed.

Why the agent made a mistake

The agent completed the local task—it wrote the HTML files—but did not check the entire pipeline. The rules existed in the context, but they had not been turned into mandatory automated checks. As a result, the system could say “done” even though the catalog, the map for LLMs, and the glossary were out of sync.

First solution: editorial rules

We established a mandatory structure: a real-world problem at the beginning, a specific outcome, difficulty level, reading time, step-by-step actions, errors, verification, and limitations. We also added a separate rule: the first technical term must link to the lab glossary.

This prevents an article from becoming a collection of generic tips. The reader understands what is broken, follows the steps, and verifies the result.

Second solution: a unified content map

Each article has several representations: an HTML page for people, an entry in guides.html, a URL in sitemap.xml, and a brief entry in llms.txt. The last file helps AI systems quickly understand the journal’s structure and choose the appropriate material.

new article
  ├── guide.html
  ├── guides.html
  ├── sitemap.xml
  ├── llms.txt
  └── glossary.js

If an article’s title or meaning changes, all related representations must be updated. A single link in Git is not enough.

Third solution: an automated publication gate

We added the scripts/check-publication.py script. It scans every guide-*.html file and checks for the required elements:

• description and canonical URL;
• level, time, and expected outcome;
• a link to the glossary;
• the article’s presence in the catalog;
• the URL’s presence in the sitemap;
• the URL’s presence in llms.txt.

cd agentlabjournal
python3 scripts/check-publication.py

PUBLICATION_GATE: OK

If a link is missing, the script returns BLOCKED and identifies the specific file. A polished response from the model cannot conceal that result.

Why this is autonomy

Autonomy does not mean “the agent does whatever it wants without oversight.” It means a system that performs permitted work on its own and stops itself when a mandatory condition has not been met. A person decides the objective, while the machine checks repeatable rules.

Conclusion

One mistake led us to build a reproducible pipeline: rule → artifact → automated check → publication. The same approach can be applied to code, R&D experiments, email, and Telegram agents.

← All guides · Lab glossary →