Skip to content
AI Primer
workflow

Practitioners turn agent feedback into reusable skill documents and lints

Practitioners describe converting feedback into persistent skill documents and generated lints. Daily transcript reviews and execution traces also inform changes to their agent harnesses.

4 min read
Practitioners turn agent feedback into reusable skill documents and lints
Practitioners turn agent feedback into reusable skill documents and lints

TL;DR

  • Skill feedback is becoming persistent context: moritzkremb’s workflow reloads a feedback document on every run.
  • Corrections become checklists, lints and CLI commands, with daily transcript reviews in ivanleomk’s setup.
  • Cross-project escalation queues filter agent activity down to items needing human attention, as shown in kevinkern’s demo.

Oxlint’s plugin API accepts custom JavaScript plugins and remains in alpha. One developer’s failure log includes an agent testing a report-protection guard against the real report, then overwriting it.

feedback-learnings.md

moritzkremb keeps a feedback file alongside SKILL.md and has the skill read it every time it runs. Corrections are added when he gives feedback on an output.

He treats it as memory for that workflow and keeps it separate from core rules so accumulated feedback does not overwrite them.

Generated lints

ivanleomk describes three parts to his setup:

  1. A low-friction way to give the agent feedback.
  2. Feedback codified into checklists, lints or CLI commands.
  3. A daily job that reviews transcripts and suggests improvements.

He also shared an example of the Oxlint rules being created. Corrections get teeth through Oxlint’s plugin configuration: jsPlugins loads a local plugin, while rules enables its individual checks.

Daily transcript review

In a follow-up, ivanleomk listed four kinds of changes his setup produces:

  • Suggestions for better prompting.
  • Antigravity rules and hooks.
  • Corrections to skills that are not working.
  • New or upgraded CLIs.

Execution traces also inform deeper changes: omarsar0 describes using traces to adjust his custom harness, including its system prompt.

Persistent instructions still leave room for repeat failures. Debashish Ghosal reports recurring violations across five weeks despite session-start rule loading, then describes moving enforcement into gates that check commands before execution.

Approval queues and actor ownership

kevinkern estimates that 80% of agent traces are useless to him and demonstrates a cross-machine queue for items needing his attention.

His approval policy changes as a project develops:

  • He defines the architecture, schemas, technology stack and boundaries at the start.
  • All merges and deploys require his approval.
  • After several days, he reviews earlier decisions and gives the orchestrator more permissions.

At the execution level, zeeg says the sandbox is shared, with instructions pinned to one actor at a time.

Desired-state control loops

dexhorthy splits the signals governing an agent into two categories:

  • Forward pressure: crashes, bug reports and performance problems that cause work to start.
  • Backpressure: linters, tests, AI review, canaries and feature flags that constrain progress toward an acceptable result.

His control loop has seven steps:

  1. Read the desired state, such as no open bugs or database performance above a threshold.
  2. Read the actual state through system queries or checks.
  3. Decide whether work is needed.
  4. Optionally obtain approval for that decision.
  5. Change code, infrastructure, documentation or processes.
  6. Optionally obtain approval for the action.
  7. Repeat.

He says the same cycle can also evolve the control loop itself.

Hermes Security confirmation

Nous Research’s official Hermes Security plugin, which Teknium says draws on mark_s’s security-review experience, makes verification an explicit stage:

  • Map attacker reachability and trusted inputs.
  • Split code across parallel reviewers and check candidate findings against the code.
  • Optionally write reproductions and execute them in a temporary repository copy.
  • Preserve evidence and rejected candidates in reports, with SARIF export.

According to the marketplace documentation, “Confirmed by test” requires both a passing reproduction and a matching negative test behaving as expected. Test confirmation executes repository code as the current user in a temporary copy, with credential-like environment variables removed but no sandbox isolation.

Further reading

Discussion across the web

Where this story is being discussed, in original context.

On X· 5 threads
feedback-learnings.md2 posts
Generated lints1 post
Daily transcript review1 post
Approval queues and actor ownership2 posts
Hermes Security confirmation1 post
Share on X