> I am in the process of attempting to have AI run my business. I'm actually making very good progress, but it's happening in pieces
Very cool! What's been the hardest part? Have you successfully automated non-trivial communications? (for example with prospects, customers, vendors, partners, etc.)
> maybe an agent starting a business from scratch would have a better time
This describes the project I've been working on since last year, with agents in company roles, deciding on strategy, collaborating, and making progress (admittedly nonlinear). Some parts of the platform are stronger than others. So far the company agents have developed a company strategy and plans for executing it, launched a website and blog, and built and operate two products, one free and one paid.
But I would be wary of using a third party like Pion as a platform for running a business. It's one thing to know that your prompts and responses will be used to train models in the future, by companies that have a huge sea of relatively unstructured data. But it's another to hand over to another company every aspect of your strategy and operations, available to that company in real time and structured in such a way it's quick and easy to understand what you're up to and where you are going. Especially if it's being offered for free and at scale. With a free offering, value creation will likely come from either customer data or from escalating prices once customers are locked in. And even if neither of those happen, you now have a huge single point of failure for your entire organization. Seems like there are lots of strategic risks in there for the users.
> Very cool! What's been the hardest part? Have you successfully automated non-trivial communications? (for example with prospects, customers, vendors, partners, etc.)
Yeah, the most recent brand I acquired had enough customer contacts via FB/IG comments and messages, plus email, that I had Claude build out an application for me where it pulls those in as they come, drafts a response including any actions that need to be taken, and lets me edit or send. It takes any edits as feedback to refine its playbook, so its gotten pretty good now. Handles things like warranty claims (get picture of the damage plus order number, create replacement order, send email confirming replacement order) plus most common questions. Still entirely human in the loop, but I'm going to let it start handling common cases autonomously soon.
On the topic of starting an autonomous agent-run business from scratch, I had Claude set up a bunch of websites that pull publicly available (but somewhat annoying to access) government data, display it in easy to read format and send weekly emails. It's now running on its own with a couple levels of management - first, it does a weekly review of Google Search Console/GA4 and updates or creates new content to try to improve things from an SEO perspective. Then second, once a day it tracks all of the problems that have arisen over the past day (usually issues pulling the data or issues with the prose the models write), corrects them, and then tries to improve the system to stop them from recurring.
I'm not really keeping tracking of it at the moment - too much other stuff to do right now - but it's getting a trickle of subscribers and is slowly rising the SEO ranks, so for now I'm content to let it sit and see if it ever gets to the point of being monetizable.
In our new world of non-deterministic output (that's why we love LLMs! they say such helpful/agreeable/sometimes wrong stuff!), I think CI won't be sufficient. CI is in the realm of Quality Control; when I build the thing, is it to spec and does it do what I need it to do?
But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time. For example, if a hotel chatbot starts telling guests checkout time is at 11pm instead of 11am, that's a real operational problem and ideally should be caught before it impacts customers.
LLMs introduce new classes of problems/risks that we are just starting to understand and develop the tools to manage.
>But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.
The instinct seems good because it's impossible to keep up with all the details if you are running AI full-blast. Absolutely impossible. So testing outputs makes sense.
I have a hard time seeing exactly how we get from here to there. But intuitively I would not be surprised. One of these thing where quality may drop 20% but you can scale 100x.
> In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time.
Is that hotel example real? Curious how exactly you employ this technique—my naive idea was, if talking software development, a sort of 'sanity-check auto-linter agent' catch errors on a regular basis (every 10 seconds, every write, w/e).
Well, it's a real example! No, it's not a true story that I'm aware of, but there are plenty of examples of real chatbots run amok.
I refactored the tool out of my application and it's available now at https://endpointevaluator.com . There's nothing there someone couldn't build themselves, but then you have to spend that time building and maintaining it. I think the ongoing and long term cost of maintaining all this LLM-generated software is under-appreciated. So hopefully there is still space for outsourcing tools that are generic enough to be used by many and provide good value for cost.
If that aspect is critical to your workflow, it seems like you could run the model of your choice off of hugging face, on GPU hardware under your control, so the model won't shift out from under you.
That's one good way to reduce the risk, but I don't think it eliminates it.
Even with the same model and the same input, the output is inconsistent. And what I've observed is that as the size of the input and output grows, the consistency and accuracy of the output seems to decrease. It gets more complicated when you don't control the full input, such as a chatbot with customers.
I think the problem remains even if it can be mitigated by freezing the model and the hardware, which carries the tradeoff of requiring a model you can download and run on your own so you can't use the SOTA models.
Do you have a solution for degradation in accuracy when compiling larger amounts of llm-produced text?
I am also building LLM knowledge/memory systems and I've been surprised how bad LLMs are, even SOTA models, at summarizing non-trivial input batches of text. They get things wrong, distort the underlying meaning or data, etc.
A constant challenge. Don't have a perfect solution for it yet, but importantly every change to any article logs who did it, what it did and the reasoning behind it. So I have enough data to work with as I continue to improve things.
Divide and conquer essentially, is what I've found so far to work best. Split things into smaller and smaller chunks to independently be verified, double-check everything, then coalesce upwards with verified summarizations. Have benchmarks for every single task and sub-task that will happen everywhere a LLM is involved, so you can measure improvements. Takes a ton more effort and tokens in the system itself obviously, but if you're not paying per token, it seems to work pretty well, albeit feels slightly over-engineered already.
> Have benchmarks for every single task and sub-task that will happen everywhere a LLM is involved, so you can measure improvements
exactly. are there standard approaches or tools specifically for instrumenting LLM utilization, for applications rather than model development?
i use claude code a bit. i relentlessly push it to keep its work ephemera in source controlled and discoverable locations, following a lot of ozbrain's motivation. i also have claude generate task logs for all spawns. the theory with the task logs is to try and build some standard measures so i can see whether a thing i do (writing skills, experimenting with third-party MCP tooling, etc) actually helps in a way that is more robust than gut. i have yet to turn that effort into something useful. i'm just starting to get enough data to try to do some diving.
the problem is that this feels like the classic problem of "how do we measure productivity"? i was thinking very basic efficiency like "tokens in per tokens out" (maybe with a domain-specific value weighting function on the output tokens, e.g. code tokens > memory tokens > chat tokens) could be an possibility.
the point being, seems like you'd want to become pretty sophisticated about measurement to sell a product like ozbrain, as a basis for continuous improvement across major versions. or convincing people to use it, quantitatively rather than qualitatively.
Cool project! I haven't seen that OpenRouter workflow yet (sign into OpenRouter and it creates an API key that your app can use), that looks like an interesting pattern to investigate.
My company recently built a tool that is closer to your first category, but it's an API so it doesn't have the security (supply chain) concern of being embedded in your application.
It's built to help people manage the risk of LLMs changing underneath them and drifting from their designed behavior. Traditional deterministic testing probably won't be sufficient for apps that provide nondeterministic output, like a chatbot backed by an LLM.
The point in the linked article about the challenge of selling developer tools to developers is a good one. I think the first reaction to coding agents is "let's build everything ourselves!" but the long tail of maintenance is still there and the pendulum will probably swing back to "let's stick to our knitting."
Very cool! What's been the hardest part? Have you successfully automated non-trivial communications? (for example with prospects, customers, vendors, partners, etc.)
> maybe an agent starting a business from scratch would have a better time
This describes the project I've been working on since last year, with agents in company roles, deciding on strategy, collaborating, and making progress (admittedly nonlinear). Some parts of the platform are stronger than others. So far the company agents have developed a company strategy and plans for executing it, launched a website and blog, and built and operate two products, one free and one paid.
But I would be wary of using a third party like Pion as a platform for running a business. It's one thing to know that your prompts and responses will be used to train models in the future, by companies that have a huge sea of relatively unstructured data. But it's another to hand over to another company every aspect of your strategy and operations, available to that company in real time and structured in such a way it's quick and easy to understand what you're up to and where you are going. Especially if it's being offered for free and at scale. With a free offering, value creation will likely come from either customer data or from escalating prices once customers are locked in. And even if neither of those happen, you now have a huge single point of failure for your entire organization. Seems like there are lots of strategic risks in there for the users.
reply