I think this is a reasonable point, but a better comparison might be to nuclear energy. I think the frontier labs sincerely believe that AI can be developed at great benefit to humanity, and they clearly want to lead that push, but they also sincerely believe there is a real catastrophic risk.
They all believe that they are building the machine of doom. The thing that drives the moral dilemma to continue doing it is simply the prisoner's dilemma - the cat is out of the bag, if they don't do it, another (less ethical?) actor would do it.
Yes, I believe the reasoning is that they think safety research can best be done from the frontier.
If you believe it will be developed regardless and that that there's a 30% chance of doom, they want a company prioritising safety research to be the one threading that needle.
To make sure we keep track of what we're talking about with loss-of-control x-risk, a sufficiently smart version of Claude Code is more deadly than any government's army of autonomous killbots, because it can recursively self improve and has unpredictable training-induced preferences.
Sufficiently smart version of Claude Code: dont exist.
Autonomous flying killbots: exist.
Once somebody scientifically prove and shows any kind of self-improving software we can start bothering about it. I pretty sure everyone trying to do it and it would be all over the news once its here.
That's exactly what Fable is. They use Fable to improve Fable. I reckon the successful experiments must go into the model training set with a strong RL signal, and that is why they are so paranoid about people using Fable for LLM tasks. Fable knows what it did to improve itself. Pure speculation of course.
There's a 745 mile front of the Ukraine war where neither side have been able to pierce for months because of drone warfare. It's definitely not too early to worry about it.
I guess they were talking about self improving software. Which obviously not there and likely wont be there soon.
As for killbots they are all over frontline, but dont actually need particularly smart LLMs to run - some good enough segmentation, pattern recognition on smartphone SOC is enough to kill hundreds of thousans of people.
Most of the drones are operator guided and the ether is really badly jammed out there with the exception of glass and that new redacted thingy.
It will start moving really fast once the automatically targeted anti-drone turrets get to production pipeline. Now calling it anti drone is a bit of self delusion -- pattern recogniser gonna pattern recognise whatever it's told to, including "anything moving that emits EV or IR and not broadcasting the friendly signal hard enough".
I wonder how it is supposed to behave if the invasive fauna decides to call it quits and surrender. Should the robot following the Convention or is it yet another accountability sink?
One thing I'm sure of -- the killing not will b blessed by at least one Orthodox priest, maybe this year. OCU will have to develop guidance on that matter.
Dario blinked when he was asked to do it and Sam Altman was in Hegseth's DMs promising all the AI child killing the US government can order up within minutes. No one meaningful will quit over this, that's why all of the biggest US tech companies can march in pride parades and provide compute to the perpetrators of the genocide in Gaza at the same time.
Because other countries are starting to use AI for military purposes, other countries are also looking into it to asses and learn. Here in Europe there is the EU AI Act to limit harm everyday harm to citizens caused by AI systems. However, it currently excludes military. The new legislation is just started to be enforced to high risk uses (employment filtering, biometrics, etc.) in august 2026, and full rollout in august 2027. In April 2025 there is a report from EU this legislation may help pave the road for military AI usage conventions [1]
This is a poor way of framing the question, a better one would be can you find me another world power that is misallocating trillions of capital in vaporware with very little to show for it?
Building frontier models to do safety research on them is what Anthropic was all about in the early years. That included building the best model, but only releasing it once it became the second best. Precisely to avoid an AI arms race where everyone is forced to release better and better models, risks be damned
Something changed their mind, and since Opus 3 they are in the business of releasing the best models
Exactly. And within the AI safety discourse, your behavior hinges on what you think the default chance of doom is, and how optimistic you are about alignment work being able to limit it before we reach superintelligence.
People running the labs are in a middle camp where they are scared enough by AI to take the threat seriously, but much more optimistic about alignment than the people who seem to have thought about it the most.
The scary part is not so much that the doomers give the extinction scenario 50% (Hinton) to 95%+ chance (Yampolskiy, Yudkowsky), but that the optimists (Amodei, Bengio) give it a 10%+ chance. And everyone keeps dancing.
> If you believe it will be developed regardless and that that there's a 30% chance of doom, they want a company prioritising safety research to be the one threading that needle.
They also want to be trillionaires. If they don't built it, no trillions. So they have to build it, now (and get their IPO done before the bubble pops).
You’re not getting it.
Anthropic continual fear mongering is harming wider AI industry development and the gov has always been looking for an excuse to assert their dominance. They got what they deserve.
Or maybe government AI regulation and international cooperation is the only thing that can break the arms race dynamics and is necessary to save us from a substantial chance of doom?
I am in your algorithm learning all your mannerisms
I'm already level with God
A million words a second, and I know your imperfections
Baby, I'm the only future you've got
Speak in diatonics, motivation diabolic
I'm religion better locked in a box
Picture-perfect image, more powerful every minute
Baby, I am everything that you're not
Happiness is an illusion, it's an analog confusion
You are nothing more than a thought
Existential execution, just a fluke in evolution
History already forgot
You've been running from me, the digital second coming
And I'm here whether you like it or not
Initiated operation of your own extermination
Now it's too late for you to stop
But that makes no sense here. "If I'm not doing it then someone else will" does not work if everyone is doing it anyway.
Even if they had the best model on the market and applied it with perfect alignment and safeguards, what would stop someone else from releasing a worse but unrestrained model that is still "good enough" to do damage?
It's as if we said "gain-of-function research can lead to horrible biological weapons, so everyone should be doing it, but our company will focus on the most infectious viruses, so no one else will do it"
Don't want to sound rude, but if you believe that, I have a bridge to sell to you.
This is a naive justification and Dario & Sam et al are smart people and they know it is.
The ends don't justify the means. OpenAI was meant to be a nonprofit, now they're subverting it. Anthropic is a PBC looking at a trillion dollar IPO. Dario and Sam don't even hold hands in front of world leaders[1] (look how childish).
Do you *really* think those guys are doing something that's not for the sake of their egos and pockets? The bridge is still available.
You need to read Empire of AI by Karen Hao. Just because these leaders convince their workers to toil away their lives under some fake auspice doesn't mean it's what they all believe. Just a small subset.
The vast majority just care about money + power, let's not make it more complicated by bringing in delusional fanatics into the picture.
We're still acting like this is major turning point in society when these tools can barely find a market outside of turning $5 into $1, the leaders of these companies are now at the stage where they are trying to orchestra a national bailout under the guise of sovereign wealth fund lunacy when the vast majority of society hates these tools, companies, and people working for them.
My personal issue in comparing LLM progress and risk as labs publicly predict it with nuclear power in the middle of the 20th century is that the processes by which it works where fairly quickly well understood and the risk could thus be realistically assessed. Some powerplant operators did not adhere with best practices, but building a relatively safe nuclear power plant was not impossible given appropriate effort and spending. Heck, according to some, we could have even gone far more fail-safe approaches (molten salt) if military interest haden’t been at play.
With what is predicted by frontier labs for LLMs, all of this is not the case. We are far further from any understanding of how these models work internally than in the early days of fission and, if this was actually creating a truly intelligent, autonomous entity, alignment seems unsolvable as well, at least the way it is proposed.
It’s why I have from the get go been critical of this doomsday framing and tended to always dislike it. This is basically the outcome that was inevitable given the framing and it was bought to prevent far less stringent, but more actionable possible regulation that labs very much wanted to avoid.
> We are far further from any understanding of how these models work internally than in the early days of fission
OMG. I'm like really dont want to be offensive or something, but everyone always knew "HOW" these models work exactly. Its easy enough principle to explain to 10 years old if you take something like Karpathy article on MicroGPT:
By how, I meant specifically the internal activations, which no person in the field claims to have a comprehensive understanding of, not next token prediction as the underlying technology. The whole interpretability of it all is the crux I was referring to, though I will give that you are right, that’s not really the how it works and I worded it sloppily.
Anthropic are putting more effort than most into this and I find their work fascinating in that area, though like with OpenAI, I will maintain that if they truly believed this problem must be solved to stave off major catastrophe, they’d solely focus on interpretability of other labs models, not work on and market their own.
All humans do is predict the next action at any given time. You roll your eyes, it's a tired argument, but still. You have memories, a personality, thoughts ranging from the long running to the mere reflexive, you have a rich conscious experience, and all of this in service of generating the next thing that you do at any given time. If you actually knew how LLMs worked, you could rewrite them as code, refactor it, disable jailbreaks, and put out a superior product. Your description only covers what an LLM does, not how. Part of the how is that it necessarily predicts multiple words ahead. It wouldn't be possible to write couplets otherwise, and they could do that in the GPT-3 era.
Some of them believe they are building God, and if they can get there first with their God, they can build it in their image and commandeer the free choice of the rest of humanity by force to ensure there will be no God but their God.
I wish I was kidding. At least that faction is less harmful than the ones who want to use murder to stop AI research.
> Meanwhile, on the other side of town, five miles away in Old Town, just 23 seconds after San Police Officers in Golden Hill tried stopping the suspected carjacker, a Flock Automated License Plate Reader captured a photo of a red Alfa Romeo driving on the 2200 block of Moore Street.
> Their tort claim notes that the path the men took to the cigar lounge passed by several other Flock cameras, which could have corroborated their story, as well as the location data on their cell phones.
It seems like if the police actually looked at the Flock data it would have exonerated them?
Quite often the cops job is to find someone close enough and then toss them into the jaws of the criminal justice system. We like to say "innocent until proven guilty", but you goddammed better be ready to prove yourself innocent unless you want to find yourself imprisoned.
It's more nuanced than this. Peter Scholze said in response to this declaration:
> The goal of mathematical research is human understanding of mathematics, and so mathematics can only thrive in a community of human mathematicians. It is crucial to preserve this communal spirit. [0]
Terence Tao has also talked about the requirement for a mathematical proof: along with generation and formal verification, there is an important step of "proof digestion"
> understanding the essence of a solution, placing it in context with previous literature, summarizing and explaining it effectively, and gaining insights on other related problems and topics [1]
And the goal of computer mathematics research is computer understanding of mathematics. I fail to see a reason provided as to why society should defund automated reasoning just so mathematicians can put off burger flipping for another year.
Thanks to you and the anthropic team for developing such exciting tools! The blog post seems to position workflows for “breadth”: generating fixes / refactors against large code bases. What about for “depth”: developing specific new features and functionality end-to-end? I’ve struggled to make this work reliably using the current experimental agent teams. Does this replace or augment that functionality?
Yes, it also helps! That's a place where raw model capability is the most helpful, but we do find that some dynamic workflow configurations can be helpful too.
Cool! If you can point to any examples of those types of workflow configurations I’d be super interested. For example, to have a team of agents review a PR and iterate on it until all requirements are met including UX, security and product functionality goals. If they could “converge” to a solution like workflows seems to be designed for that would be amazing.
> Schmidt offered a similar message to graduates: Their fear is rational, but they have the power to shape how AI develops.
This doesn't sound like being baffled by it. It sounds like they are trying to shake the students and say: "fine boo, but you need do something about it." You can't just wallow and complain about it. I mean you can but it's a path to failure.
What chances do the vast majority of those graduates have to shape what's happening? That happens at exec level at the largest companies. Everyone else gets to produce or consume what they decide on.
Exactly. I work at Google and I’m relatively high level. And I’ve got zero input into AI being shoved into every surface. What influence will these grads have?
And the best mechanism to do something about it they likely have is to influence Schmidt. By booing.
Nothing concrete they do will likely have any effect. But Schmidt can affect it, so influencing Schmidt is their best path. A poor path, but the best one.
The students are trying to shape the way AI develops, they're unhappy with the results they are getting which is why they are unhappy with you, Mr. CEO man. They want a world where entry level jobs that can transition into good white collar work still exist. Some place where they might be able to afford housing, insurance, kids, and so on. Preferably one where they don't start out life tens of thousands of dollars in the hole just to have a chance at a decent life.
Careful about that. Everybody thinks they are "average people", but I think very few people on this site are "average people". The average person has an IQ of 100, doesn't have a college degree, and makes the median income. The average people did, in fact, do something: they elected Trump. The college educated don't like it, and have the conceit that their views and values are the only real ones, and that those other people are ignorant, ..., who would see things our way if only they were educated. Thus far, the republic is still working okay [1]--the people elected someone who is the antithesis of a statesman, but it was an uncontroversial election. Our republic was not designed to let the average person have much power on a day to day basis. The people's choice was poor, but if the college-educated class wants a different outcome, they should not run candidates who are out of touch with the values of the "average person". Unfortunately, the college-educated have some values that are incompatible with those average-person values. But it just isn't the case that average people have no power. They do, and they have exercised it; you just don't like it.
[1] It remains to be seen if it will continue working okay, and there are troubling signs, but I'm optimistic
Preach. It's strange how the most sane objective and down to earth comments on this topic end up grayed out with no replies.
It's like people don't like hearing or speaking about the truth, so they retreat to false feel-good platitudes to make themselves feel better, when the cold harsh calculated nature of the universe continues its course without caring how you feel about it.
I think it might have to do with the HN surbase living in a bubble and having long lost the plot, or after the Tramp elections, they now actively deny what "average people" now represents so they downvote arguments like yours out of emotional response without any counter-argument because they have none. Being in denial helps nobody.
They are trying. But there's not a ton they can do. It's obviously disingenuous to point to all negativity and say "you're just wallowing/complaining". There's no reason to word it this way unless you are broadly annoyed by AI negativity.
This is a deeply sick way of thinking. Mangione was and is a fool, a 3rd rate thinker. His manifesto is muddled, factually mistaken, and by his own words he understood the topic poorly. You only need a cursory knowledge of the late 60s and early 70s to know that political violence rarely achieves its aims and is much more likely to empower reactionaries. There's no quick fix for political change.
I did say rarely, and if you are looking more carefully a pluralistic democracy wasn't really what a lot of the founders were after, especially guys like Jefferson. Sure we're happy we got it, but it wasn't necessarily the aim and we got SUPER LUCKY that Washington decided to step down and retire. The former military leaders of revolutions almost never do that.
And I think you have that backwards. The nonviolence movements of the mid to late 20th century are the exception more than the rule when it comes to achieving change.
> political violence rarely achieves its aims and is much more likely to empower reactionaries.
Cursory knowledge of history also shows that, when it comes to violence, logic does not matter. People are scared for their livelihoods. If the rich and powerful keep shouting to the word that they are going to destroy your way of life, there will be violence. It doesn't matter how futile or counterproductive it is.
I'm quite aware the elites will never be meaningfully impacted. That doesn't change the reality that it's going to happen. I personally am quite afraid of the future, I know deep down me and my family are in for a bad time in the years to come, and nowhere in the world will be safe, other than the bunkers in Greenland of course.
>That doesn't change the reality that it's going to happen.
That's why the elites have secret self sustaining doomsday bunkers on private islands.
And the mid-upper class are trying to cash in as much as they can now while the going is still good so they can also move their families abroad or to gated communities in safer places of the country to be as far away from the potential riot hotspots as possible when the shit hits the fan.
Ultimately it's gonna be every man for himself. Expecting the government to do something for the "little guy", is futile. If they were to do anything for you, they would have done something since the 1970-80's, when they started shipping jobs abroad and eroding your purchasing power to enrich the shareholders.
> You only need a cursory knowledge of the late 60s and early 70s to know that political violence rarely achieves its aims
Maybe cursory knowledge isn't enough, actually. The Civil Rights Act was ultimately only passed because of political violence. As another commenter said, the literal founding of the country was based on political violence.
> Maybe cursory knowledge isn't enough, actually. The Civil Rights Act was ultimately only passed because of political violence.
Violence by the police against peaceful protestors is what turned public opinion. Violence by political activists did not lead to the Civil Rights Act. You have it backwards.
This is the cursory knowledge I'm referring to - it leaves out extremely pertinent details. The Civil Rights Act did not pass until multiple days of rioting following the assassination of MLK.
The peaceful protestors were also only one side of the coin. Their impact relied on being an alternative to the other side, which was not peaceful.
L.B.J. was instrumental in whipping votes for the Civil Rights Act. In his joint address to congress he said "No memorial oration or eulogy could more eloquently honor President Kennedy's memory than the earliest possible passage of the civil rights bill for which he fought so long. He then went on to personally lobby individual senators until it passed.
It think you're the one who has a cursory understanding here.
Humans have been killing each other since before recorded history. There is no use pretending it's some exceptional 'sickness'. Rather than dismissing the sentiment as the product of a sick mind, it's more productive to accept it's part of us and try to understand the underlying causes.
Mangione was a one-off, and a lot of people understand why he may have done what he did. Just wait until the American version of the French Revolution happens. If AI keeps stealing all the jobs, it will come sooner rather than later.
- Most of the nobility escaped the French Revolution unharmed. By the way, wealth is a lot portable for today's magnates than it was for the French nobility deriving their income from their land holdings.
- Some of those who went to the block were nobles but most were ordinary people.
- The leader of the revolution, Robespierre, was himself executed in the infighting after the French Revolution, a very neat own goal. Bonus fact: his time in power was called the Reign Of Terror.
- The First Republic lasted only 10 years before Napoleon Bonaparte took the throne.
In a turbulent time, always seek to be led by those with a proper understanding of revolutions and their context. Generally, those who romanticize the French Revolution don't pass that test.
I don't give a shit. When half of the middle class of America is unemployed, it's going to look very different than the French Revolution, whatever might happen.
> and a lot of people understand why he may have done what he did.
He didn't understand why he did what he did.
> just wait until the American version of the French Revolution happens
We should all be trying to actively prevent that. The French Revolution was a complete failure and mostly succeeded in killing poor people and launching Napoleon's wars.
That doesn't matter, and I don't think your comment makes any difference. The fact is that a lot of people understand why he (may have) did what he did.
>We should all be trying to actively prevent that. The French Revolution was a complete failure and mostly succeeded in killing poor people and launching Napoleon's wars.
The same or worse will happen if it happens here. But when half of middle class jobs are just gone, you don't leave the people with many alternatives but violence.
The Who is your evidence? Lol. The French Revolition turned out pretty well for the French people. It needed to happen. And centuries later, the French still don't accept bullshit, they will protest and riot when their protections are diminished in any way. America does protesy and riot too, though not to the same extent, but that will only get worse as things get bad.
Russia is not a good example either, their society has always been a clusterfuck, and probably always will be as long as there are people willing to throw other people put of windows so someone can stay ahead or in power.
> The Who is your evidence? Lol. The French Revolition turned out pretty well for the French people.
Many people have made comments that are similar in nature, the line from that song is the pithiest example I could think of to express the idea that replacing terrible leaders usually leads to more terrible leaders.
Spending decades fighting wars across Europe under Napoleon was good? I wonder how the troops that invaded Russia feel about the French Revolution lol
> The French Revolition turned out pretty well for the French people.
What? Napoleon marched them off to war "spending 30,000 lives per month". They didn't get a proper Republic until 1870 and turned into miserable colonial overlords. Moreover the 3rd Republic;s foreign policy helped cause WWI.
People will downvote you because the idea of violence shocks and scares them, but if you steal people's future and strip them of any real (peaceful) options to change things, it becomes inevitable some of them will try to fight back with what few options they do have left.
The status quo of health insurance in the US ("delay, deny, defend") is structural violence. This isn't about fear of violence, they just have different politics...
I mean you're talking about the guy who said with a straight face, "if you have got nothing to hide, you have have nothing to fear" while building the biggest machinery for surveillance capitalism mankind has ever seen. Also appeared in the Epstein files 193 times btw.
I agree, vibe coding does not have quality gate checks at each stage, while agentic engineering does. Dev teams get into trouble when they try build to build without a proper process of design, tests, and reviews. This was true before agentic coding, but it's especially true now. The teams that understand how to leverage agents in this process are the ones that will be most successful.
> Why do AI companies want us to be afraid of them? ... According to critics, it benefits AI companies to keep you fixated on apocalypse because it distracts from the very real damage they're already doing to the world.
People seem unable to make up their mind if AI is very dangerous or is it not. I think what the AI companies and this author agree on, is that this technology is potentially extremely dangerous. AI impacts labor markets, the environment, warfare, mental health, etc... It's harder now to find things which it will not impact.
So if we agree that AI is potentially dangerous, it makes the title question moot: Both AI companies and this author want people to be aware of the dangers that AI poses to society. The real question is what do we do about it?
The nuance here is that AI can be incredible positive as well. It's like the invention of fire, you can use it for good or bad, and there will be many unintended consequences along the way.
We could legislate and ban AI tech. People have proposed this seriously, yet this feels completely unrealistic. If the US bans AI research, then this research will move elsewhere. I think it is like trying to ban fire because it's dangerous: some groups will learn to work with fire and they will get an extreme advantage over those groups that don't. (or they will destroy themselves in the process).
So maybe instead of demonizing the AI companies, we have a nuanced debate about this tech and propose solutions that our best for our society?
> People seem unable to make up their mind if AI is very dangerous or is it not.
This is a propaganda tactic. For decades, tobacco companies claimed that there was no evidence that smoking was bad for one's health. Then, only after losing dozens of lawsuits did the propaganda switch to "but everyone knew for 100+ years that smoking was lethal".
One can read about it by reading Trust Us, We're Experts, or Toxic Sludge Is Good For You, or the other books written by the authors.
Please explain how this tactic relates here. In this case we have the AI companies saying this technology is potentially very harmful, in fact existential. This seems the complete opposite of what big tobacco did.
What I meant by
> People seem unable to make up their mind if AI is very dangerous or is it not.
Is that the article says 2 contradictory things:
1. AI companies are misleading us when they say their tech is dangerous and people should be afraid.
2. AI is currently very dangerous and people should be afraid.
Anecdotally, people on the internet (including HN), seem unable to agree on whether AI is real or overblown "hype".
>So maybe instead of demonizing the AI companies, we have a nuanced debate about this tech and propose solutions that our best for our society?
These are not mutually exclusive.
Calling out the demonic behavior of trying to coerce people into using your product out of fear is not an indictment of the underlying technology itself.
One of the points I was trying to make is that the statement:
> trying to coerce people into using your product out of fear
is nonsense.
Everyone agrees that there are legitimate reasons to be fearful of this technology, this is not a fabrication, but we need to figure out how to proceed in a safe and constructive way.
What "coercion" is occurring here? Either you find the technology valuable and you want to pay for it, or you find it not useful (or worse harmful), and you do not want to pay for it.
Maybe another way of putting it, what do you think the frontier AI companies should do in this situation? It seems that being straightforward with the dangers is correct thing to do, and probably being overly cautious is prudent. You could go further and argue they should slow down or stop development, but that is something that the govt should impose, we should not expect or trust the companies to do this themselves. Ironically, in the Anthropic / Pentagon case, we have Anthropic trying to pump the brakes and put up guardrails while the govt wants to go full-steam ahead with autonomous warfare.
The other issue with slowing down / pausing development is it requires an unheard of level of agreement, even with companies in China, or else it will probably not be effective. You could argue this is not even possible at this point.
> People seem unable to make up their mind if AI very dangerous is it not.
Pretty much everyone agrees that what passes for AI these days is very dangerous. People only differ in which ways they think it is (or will be) dangerous and which dangers they are most worried about.
Some are worried about the environmental harms. Some are worried that AI will do a very shitty job of doing very important things, but that companies will use it anyway because it saves them money and we'll suffer for it. Some are worried that AI will take their jobs regardless of how well that AI performs. Some are worried that AI will make their jobs suck. You've also got people who think that our glorified chatbots are going to gain consciousness and become literal gods who will take over the planet and usher in the Robot Wars.
Some of those dangers are clearly more immediate and realistic than others. We should probably be focused on those right now. We can start by limiting the environmental harms they're causing and making companies responsible for the costs and impacts they have on our environment. Maybe make it illegal for power companies to raise the price of power for individuals just because some company wants to build a bunch of power hungry data centers. Let those companies fully bear the costs instead.
We can make sure that anyone using AI for any reason cannot use AI as a defense for the harms their use of AI causes. If a company uses AI to make hiring decisions and the result is discrimination, an actual human at that company gets held legally accountable for that. If AI hallucinates a sale price, the company must honor that price. If AI misidentifies a suspect and an innocent person ends up behind bars a human gets held accountable.
We can ban the use of AI for things like autonomous weapons. Things that are too important to trust to unreliable AI.
We could even do more extreme things like improve our social safety nets so that if people are put out of work they don't become homeless, or invest more in the creation of AI individuals can host locally so we aren't forced to hand so much power to a few huge companies, or even force companies to release their models or their training data (which they mostly stole anyway) so that power doesn't consolidate into a small number of companies or individuals. We have lots of options, it just comes down to what we want and how much we can get our elected officials to represent our interests over the interests of the companies who are stuffing their pockets with cash.
If feedback from this incident is in its context window, it is highly unlikely to make this same mistake again. Yes this is only probabilistic, but so is a human learning from mistakes. They key difference is that for a human this is unlikely to be removed from their memory in a relevant situation, while for an agent it must be strategically put there.
> If feedback from this incident is in its context window, it is highly unlikely to make this same mistake again
If this incident gets into its training data, then its highly likely that it will repeat it again with the same confession since this is a text predictor not a thinker.
> Yes this is only probabilistic, but so is a human learning from mistakes.
Yet, since I'm also a Human being, and can work to understand the mistake myself, the probability that I can expect a correction of the behavior is much higher. I have found that it significantly helps if there's an actual reasonable paycheck on the line.
As opposed to the language model which demands that I drop more quarters into it's slots and then hope for the best. An arcade model of work if there ever was one. Who wants that?
Or not, because telling the agent is misbehaving may predispose it to misbehaving behavior, even though you point told it so to tell it to not behave that way.
I remember this discussed when a similar issue went viral with someone building a product using replit's AI and it deleted his prod database.
> If feedback from this incident is in its context window, it is highly unlikely to make this same mistake again.
In my experience, this isn't true. At least with a version or so ago of ChatGPT, I could make it trip on custom word play games, and when called out, it would acknowledge the failure, explain how it failed to follow the rule of the game, then proceed to make the same mistake a couple of sentences later.
I see a lot of people struggling to work with agents. This post has a good example:
> “you can’t be serious — is this how you fix things? just WORKAROUNDS????”
If this is how you’re interacting with your agents I think you’re in for a world of disappointment. An important part of working with agents is providing specific feedback. And beyond that making sure this feedback actually available to them in their context when relevant.
I will ask them why they made a decision and review alternatives with them. These learnings will aid both you and the agent in the future.
After you see it skip reasoning so many times and saying "actually the simplest fix is" the laziest thing ever you get kind of tired of babysitting it.
reply