Computers started as massively costly, corporation / government only mainframes, that much is clear. They've been productivity machines from the get go, they've been machines made for war.
What computers "are" has been ebbing and flowing decade after decade. They've been repurposed to something that they initially weren't. It was originally niche, and not at all the norm.
Then, there was a brief period when counter culture was pop culture. That was reflected in everything in the 90's, from music to computers.
To make it clearer: He's one of the founders of the company that thrives in this sort of system, World (FKA Worldcoin). People were sort of making fun of the whole company and the dystopian premise a handful of years back... But here we are. Their latest "manifesto" was posted earlier this week, called The Simple Plan.
> You know what else I’d like to do besides becoming a great snowboarder? I want to learn kung fu. I’d also love to be a lot better at video games, get my Yu-Gi-Oh! hobby back on, and become at least fluent enough for everyday conversation in oh, I don’t know, eight more languages.
I think this sort of underplays the feeling of "lives unlived, paths not taken" that everyone gets hit with. Just flattens the whole thing that had been building up to that point, instead of allowing it to open up further.
Being 10x (or whatever multiplier) faster at programming doesn't mean you're going to be 10x faster in designing a product or any other aspect that goes into making a good product.
Even if you hired an actual programmer, it'd take a massive amount of time to build a Photoshop clone.
Of course, at the end Photoshop is lines of code and it could be output as is, end to end. One problem is that users aren't generally giving very precise design documents which would narrow the way to interpret them into code in precisely one way. Or that a design document at any level of precision, other than code, couldn't be interpreted in multiple ways when it comes to a specific implementation.
LLMs also take a relatively long time to output acceptable code, often taking tens of minutes before giving you a small diff. The larger the codebase, the longer it usually takes to start producing code, even over an hour.
> Of course, at the end Photoshop is lines of code and it could be output as is, end to end
And yet it’s not.
> LLMs also take a relatively long time to output acceptable code, often taking tens of minutes before giving you a small diff. The larger the codebase, the longer it usually takes to start producing code, even over an hour.
The problem is that they don’t generate acceptable code, they generate code that needs to be edited to be acceptable. That has always been the slow part of engineering. Waiting an hour for a bugfix even if it cost $75 in tokens would be cheaper than hiring an engineer but only if it worked. And it’s a bit like hiring a snake oil salesman - it passes the sniff test but it’s only when you’re drowning in the fact that your ai now takes 4 hours to fix the same bugs, and it introduces new bugs _and_ you don’t have anyone who can reduce that complexity that you see the reality. For a lot of us, that is immediately clear from first glance at the output of Claude and codex and the likes.
That was my point as well. That it hasn't been output, even though it could be done by a talented solo developer given enough time, and that current LLMs definitely aren't able to do so.
> The problem is that they don’t generate acceptable code, they generate code that needs to be edited to be acceptable.
You've never had an LLM output a one line bugfix that is correct to the point where you don't have to edit it?
To make things more concrete, here's an example from the creator of Redis on how he utilizes LLMs in programming: https://antirez.com/news/164
> You've never had an LLM output a one line bugfix that is correct to the point where you don't have to edit it?
I have. I’ve also had IDEs and static analysers do the same thing. I can also take my car out of gear and have it roll down a hill but that doesn’t mean it can run without fuel. Only a sith deals in absolutes, and in the general cases LLMs don’t generate acceptable code.
My experience is that when I ask for a clear and well defined problem, of the scale of "add motion blur (linear, spin, and zoom) to the filters menu; include standard dialog box (see existing design) for user input on all options", this works something like 90% of the time, is obvious garbage 5% of the time (in my experience, when it claimed to be writing "unit tests" it was actually performing regexes on the source code), and is subtly wrong the other 5% of the time.
If you use the planning mode, and your first move in the project is "write plan to reimplement photoshop" then you blindly say "continue" until the plan is done, then you get 0.9^{number of features} success, which of course on the scale of photoshop is going to be a failure. But this is still in one sense a 10x speedup in that 9 times out of 10 you're only doing code review, not having to re-write it. But code review is a real thing, so it's 10x on writing code not 10x on delivery.
Try it out, like seriously, learn to use it well. Spent a few weeks with it. You will not say these things if you were an experienced user of these tools. Saying their code is "unacceptable" is a skill issue. Describe what you consider "acceptable code" and watch it produce it in copious amounts. They don't have one mode, one setting, they can generate whatever the F you want in whatever style you deem "acceptable". You're completely in control.
That said, I watched many of my - generally pretty clever - colleagues struggle mightily with this. I can't put my finger on it yet. Regular "programming" - typing BS syntax one character at a time - always felt astronomically boring to me so I'm one of the guys happy with these tools. Not happy with how it will fuck up society though, but that's uh.. yeah.
As always in these discussions, I think people compare apples and oranges. LLMs are great with in-distribution solutions for solved problems with a lot of relevant prior material and in established technologies. Frontend stuff works great, for instance.
But for novel solutions, complex business logic, things deeply integrated with external systems... the code generation quickly turns terrible and useless. Especially if it's in anything but Python, Java, or JS.
Most of these differences in results end up being about differences in application. LLMs suck at out-of-distribution material, inherently.
The part about positional encoding is not correct.
> The intuition: instead of adding position info to each token’s vector, RoPE rotates the vector by an angle that depends on its position
You can't rotate the token's entire vector (or all three vectors, whatever is being implied is unclear). You rotate each token's Query and Key vectors only, so dot product can be used to tell how far apart the tokens are when comparing token 1's Query vector to token 2's Key vector.
Positional embedding should just be explained after explaining the Query, Key and Value vectors. When the article explains those only after that, the reader is building up on a wrong intuition and it gets confusing.
Yep, you’re correct. I got to that bit and thought that can’t be right. It’s obviously wrong. If you rotate a semantic vector, you change the semantics of it. You don’t want that.
If a project has a rule to not submit AI generated PRs, people should never submit AI generated PRs to that project. It's spam. Or if the rule is more nuanced than that in relation to AI, it should be respected.
It's 100% an issue with the people with submitting these PRs. So, if someone has a history of having no issue with breaking project rules (let alone being arrogant about it), it should be a massive red flag about the person for any possible employer or future collaborator checking their profile, etc.
Why people are wilfully destroying their own reputation like that is beyond me. It's infinitely better to have no activity at all on your profile than to create a track record of bad behaviour.
I don't really know if this would be the best solution, but we could maybe define a sliding LoC limit per PR? Also a limit on the number of open PRs per contributor. This way those small, reviewable contributions would get a fair shot at being landed instead of being blanket banned, and the maintainer overhead would be more reasonable.
Even that small one-line contribution is not the contributor’s own work. Since the contributor does not hold the copyright to it, they do not have the right to contribute it to the project.
What computers "are" has been ebbing and flowing decade after decade. They've been repurposed to something that they initially weren't. It was originally niche, and not at all the norm.
Then, there was a brief period when counter culture was pop culture. That was reflected in everything in the 90's, from music to computers.
reply