Retool, a development platform for business software, recently published the results of its State of AI survey. Over 1,500 people took part, all from the tech industry:...
Over half of all tech industry workers view AI as overrated::undefined
I’ve found that relying on it is a mistake anyhow, the amount of incorrect information I’ve seen from chatgpt has been crazy. It’s not a bad thing to get started with it but it’s like reading a grade school kids homework, you need to proofread the heck out of it.
What always strikes me as weird is how trusting people are of inherently unreliable sources. Like why the fuck does a robot get trust automatically? It’s a fuckin miracle it works in the first place. You double check that robot’s work for years and it’s right every time? Yeah okay maybe then start to trust it. Until then, what reason is there not to be skeptical of everything it says?
People who Google something and then accept whatever Google pulls out of webpages and puts at the top as fact… confuse me. Like all machines, there are failures. Why would we trust that the opposite is true?
At least a Google search gets you a reference you can point at. It might be wrong, it might not. Maybe it points to other references that you can verify.
ChatGPT outright makes shit up and there’s no way to see how it came to a given conclusion.
Because the average person hears “AI” and thinks Cortana/Terminator, not a bunch of if statements.
People are dumb when it comes to things they don’t understand. I’m dumb when it comes to mechanical engineering of any kind, but I’m competent with software. It’s all about where people’s strengths lie, but some people aren’t aware enough to know they don’t know something
My guess, wholly lacking any scientifc rigor, is that humans naturally trust each other. We don’t assume the info someone shares with us as wrong, unless there’s “a reason” to doubt. Chatting with any of these LLM bots feels like talking to a person (most of the time), so there’s usually “no reason” to doubt what it spews.
If human trust wasn’t so easy to get and abuse, many scams would be much harder to pull.
That 80% also doesn’t take that much effort. Automation can still be helpful depending on how much effort it is to repeatedly do it, but that 20% is really where we need to see progress for a massive innovation to happen.
I actually disagree. Ai is great at doing the parts that are easy to do mentally but still take time to do. This “fancy autocomplete” is where it shines and can accelerate the work of a professional by an order of magnitude
I have found that it’s like having a junior programmer assistant. It’s great for “write me python code for opening an in file from a command line argument, reading the contents into a key/value dict array, then closing the file.” It’s terrible for “write me a python code for pulling data into a redis database.”
I find it’s wrong 50% of the time for certain command line switches, Linux file structure, and aws cli.
I find it’s terrible for advanced stuff like, “using aws cli and jq, take all volumes in a vpc, and display the volume id, volume size in gb, instance id it’s attached to, private IP address of the instance, whether is a gp3 or gp2, and the vpc id in a comma separated format, sorted by volume size.”
Even worse at, “take all my gp2 volumes and make them gp3.”
I recently used it to update my resume with great success. But I also didn’t just blindly trust it.
Gave it my resume and then asked it to edit my resume to more closely align with a guide I found on Harvards website. Gave it the guide as well and it spit out a version of mine that much more closely resembled the provided guide.
Spent roughly 5 minutes editing the new version to correct for any problems it had and boom. Half an hour of worked parsed down to sub 10
I then had it use my new resume (I gave it a copy of the edited version) and asked it to write me a cover letter for a job (I provided the job description)
Boom. Cover letter. I spent about 10 minutes editing that piece. And then that new resume and cover letter lead to an interview and subsequent job offer.
No. It’s not and hasn’t been for at least a year. Maybe the ai your dealing with is, but it’s shown understanding of concepts in ways that make no sense for how it was created.
Gotta go.
It does a shockingly good analogue of “understanding” at the very least. Have you tried asking chatgpt to solve analogies? Those show up in all kinds of intelligence tests.
We don’t have agi, definitely, but this stuff has come a very long way and it’s quite close to being genuinely useful.
Even if we completely reject the “it’s ai,” we more or less have a natural language interface for computers that isn’t a shallow trick and that’s awesome.
Have you tried asking chatgpt to solve analogies? Those show up in all kinds of intelligence tests.
This two statements are causal to each other. And it actually gets them wrong with some frequency in ways that humans wouldn’t, forgets stuff it has already “learned”, or changes to opposite stances midways sentences. Because it is just an excel sheet on steroids.
“Excel sheet on steroids” isn’t oversimplification: it’s just incorrect. But it doesn’t really sound like you’re particularly open to honest discussion about this so whatever.
Well here’s the question. Is it solving them, or just regurgitating the answer? If it solves them it should be able to accurately solve completely novel analogies.
yes, it’s has. the most famous example is the stacking of the laptop and the markers. you may not have access but it’s about to eclipse us imho. I’m no technological fanboy either. 20 years ago I argued that I wouldn’t be possible to understand human speech. now that is a everyday occurrence.
As you can see, the past year has shed a lot of light on the topic.
One of my favorite facts is that it takes on average 17 years before discoveries in research find their way to the average practitioner in the medical field. While tech as a discipline may be more quick to update itself, it’s still not sub-12 months, and as a result a lot of people are continuing to confidently parrot things that have recently been shown in research circles to be BS.
Best assessment I’ve heard: Current AI is an aggressive autocomplete.
I’ve found that relying on it is a mistake anyhow, the amount of incorrect information I’ve seen from chatgpt has been crazy. It’s not a bad thing to get started with it but it’s like reading a grade school kids homework, you need to proofread the heck out of it.
What always strikes me as weird is how trusting people are of inherently unreliable sources. Like why the fuck does a robot get trust automatically? It’s a fuckin miracle it works in the first place. You double check that robot’s work for years and it’s right every time? Yeah okay maybe then start to trust it. Until then, what reason is there not to be skeptical of everything it says?
People who Google something and then accept whatever Google pulls out of webpages and puts at the top as fact… confuse me. Like all machines, there are failures. Why would we trust that the opposite is true?
At least a Google search gets you a reference you can point at. It might be wrong, it might not. Maybe it points to other references that you can verify.
ChatGPT outright makes shit up and there’s no way to see how it came to a given conclusion.
That’s a good point… So long as you follow the links and read more. My girlfriend for example, often doesn’t
Because the average person hears “AI” and thinks Cortana/Terminator, not a bunch of if statements.
People are dumb when it comes to things they don’t understand. I’m dumb when it comes to mechanical engineering of any kind, but I’m competent with software. It’s all about where people’s strengths lie, but some people aren’t aware enough to know they don’t know something
My guess, wholly lacking any scientifc rigor, is that humans naturally trust each other. We don’t assume the info someone shares with us as wrong, unless there’s “a reason” to doubt. Chatting with any of these LLM bots feels like talking to a person (most of the time), so there’s usually “no reason” to doubt what it spews.
If human trust wasn’t so easy to get and abuse, many scams would be much harder to pull.
I think you might be onto something. Thanks for sharing!
People trust a squid predicting football matches.
I feel like the AI in self-driving cars is the same way. They’re like driving with a 15 year old that just got their learners permit.
Turns out that getting a computer to do 80% of a good job isn’t so great. It’s that extra 20% that makes all the difference.
That 80% also doesn’t take that much effort. Automation can still be helpful depending on how much effort it is to repeatedly do it, but that 20% is really where we need to see progress for a massive innovation to happen.
I actually disagree. Ai is great at doing the parts that are easy to do mentally but still take time to do. This “fancy autocomplete” is where it shines and can accelerate the work of a professional by an order of magnitude
I just reviewed a PR today and the code was… bad, like unusually bad for mycoworkers and left some comments.
Then my coworker said he used chatgpt without really thinking on what he was copypasting.
I have found that it’s like having a junior programmer assistant. It’s great for “write me python code for opening an in file from a command line argument, reading the contents into a key/value dict array, then closing the file.” It’s terrible for “write me a python code for pulling data into a redis database.”
I find it’s wrong 50% of the time for certain command line switches, Linux file structure, and aws cli.
I find it’s terrible for advanced stuff like, “using aws cli and jq, take all volumes in a vpc, and display the volume id, volume size in gb, instance id it’s attached to, private IP address of the instance, whether is a gp3 or gp2, and the vpc id in a comma separated format, sorted by volume size.”
Even worse at, “take all my gp2 volumes and make them gp3.”
I recently used it to update my resume with great success. But I also didn’t just blindly trust it.
Gave it my resume and then asked it to edit my resume to more closely align with a guide I found on Harvards website. Gave it the guide as well and it spit out a version of mine that much more closely resembled the provided guide.
Spent roughly 5 minutes editing the new version to correct for any problems it had and boom. Half an hour of worked parsed down to sub 10
I then had it use my new resume (I gave it a copy of the edited version) and asked it to write me a cover letter for a job (I provided the job description)
Boom. Cover letter. I spent about 10 minutes editing that piece. And then that new resume and cover letter lead to an interview and subsequent job offer.
AI is a tool not an all in one solution.
https://www.cs.cornell.edu/info/people/fcc/humor/history.html
Nice one! I have heard it called a fuzzy JPG of the internet.
And that’s entirely correct
No. It’s not and hasn’t been for at least a year. Maybe the ai your dealing with is, but it’s shown understanding of concepts in ways that make no sense for how it was created. Gotta go.
No it hasn’t.
It does a shockingly good analogue of “understanding” at the very least. Have you tried asking chatgpt to solve analogies? Those show up in all kinds of intelligence tests.
We don’t have agi, definitely, but this stuff has come a very long way and it’s quite close to being genuinely useful.
Even if we completely reject the “it’s ai,” we more or less have a natural language interface for computers that isn’t a shallow trick and that’s awesome.
This two statements are causal to each other. And it actually gets them wrong with some frequency in ways that humans wouldn’t, forgets stuff it has already “learned”, or changes to opposite stances midways sentences. Because it is just an excel sheet on steroids.
It is, in my opinion, a shallow trick indeed.
“Excel sheet on steroids” isn’t oversimplification: it’s just incorrect. But it doesn’t really sound like you’re particularly open to honest discussion about this so whatever.
Well here’s the question. Is it solving them, or just regurgitating the answer? If it solves them it should be able to accurately solve completely novel analogies.
Novel analogies. Very easy to prove this independently for yourself.
yes, it’s has. the most famous example is the stacking of the laptop and the markers. you may not have access but it’s about to eclipse us imho. I’m no technological fanboy either. 20 years ago I argued that I wouldn’t be possible to understand human speech. now that is a everyday occurrence.
Depends on how you define understanding and how you test for it.
I assume we are talking LLM here?
Maybe if you Interpret it’s output as such.
It’s a tool. And like any tool it’s only as good as the person using it. I don’t think these people are very good at using it.
Too bad it’s bullshit.
If you are actually interested in the topic, here’s a few good reads:
Do Large Language Models learn world models or just surface statistics? (Jan 2023)
Actually, Othello-GPT Has A Linear Emergent World Representation (Mar 2023)
Eight Things to Know about Large Language Models (April 2023)
Playing chess with large language models (Aug 2023)
Language Models Represent Space and Time (Oct 2023)
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets (Oct 2023)
As you can see, the past year has shed a lot of light on the topic.
One of my favorite facts is that it takes on average 17 years before discoveries in research find their way to the average practitioner in the medical field. While tech as a discipline may be more quick to update itself, it’s still not sub-12 months, and as a result a lot of people are continuing to confidently parrot things that have recently been shown in research circles to be BS.