AI and it's effect on your music, your job and the future

[crushingpetal]

I missed this earlier, but I'm not sure all the text of the internet quite portrays a meaningful or complete view of the world. It certainly portrays a good chunk of how online folks like to talk about the world, but if we're scraping the reddits and x/twitters of the world for a large chunk of that, we're not exactly starting from a very reliable source of "understanding" the world to begin with.

If the goal is to plausibly sound like someone who would post on the internet, that might be enough, but I don't think that satisfies a model of understanding the world.

Again, +1. You're hitting it out of the park today. (Or pick your favorite sports analogy.)

 

[StevenC]

The bad parts of AI: planet is burned because of capitalist hellscape

The good parts of AI: given an infinite amount of time a monkey hitting a keyboard randomly will almost surely solve cancer

 

[narad]

Agree to disagree on these points.

>I think we have basically all the text-of-the-internet data we're ever going to need when it comes from gleaning an >understanding of the world from raw text.

Try a thought experiment on a scenario where we grabbed the same amount of text from people in the 1970s.

I don't quite understand. People in the 1970s had human level intelligence, no?

I missed this earlier, but I'm not sure all the text of the internet quite portrays a meaningful or complete view of the world. It certainly portrays a good chunk of how online folks like to talk about the world, but if we're scraping the reddits and x/twitters of the world for a large chunk of that, we're not exactly starting from a very reliable source of "understanding" the world to begin with.

If the goal is to plausibly sound like someone who would post on the internet, that might be enough, but I don't think that satisfies a model of understanding the world.

I didn't say that we had enough text to have an understanding of the world. I said we had enough text to gain as much of an understanding of the world as we can "from raw text", i.e., as much as we're going to need from that modality. The current / next-gen of models are getting a lot of their comparatively new data from other modalities directly (audio data in the case of GPT4o, transcriptions of audio in the case of a lot of LLMs, and video data in the case of sora and some gemini models). So if you're talking about model collapse, there's this assumption that the internet keeps growing in size, AI-generated content makes up a lot of it, and we still train naively and uniformly on that web scrape. In reality, the web scrape's usefulness is not going to keep growing proportional to its size, future models will be trained on huge scrapes of data in other modalities, models will be used to score the likelihood of data coming from previous models, and data will be sampled selectively to maximize the utility to the new model.

 

[TedEH]

It's very hard not to read your comments as coming from a place where you're so invested in seeing AI succeed that you don't want to consider any critical perspective. Which is fine, you do you. The cutting edge is exciting. Optimism is good. I'd actually really like to hear what critical takes could come from someone who is following the topic so closely and gets it. But it's got to be balanced out with at least some amount of staying critical or skeptical, because there's very likely a ceiling to this progress, or at least this rate of progress. Remember when blockchain was going to solve all the worlds problems? It solved exactly zero problems that it didn't create for itself.

Nobody is claiming that AI is useless, but it has its limits and its downsides, and will always have limits and downsides, and I think it would be naive and maybe dangerous to culturally dive head first into disruption without first grounding ourselves in the kind of skepticism that would prevent us from shooting ourselves in the foot with technology just for its own sake, as we have a habit of doing. When you "move fast and break things", there's a significant risk of breaking something very important.

there's this assumption that the internet keeps growing in size,

AI-generated content makes up a lot of it

and we still train naively and uniformly on that web scrape

I think these are all reasonable assumptions a person could make.

future models will be trained on huge scrapes of data in other modalities

What are these other modalities? If we're not feeding models more text, images, sound, etc. from the internet, what are we talking about? Setting up mics and cameras everywhere and letting the AI just sort itself out? Let it literally watch us and decide for itself? Like you keep saying, AI is not just chat bots - we've been feeding images, sounds, etc. into different models for a good amount of time now.

models will be used to score the likelihood of data coming from previous models, and data will be sampled selectively to maximize the utility to the new model.

This hasn't really worked so far. All evidence I've seen so far shows that AI is bad at detecting AI.

 

[gabito]

The good parts of AI: given an infinite amount of time a monkey hitting a keyboard randomly will almost surely solve cancer

That will never happen. Monkeys are too busy buying guitars they'll never use because they spend an infinite amount of time typing random stuff in guitar forums.

 

[SalsaWood]

The word "need" is carrying a lot of weight here. There's no need for most of the AI out there. ChatGPT is not a need. Smart appliances are not a need. Really advanced minecraft bots are not a need. 400 bazillion propaganda photos with too many fingers are not a need. Medical and science advancements could be had using machine learning without abstracting it out to chat bots and generalized job automation and trying to humanize computers.


No it hasn't. Not generally enough to put it that way. There are some folks trying to make this happen, or lying about how much "AI" is in their software, and they're being called out for it.

I mean, autocomplete is software and that's all these AI are. They fill in the blank or do generative modeling in different forms. That's what it all is. Algorithms within software don't make it an entirely new thing, or even a very different thing at all. Are you saying people are talking about it in such ways, but that they shouldn't because of some non-technical colloquialism?

 

[TedEH]

I mean, autocomplete is software and that's all these AI are.

Autocomplete, as I understand it, doesn't typically work using AI in any sense of the word - it's generally just a lookup tree of some kind. I've never heard anyone draw that comparison before - in part because it's not a good comparison.

Are you saying people are talking about it in such ways, but that they shouldn't because of some non-technical colloquialism?

I'm saying that fewer people than you implied are calling all algorithms AI, and those who are doing so are mistaken, whether it's literally, technically, or colloquially. It would be like calling your shoes a car. They both get you from A-to-B, but there's about as far as the comparison goes.

 

[SalsaWood]

Autocomplete, as I understand it, doesn't typically work using AI in any sense of the word - it's generally just a lookup tree of some kind. I've never heard anyone draw that comparison before - in part because it's not a good comparison.


I'm saying that fewer people than you implied are calling all algorithms AI, and those who are doing so are mistaken, whether it's literally, technically, or colloquially. It would be like calling your shoes a car. They both get you from A-to-B, but there's about as far as the comparison goes.

What is the literal difference?

I gather that you're saying all AI is software but not all software is AI. I think I get that far at least.

 

[narad]

It's very hard not to read your comments as coming from a place where you're so invested in seeing AI succeed that you don't want to consider any critical perspective. Which is fine, you do you.

I can understand why you might be considering that but I think the reality is different. I have four jobs: I work at a video game company, and I teach ML at the university. I also consult for a music company and for the US government. None of these is the hype-driven venture capital backed high-equity job that gives me any skin-in-the-game on whether AI "succeeds" or fails. For my purposes, the LLMs of today/recent future are more than enough to accomplish my company goals from a technological perspective. In fact, a fastly improving AI from some big-tech company only hurts me, as it minimizes my contributions, while also frustrating me that most of my peers and previous students have TCs in the $300-800k range while I don't because that's not how Japan rolls.

Also, you have to acknowledge that you setup a catch 22: the people who know the technology and its limitations the most are by virtue of that typically the most invested in its success. To draw an analogy, climate scientists are the most invested in people paying attention to depressing climate model predictions and warnings from the climate science community. They need grants to do work, the grants are largely contingent on government policy, government policy is often in large part dictated by public sentiment. This exact criticism has been raised by climate skeptics for decades. And in this case almost all AI experts are saying, yes, this technology will be disruptive and you would be justified to be worried about large scale job displacement. In this analogy you're basically saying, in as many articles you've read about how the weather works, and the fact you had a colder summer this year, that all this global warming stuff is extremely exaggerated.

Or even worse than that, because I'm not asking you to change your lifestyle or anything. I'm basically just recommending that people accurately appreciate the risks of AI impacting their jobs, or jobs in their industry, while simultaneously remaining open-minded about the positives of AI, which is poised to improve the advancement of civilization in many ways, even if not to the degree that I personally believe.

Of course I'm open to criticisms of AI (and acknowledge many current limitations -- i.e., my job for the most part), but they have to come from informed opinions. They can't come from pointing to glitched out images or talking about AI being autocomplete.

I think these are all reasonable assumptions a person could make.

The last one is not one that AI researchers are going to make. We're not going to be naively training on text uniformly in the future. There's currently many lines of research here, branching out in different ways, all moving away from this. So in that sense the rest of those assumptions don't matter.

 

[TedEH]

I gather that you're saying all AI is software but not all software is AI.

Nail on the head. I mean, it's in the name - any definition of what AI necessarily needs to be trying to simulate/emulate/appear intelligent.

Autocomplete, in a lot of implementations, doesn't fit any definition of AI. It's not learning, it's not planning, it's not emulating intelligence, it's not trying to appear intelligent or doing any language processing, etc., it really often just has a big ol' pre-sorted list of words in a big ol' tree and looks em up as you type. It can use AI, but it doesn't have to. And some of the things we call AI can be in part implemented by using a bunch of trees (especially if you're talking "video game AI"), but it's not the tree that makes it AI.

but they have to come from informed opinions. They can't come from pointing to glitched out images or talking about AI being autocomplete.

I'd love to point at better examples, but until we have them, it's speculation.

If something is poised to enter into everyone's daily life, I don't think we can meaningfully benefit from it in an "advancement of society" kind of way if nobody understands it, let alone barely being able to identify what and where it is. Say, for example, that crypto had proven to actually be a much better form of currency instead of a garbage one - nobody understood it. It was impenetrable for the layman. Joe Blow who needs to pay his landlord is not equipped to figure out how to get crypto currencies out of his "wallet" (if he even understands what a wallet is in this context). In the same way, AI that isn't packaged up into a product someone can just "prompt engineer" or whatever isn't meaningful to the average person.

It's convinced it's going to be stuck, for a good while, relegated to being useful in niche applications to those who have the kind of specialist knowledge it takes to actually use it. Just like every technology. And a bunch of generative slop.

The last one is not one that AI researchers are going to make. We're not going to be naively training on text uniformly in the future.

But it has to train on something, and that data has to come from somewhere. I don't understand what you're claiming these new sources of data will be. It's not about the form of the data, it's about the source. The same criticism has been leveled at art generation, because once you've trained on all the available art, where do you go from there? You're permanently stuck in a loop of derivative work at that point. You can't just say "well, at that point, AI understands art well enough, so we don't need new art". That's just a nonsensical statement, because it defies what art is. And it's not like sound generation is faring any better.

 

[narad]

But it has to train on something, and that data has to come from somewhere. I don't understand what you're claiming these new sources of data will be. It's not about the form of the data, it's about the source. The same criticism has been leveled at art generation, because once you've trained on all the available art, where do you go from there? You're permanently stuck in a loop of derivative work at that point. You can't just say "well, at that point, AI understands art well enough, so we don't need new art". That's just a nonsensical statement, because it defies what art is. And it's not like sound generation is faring any better.

It doesn't necessarily need more data to be better in a certain modality. Like in text, you have the sum of mankind's knowledge on the internet now. If you can't figure out how to read and reason from that much signal, the problem is the model and not the data. So there is work that tries to optimize which examples are chosen, then there is work that tries to understand more general relationships between data in a different way (open-endedness), and then there is work on utilizing the model to generate its own training data. There is progress in all of these fields.

Open-endedness in particular -- when people talk about stuff like art and think, well, once you've trained on all the available art, where do you go from there? Yea, (a) that's a lot of art and it's going to cover generative capabilities for most use cases, but (b) when you talk about innovating within the art space, that is open-endedness, and LLMs are an important breakthrough for making those systems work, as they are good general models for scoring what is valuable to humans.

And then there is also data from audio and video which is not going to be saturated with AI-generated content, or indiscernibly so, for a long time.

So again, from the pov of some random reporter doing an AI hit-piece, these questions might sound like some effective criticism. But from within the AI community, these things are all plodding along, and have been for a while.