Why is Searle's Room the seemingly de facto thought experiment for this stuff? You always have to take an extra couple of steps to make it work, and it never really sells it to me. Its true he was distinctly concerned with AI, but he was not at all speaking to the same context, i.e., before "the bitter lesson."
I humbly suggest looking into guys more like Quine. His problem of "radical translation" is much more easily mapped to LLMs. (Thinking specifically here of the model as the "translator"). Its maybe a little harder to grasp for non-domain experts, but at least there is no need for hyperstitional armchair interpretations of old problems in order to make it relevant.
People jump straight into cognitive science/philosophy with this stuff, I just want to be like "whoa, slow down! So much to establish before that.."
I humbly suggest looking into guys more like Quine. His problem of "radical translation" is much more easily mapped to LLMs. (Thinking specifically here of the model as the "translator"). Its maybe a little harder to grasp for non-domain experts, but at least there is no need for hyperstitional armchair interpretations of old problems in order to make it relevant.
People jump straight into cognitive science/philosophy with this stuff, I just want to be like "whoa, slow down! So much to establish before that.."