About this episode
Current text generators, such as ChatGPT, are highly unreliable, difficult to use effectively, unable to do many things we might want them to, and extremely expensive to develop and run. These defects are inherent in their underlying technology. Quite different methods could plausibly remedy all these defects. Would that be good, or bad?
https://betterwithout.ai/better-text-generators
John McCarthy's paper "Programs with common sense": http://www-formal.stanford.edu/jmc/mcc59/mcc59.html
Harry Frankfurt, "On Bullshit": https://www.amazon.com/dp/B001EQ4OJW/?tag=meaningness-20
Petroni et al., "Language Models as Knowledge Bases?": https://aclanthology.org/D19-1250/
Gwern Branwen, "The Scaling Hypothesis": gwern.net/scaling-hypothesis
Rich Sutton's "Bitter Lesson": www.incompleteideas.net/IncIdeas/BitterLesson.html
Guu et al.'s "Retrieval augmented language model pre-training" (REALM): http://proceedings.mlr.press/v119/guu20a/guu20a.pdf
Borgeaud et al.'s "Improving language models by retrieving from trillions of tokens" (RETRO): https://arxiv.org/pdf/2112.04426.pdf
Izacard et al., "Few-shot Learning with Retrieval Augmented Language Models": https://arxiv.org/pdf/2208.03299.pdf
Chirag Shah and Emily M. Bender, "Situating Search": https://dl.acm.org/doi/10.1145/3498366.3505816
David Chapman's original version of the proposal he puts forth in this episode: twitter.com/Meaningness/status/1576195630891819008
Lan et al. "Copy Is All You Need": https://arxiv.org/abs/2307.06962
Mitchell A. Gordon's "RETRO Is Blazingly Fast": https://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html
Min et al.'s "Silo Language Models": https://arxiv.org/pdf/2308.04430.pdf
W. Daniel Hillis, The Connection Machine, 1986: https://www.amazon.com/dp/0262081571/?tag=meaningness-20
Ouyang et al., "Training language models to follow instructions with human feedback": https://arxiv.org/abs/2203.02155
Ronen Eldan and Yuanzhi Li, "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?": https://arxiv.org/pdf/2305.07759.pdf
Li et al., "Textbooks Are All You Need II: phi-1.5 technical report": https://arxiv.org/abs/2309.05463
Henderson et al., "Foundation Models and Fair Use": https://arxiv.org/abs/2303.15715
Authors Guild v. Google: https://en.wikipedia.org/wiki/Authors_Guild%2C_Inc._v._Google%2C_Inc.
Abhishek Nagaraj and Imke Reimers, "Digitization and the Market for Physical Works: Evidence from the Google Books Project": https://www.aeaweb.org/articles?id=10.1257/pol.20210702
Get every episode summarized
Each time Fluidity publishes, we email you a written briefing from the transcript — the topics, who appeared, and any specific claims, with the ad reads skipped.
Email me new episodesFree for 3 shows. No card needed.
Hosts & guests
No transcript yet
This episode has not been transcribed. Request it and it moves to the front of the queue.
More episodes
