So Long And Thanks For All The Tokens

I had the priviledge of attending some cool talks on GPT-3 at the Startup Village in Amsterdam last week. I won't name-and-shame but there were three speakers who gave their personal take on the opportunities and challenges posed by this technology and while some of the discussions were very technical, their bottom line was that GPT-3 is just a fancy word generator—not necessarily any more disruptive than predictive-text software.

As far as these speakers believed, Large Language Models (LLMs) are not capable of cognitive reasoning or constructing logical arguments, a crucial requirement for true artificial general intelligence. This came as a surprise to me, as my experience thus far has been wildly more interesting. Maybe I've just dug deeper, and have smarter and more open-minded people around me, so wanted to give you my take:

Is it really "intelligent?"

It is of course true that an LLMs basic function is to predict the next word in a string of text, and that seems so incredibly arbitrary. However it doesn't take much experimentation with ChatGPT to see a few example responses which are eerily intelligent and I find it difficult to reconcile as simply generated language magic. The more I work with these models, the more I realize that these models are probably closer to the structure of the human brain than we think.

GPT-3 uses a sophisticated probabilistic model based on a neural network of weighted synapses to link "tokens" (words or concept) together. When given an input like "the cat sat on the ..." the model predicts the most likely next word based on probabilities (e.g., "mat" with 99% likelihood, "floor" with 0.5%, "carpet" with 0.4%, etc.), and that generally comes up with approximately the right response.

While GPT-3 uses words to communicate the internal structure of language stored in its model, us humans use words to communicate the internal structure of concepts stored in our own brains. Both of these these structures could be considered conceptual knowledge in some sense, and if we look deeper into the neural network we may even be able to find particular concepts represented physically in the shape of the model too. So, although GPT-3 uses connected tokens to represent "concepts", the linkage and traversal of these leads to it being able to produce "ideas" in an unstructured way (probably not dissimmiar to the way our brains traverse and weave together coherent thoughts!).

An example of this could be seen in GPT-3 being able to play chess at a fairly decent level. It does not have a chess engine embedded in it, and it has only even been exposed to chess through reading the rules and seeing a number of game being played. However, there is evidence to suggest that GPT-3 is coming up with novel strategies to outwit opponents which can only be described as being logical or strategic in nature. If we looked deep enough, would we find a simple chess engine encoded deep inside the internals of the language model? My money is on yes, and that should be pretty mindblowing for anyone looking for evidence of general intelligence. We may even find the emergence of regions in these huge models that are similar to other functional regions of our brain: a visual cortex, and audio cortex; and structures like the neocortex too.

The Most Sophisticated Goldfish Ever Built

The biggest constraint in my opinion is not whether GPT is the correct computational architecture, but rather its very small working memory (which we can call "contextual memory"). For the latest GPT-3.5 family this is limited to approx 4,000 "tokens".

The tiny token limit is unfortunately a massive limitation for use-cases which require large amounts of context and an area in which humans are fairly proficient. Although humans don't have endless working memory either (maybe even less than 4000 tokens!), we are able to build systems to manage and extend this working capacity—whereas GPT-3 doesn't yet have a reliable way to achieve this. This constraint means working with GPT-3 can sometimes feel like you're working with the actor trapped in the movie "Groundhog Day", combined with the plot of "Memento" where every few iterations you have your memory wiped and hope that you've written enough down to figure out where you were the next day.

There are however some "hacks" that attempt to compact and summarize previous responses to keep memory usage low, and attempt to create feedback loops to retain information in working memory. However, these comes with a cost, reducing the model's impact in various use-cases such as large project code generation, or document understanding, or contextual search. GPT-3 cannot easily build a holistic view of the subject and forces prompt engineers to focus on smaller, more incremental approaches rather than unleashing the model's full potential on more adventurously sized tasks. The model's limited memory capacity can lead to some crazy responses (inclusive of accompanying mad-scientist laughter), adding an unpredictable and frustrating element to the experience.

TL;DR (and less than 4000 tokens I promise)

While some experts may view GPT-3 as nothing more than a sophisticated word generator, it's clear that this tech is here to stay and will be finding its way into everything. As developers we will be working more and more with large-language-models, and we will need to become proficient in architecting, designing and building systems which integrate with them.

The ecosystem is exploding, and its worth keeping an eye on projects like langchain and LlamaIndex to see examples of how the community is building interesting tools for managing the complexity and non-determinism in these use-cases. Its entirely possible that these tools are very short-lived, and replaced by much better architectures, or even fall away entirely as the models become more capable and less management is required. An exciting, scary, time ahead for builders!

BREAKING NEWS: GPT-4 - announced yesterday - raises the token limit to between 8k and 32k (equivalent to about 52 pages)! Another huge step forward and expect see a much more powerful use-cases with the ability to understand much larger tasks.

Article
  { slug: "so-long-and-thanks-for-all-the-tokens"
  , originalUrl: Nothing
  , title: "So Long And Thanks For All The Tokens"
  , date: constructDate 2023 3 15
  , keywords: ["GPT","AI","context"]
  , cover: { alt: "Code", caption: Nothing, src: "/images/covers/code.png" }
  , summary: "GPT-3, intelligence, and the most sophisticated goldfish ever built."
  , author: { bio: "Towards an AI-first, hyper-scaled, functional reactive universe!", email: "tom@tomwells.org", name: "Tom Wells" }
  , body:
      [ FlowParagraph "I had the priviledge of attending some cool talks on GPT-3 at the Startup Village in Amsterdam last week. I won't name-and-shame but there were three speakers who gave their personal take on the opportunities and challenges posed by this technology and while some of the discussions were very technical, their bottom line was that GPT-3 is just a fancy word generator—not necessarily any more disruptive than predictive-text software."
      , FlowParagraph "As far as these speakers believed, Large Language Models (LLMs) are not capable of cognitive reasoning or constructing logical arguments, a crucial requirement for true artificial general intelligence. This came as a surprise to me, as my experience thus far has been wildly more interesting. Maybe I've just dug deeper, and have smarter and more open-minded people around me, so wanted to give you my take:"
      , FlowSection "Is it really \"intelligent?\""
      , FlowParagraph "It is of course true that an LLMs basic function is to predict the next word in a string of text, and that seems so incredibly arbitrary. However it doesn't take much experimentation with ChatGPT to see a few example responses which are eerily intelligent and I find it difficult to reconcile as simply generated language magic. The more I work with these models, the more I realize that these models are probably closer to the structure of the human brain than we think."
      , FlowParagraph "GPT-3 uses a sophisticated probabilistic model based on a neural network of weighted synapses to link \"tokens\" (words or concept) together. When given an input like \"the cat sat on the ...\" the model predicts the most likely next word based on probabilities (e.g., \"mat\" with 99% likelihood, \"floor\" with 0.5%, \"carpet\" with 0.4%, etc.), and that generally comes up with approximately the right response."
      , FlowParagraph "While GPT-3 uses words to communicate the internal structure of language stored in its model, us humans use words to communicate the internal structure of concepts stored in our own brains. Both of these these structures could be considered conceptual knowledge in some sense, and if we look deeper into the neural network we may even be able to find particular concepts represented physically in the shape of the model too. So, although GPT-3 uses connected tokens to represent \"concepts\", the linkage and traversal of these leads to it being able to produce \"ideas\" in an unstructured way (probably not dissimmiar to the way our brains traverse and weave together coherent thoughts!)."
      , FlowParagraph "An example of this could be seen in GPT-3 being able to play chess at a fairly decent level. It does not have a chess engine embedded in it, and it has only even been exposed to chess through reading the rules and seeing a number of game being played. However, there is evidence to suggest that GPT-3 is coming up with novel strategies to outwit opponents which can only be described as being logical or strategic in nature. If we looked deep enough, would we find a simple chess engine encoded deep inside the internals of the language model? My money is on yes, and that should be pretty mindblowing for anyone looking for evidence of general intelligence. We may even find the emergence of regions in these huge models that are similar to other functional regions of our brain: a visual cortex, and audio cortex; and structures like the neocortex too."
      , FlowImage { alt: "GPT plays chess", caption: (Just "Screenshot of Zack Witten (@zswitten), 2 March 2023."), src: "/images/articles/so-long-and-thanks-for-all-the-tokens/gpt3-chess.png" }
      , FlowSection "The Most Sophisticated Goldfish Ever Built"
      , FlowParagraph "The biggest constraint in my opinion is not whether GPT is the correct computational architecture, but rather its very small working memory (which we can call \"contextual memory\"). For the latest GPT-3.5 family this is limited to approx 4,000 \"tokens\"."
      , FlowParagraph "The tiny token limit is unfortunately a massive limitation for use-cases which require large amounts of context and an area in which humans are fairly proficient. Although humans don't have endless working memory either (maybe even less than 4000 tokens!), we are able to build systems to manage and extend this working capacity—whereas GPT-3 doesn't yet have a reliable way to achieve this. This constraint means working with GPT-3 can sometimes feel like you're working with the actor trapped in the movie \"Groundhog Day\", combined with the plot of \"Memento\" where every few iterations you have your memory wiped and hope that you've written enough down to figure out where you were the next day."
      , FlowParagraph "There are however some \"hacks\" that attempt to compact and summarize previous responses to keep memory usage low, and attempt to create feedback loops to retain information in working memory. However, these comes with a cost, reducing the model's impact in various use-cases such as large project code generation, or document understanding, or contextual search. GPT-3 cannot easily build a holistic view of the subject and forces prompt engineers to focus on smaller, more incremental approaches rather than unleashing the model's full potential on more adventurously sized tasks. The model's limited memory capacity can lead to some crazy responses (inclusive of accompanying mad-scientist laughter), adding an unpredictable and frustrating element to the experience."
      , FlowSection "TL;DR (and less than 4000 tokens I promise)"
      , FlowParagraph "While some experts may view GPT-3 as nothing more than a sophisticated word generator, it's clear that this tech is here to stay and will be finding its way into everything. As developers we will be working more and more with large-language-models, and we will need to become proficient in architecting, designing and building systems which integrate with them."
      , FlowParagraph "The ecosystem is exploding, and its worth keeping an eye on projects like langchain and LlamaIndex to see examples of how the community is building interesting tools for managing the complexity and non-determinism in these use-cases. Its entirely possible that these tools are very short-lived, and replaced by much better architectures, or even fall away entirely as the models become more capable and less management is required. An exciting, scary, time ahead for builders!"
      , FlowRichParagraph [ InlineEmphasis "BREAKING NEWS: GPT-4 - announced yesterday - raises the token limit to between 8k and 32k (equivalent to about 52 pages)! Another huge step forward and expect see a much more powerful use-cases with the ability to understand much larger tasks." ]
      ]
  }