• Even_Adder@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    5
    ·
    1 year ago

    It can’t see what tokens it puts out, you would need additional passes on the output for it to get it right. It’s computationally expensive, so I’m pretty sure that didn’t happen here.

    • Jesusaurus@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      With the amount of processing it takes to generate the output, a simple pass over the to-be final output would make sense…

      • adderaline@beehaw.org
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        it chunks text up into tokens, so it isn’t processing the words as if they were composed from letters.