Jul 22, 2023
Huh. Thanks, Peter. I've seen the little demos where letters are changed randomly but we can still easily read the sentences. I guess you can still get a lot of the meaning when you drop a bunch of the tokens in a corpus. A large percentage of the dropped tokens are probably (probabilistically) articles, prepositions, adjective, and adverbs that aren't essential to the overall meaning of the text.