why is <|startoftext|> not inserted when read in plain text files?

[fzhang612]

I am having some difficulities to understand the logic of load_dataset(). When it's csv file, every line is padded with start_token and end_token. However for other plain text file, only end_token is used. Is there any particular reason to omit start_token for plain text file?

Thanks

[minimaxir]

That is the way it was implemented in the original dataset the GPT-2 model was trained on. If you do not provide a prefix, it will start generating as if it was after an <|endoftext|> token.

Text immediately after <|endoftext|> can infer the "beginning" of text, but IMO it's cleaner to make that more explicit, especially if training on more document-oriented datasets like CSVs.

[fzhang612]

Thanks, that makes sense.

A follow-up question: currently, end_token is not inserted when the length of text is above combine. Should one end_token be appended here as well?

[woctezuma]

The initial question was related to #49.