This repository contains the dataset splits used in Pitfalls of Static Language Modelling (Lazaridou, Kuncoro, Gribovskaya et al., 2021).

We provide splits of two public datasets used in the paper: WMT News Crawl and arXiv abstracts.

Each subset is stored on Google Cloud Storage as gzipped text file specifying publication dates (in the YYYYMMDD format) and IDs of documents contained in the subset.

arXiv abstracts and publication dates were obtained through arXiv's OAI-PMH service on January 2, 2021. We used the value in the created field as the article's publication date. The arXiv dataset can also be downloaded from Kaggle.

We downloaded document-split versions of the English and German WMT News Crawl dataset. As the dataset does not provide document IDs, we used SHA256 hashes of the Base64 encoded unsplit texts of articles as their IDs, i.e.:

import gzip
import hashlib

with gzip.open('news-docs.2007.en.filtered.gz', 'rb') as gz_file:
  for line in gz_file:
    date, sentence_split_text, unsplit_text = line.decode('utf-8').strip().split('\t')
    docid = hashlib.sha256(unsplit_text.encode('utf-8')).hexdigest()
    yield docid, (date, sentence_split_text, unsplit_text)

We trained models on sentence split article texts. Some articles may appear multiple times in the dataset with different publication dates; we used each article's earliest publication date.