Model Description

Fairseq-dense 6.7B-Janeway is a finetune created using Fairseq’s MoE dense model.

Training data

The training data contains around 2210 ebooks, mostly in the sci-fi and fantasy genres. The dataset is identical as dataset used by GPT-Neo-2.7B-Janeway. Some parts of the dataset have been prepended using the following text: [Genre: <genre1>,<genre2>]

Limitations and Biases

Based on known problems with NLP technology, potential relevant factors include bias (gender, profession, race and religion).