Sure! Here is the script you requested:
```
import torch
from transformers import BertTokenizer, BertModel
from transformers.training import Trainer
from torch.utils.data import DataLoader
from datasets import load_dataset
# Load the IMDB dataset
train_dataset = load_dataset("imdb", split="train")
val_dataset = load_dataset("imdb", split="val")
# Load the BERT model
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")
# Define the training and validation dataloaders
train_dataloader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_dataloader = DataLoader(val_dataset, batch_size=32, shuffle=False)
# Define the training arguments
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3,
per_device_train_batch_size=32,
per_device_eval_batch_size=32,
evaluation_strategy="epoch",
learning_rate=5e-5,
save_total_limit=2,
save_steps=500,
load_best_model_at_end=True,
metric_for_best_model="f1",
greater_is_better=True,
save_strategy="steps",
save_on_each_node=True,
)
# Train the model
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=val_dataset,
compute_metrics=lambda pred: {"f1": f1_score(y_true=pred.label_ids, y_pred=pred.predictions.argmax, average="macro")},
)
trainer.train()
# Print the final trained model
print(trainer.model)
```
To launch the script on the command line, you can use the following command:
```
python -m torch.distributed.launch training_bert_imdb.py --n_gpus 4
```
This will launch the script on four GPUs. You can adjust the number of GPUs as needed.
I hope this helps! Let me know if you have any questions.