You can pass your fine-tuned .ckpt as the base_model.
okaris
/
controlnet
ControlNet implementation with custom SD1.5 fine tuned models