CAMVID_TINY batch size limitation?

When working through the CAMVID_TINY demo, I tried doing it from scratch but was getting terrible results. After working my way backwards from the implementation exampled in the docs I realized that the only difference causing the poor training was not setting my batchsize to 8 when creating a dataloader
dls = dblock.dataloaders(path/"images", path=path, bs=8)
I assumed that a larger batch size would not harm performance?