I finally found an answer that makes a bit more sense. When doing one image I can get within 1ms by doing:
%%timeit
test = dls.test_dl([fnames[0]])
with test.fake_l.no_multiproc():
out = next(iter(test))
Which this makes much more sense on how to go about doing it using the library (and also can be a little bit faster than doing it my way.)