The visitors fell behind from a corner in the first half when Fabian Schar headed in but Spurs were left furious as their players surrounded the referee and Frank confronted the fourth official.
As soon as it was on and Spence got up, Sandro Tonali swung the corner in before the Spurs man had got back into the box, and Schar - whom Spence was supposed to mark - headed home.
"Unfortunately, the ref was part of the mistake in the first one. I'm not saying there's things we can't do better, but his boot is off, we need to tie his shoelace and tie his boot.
"And you can clearly see on the footage afterwards that the whole boot is off, so we need to do it. And it's minute 23, it's not like we delayed the game for one minutes and we have one goal up. It's common sense to allow him time to get in. So as soon as he had the boot on, the ref let it go."
Frank admitted his side were to blame for the second goal, when Antonin Kinsky came for a cross and did not get there to allow Nick Woltemade to head into an empty net.
"So that's why we were very animated. I think that was the biggest one. I think the rest was fine. There's always small bits you can be irritated about, but it's very difficult to ref a game.
"But this for me was unfortunately a big mistake. And then we made a mistake ourselves on the second one. So small margins decided the game today."