A spirited second-half rally could not prevent Spurs suffering another early exit in a domestic cup to increase the pressure on Frank, who has lost seven of his last 13 matches after this 2-1 defeat.
Frank had endured a nightmare week after he inadvertently used a coffee cup with the logo of Spursâ rivals Arsenal on before a 3-2 loss at Bournemouth on Wednesday, and first-half goals by Villa duo Emiliano Buendia and Morgan Rogers denied him a much-needed cup run.
Wilson Odobert reduced the deficit in the 55th minute and Spurs created further second-half chances, but they ultimately lost again to spark more boos for Frank with Villaâs away support poking fun at the Danish coach with chants about him being an Arsenal fan.
Tempers spilled over at full-time after Frankâs old Brentford forward Watkins celebrated in front of Palhinha, who went on to clash with Morgan Rogers and Ezri Konsa as players from both teams piled in to produce ugly scenes, but Villa progress.
âWe are all hurt. There is nothing we want to do more than to get through to the next round no matter who we face. We play at home, we had a good opportunity and unfortunately we couldnât go through,â Frank said.
Quizzed on if his players were losing control after they clashed with Tottenham fans away to Bournemouth, Frank said: âI donât think so.
âI guess you have seen the situation through? I just saw it to be sure I saw what I saw. Of course itâs all about keeping a cool head.
âThe players gave everything, everything out there, (but) losing a tight game, season not going perfect and I think Ollie is very provoking.
âThe way he is going down to celebrate in front of the Villa fans and he is walking into Joao, he can just easily walk around. I think everyone that has been in a competitive nature, that is difficult and can trigger things.
âYou need to play with passion and when you play with passion then sometimes you play to the line and sometimes a little bit borderline.
âI see players with big passion that want to do very well for the club.â
Unai Emery made six changes after a gruelling schedule for Villa, but they sealed a 13th win in 15 matches.
âAfter 90 minutes, with our momentum, with their momentum, I think we deserved to win,â Emery saidbefore he decided to praise referee Craig Pawson.
âOf course we must respect the referee, we must respect the competition. Only we were a little bit angry because (Boubacar) Kamara was getting injured with one action with Palhinha, but I think the referee today did a fantastic job.
âSome moments happened, with some players, and I think we can understand in the circumstances it happens on the field.â