The result added to a disappointing season for the north London club, who have won just two of their last 12 Premier League matches.
Mr Van de Ven said: "Gutted obviously that we’re out of the cup.
"First half nowhere near our level, nowhere near where we need to be and second half way better.
"I think we showed also some mentality, but if we showed it from the first minute, this game is totally different."
Tottenham saw the FA Cup as a chance to salvage their season, but first-half goals from Emiliano Buendia and Morgan Rogers secured Villa's progress to the next round.
Despite a stronger second-half performance and a goal from Wilson Odobert, Spurs were unable to mount a full comeback.
The defeat increases pressure on manager Frank, who has secured only five wins in his last 20 matches across all competitions.
Mr Frank said: "Of course that’s disappointing.
"We all know there’s only one way to have everyone happy.
"That is performing consistently and winning enough games.
"That’s the only way.
"And we could see, second half especially, the energy, how they feed off each other, the players and the fans.
"It was a fantastic experience to be in the middle of it.
"Unfortunately, we couldn’t get that fantastic comeback, which sometimes kick-starts a momentum and that’s what we are working very hard to do."
The manager now faces growing scrutiny from fans, who are frustrated by poor results and an inconsistent attacking approach.
Despite this, the club’s hierarchy currently continues to back Mr Frank.
Tottenham will host West Ham in the Premier League, before a crucial Champions League fixture at home to Borussia Dortmund.
The team continues to struggle with injuries. Dominic Solanke returned on Saturday to make his first appearance since August 23.
Injuries to Richarlison and others have only added to the team’s difficulties.
Mr Frank also criticised the fixture schedule, arguing it has played a part in his squad’s fitness issues.
He said: "I think it’s something for the football authorities to look into as well.
"We played five games in 13 days, four in 10.
"One of the few clubs who did that.
"This is our third time this season.
"We’re doing everything we can to compete in it, but that’s a tough schedule."