The Tottenham full-back was withdrawn just before the hour mark in the 3-0 defeat to Nottingham Forest at the City Ground.
Spurs were by that point two goals down and well on their way to a chastening loss, one that piles the pressure back on Thomas Frank.
Spence, who had been struggling with a knock before the break but appeared to run it off, was visibly frustrated as his number went up on the board.
After briefly acknowledging the applause from the fans of his former club, Spence then trudged to the bench and threw the jacket handed to him to the floor.
Frank was unaware of that reaction at the time but confirmed he would discuss it with Spence.
“You're the third one that said that to me today,” Frank said at his press conference after the match. “I didn't see it. Of course I will look back to be aware of it.
“I think there can be three reasons. He can be disappointed with his own performance, the team's performance, he can be disappointed with being subbed off.
Pedro Porro had to be told to calm down by Frank after squaring up to Elliot Anderson before half-time.
“I think in general, no matter what, to have that coolness on the pitch and relentless concentration and focus, that's a big thing of being a top team.”