The 20-year-old striker was recalled from his loan spell at Oxford United on January 20, having scored four goals in 22 appearances across all competitions.
Just seven of his games were starts in Sky Bet Championship, while Scarlett featured in only three league games out of a possible six after Rowett was appointed U’s head coach.
READ ALSO: How to get unlimited access to all your U’s news
Scarlett has since netted his first goal for Spurs, in a Europa League win at home against Elfsborg, and was a substitute for Richarlison in a Premier League victory at Brentford earlier this month.
Having featured for Spurs, it meant Scarlett could only play for either Ange Postecoglou’s side or United between now and the end of the season.
Asked if there was a chance of Scarlett returning to the U’s following his recall, Rowett responded: “Decisions were made by other clubs. Decisions were made by us. You roll with that.
“At that point, we would’ve had potentially four or five players who can play as a number nine.
“Is there such a huge difference between all of those players in terms of one being so much better than the other? I’m not so sure.
“You can always look back when you pick an injury up and say what if, but ultimately you have to make those decisions, and also you would’ve ended up with two or three players, good players, out of the squad every week, and I don’t think that’s fair.
“It is what it is, and we’ll make do with what we’ve got.”