The whole premise was that the professor only gave a crap about him when he found out he was a math genius, this scene really shows that before he knew who he was he thought less of him because he was a janitor.