Actually, the empathy could be what sets Brandon off, since Loki is kind of a crazypants mess. So as Agent Ginna, the deeper emotions are only very rarely going to match what appears to be going on.

Though really at SHIELD, most of the time Loki is just having to try his hardest not to burst out laughing constantly. When he's not being a completely cold fish. But arguably, the emotions might feel a bit different from a normal human anyway.