The older I get, the more I appreciate what BS it is to think other people are these magical mind readers. I try to combat it myself by making more of an effort to tell people what's going on in my head, particularly when it's something nice about them.
This also reminds me... my mom is so convinced that she's so good at this that, when I still lived at home, she would be certain I'd be upset about something and would ask me "what's wrong?" over and over until the thing that was wrong was her bugging me about something being wrong. She also thinks my dad secretly ~knows~ what she wants him to do, and then vents to me about how he doesn't do those things, and boy howdy, thinking other people just understood things was such a fuckup mistake I made in my earlier relationships.
Anyway, tangent aside, I think even people who are good at reading social cues in a very specific setting don't actually pick up on this broad stuff either. I mean, you most likely did a lot of things just living your life, how tf are you supposed to know which, specifically, set someone off? Unless "you know what you did" is being made as a shocked exclamation to you sharing that you just did a fucked up thing, I don't think most people actually magically know either.