I think that when this is said, there is some sense that one has gone a little bit beyond the boundaries, just enough to cause comment or subtle criticism. If a father tells his teenage son "You're really pushing the boundaries!" he is implying that while the son isn't in big trouble yet, if he keeps going, he will be.
I sort of see it as a balloon. You can press against the balloon and cause it to expand and as long as you don't push too hard, it comes back to its original shape. If one pushes too hard, the balloon bursts.
So while you may have pushed slightly beyond the boundaries/limits, causing comment, no harm is done yet. Generally staying entirely inside the limits wouldn't trigger this kind of comment.
In other circumstances, one might say, "One thing I like about
Ted is that he's always pushing the limits. Jane would never have altered the leave policy if Ted hadn't asked for so many personal days." In this case, going slightly beyond what is strictly permitted, or pushing the envelope, causes the rules to be changed.
This is just my feeling as a native speaker, it's not based on an analysis of actual usage or definitions.