"To fathom" has a long history and several meanings, but the most common one lately is "to get to the bottom of, dive into, penetrate, see through, thoroughly understand."
I have never heard of "unfathom," and it is not to be found in the OED. I won't say it doesn't exist, because then someone would google it and find an infinite number of occurrences of it. Let's just say I'm out of my depth on that issue.
However, "unfathomable" and a few variations on it does exist and it means (of feelings, qualities, conditions, etc.) "incapable of being fully ascertained, explored, exhausted, etc."
Of space, the oceans, and other very large things, it means "incapable of being fathomed or measured; unsoundable, immeasurable, vast."
Do you require the derivation of the expression?