The boundaries are somewhat fuzzy and overlapping, so there's probably no hard and fast rule, but I think this is a question of gradability which applies to the sense of the word, rather than the word itself.
Some adjectives are non-gradable, e.g. annual, medical. Comparatives and adverbs of degree are not normally appropriate.
"Typical" adjectives are gradable: they express a quality that can be possessed in varying degrees; they can usually be modified by adverbs like "very" or "extremely", e.g. high, large, tired. Even these adjectives can often be treated as "binary" and can be negated, when there is an implied binary division of the scale in context, e.g. "not long", unclean, not tired.
And some adjectives are usually classified as "absolute" but have some characteristics of gradable adjectives: they may, for instance, be used to express an extreme condition and be gradable in the extreme, permitting adverbs like "absolutely", e.g. exhausted, dead, equal, essential, unique.
Other adjectives, like colours, may reflect the sub-division of a scale or scales (electromagnetic frequency, intensity), where the degree of approximation to an "ideal" may permit some gradability, e.g. "greener".
[Some ideas and examples from CGEL but the errors are mine]