Sagging around eyes happens as we age due to volume loss. However, in some instances it could also be due to the side and upper muscles surrounding the eye are always working against one another. This pulls eyelids in a downward direction gradually..