sand

(picture by: http://sheikhrajput.blogspot.com/2009/11/from-grains-lost-in-time.html)

If you let me go, I will go

Don't worry, I'm not the mud

I'm just  the sand

Then, you will never get it back