that my name may be there forever

"that I may be there forever" or "for people to worship me there forever"

My eyes and my heart will be there every day

"I will watch and protect it forever". This implies that he will protect his temple.