"Let's Be Friends"

It started Monday night. A older gentleman (between 50-70? best guess) came into our store. I greeted him and asked if he needed help - nothing out of the ordinary. He proceeded to tell me it was his first time in the store and why he came in. He was looking for a necklace to give a friend of his who had helped him through losing his wife two years ago. As I empathized with him, I gave examples of how I understood. This led to a lengthy conversation. During this conversation, he asked me my name (Rachel,) and I told him. He, then, asked for pen and paper. I wasn't sure what he was doing, but I gave it to him anyway. He asked me if I "was okay with this." I told him I didn't know, since I didn't know what he was doing. He wrote down his name and phone number and gave it to me. He told me he liked me and that he would love to keep talking and maybe form a friendship. He wanted me to text him that night so he knew it was me. I told him I would when I was ready (which would be never.) That conditional phrase was apparently lost in the communication. He asked me if it was too fast, too soon. I could do nothing but be silent. He walked around the store a little more and then "excitedly" came back up to me and asked me if I was working the following day. I told him "no." He was only slightly disappointed that his friend wouldn't be meeting me...
Fast forward four days. Friday night. About a half hour before we closed, this same gentleman entered the store. I quickly swapped places with my CLT and went to put things away in hopes of "avoiding" him. She said he sped up to catch up to me. He, then, proceeded to ask me if he had given me the right number. I told him I didn't know, I hadn't checked. He continued telling me he was concerned I hadn't been given the right number (since he hadn't heard from me yet.) I went up front to my CLT, asking her questions about my job at hand - legitimately - and then grabbed a box of items that were to go on the opposite side of the store. I wound up helping a woman over there until the man left. My CLT, then, notified me that this man had told her to tell me, "hello." I'm guessing his greeting to me previously wasn't sufficient?
I'm afraid this man will come back into the store and continue to try and "befriend" me. I'm very uncomfortable with how he has approached me and spoken to me. He had even acknowledged "Oh, you're probably working... BUT - " and hindered me from doing my job. This feels like a very gray situation where boundaries are being crossed. I don't know if I could've prevented this from happening or if it could've happened to any one of us. Should I have said "no?" Maybe. I don't know if that would have changed things. But I certainly didn't say "yes." And THAT is what should have been respected - not the "no," but the absence of "yes."