Why would a church leader say this to me? It really upset me!?

Im 23 and I go to church on sundays. I used to go to church more often (On thursday and fridays) but since I am very busy with school and other studies I am only able to attend church on sundays. So today after church I come up to a friend (same age as me) that i used to hang out a lot (he's married now so i don't see him often) and just wanted to say hello and tell him "God bless you" He currently more involved in church than me. So i shake his hand and say hello and he just holds it and tells me "hmmmm...Why do i not see you at church on thursdays and fridays, you keep saying that you want to grow spiritually but I don't see that!" And i got so upset that he would have the nerve to say that to me. I told him "just because i don't go church more doesnt mean im not growing spiritually" plus nowhere in the bible it says that your christianity is measured by how many times a week you go to church. This is not the first time he has said this to me. It makes me feel like he thinks im completely unchristian!