The challenge, I believe is that the more you say it is no big deal, the more you mean it is a big deal. This was not a conversation to have over text. It was one to have in person or by phone. 2 am? Oh sweetie!
Also , he knows you like him more than the fwb, so you are compromising yourself. Good men will not mess around with women who like them more.
He is showing you with his non response that he is not interested.