Do you think I would scare him away?

I don't think there is any harm at all in just adding him on facebook. Even if nothing comes out of it, there's always a chance that there is an attraction and the two of you may get talking again and who knows where that may lead. You have nothing to lose lovely :)