From a time standpoint, no. Just do a break even analysis of what you would be saving vs. any out of pocket expenses. I have lived in my home for almost 2 years and have already done 2 refinances and am on the verge of pulling a 3rd after the rates dropped from the Brexit vote. I always do a slightly higher rate with a no cost refi since they typically ladder the different rate tiers at an 8 year break even.
Agreed with
@ekREV98
. Go for the 20. You're used to making the payment now. Don't go backwards and reset the clock when you can lock in a great rate.
If I had to bet, I would say rates will go even lower within the next year, so you might even be able to get a better deal if the economy goes south.