Peanut butter & jelly cupcakes can stay at room temperature for a day or two, since they will not go bad but rather soft.
For longer storage keep refrigerated for up to 5 days, I like to bring them to room temperature before serving.
Freeze (iced or un-iced) for up to 1 month wrapped well