It depends on what you're allergic to and where you live.
If you have seasonal allergies or hay fever, tree pollens can trigger symptoms in the late winter or spring. Ragweed releases pollen in the summer and fall. The specifics also depend on where you live. Allergy season can start as early as January in Southern states and linger into November.
Putzing in the garden is nothing less than therapy. It's even good exercise, if you exert enough effort. But the sneezing and stuffy-headed feeling that lingers afterwards -- that's the downside of gardening with allergies.