If it happens only occasionally, it's not likely to be serious. All men have problems with erections at some time in their lives.
If it slowly but consistently gets worse, there's probably a physical cause. This is generally what happens with chronic impotence.
If it happens suddenly but you're still stiff early in the morning and can get an erection while masturbating, that suggests your mind is involved. There could be something going on physically, too.
When to Call Your Doctor
Pick up the phone if your ED worries you so much that it causes anxiety or threatens your sexual relationship. At the very least, your doctor can clear up misinformation, which often makes sexual problems worse. Sometimes taking medication for a short time can get you through a rough patch, too.
To help your erections, your doctor may suggest taking medication as a pill or as an injection in your penis, or using a mechanical device. And you'll need to deal with the underlying medical condition, too.