Most people have the greatest difficulty with impromptu speaking. Even some of the most seasoned public speakers struggle when it comes to speaking off-the-cuff.
When a person has a well-crafted speech in hand which they know inside out and have delivered dozens of times, of course it makes sense that their speech comes across with confidence.
It's a completely different situation however when a person is called upon to speak extemporaneously on a topic they may know nothing about. To do this with an air of grace and authority, a skill set is needed which, in some ways, is quite far removed from the comforts of a written and rehearsed speech.
Here's an analogy:
Let's imagine you are camping in an area you know very well. It's a wooded area with many trees, almost forest-like. There's a stream a few hundred yards away, you're very well acquainted with the underbrush. During the day, when you need water, you wander down to the stream and fill up your container. No problem. You feel slightly vulnerable at times (there's a remote chance of wildlife in any wooded area), but not so much so that it inhibits your enjoyment of the camping experience.
The next weekend you join your friends for another camping experience. It's about 25 miles away from the spot you know. They're running late and it's well into dusk by the time you get there. By the time the tent is set up its completely dark. Now you are asked to go to the creek to get some water.
Same task. Same general area. Same tree's, same underbrush. Virtually everything is the same except for the absence of light.
Consider that light to be the comfort of a rehearsed speech.
The task is still attainable. Everything you need to have is still there. You just have to go about it in a different manner. More carefully, a little more slowly at first. You have to check-in with your intuition more readily.
Impromptu speaking still works within the basic framework of any other speech, yes. It requires, however, a keener sense of navigation. This can be strengthened through checking-in, not so much with the audience as you would with a rehearsed speech. Rather, it requires a checking-in with yourself. Allowing your intuition to guide you to a place where where you can find the clearing to give the audience something worthwhile they can take home.
There are several tips that can help as a gateway to master impromptu speaking. Mostly though, just allow your mind, and your heart-rate, to slow down.