Are all 88 constellations within the Milky Way?

The modern astronomical concept of constellations is simply the division of the sky into 88 different regions, borrowing the names of the ancient constellations (e.g., the large Andromeda galaxy M31, which you mentioned, is located in the Andromeda region). Each object in the region may be at a different distance from the Earth, just too far away for our naked eye to discern their relative distances.

Objects outside the Milky Way are simply too far away (such as the large galaxy M31 in Andromeda, which you mentioned, is 2.5 million light-years away), and because of the limited ability of our naked eye to discriminate between stars in extragalactic galaxies, even if we can see the galaxy (even if there is a supernova outburst in the extragalactic galaxy, the naked eye can see just a little bit of bright light, and can't distinguish details), the 88 constellations in the region of 88 constellations can be different distances from Earth. Therefore the stars visible to the naked eye in the 88 constellations can all be said to be within the Milky Way. It is rumored that the farthest star visible to the naked eye is HIP 5926 in Cassiopeia, which is 14,818 light-years away from Earth.