What does it mean to say, "We're all at the same end of the world, and we've met before"?

This quote is from Pipa Xing (琵琶行) and its preface by Bai Juyi (白居易).

The author was relegated to a remote place and was depressed; he met a pipa girl who had been abandoned by a merchant, and learned that the other girl was in the same miserable situation, so his heart sank

The meaning is that we are all in the same boat, and since we meet, whether we have known each other before or not, we will feel sorry for each other and respond to each other's voices. So why should we care if we have known each other before?