It’s time to put this failed thought experiment to sleep for good.  Here’s a simple way to remove 99% of the cases:

Consider the color spectrum (ROYGBV).  Now consider a bizarro person who has two colors randomly flipped in their own private spectrum (such that they perceive ROYGBV to be RGYOBV).  WTF?  Now we have green in between yellow and red , when, by definition, any transition from red to green must include yellow.

Similar arguments exclude all but the complete inversion case (I got this far in fifth grade.  Science is finally catching up!)  And for that, we turn to Wikipedia:

Realistically, since the spectrum of color is only a section of the much larger electromagnetic spectrum, an inverted frequency of electromagnetic radiation (not just visible light) would cause catastrophic physical problems far greater than an altered perception of color. Most supporters of the “Inverted Spectrum” argument have an illogical understanding of colors and their corresponding frequencies.

And check out this ridiculous quote:

In his book I Am A Strange Loop, Douglas Hofstadter argues that the inverted spectrum argument entails a form of solipsism in which people can have no idea about what goes on in the minds of others– contrary to the central theme of his work. He presents several variants to demonstrate the absurdity of this idea: the “inverted political spectrum”, in which one person’s concept of liberty is identical to another’s concept of imprisonment; an inverted “sonic spectrum” in which low musical notes sound like “high” ones and vice versa (which he claims is impossible because low sounds can be felt physically as vibrations); and a version in which random, complex qualia such as riding a roller coaster or opening presents are reversed, so that everyone perceives the world in radically different, unknowable way

And then this guy completely runs away with it, and finally explains that crazy image I always see in the Mac color calibrator:

So is color real? Well, photons with specific wavelengths seem to correspond to specific colors. But the interior of the CIE 1931 color space is a representation of the a most ridiculously abstract concept, labels that aren’t even labels, something our brain experiences and calculates from averaged photon wavelengths. It is an example of what philosophers call qualia – a subjective quality of consciousness.