Although color term ‘Gold’ is commonly used, traditional color science cannot deal with ‘Gold’ because there is no region corresponding to ‘Gold’ in the chromaticity diagram generated based on the color matching experiments. Appearance of an object changes from ‘Yellow’ to ‘Gold’ with an increase in the specular reflectance, and understanding how we discriminate ‘Gold’ from ‘Yellow’ is tightly related to an important problem of how we perceive surface reflectance or gloss of objects. To understand neural processes underlying gloss perception, we conducted a series of experiments. When we compared neural activities evoked by objects with specular and matte surfaces using functional magnetic resonance imaging in monkeys, stronger activities to specular surface were observed in areas along the ventral pathway of the visual cortex including the inferior temporal (IT) cortex that plays an essential role in object discrimination. We also recorded single neuron activities from the IT cortex and found that there exist neurons that are selectively responding to specific gloss, and that as a population, these neurons systematically represented a variety of glosses. We speculate that visual features distinguishing surface glosses are detected in early visual areas and this information is integrated along the ventral visual pathway to form neural representation of a variety of glosses of object images in the IT cortex. Neural mechanisms underlying discrimination between ‘Gold’ and ‘Yellow’ should at least in part lie in this process.