Outside of the semantic dementia literature, most of the discussion of conceptual/semantic organization in the brain centers around category specific deficits. There is, of course, good evidence for dissociations in the ability to perform tasks (typically, but not limited to, naming) involving one conceptual category versus another. The primary cleavage in conceptual categories in this research seems to be living things vs. artifacts.
The readings we surveyed provided nice overviews of both the data, and the various theories put forth to account for these data. Caramazza & Mahon (2003, TICS, 7:354-61), for example, discussed the relative strengths and weaknesses of the Sensory/Functional Theory (categories are organized around sensory and functional systems), the Domain-Specific Hypothesis (some categories are organized into separate modules as a function of evolutionary pressures to process information in those categories), the Conceptual-Structure account (a semantic feature-based account). Hart et al. (2007, J. Int. Neuropsych. Soc., 13: 865-80) provide an even more exhaustive review of the various theories, complete with handy crib sheet tables. And Martin (2007, Annu. Rev. Psychol. 58:25-45) gives a nice summary of the range of imaging data on the topic. The Caramazza & Mahon (neuropsych oriented) and Martin (imaging oriented) papers are particularly useful if you want to get a quick overview of the field.
The conclusion one gets though, after pouring over all these papers is that there are a number of different ways to account for the available data, and no obvious way to choose between them. For example, Martin summarizes an impressive range of findings from neuroimaging that seems to show that the same sensory-motor systems involved in processing a given bit of information is involved in representing that bit of information in conceptual memory. But from functional imaging alone, it is very hard to know whether these activations reflect the substrate for semantic memory, or merely the sensory-motor associates of the "real" concepts stored somewhere else. To compound matters, Caramazza & Mahon argue explicitly that the neuropsychological data do not support a sensory-functional hypothesis, and suggest instead that "the first-order constraint on the organization of conceptual knowledge is object domain [i.e., animal, fruits/veggies, conspecifics, a possibly tools]" p. 356. But even these authors admit a possible "fine-grained" organization involving other properties, or that there may be two independent organizational levels.
If you think about this issue hard enough, it all seems to come back to the question, what is a concept? If you believe a concept is nothing more than its sensory-motor/functional properties (for concepts that fit into those dimensions) then you can point to imaging data showing activation of the lip area of motor cortex when you read the word "kiss" and many other findings of this sort. If you believe a concept is something more complicated, you can point to complex patterns of associations/dissociations in the neuropsychological realm, and you could write off the imaging data as a peripheral association of the core of the concept: lip movements are associated with the concept KISS but don't define it, or even contribute substantively to it.
What kind of evidence do we need to really settle the issue? I would say that a convincing finding would be something like lesion (or TMS) evidence that when a person damages the lip area of motor cortex, they lose the concept KISS. Is there any such evidence out there?