Could someone advise me about which of the two sentences have the correct verb agreement and why? Also, what is the subject of the sentence--is it "500 cm3" or "Herschelite"?
In this example, strictly it should be should be was, although some usages vary (see below).
The way to think about it is as if you were saying, ‘This quantity of Herschelite was hydrated.’
If you were to say, ‘500 cm3 of the Herschelite were hydrated,’ your statement works to distinguish these cm3s from other cm3s, which is not the point of what you are saying.
You might very well say that, ‘500 ducks were observed over several days,’ because the individual ducks then seem important. You might say that 500 kilos of something were treated in a certain way, if the important concept is that there only a limited number of kilos are available, of which these 500 went down a certain route, leaving whatever number remains.
This is a matter of perspective, and can get more uncertain in other cases. For example, an engineering firm might declare that 20 miles of motorway was widened last week, indicating that 20 miles is regarded as a single job, or a quantity of work to compare to other weeks (then standing for the idea ‘a total of 20 miles’).
A news programme, however, is likely to say that 20 miles of motorway were widened, given the likelihood of listeners picturing the motorway as a number of successive miles of road (and perhaps the entire length of the motorway seeming like a known target distance): in this case, the prevalent concept then becomes one of each ‘mile’ having been individually widened.
In the experimental context that you describe, however, if you are not discussing palpably discrete elements (ducks, ball bearings, cans of cat food) the convention is to say that ‘[a given quantity] was [treated in a certain way].’
