I would mainly be concerned only if you noticed a significant change (i.e., decrease) in size.
When talking about normal/average testiclular size, it is most accurate to consider volume. Measurements can be taken using a ruler or with ultrasound. Then, the following equation can be used to calculate volume:
Volume = constant x length x width x height
(The constant depends on the shape of the testes; 0.52 or 0.71 are commonly used as compromise values for the constant. Current studies have shown that 0.71 is probably more accurate).
The average textbook values:
Average volume = 18.6 cm +/- 4.6 cm.
Average length = 4.6 cm (range 3.6 cm - 5.5 cm)
Average width = 2.6 cm (range 2.1 cm - 3.2 cm)
A recent study found that testicles of young men averaged 15 mL per testicle and those of older men were closer to 10 mL per testicle (they used the formula: Volume = 0.52 x length x width x height).
Testicular Volume in Relation to Hormonal Indices of Gonadal Function in Community-Dwelling Elderly Men -- Mahmoud et al. 88 (1): 179 -- Journal of Clinical Endocrinology & Metabolism
Testicles with a length of about 4 cm are within the normal range (and probably correspond to around a 13 - 16 mL volume depending on width). Plus it's also important to remember that testicles generally increase in size to some degree with arousal, so the measurement may differ depending on when you measure.