Yield maps give farmers information about growth conditions and can be a tool for site-specific crop management. Combine harvesters may provide farmers with detailed yield maps if there is a constant flow of a certain amount of biomass through the yield sensor. This is unachievable for grass seeds because the weight of the intake is generally too small to record the variation. Therefore, there is a need to find another way to make grass seed yield maps. We studied seed yield variation in two red fescue (Festuca rubra) fields with variation in management and soil fertility, respectively. We estimated five vegetation indices (VI) based on RGB images taken from a drone to describe yield variation, and trained prediction models based on relatively few harvested plots. Only results from the VI showing the strongest correlation between the index and the yield are presented (Normalized Excess Green Index (ExG) and Normalized Green/Red Difference Index (NGRDI)). The study indicates that it is possible to predict the yield variation in a grass field based on relatively few harvested plots, provided the plots represent contrasting yield levels. The prediction errors in yield (RMSE) ranged from 171 kg ha−1 to 231 kg ha−1, with no clear influence of the size of the training data set. Using random selection of plots instead of selecting plots representing contrasting yield levels resulted in slightly better predictions when evaluated on an average of ten random selections. However, using random selection of plots came with a risk of poor predictions due to the occasional lack of correlation between yield and VI. The exact timing of unmanned aerial vehicles (UAVs) image capture showed to be unimportant in the weeks before harvest.