Soil organic carbon (SOC) is broadly recognised as an important parameter affecting soil quality, and can therefore contribute to improving a number of soil properties that influence crop yield. Previous research generally indicates that soil organic carbon has positive effects on crop yields, but in many studies it is difficult to separate the effect of nutrients from the effect of SOC in itself. The aim of this study was to analyze whether the SOC content, in itself, has a significant effect on potential yields of commonly grown cereals across a wider range of soil types in Denmark. The study draws on historical data sets from the Danish national field trials consisting of 560 winter wheat (Triticum aestivum L.) trials and 309 spring barley (Hordeum vulgare L.) trials conducted over the past 20 and 17 years, respectively. We hypothesised that for these two crops, the potential grain yield, the yield with no fertiliser N application and the N use efficiency would be positively affected by SOC level. A statistical model was developed to explore relationships between SOC and potential yield, yields at zero N application and N use efficiency (NUE). The model included a variety of variables and aimed to elucidate the sole effect of SOC by controlling for potential confounding variables. No significant effect of SOC on potential winter wheat was found, whilst for spring barley, only for the course sandy loam soil type was a borderline significantly positive effect of SOC on potential yields found. The relationship between unfertilized plot yields and SOC was positive for winter wheat, although not significant, whilst for spring barley a significant positive effect of SOC was found only for the coarse sandy soil type, and a borderline significant positive effect of SOC was found for the coarse sandy loam soil type. A significant negative relationship was found between SOC and NUE for both winter wheat and spring barley. Based on the large dataset analyzed, we cautiously challenge the importance of SOC in contributing to crop productivity in contexts with similar soils and climate, and we speculate that in situations where nutrient limitation does not occur, SOC levels above 1% may be sufficient to sustain yields. In light of the findings presented in this study, further work should be conducted which can further elucidate the effect of SOC on yields.