In this study, we draw a theoretical model to demonstrate that small farms achieve lower total factor productivity (TFP) compared to large farms, even though their yield may be higher. We argue that taking into account family labor modifies the farm size-productivity relationship. We test our hypotheses on geocoded data from 5,645 agriculture farms in Pakistan using Pakistan household integrated economic survey 2018-19 and labor force survey 2018 combined with remote sensing data to account for farm-specific topographic features. We base our analysis on OLS and stochastic frontier analysis. We find that family labor is the key to understanding the nature and strength of the farm size-productivity relationship. Farm size's association, both with yield and TFP, turns positive when we measure family labor in terms of market wage rate rather than marginal product. Farm yield decreases by-0.07% with a one percent increase in farm size but gets insignificant or increases by 0.034% when family labor cost is measured at market wages rather than the marginal product. We find that higher family labor intensity, labor market distortion due to the notion of family dishonor, and suboptimal crop selection by small farms play a crucial role in this context.