1st one needs to understand the makeup of the fishery you are on. If the cycle is balanced and the water has good populations of all size fish then there is no reason to keep the small ones. But if it is out of wack and there are abnormal amounts of small fish then there is a problem. Either fishing pressure is eliminating the large fish or there is a natural reason the fish do not attain size. Such as lack of forage, repeated winter kills, or a lack of preditory fish to reduce the population of smaller fish which all compete for the same food.
An example of a few small lakes in my area. These are small bodies, less than 150 acres, they are shallow and have real problems with over vegetation throughout the lake except in many of the sandy bottom areas that are less than a knee deep. The bluegills thrive in these areas and the bass and northern rarely come in there to feed.
The gills then reproduce rampantly, almost to a point of overpopulation to match the forage base. Not enough food and they quit growing. Then from what I've seen those small fish that venture into the weeds are soon eaten though not at a rate that would bring the population back into check.
One of these lakes had an extreme winter kill one year and 2 years later the gills were starting to recover which again only lasted about 2 seasons and the lake was right back to were it was previously. This lake gets very little pressure because of it's long time reputation as only having small fish.