Over my turkey hunting career, I have gotten half my birds from 4-7PM. This year our group got 3 of their 4 birds in the afternoon, all around 4-5pm. You should do well. I seem to see more single toms and groups of just toms in the afternoons compared to the morning mix of birds. One consistent pattern by me is that they seem to hit food sources just prior to roosting and if you know what they are eating in your area you can intercept them.
Don't give up hope, we have had great luck in the afternoons. Mornings have more suspense around them because of all the chatter on the roost and the challenge of setting up right and getting one right away, but afternoons can be just as good.