I can shed some first hand experience from the other side of the fence on this issue, as my family deals with this on an annual basis with more than one neighbor. My uncle owns 380 acres where we deer hunt. His neighbor to the south owns about 60 acres (~40 acres of ag field and a 15-20 acre strip of woods that leads from a creek bottom in the valley up to the 20,000 pine trees we planted when I was 12 years old that offers GREAT bedding / cover when everyone starts shooting). This neighbor has 3-4 guys that hunt his land (again 3-4 guys hunting 15-20 acres of woods)...two of them sit within 15 yards of my uncle's fence line, neither one of which is a very good shot. Between bow season and gun season this year ALONE, they have asked to trail 6 wounded deer into by uncle's property. 2 bucks and 4 antlerless. This is an EVERY YEAR occurrence. And the sad part is they use a crossbow during archery (have nothing against using a crossbow, but there is ZERO excuse to be wounding that many deer with ANY bow, let alone a crossbow!). They are nice to him year-round, stop and BS, offer to help with things, etc. This year was the final straw though. We have always asked them to call or stop at the house to ask, and when they do, permission was granted but they had to wait until after dark to go look for the deer. The owner of the property came up after trailing a wounded buck 300 yards into my uncle's land after dark to ask for permission to keep going. Basically, he was to the point that if he kept trailing it, my uncle would have been able to see his flashlight in the valley across the road from his house. An argument ensued and basically my uncle told him to learn how to shoot or get off the fence line. Like I said, there is a 15+ year history of this poor/unethical shooting and hunting, and enough is enough. The neighbor to the NW also sit on the fence line and every year we usually find 3-4 deer dead in that valley as well, usually with arrows in them from archery season. This group can at least shoot a rifle fairly well. We understand that it happens, but when it happens every week or every other week from September through November, it gets old. Fast. Anything more than once a year is not acceptable in my eyes. You are either doing something wrong or sitting too close to the fence.
Having a good relationship with your neighbors is crucial in the event that you would need to trail a wounded animal onto their property, but you cannot abuse that privilege. It should be the exception, not the rule. It sounds like you may be in the same boat as my uncle's neighbors. Either learn how to put that deer down before it gets to the property line, wait for the deer to get further into your property or move farther from the property line. Probably not what you wanted to hear, but the way you make this sound, they probably have a valid argument. This sounds like an annual problem and, like them, I would have let the first couple slide. But these neighbors are paying tens of thousands of dollars to buy their land, plus annual property taxes, food plots, stands, licenses, etc. and that is just the MONETARY 'costs'...then there is the maintenance of the property, fence lines, buildings etc. that is labor-intensive. If the tables were turned and he were knocking on your door every year wanting to trail another deer into that bedding area that is so productive for you, would you not be frustrated? Maybe the area of land adjacent to your bedding area is one of his most productive feeding areas and you keep tracking deer through it? Just food for thought...I obviously have never seen your property, but there has to be other places on your land that deer pass through, feed or would otherwise offer shots? If not, maybe it is time to start making improvements through food plots, watering holes, plating trees for mast production and additional bedding area, etc. to draw those deer away from the fence line where you can put them down before they make it to the fence? There are two sides to every story, but my guess is your neighbor values the time, money, effort and labor he has invested in his property and hunt and would like to try to reap the benefits of his labor.
I have bow hunted a small (15 acre) block of woods for 12 years and shot several nice bucks there and never had to track a deer onto the neighbor's land until this year when I hit my buck in the liver. When we found my buck, he helped me drag it to the road and then jokingly said it is unfortunate we had to track that buck so far (died 50 yards onto his property), but he was glad that he finally got to meet the 'mystery guy' whose truck he always had seen parked there, but never saw. He did give me permission to track any deer I shot onto his land, but I asked for his phone number and told him that although I appreciated his offer, I will not set foot on his property without contacting him first. I also dropped a 12 pack off at his place during gun season as a thank you, as I saw a few empty bottles in the bed of his truck so I knew what he liked. I hope I never need to call him for that favor again, but if it would happen in a few years, I know he would allow me to go on his land again.
Like a couple others have suggested, it may be in your best interest going forward to swallow your pride, go talk to him well after hunting season (to allow him time to cool down if needed) and then have an honest conversation about your expectations of each other and find out EXACTLY what his feelings are and what is bothering him. Then adjust your hunting strategy accordingly. Sorry this was long-winded, but if you step back and take a look at the big picture, you both probably want the same thing...to have successful hunts. Now it is a matter of finding out how to do that without interfering with one another. Good luck!