Alright I'm looking for opinions  on how this situation should be handled. I drew a picture to help picture the situation.

My family and I have a property that consist of 55 acres. We bought it in 2010 and in 2012 I put a stand up (gun stand) on the north end of the property about 100 yards away from the the north boarder. It over looks a 2-3 acre little point on the creek that the deer bed in. It's a textbook spot and in the last 6 years, we shot 6-7 good quality mature deer from this spot. See deer all day long. Just a spot that everybody wish they could hunt and we are fortunate to have it.

Here is the problem, the deer come off the two surrounding properties to enter this bedding area and that's when we get our opportunity. Well we shoot the deer on our property (and have passed many on the neighbors land or waited til they get to ours) they run on to the neighbors, weather it be 10 yards to maybe 100 yards. We always text or call asking to retrieve the deer. Well he seems to think this is a problem and is getting to the point where he may not let us in anymore to get the deer.

I feel like we have done nothing wrong and we just have a unique situation here. 1) the stand is 100 yard off the board and facing into our property 2) all deer are shot on our land and passed if not 3) we always ask permission to retrieve deer no matter what

Am I missing something, how should I handle this and what is the best way to go about hunting our best spot. Just puzzled