Is webcam image classification afool’s errand? [N]

I’ve been bashing away at this on and off for a year now, and I just seem to be chasing my tail. I am using TensorFlow to try to determine sea state from webcam stills, but I don’t seem to be getting any closer to a useful model. Training accuracy for a few models is around 97% and I have tried to prevent overtraining – but to be honest, whatever I try doesn’t make much difference. My predicted classification on unseen images is only slightly better than a guess, and dumb things seem to throw it. For example, one of the camera angles has a telegraph pole in shot… so when the models sees a telegraph pole, it just ignores everything else and classifies it based on that. “Ohhh there’s that pole again! Must be a 3m swell!”. Another view has a fence, which also seems to determine how the image is classified over and above everything else.

Are these things I can get the model to ignore, or are my expectations of what it can do just waaaaaaay too high?

Edit: can’t edit title typo. Don’t judge me.