VF500 rear wheel bearing diagnosis

Hope I am posting in the right spot and sorry in advance if I am not.
Here's my question:
Bought an 85 with lovely spray bomber red wheels. I picked up a clean set of stock wheels as well.
A test ride showed no slackness in the rear wheel at all. I hade the tires swapped from the red wheel to the black ones as the tires on the black wheels were worn.
I noticed when removing the axle and also installing on both wheels that the axle on the red wheel seemed to come out and go in easier. I swapped axles to make sure there were no warps and nothing changed so the axle in the rear wheel definitely needed more force, via a rubber mallet, to remove and install.
Neither wheel showed any slackness or looseness.

The black wheel is mounted and definitely has a slight drag compared to the red wheel. I will check the caliper again but my question is: should the axle be tight through the bearings? If I recall my basic mechanics, a wheel bearing spins internally against its faces but should not rotate on the axle or inside the hub as it will cause axle and hub wear.

So, can I assume that the bearing in the red wheel was starting to wear and the stiffer one is normal?

I don;t want to start pulling bearings again as the time will likely eat up what little remains of my riding season... nor do I want to have a bearing seize at speed. Pulling and installing that wheel was a ***** as I kept on forgetting one item (install axle, forget caliper bracket on swingarm, remove axle, install bracket, install axle, forgot to mount caliper (as I could not get my right pipe off), remove axle, mount caliper, install axle, forgot tensioner bracket, remove axle, get it all on, time for dinner).

Regards