yes i have just yesterday i found that my swing arm i just installed was not lined up correctly with the front sprocket. there was play just as you described so i replaced the swing arm pivot bushings with some others and it was fixed. my guess would be those long metal spacer/bushings at the front of the swing arm are worn.
2007 OUTLAW 525 IRS
2001 400 TWO STROKE SCRAMBLER