It could possibly be just residue from the rouge compound and the sulphides you are removing. You might try using soapy ammonia to temove the smudges.
Is the finish on you silver smooth after you have spun it. Rough does little to remove deep scratches and perhaps you are buffing with a bit too much force. If you wish to remove scratches perhaps you should start with a more abrassive compound such as tripoly or white diamond. And then finish off with the rough and with a lighter touch.
Dear Mr. Zweig: Thank you for your very kind response. After I have spun the piece I do not have and spinning lines as I smooth the piece. I will take your suggestion and reduce the amount of the rouge compound I use. I also plan on tripoling the piece first as there were some minor scratches when I received the metal from the supplier. Nothing major at all. This is my first attempt at polishing sterling. Will keep you informed. Thank you. Richard Alexander