Klingspor’s Sandflex® Blocks are useful tools for quickly removing rust. Made of some sort of rubbery, eraser-like material impregnated with abrasive, they come in fine, medium and coarse. Most of my current armor is not a mirror finish, but the finish is fine enough that it will show scratches from a Scotch-Brite scouring pad. By ending with the fine block I can remove rust without marring the finish. I also use them to remove rust from sword and dagger hilts. The blocks are also useful for removing corrosion from copper alloy fittings and returning them to a bright finish.
You can read more here.
You can save yourself a lot of later trouble by removing rust before the metal becomes pitted.