Lookin' GOOD! Take your time and get it right now, and you can enjoy many trouble free miles. These sleds were very reliable when new....most all of them from the late '70s- early '80s were, from all of the manufacturers. But, we have to contend with the 35 plus years since then.
Polishing the rough surface of the factory castings has to help some. I cleaned up the roughness on my Invader cylinders many years ago. I don't have a clue how much (if any) it helped.
Absolutely avoid getting too close to the inner chrome liner on the cylinder. Without the proper high speed grinding equipment, it's possible to chip the chrome lining.
I wouldn't spend a great deal of time, other than smoothing the surfaces, especially the exhaust side. Matching the exhaust port dimension to the gasket, and the y pipe would certainly help.....but the gains that come from having the carbs very clean, the clutches perfectly aligned, and all of the bearings in the drive train in good shape probably outweigh any tiny gains from that precision matching of the ports.