I think that is probably your best bet, yes. You could go through just the ContactDown and ContactUp, and calculate the difference in their relative positions, for example, but Manipulations essentially give you that.
Also, at the risk of making some assumptions without all of the information, it sounds a little like you are wanting to get information about what a user is doing independent of that input's effect on the actual visuals in your app. Keep in mind that
Surface is primarily about direct user interaction with content, versus performing some gesture and then having that create some effect after the fact. I can't tell for sure what your design is, but if you are not currently using direct manipulations, consider
altering your design in favor of that approach.
Doug