Does each image have to be a post with an assigned category?
Yes, the slider will be shown the posts(featured image) by category. It is the only way.
how can I avoid having those image posts appear when scrolling between portfolio posts when using the Prev/Next buttons?
Please set all portfolio posts in one category, go to: Appearance > Theme Options > Layout: Category for More Project Button On The Bottom of Gallery Post, choose the category. The Prev/Next would loop all portfolio posts in this category.