Yes, gift exchange organizers may remove invitees, even after the invitations have been sent. From the gift exchange event page, click on the Participants tab and click the arrow to expand the box for the person you would like to remove. Click on the Decline or Remove button for that participant to change their status. A declined status removes this person from the exchange, but will keep them part of your group. A removed status will remove that person entirely.