Using abrasives on your carving is usually the last step before painting, its generally not a good idea to use your tools on the work again as there may be particles left behind which will ruin your cutting edges..
Sometimes however it is necessary to do this, so If you have to then always use cloth backed abrasives. I almost exclusively use Hermes RB406 J-FLex which doesn't clog or drop particles excessivly. It's a little more expensive than some but it lasts a long time before wearing out and you can clean it in water too.
If you do have to add final detail after sanding, give the wood a wipe/rinse with white spirit, this should clear anything that might be left behind. Don't wipe the wood with water if you can avoid it as it will raise the grain.