If the select by color isn't accurate, then there's always the eraser tool. Add an alpha channel as I mentioned below, then just use the eraser tool to remove all the gray. Big brush for bulk cleanup, then small brush for detailed work near the image art.

Labor intensive, but you have exact control.