1) I've seen issues some years back where BRIGHT would have a hard time compressing textures that had a very wide range of colors in the template palette. The most common result was odd colored banding appearing, where there simply was not enough coloring to simulate shading.
Is this the kind of thing you are seeing?
2) Imported vector art will be limited in resolution by the size of the template in Photoshop you import it into. For example, if you've got a template in Photohsop that is 1024 pixels square and paste a vector layer in, that vector layer will be converted to a 1024 resolution pixel art when it's saved as a BMP or JPG (or any non-PSD format).
While you can have the vector art as a layer in Photoshop, it will be reduced to pixels when you save it as an image file.