There are a couple things that are different about a jacket made to be worn as an odd piece and a jacket that is made as a part of a suit. The material used to make suit jackets is different from that of sport coats. Essentially, suiting material is usually a soft textured fabric like wool, while a sports coat is generally made from a rougher textured fabric like flannel. Sports jackets are also often made of thicker fabrics like cashmere or cotton.
A sports coat will look good with other casual attire, because of the general roughness of its material. A suit may not look as good, and in fact may look very out of place.
Please note that not all suites are made from soft material. Even though some suits can be made from a rougher material, this is generally not the case. Check the texture and the color of your suit jacket and let that be your guide. The rougher the suit material or paler the color, the most likely it is that it could be passed off as casual. A checkered pattern may also be easier to dress down.
When deciding if you should try dressing down your suit jacket, take into consideration the material, then the pattern then the color. Remember a pinstripe is a very formal pattern and is hard to dress down no matter the material.
So the no, now becomes a maybe. It may be that you have a suit that has the perfect texture, pattern and color that allows it to be dressed down appropriately. If you think that it so you can try it. Just keep in mind the concept of textures – smooth and rough, patterns – checkered and pinstripe and color – cream versus the very formal navy blue, when deciding to dress down your suit jacket.
You are a fashionable individual and you have invested in a good suit. Somehow you have torn the pants or it has just become un-wearable so you are thinking that you should be able to wear the jacket as a sports coat on occasions. This may seem logical, but from a style perspective this is a big risk.