The problems usually arise from using too thin of a material. Did you use 3/4" material or 1" stock? Did you use MDF or plywood.? Are your baffles floor standing or do they rest on the subs?
Most people make the mistake of using 3/4" material because that is what is commonly available. Please contact me with details of your setup and I can let you know what you can do about it. Usually though, once wood or even MDF has warped that's the end of the ball game. I can send you photos of my thicker baffles that have a routed relief from the back so that the driver is at the proper depth. At anything more than 1" thick you have to be able to put a relief in the back.