It seems that with some (almost all) ale have a strange bug. When you pass your fortitude save you take a hit to int. But when you fail your saving throw you take no hit to int.

Fixed.