Any Walking Dead fan will tell you, if there is one guy you don’t want to piss off, it is the dude known for carrying around the barbed wire-covered baseball bat. But that's exactly what the cast of Destination X did when they got on host Jeffrey Dean Morgan ’s bad side at one point during filming.

I said, 'I think you have the wrong person,'" Morgan tells PEOPLE exclusively. "I live on a farm, and I kill zombies, and I'm a dad, and I'm kind of a pain in the a--."Throughout his decades-long career as a successful actor,

Destination X whisks contestants away on the journey of a lifetime as they travel through Europe and put their geography knowledge to the test in hopes of winning it all and taking home $250,000 – but where would the players be without their host?