The brand known for its khaki chinos, and credited as integral to Casual Fridays, has struggled in recent years.