By 1957, the bloom was beginning to leave the rose. ABC-TV had given Freed a nationally televised show. On one episode, black singer Frankie Lymon danced with a white girl, virtually a no-no for that era. The ABC southern affiliates were incensed by this and the show was canceled. Next came a show he promoted in Boston where he was charged with inciting a riot. Although cleared of the charges, WINS let him go.
Freed was now mired in bankruptcy and his troubles were just beginning. He was convicted in the payola scandals of the late 1950's where disc jockeys were accused of accepting money and gifts for playing certain records. Although his sentence was a small fine and no jail time his career was basically finished.
He worked briefly at stations in Los Angeles and Miami and tried to cash in on the "twist" craze of the early 1960's. When that fad passed, he was out of work again and drinking heavily. In 1964, while broke and barely existing in Palm Springs, California, the Federal Government charged him with income tax evasion. Shortly afterward, Freed entered a hospital suffering from uremia and eventually died on January 20, 1965 at the age of 43.