Pizzarelli began his professional career at 17 when he joined the Vaughn Monroe dance band in 1944.

In 1951, he did his first recording as a sideman outside the Monroe orchestra with Joe Mooney.

In 1952 Pizzarelli became a staff musician for NBC, playing with Skitch Henderson. In 1964, he became a member of The Tonight Show Band on The Tonight Show Starring Johnny Carson. During his time spent performing for the Tonight Show, he accompanied guest bands and musicians playing through a variety of musical genres, including playing with Tiny Tim (after tuning the performer's ukulele) on the day that Tiny Tim married Miss Vicki on Carson's show.

From 1956 to 1957, Pizzarelli used the stage name "Johnny Buck" and performed with The Three Suns pop music trio. During the following year, he and guitarist George Barnes formed a duo and recorded two albums, including a live performance in August 1971, at The Town Hall in New York City. Beginning in the 1970s, he began recording as a leader, issuing many tributes to musicians of the 1930s. He toured several times with Benny Goodman until Goodman's death in 1986. He performed with Benny Goodman at the White House in Washington, D.C., and he performed for presidents Ronald Reagan, Bill Clinton, and First Lady Pat Nixon.

"Jersey Jazz Guitars" was the name of a 1985 concert held at the Rutgers University Nicholas Music Center in New Brunswick, New Jersey. The ticket featured Pizzarelli, Les Paul, Tal Farlow, and Pizzarelli's son, John. The concert was aired on New Jersey's public radio station as part of their three-part New Jersey Summerfare Series. Pizzarelli and Les Paul had performed together before, as they were neighbors and friends. The show aired for one hour in August 1985, with son John adding his vocals on two selections.

Pizzarelli continued to play into his 90s, making several appearances even after a stroke in 2016, officially retiring after a final brief appearance with Michael Feinstein in 2018. He died of COVID-19 on April 1, 2020, in Saddle River, New Jersey. He had been battling several serious health problems in recent years.