Joseph Luther Baldwin was born July 15, 1910 in Menan, Jefferson County, Idaho, the fifth child of John Alpheus Baldwin and Ruth Alice Tanner Baldwin.

In 1926, at the age of 16, he started a boxing career, with his father as his manager. He retired from boxing when he met and married Sophie Jozefoski.  They married on November 29, 1933, and had four children; three sons and one daughter.

He passed away on April 11, 1995 in Olympia, Thurston County, Washington and is buried next to his beloved wife, Sophie, in Sunset Memorial Gardens Cemetery, Richland, Benton County, Washington.