The Portland Sugar Company suffered in the Great Fire of Portland on July 4, 1866, and never returned to its previous success. Even before the fire, John Bundy Brown turned his attention to real estate and investment ventures, initially as Churchill, Browns & Manson, followed by the very successful Portland firm JB Brown & Sons.

John Marshall Brown fought the fire, later recounting the experience to his then-fiancé, Alida Carroll and noting concern for his father, John Bundy Brown,
"When we reached the scene the whole of the immense building was in flames. We saw that all attempts to save it were fruitless & therefore devoted ourselves to that portion of the building where our office was and all our private papers, money & other valuables. For five hours we fought the fire but to no purpose. Everything was destroyed… the magnificent building was entirely consumed with its contents & the labor of 25 years seems blotted out altogether. Father was very cool and collected although we felt very anxious about his ability to bear the blow."