When Joseph retired, he turned the business over to his remaining sons in 1865. One of these sons, Colonel William Rice Milward, would eventually assume the leadership role.

It led to a key decision in 1887, when Colonel Milward permanently separated the funeral directing from the retail furniture store. The business first moved to Main Street, and then, in 1896, to what is now 267 South Broadway.

Nine years later, in 1905, Colonel Milward had a three-story site built at 159 North Broadway. The building was 66 feet in width and 190 feet in depth. Offices were in the front of the first floor with hearses and vehicles stored in the rear. Horses were kept on the second floor and the third floor was storage.

He was a man of extraordinary vision and understanding — becoming an honorary member of the Lexington Chapter of the United Confederate Veterans despite his history as a Union officer. This ability to extend the hand of reconciliation was typical of Colonel Milward.

Upon his death in 1915, Major P.P. Johnston, chapter president of the Confederate veterans, issued an appeal in the Lexington Leader to all members to attend the funeral “…of our lifelong friend, Col. W.R. Milward, a gallant and uncompromising Union Soldier, always loyal to all that his service implied. The war over, he extended the hand of friendship to his fellow citizens, who retired from the fight and gave them recognition due to brave men…”