"Jackie was a great player, and what a competitor," said Zimmer, a senior advisor for the Rays. "Then, to go through what he went through without ever complaining. Not many people could have done that. He just took it that way. He was special, no doubt.
"It was a shame when we'd pull up at the hotel and the black players would have to be dropped off at a different hotel. I never really understood it, but that's the way it was.
"Branch Rickey had to be the smartest man in the world.
"To take one black man out of the world to make a Major League ballplayer and pick Jackie. And for Jackie to be able to do what he did. He went through hell. That's the way it was until more black players came in and Jackie was finally able to fight back."
Zimmer's American Legion team from Cincinnati won a national championship in 1947. Part of the team's reward was a trip to the World Series, where the Brooklyn Dodgers played the New York Yankees.
"We got to meet Babe Ruth when we were there, and that's the first time I saw Jackie play," Zimmer recalled with a grin.
Upon graduating from high school in 1949, Zimmer signed with the Dodgers, and that afforded him the opportunity to meet Robinson. He was awestruck by Robinson's athleticism and physical attributes.
Robinsons are great ambassadors
Justice: Jackie's courage immeasurable
Rickey's foresight shaped game
RBI, UYA, CRG embody Jackie's spirit
More on Jackie Robinson Day
Jackie Robinson Foundation
A look back at barrier breakers
Jackie Robinson Day
Jackie Robinson's debut in 1947
MLB Network examines Jackie's life
MLB.com's looks at No. 42
Shop the Jackie Robinson collection
"He had legs on him," he said. "Calves like that." (Zimmer used his hands to illustrate the enormity of Robinson's calves.) "He could have played any sport."
By 1953, Zimmer found himself in the Dodgers' camp during Spring Training, and that's when he really started to know the man.
"Most of his teammates loved him," Zimmer said. "But a guy like Dixie Walker, before I got there, he refused to play with him. And Dixie Walker didn't last long. They got rid of him.
"Jackie was liked by most of his teammates, but he never let anybody get too close to him. Pee Wee [Reese] was probably the closest. Jackie took a liking to me. I wasn't a good player, but he liked the way I played. And I think he respected that."
Zimmer calls Robinson a "great hitter."
"He hit 20 home runs a year, 19 a year, he hit line drives," he said. "When he hit a line-drive home run, it was something. And the next time up he'd lay down a bunt, run pigeon-toed. He could fly."
And almost in an afterthought, Zimmer added, "And he played the horses like I did. He bummed around with us guys."
Zimmer got a twinkle in his eye recalling the group of Dodgers who would go to the horse track together, a group that included Duke Snider, Reese, Gil Hodges and Robinson.
"We had a lot of guys who liked to go to the horses," he said. "We'd go to Aqueduct or Belmont. On an off-day, that's where we went. We all liked the same thing."
Zimmer chuckled when asked if any among the group was a good handicapper.
"We were all in the same boat, one winner and 10 losers," he said.
One of Zimmer's favorite memories of Robinson came in 1957.
"I can remember when Jackie was traded to the Giants and refused to go, and quit to work for Chock full o' Nuts," he said. "And he said to me, 'One of my stops is in Tampa, could you get us a golf game if I came down?'"
Zimmer thought Robinson was just making conversation, then he got a phone call from him.
"He said, 'I'm coming down Wednesday, can you get us a golf game for Thursday?'"
Zimmer had a regular golf game in St. Petersburg with Jimmy Mann, a sportswriter for The Evening Independent.
"I called Jimmy and said, 'We've got a golf game for Wednesday,'" Zimmer said. "He said, 'Who with?' When I told him Jackie Robinson, well, he liked to drop the phone. That was one of the biggest thrills I've had. That Jackie Robinson would ask me to get him a golf game. That was quite a thrill. He was like us, a hacker. He just enjoyed being out on the course playing."
One of Zimmer's most prized possessions is a photograph of him and Robinson taken on the golf course that day.
Zimmer noted that the annual recognition for Robinson is appropriate, and that the recognition of his friend means a lot to him.
"And I'm sure it means a lot to a lot of the Dodgers who are still alive," he said. "He was a great man."
Bill Chastain is a reporter for MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.