Just checked out the 1941 travelogue. Some pretty bizarre history in there.
Interestingly, there's a shot of Taylor and Jefferson at 5:51 that matches exactly the view in the colorized footage -- except there's no sign prohibiting photography in the travelogue version. Obviously, no wartime restrictions at the Wharf.
This also explains why there were still statues and cannon on the Sutro Heights parapet. The army hadn't taken over the Heights in 1941.
I'm guessing the restored 1940s footage was shot at various times in the early '40s and later combined into the colorized and speed adjusted version we're discussing.