There are many different "analog video signal" formats. The format of the transmitter and the receiver need to match, in terms of electronic signal format as well as mechanical connection. Simply placing a wrong electronic format of the rigt mechanical connector will not result in a successful video connection.
Not if the video formats of transmitter and receiver do not match. This seems to be the case here. From what little information you have provided, the camera likely transmits a composite video signal. The monitor expects VGA which implies RGB and separate sync signals.
Always at the transmitting side. The camera in your case.
Assumprions will not help much here. You need to find which format you camera delivers. Read the manual or look up a datasheet.
Right. Start by telling us which model of camera and monitor you use. That may already answer some of our questions.
Your best option is imho a commercial composite to VGA adapter as for example
this one.