Correct option is A
The Nyquist plot of a loop transfer function G(jω)H(jω) of a system encloses the (-1,j0) point.
That means the given system is unstable, hence the gain margin is negative.
For a stable system, gain margin and phase margin are positive
For an unstable system, gain margin and phase margin are negative
For a marginally stable system, gain margin and phase margin are equal to zero.
