Yes "ambiguë" is pronounced exactly as if it was written "ambigu".

The thing here for those wondering is the masculine and feminine in French, with the feminine created by adding an 'e' (often silent!). "Ambigu" is masculine and "ambiguë" is feminine but as you said without the diaeresis that final 'e' would completely change the pronounciation of the word.