How make the lookup wizard prevent duplicate value of more than one field

This can be done and you need to do this manually using field events. I understand that you use two lookup wizards where users enter their own values. You need to implement 'change' field event for both fields, get the value of both, check for a duplicate and display some message or clear the field.
https://xlinesoft.com/phprunner/docs/field_events.htm