'VM scale set cannot reference a load balancer when LargeScale property is enabled.'
The error may well leave you as puzzled as I was!
There is a checklist of things you require
STANDARD not BASIC for the Load-Balancer is the first thing to check..
and you probably need a Standard SKU for your Public IP too.