Custom name servers can be used to customize how your registered domain name information is displayed. They are primarily seen in a "Whois" lookup on your domain name. For example, the nameservers for your domain are normally listed as such: ns1.hostinger.com and ns2.hostinger.com. With custom name servers, you might have your information like this: ns1.yourdomain.com ns2.yourdomain.com.
Basically speaking, custom nameservers are usual nameservers, they just use your actual domain name.