Hi,
According to your description, my understanding is that you want to create a windows authentication custom login page and automatically log in without promote popup window.
For the error message, I suggest you can use Fiddler to trace the detailed error message and also you can check ULS log for more information.
For passing the popup window, you can set the Internet Explorer Setting by go to Tools -> Internet Option -> Security -> Custom Level -> Scroll the whole way to the bottom and select "Automatic Login with current user name and password”
-> OK -> OK -> Close .
Here is a detailed article for your reference:
How do I make our SharePoint site stop asking me to login?
About how to create a custom login page specific for windows authentication,please refer the article:
Creating a Custom Login Page for Windows Authentication
Thanks
Best Regards
TechNet Community Support
Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact
tnmff@microsoft.com.
