Your code is configuring the HTTP response so that the header indicates that it is an excel file and the body contains the binary content of the file. When such an HTTP response is sent to a browser, the browser responds by popping up a window enquiring
what it should do with that file. However, if you write the same code in a service, the response does not go to the browser, but to whatever application is calling the service. Even if it is a javascript application running inside the browser, the Response
does not go to the browser, it goes to the application. Therefore, it is the responsibility of the application to parse the headers and take appropiate action. It will not happen automatically.
Edit: You can change your javascript code so that instead of sending an ajax query to the service, it
navigates to the same url where the REST service is located. In this case, the browser will pop up the file download window.