There is a soft limit of 20 cores per subscription - with the limit in place to protect both you and Microsoft in the event of an attempt to launch a large number of instances. You can apply for a quota increase
here. A small instance
consumes one core and the larger sizes double the core count up to eight cores for extra--large.
The Windows Azure Service Mangement REST API can be used to retrieve various properties of a hosted service - but I believe instance size (and, by implication, core count) is not
Adding to what Neil has mentioned, one way to programmatically do it would be making use of Service Management REST API and invoking change deployment configuration (http://msdn.microsoft.com/en-us/library/ee460809.aspx).
This is where you would change the configuration file contents and specify the new instance count. When you exceed the quota, this operation will throw an error and you could catch that error to see if you have indeed exceeded the quota. However this is not
a foolproof mechanism as this operation could also throw error if there is something else wrong with changing configuration file contents (may be invalid data or something else), so you will have to parse the error message to be sure that the error is thrown
because you are exceeding the quota.