none
PDOException: SQLSTATE[IMSSP]: error occurred translating string to UCS-2, using drupal 7 at azure and sql azure backend

    Question

  • Greetings,

    I recently started setting up a drupal 7 website on Azure cloud using a web role and SQL azure backend. I was very much pleased until I hit a character set brickwall.

    I have a virtual machine with Windows Server 2008 R2, IIS 7.5, SQL Server 2008 R2 as a development server, trying changes before I upload them to the cloud. I have some extra tables in drupal database to support my custom module for the application. Among others, at some point I need to run some code like that:

    $r = db_select('my_creators', 'c') ->fields('c') ->condition('my_url', $creator_url) ->execute() ->fetchAll(PDO::FETCH_ASSOC); $creator = $r[0]; foreach ($creator as $k => $vv) $s.= t('@key -->> @value<br>', array('@key' => $k, '@value' => $v));

    print $s;

    I have cut out a lot of extra code, but this is the code having the problem, when $creator_url has a value with some special character (e.g. André_Breton).

    This runs perfectly on my development machine, fetching the record and displaying the data so I was very surprised when I uploaded to the cloud and got this error

    PDOException: SQLSTATE[IMSSP]: An error occurred translating string for input param 7 to UCS-2: No mapping for the Unicode character exists in the target multi-byte code page. in dblog_watchdog() (line 157 ofE:\sitesroot\0\modules\dblog\dblog.module).

    The underlying databases are identical, actually I data-sync the sql azure database to the development sql server database daily. All character data fields are nvarchar.

    I cross-checked all versions of IIS, php, php extensions, php.ini and all are identical.

    I managed to kind of work around this by adding

    Database::getConnection()->setAttribute(PDO::SQLSRV_ATTR_ENCODING, PDO::SQLSRV_ENCODING_SYSTEM);
    
    $creator_url = iconv('UTF-8', 'UCS-2', $creator_url);

    before the select, and then for the print

    foreach ($creator as $k => $vv) {
        $v=iconv('ISO8859-1', 'UTF-8', $vv);
    
        $s.= t('@key -->> @value<br>', array('@key' => $k, '@value' => $v));
    }

    Using this workaround the select goes through, the correct record is selected and displayed, but then I get several warnings

    • Warning: htmlspecialchars(): Invalid multibyte sequence in argument in check_plain() (line 1572 ofE:\sitesroot\0\includes\bootstrap.inc).
    • Warning: htmlspecialchars(): Invalid multibyte sequence in argument in check_plain() (line 1572 ofE:\sitesroot\0\includes\bootstrap.inc).

    which makes sense since usually drupal functions expect all strings to be UTF-8.

    I am hitting my head against a wall for a couple of days now, any help will be deeply appreciated.

    Thank you!


    Fistix


    • Modifié fistikis vendredi 20 avril 2012 10:29
    vendredi 20 avril 2012 10:27

Réponses

  • Thanks Fistix, that was a good bit of testing.

    It looks like you have received the ISO-8859-1 encoded URL... I would have preferred to see "Andr%C3%A9", so I ran a similar test with a few different browsers:

    1. Firefox converts André to Andr%C3%A9
    2. So does Safari
    3. So does Chrome

      Now the bad news:
    4. Internet Explorer 8 sends André, ignoring the illegal character, passing it verbatim.
    5. Opera does the same as IE8 (p.s. never use Opera for more than 5 minutes on XP, it leaks memory constantly, they've stumbled while trying to keep up with Google)

      Now the inexplicable:
    6. IE9 - just don't ask! (which IE did you use, Fistix?)

    You could try one more test of the rewriter with this configuration string - just to see what gets printed in the IIS log file after the URL is re-written:

      <action type="Rewrite" url="index.php?q={UrlEncode:{R:1}}" appendQueryString="true" logRewrittenUrl="true" />

    I don't think it will help, it might just encode the é as %E9, which won't work, but is better than an illegal character.

    There might be other rewrite rules that could solve it, but I'm sorry to say that after all this time, you probably have to keep your workarounds in your code, so that you are prepared to receive both:

    1. well-formed URL's in url-encoded UTF-8
    2. badly formed ISO-88591 or ASCII.

    Sorry that after all this time I couldn't come up with a better solution.  Url's are tricky because there was never a rule about encoding beyond the standard printable characters in the first 127 with different character sets, but as a web site owner, you can dictate to your clients what they are and aren't allowed to type as valid URL's.  You could recommend they use Firefox, Chrome or Safari (sorry MSFT)

    On the php side, you could try something like this:

    if ( !mb_detect_encoding($creator_url, 'UTF-8', true) )
    {
        $creator_url = mb_convert_encoding($creator_url, 'UTF-8', 'ISO-8859-1');
    }
    
    $r = db_select('my_creators', 'c')
            ->fields('c')
            ->condition('my_url', $creator_url)
            ->execute()
            ->fetchAll(PDO::FETCH_NUM);

    If your clients want to use IE9 they will not succeed whatever you do at your end, according to my tests on the Windows 7 VHD with IE9.  Disappointed but not really surprised by that result.


    Rob

    • Marqué comme réponse fistikis samedi 5 mai 2012 05:21
    vendredi 4 mai 2012 11:35
  • Ok, I don't really want to install Drupal, and we're going a bit off-topic, so this is going to be my last post.

    It looks like the URL would need to go through a rewriter... have you got Microsoft's URL Rewriter 2.0 installed in IIS? Also you might need to update FastCGI.

    Here are some links I found by searching Microsoft's great iis.net web site:

    These are just 3 links I found - I think site www.iis.net will have the solution to the problem somewhere, if not one of these.


    Rob

    jeudi 26 avril 2012 13:37

Toutes les réponses

  • Following up with my troubleshooting efforts, I made a new drupal scaffold following the guide from http://azurephp.interoperabilitybridges.com/articles/how-to-deploy-drupal-to-windows-azure-using-the-drupal-scaffold and deployed it to a new hosted service / staging deployment.

    When I try to access a non-existent url with some non plain alphabetic character like 'é' I am getting the same error I state above instead of the normal 'Page not found' page.

    Error

    The website encountered an unexpected error. Please try again later.

    Error message

    PDOException: SQLSTATE[IMSSP]: An error occurred translating string for input param 7 to UCS-2: No mapping for the Unicode character exists in the target multi-byte code page. indblog_watchdog() (line 157 of E:\sitesroot\0\modules\dblog\dblog.module).

    So it has nothing to do with my code, but more something about the IIS, PHP, SQL azure configuration?

    Thanks...


    Fistix

    vendredi 20 avril 2012 14:37
  • It means that whatever data you are storing in variable $creator_url is not properly encoded as UTF-8.

    How are you assigning that variable and what does the class method 'condition('my_url', $creator_url)' do?


    Rob

    vendredi 20 avril 2012 15:12
  • First of all, thank you for your reply.

    Tha variable $creator_id gets assigned by drupal and contains the part of the url after the last / (eg. in http://site.com/creator/André_Breton it is the André_Breton part) and class method 'condition' adds the "where" clause to the query generated by db_select as documented by Drupal API at http://api.drupal.org/api/drupal/includes!database!database.inc/function/db_select/7

    Also, following my second post above, I guess somehow IIS messes up the urls and does not encode them properly as UTF-8

    Thank you


    Fistix

    vendredi 20 avril 2012 16:28
  • Thanks for the update - so do you think that $creator_url could be the cause of the problem?

    Maybe one of these modest changes could fix it - try one of these lines depending on what the encoding is of the URL, these are my guesses:

    // Notice fix of typo from ISO8859-1 to ISO-8859-1:
    $creator_url = iconv('ISO-8859-1', 'UTF-8', $creator_url);

    * or *

    $creator_url = iconv('Windows-1252', 'UTF-8', $creator_url);

    then call....

    $r = db_select('my_creators', 'c')
            ->fields('c')
            ->condition('my_url', $creator_url)
            ->execute()
            ->fetchAll(PDO::FETCH_NUM);
    
    $creator = $r[0]; // $r[0] would require option PDO::FETCH_NUM... changed above.

    Rob

    Late edit:

    If you're finding that PHP is assuming a different character set than the intended one used to encode url arguments in $_GET, you can access the raw command line with this $_SERVER variable:

    $cmdline = _SERVER["QUERY_STRING"];
    
    /* 
       Now look at the raw arguments, but don't use php 
       function parse_str(), because that will return 
       exactly the same as the existing $_GET array.
    
       This handles a basic command line...
    */
    
    $pieces = explode('&', $cmdline);
    $myArgs = array();
    
    foreach( $pieces as $piece )
    {
        $arg = explode('=', $piece, 2);
        $key = rawurldecode($arg[0]);
        $value = rawurldecode($arg[1]);
    
        // now adjust the encoding yourself...
        $key = iconv('whatever', 'UTF-8', $key);
        $value = iconv('whatever', 'UTF-8', $value);
    
        // put it in a handy $GET-style array
        $myArgs[ $key ] = $value;
    }

    • Modifié Robert Johnson lundi 23 avril 2012 10:02 Error: $_SERVER['QUERY_STRING'] instead of $_SERVER['REQUEST_URI']
    lundi 23 avril 2012 09:23
  • Hi fistikis,

    Did Robert's answer help you out?

    Thanks,

    Jonathan


    This posting is provided 'AS IS' with no warranties, and confers no rights.

    mercredi 25 avril 2012 17:12
  • Many thanks Robert for your detailed answer.

    Your answer was indeed helpful to find a workaround for my application, but since it is drupal I am working on, I would not want to "hack" the way drupal parses the query string and converts it into variables. However I think this is the only way to go until now.

    Actually the main reason to use a CMS to implement an application, is to avoid all the fuss of needing even to access the $_GET array explicitly.

    So right now I am using this workaround where I can't avoid special characters in query string, and I try not to use such characters in the query strings from now on just to be on the safe side.

    What I am looking for, is a way to make IIS server parse the parameters correctly and pass them as UTF8 to the app, instead of me having to do all the iconv stuff.

    Maybe it is just an IIS bug waiting for a hotfix, since apache works fine with that and even SQL server is ok when I pull data from the db connection (no iconv or anything required). What makes this even more obvious, is the fact that this problem exists even in a drupal default installation, only on IIS web server.

    So I am just waiting for a solution to my problem in order to avoid code patches. I don't like building a new app based on code patches.

    Thank you


    Fistix

    mercredi 25 avril 2012 17:33
  • Are you using the Drupal SQLSRV Module? http://drupal.org/project/sqlsrv

    Cheers,

    Jonathan


    This posting is provided 'AS IS' with no warranties, and confers no rights.

    mercredi 25 avril 2012 18:11
  • Please could you post the URL that's causing a problem?

    The encoding of the URL will provide clues.  If the URL has the literal character é, or %E9, then it's probably encoded as ISO-8859-1.

    If the URL is encoded as UTF-8, it should contain %C3%A9.

    There are a few possible explanations why Apache is working...:

    1. Do IIS and Apache use the same installation of PHP?
    2. If you're using PHP 5.4, the php.ini value 'default_charset' will be UTF-8, and in previous versions it is 'ISO-8859-1'.  You might need to set this value.
    3. It's unlikely to be IIS making a mistake with the URL, but Apache might be able to decode the URL with an AddCharset/AddDefaultCharset directive.


    Rob

    p.s..

    To get the URL as PHP sees it, you can do something like this with a php file that calls phpinfo() e.g.:

    http://localhost/phpinfo.php?test=André

    then inspect the value in variable _SERVER["REQUEST_URI"]... you can run the same test on IIS and Apache.

    mercredi 25 avril 2012 20:24
  • Yes at the azure drupal installation guide I mentioned above, sqlsrv is included

    Fistix

    mercredi 25 avril 2012 22:02
  • It is very easy to reproduce the error. Just set up the default installation of drupal 7 at whatever RDBMS and try to access the url http://mysite.com/André. On apache you get the normal drupal "page not found" error. On IIS you get the error we are discussing here.

    I am away from work atm, when I get the chance I will check _SERVER["REQUEST_URI"] just to be sure how the é is encoded, though I am pretty sure it won't be UTF-8.

    Thank you all for your time, I really appreciate it!


    Fistix

    mercredi 25 avril 2012 22:07
  • Ok, I don't really want to install Drupal, and we're going a bit off-topic, so this is going to be my last post.

    It looks like the URL would need to go through a rewriter... have you got Microsoft's URL Rewriter 2.0 installed in IIS? Also you might need to update FastCGI.

    Here are some links I found by searching Microsoft's great iis.net web site:

    These are just 3 links I found - I think site www.iis.net will have the solution to the problem somewhere, if not one of these.


    Rob

    jeudi 26 avril 2012 13:37
  • FastCGI is already updated, and I cannot find a solution to these posts. On top of this, deployment is on Azure, where FastCGI update should not be my concern.

    Sorry but my problem is still there. I repeat I have worked around it avoiding all "problem-causing" characters and iconv-ing back and forth where needed for the time being, but I do not accept this as a permanent solution.

    mercredi 2 mai 2012 07:43
  • Ok, thanks Fistix.

    Good point about Azure, I overlooked that, sorry... it makes most of my previous post irrelevant, but the rewriter could still the key...

    I'm interested to know why this isn't working, so we should carry on trying to solve it.  In the end, maybe Jonathan can put the thread in a different IIS or Azure forum.

    Does your site work when you enter a URL that does not need rewriting, and also contains special characters?  This will indicate if the rewriter needs to be configured.  If it's not the rewriter, let's look at something else, let us know.

    The IIS Rewriter is automatically enabled on Azure - are you able to view or change the configuration for the rewriter?

    Here is a post from Drupal.org about how to configure it (ignore if you've already got this, it looks like you have): http://drupal.org/node/3854. Interestingly this link does not show any use of the UrlEncode functions for the IIS Rewriter...

    Take a look at this excerpt from http://learn.iis.net/page.aspx/465/url-rewrite-module-configuration-reference/#String_functions:, the section titled String Functions starts with this text and includes some simplified examples:

    There are three string functions available for changing the values within a rewrite rule action, as well as any conditions:

    • ToLower - returns the input string converted to lower case.
    • UrlEncode - returns the input string converted to URL-encoded format. This function can be used if the substitution URL in rewrite rule contains special characters (for example non-ASCII or URI-unsafe characters).
    • UrlDecode - decodes the URL-encoded input string. This function can be used to decode a condition input before matching it against a pattern.

    I would have assumed that this would already be configured correctly by Microsoft, but perhaps in some cases some knowledge of the URLs' structures must be known to correctly encode or decode them, in which case you will need to supply the rewrite rules.  The link above from Ruslan Yakushev is a really good reference for the rewriter.


    Rob

    mercredi 2 mai 2012 10:53
  • Hello Rob and thank you for your time!

    I tested it with query string /q?a=André I get the normal "page not found drupal error" with text "The requested page "/q?a=Andr%C3%A9" could not be found.". So I guess rewrite is the problem indeed.

    Below is the rewrite configuration on azure, taken from drupal's web.config

        <rewrite>
          <rules>
            <rule name="Protect files and directories from prying eyes" stopProcessing="true">
              <match url="\.(engine|inc|info|install|make|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)$|^(\..*|Entries.*|Repository|Root|Tag|Template)$" />
              <action type="CustomResponse" statusCode="403" subStatusCode="0" statusReason="Forbidden" statusDescription="Access is forbidden." />
            </rule>
            <rule name="Force simple error message for requests for non-existent favicon.ico" stopProcessing="true">
              <match url="favicon\.ico" />
              <action type="CustomResponse" statusCode="404" subStatusCode="1" statusReason="File Not Found" statusDescription="The requested file favicon.ico was not found" />
            </rule>
            <!-- Rewrite URLs of the form 'x' to the form 'index.php?q=x'. -->
            <rule name="Short URLs" stopProcessing="true">
              <match url="^(.*)$" ignoreCase="false" />
              <conditions>
                <add input="{REQUEST_FILENAME}" matchType="IsFile" ignoreCase="false" negate="true" />
                <add input="{REQUEST_FILENAME}" matchType="IsDirectory" ignoreCase="false" negate="true" />
                <add input="{URL}" pattern="^/favicon.ico$" ignoreCase="false" negate="true" />
              </conditions>
              <action type="Rewrite" url="index.php?q={R:1}" appendQueryString="true" />
            </rule>
          </rules>
        </rewrite>
    

    I think we are closing in on it !

    Thank you again!

    mercredi 2 mai 2012 12:29
  • I'm not sure actually - if it worked without rewriting, then the problem is the rewriter, but it didn't work...

    The URL might still be slightly wrong though, according to the rules it should look something like one of these - does it work with:

    http://mysite.com/index.php?q=André

    or this one? :

    http://mysite.com/index.php?q=Andr%C3%A9

    If one of these works, then we can reconfigure the rewriter to fix the problem...


    Rob

    mercredi 2 mai 2012 12:42
  • Both of these urls return the same "normal" page not found error. Since none of these causes the PDOException I mentioned in the first post, I think the parameters are passed normally as UTF8, the query to the SQL server works fine and since no such page is registered in drupal DB I get the page not found error.

    On the other hand, when parameter is passed directly so that url rewriting is required, the query can't be executed and the exception occurs.

    I guess, when rewriting is needed, the parameter gets misinterpreted and somehow UTF8 breaks and can't be mapped to UCS2 (and the rest of the exception above).

    You think this can be fixed just with rewriter reconfiguration?

    mercredi 2 mai 2012 12:54
  • Ok, so "page not found" is what we want in this case?

    Try changing this line of your rewriter config (4th from bottom):

    <action type="Rewrite" url="index.php?q={UrlEncode:{R:1}}" appendQueryString="true" />

    Then do whatever you need to restart IIS on Azure and try it out...


    Rob

    mercredi 2 mai 2012 12:57
  • Yes "page not found" is good, and PDOException is bad.

    Changed web.config, restart IIS, even virtual machine, same results with all URLs. Exceptions for /Andr%C3%A9 and /André, "page not found" for /index.php?q=André and /index.php?q=Andr%C3%A9

    New rewrite section of web.config:

        <rewrite>
          <rules>
            <rule name="Protect files and directories from prying eyes" stopProcessing="true">
              <match url="\.(engine|inc|info|install|make|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)$|^(\..*|Entries.*|Repository|Root|Tag|Template)$" />
              <action type="CustomResponse" statusCode="403" subStatusCode="0" statusReason="Forbidden" statusDescription="Access is forbidden." />
            </rule>
            <rule name="Force simple error message for requests for non-existent favicon.ico" stopProcessing="true">
              <match url="favicon\.ico" />
              <action type="CustomResponse" statusCode="404" subStatusCode="1" statusReason="File Not Found" statusDescription="The requested file favicon.ico was not found" />
            </rule>
            <!-- Rewrite URLs of the form 'x' to the form 'index.php?q=x'. -->
            <rule name="Short URLs" stopProcessing="true">
              <match url="^(.*)$" ignoreCase="false" />
              <conditions>
                <add input="{REQUEST_FILENAME}" matchType="IsFile" ignoreCase="false" negate="true" />
                <add input="{REQUEST_FILENAME}" matchType="IsDirectory" ignoreCase="false" negate="true" />
                <add input="{URL}" pattern="^/favicon.ico$" ignoreCase="false" negate="true" />
              </conditions>
              <action type="Rewrite" url="index.php?q={UrlEncode:{R:1}}" appendQueryString="true" />
            </rule>
          </rules>
        </rewrite>

    Thanks!

    mercredi 2 mai 2012 14:12
  • To save time, when you change web.config, your site should automatically restart and reload its config - you should be able to just edit the file then test the change straight after...  theoretically...

    Try this - it's just a guess:

    <action type="Rewrite" url="index.php?q={UrlEncode:{UrlDecode:{R:1}}}" appendQueryString="true" />

    Then try this, obviously useless but lets us know if we're on the wrong lines:

    <action type="Rewrite" url="index.php?q=Andr%C3%A9" appendQueryString="true" />

    Rob


    mercredi 2 mai 2012 14:50
  • No change for the results with any of these lines and restarting IIS after each change.

    All my tries had been on Chrome so now I tried on Internet Explorer, just to be on the safe side. I noticed that the error for the URL /index.php?q=André was "The requested page "/index.php?q=Andre" could not be found" instead of "The requested page "/index.php?q=Andr%C3%A9" could not be found." with Chrome. If this could be helpful in any way... Rest results were the same.

    Thanks!


    Fistix

    mercredi 2 mai 2012 17:20
  • Sorry - "page not found" is the right result, so just to be sure, did you get the usual PDO exception when the rewriter config was set to either of the 2 lines in my post before this one, and you navigated to the URL http://mysite.com/André ? (to perform the rewriter test.)

    We don't need to test this URL in the browser at the moment: index.php?q=André


    Rob

    mercredi 2 mai 2012 18:46
  • Yes, with both configurations when I navigated to http://mysite.com/André I got the usual PDO exception.

    Fistix

    mercredi 2 mai 2012 20:31
  • The strange thing is that with the 2nd dummy rewriter setting, the URL was hard-coded to one that works when when entered into the browser.

    Put your rewrite configuration back to its original settings, removing UrlEncode etc.

    I just found a second post from Ruslan Yakushev about configuration for IIS Rewrite 2.0 here: http://learn.iis.net/page.aspx/665/url-rewrite-module-20-configuration-reference/

    There is no mention of UrlEncode etc in his later instructions, I suppose it shouldn't be necessary - if it was then the original URL would have to be badly formed.

    Here's what I would do now:

    1. In this bit of code, have a look at the contents of $creator_url.  If necessary, add another line of code so that you can check what it contains, in this example, error_log adds a line to the PHP log file:
      error_log('$creator_url is: [' . $creator_url . ']');

      $r
      = db_select('my_creators', 'c') ->fields('c') ->condition('my_url', $creator_url) ->execute() ->fetchAll(PDO::FETCH_NUM);

    2. Type in the short URL into your browser, i.e. mysite.com/André.  Do not type any URL that already works.

      • Have a look for your IIS logs, and see what requests were handled during from your latest request.  You should see the original URL from the browser recorded, e.g. http://mysite.com/André.
      • Look at the PHP log to see what $creator_url was.

    3. From the link to Ruslan's guide above, add 'logRewrittenUrl' to your rewrite rules so that the rewritten URL is logged:

          <action type="Rewrite" url="index.php?q={R:1}" appendQueryString="true" logRewrittenUrl="true" />

      Then try the URL and check the log to compare the URL's that are written to see what the rewriter has done.  Hopefully it will show something like http://mysite.com/index.php?q=André.

      Check your php log to see what $creator_url was.

    Let us know of any findings you think might help...


    Rob

    jeudi 3 mai 2012 09:34
  • Your suggestions are very correct, followed them to the letter and I post the results that confused me more...

    When invoking http://mysite.com/creator/André (a url that doesn't work):

    php log has: [03-May-2012 22:14:32 UTC] $creator_url is: [André]

    iis log with logRewrittenUrl="false" has: blablabla GET /creator/André - 80 - blablabla

    iis log with logRewrittenUrl="true" has: blablabla GET /index.php q=creator/André 80 - blablabla

    All look perfectly correct to me, the exception still exists though...

    Thank you.


    Fistix

    jeudi 3 mai 2012 22:22
  • Thanks Fistix, that was a good bit of testing.

    It looks like you have received the ISO-8859-1 encoded URL... I would have preferred to see "Andr%C3%A9", so I ran a similar test with a few different browsers:

    1. Firefox converts André to Andr%C3%A9
    2. So does Safari
    3. So does Chrome

      Now the bad news:
    4. Internet Explorer 8 sends André, ignoring the illegal character, passing it verbatim.
    5. Opera does the same as IE8 (p.s. never use Opera for more than 5 minutes on XP, it leaks memory constantly, they've stumbled while trying to keep up with Google)

      Now the inexplicable:
    6. IE9 - just don't ask! (which IE did you use, Fistix?)

    You could try one more test of the rewriter with this configuration string - just to see what gets printed in the IIS log file after the URL is re-written:

      <action type="Rewrite" url="index.php?q={UrlEncode:{R:1}}" appendQueryString="true" logRewrittenUrl="true" />

    I don't think it will help, it might just encode the é as %E9, which won't work, but is better than an illegal character.

    There might be other rewrite rules that could solve it, but I'm sorry to say that after all this time, you probably have to keep your workarounds in your code, so that you are prepared to receive both:

    1. well-formed URL's in url-encoded UTF-8
    2. badly formed ISO-88591 or ASCII.

    Sorry that after all this time I couldn't come up with a better solution.  Url's are tricky because there was never a rule about encoding beyond the standard printable characters in the first 127 with different character sets, but as a web site owner, you can dictate to your clients what they are and aren't allowed to type as valid URL's.  You could recommend they use Firefox, Chrome or Safari (sorry MSFT)

    On the php side, you could try something like this:

    if ( !mb_detect_encoding($creator_url, 'UTF-8', true) )
    {
        $creator_url = mb_convert_encoding($creator_url, 'UTF-8', 'ISO-8859-1');
    }
    
    $r = db_select('my_creators', 'c')
            ->fields('c')
            ->condition('my_url', $creator_url)
            ->execute()
            ->fetchAll(PDO::FETCH_NUM);

    If your clients want to use IE9 they will not succeed whatever you do at your end, according to my tests on the Windows 7 VHD with IE9.  Disappointed but not really surprised by that result.


    Rob

    • Marqué comme réponse fistikis samedi 5 mai 2012 05:21
    vendredi 4 mai 2012 11:35
  • I was already in the process (since I started this post) of removing the need for URLs with special characters from my site, so I will stick to that since it seems the only safe way to go, if I don't want to restrict the browser selection of my users. I guess, the workarounds will stay for some more time, but I hope to get rid of them soon.

    Thank you for all your time and efforts Rob, your debugging was invaluable. I consider this case closed, and I am happy we finally discovered who the culprit was, even though we could not "get him out of the way".

    Cheers!


    Fistix

    samedi 5 mai 2012 05:21
  • Somehow I managed to have this problem again and this time I needed to have a solution that would work for everything. So based on the great help of Robert Johnson which I marked as answer and very helpful, I "patched" Drupal 7 so it is now working as expected. Here is what to do:

    You just have to add few lines of code at two specific drupal distribution files.

    First the file modules\dblog\dblog.module, go to function dblog_watchdog(array $log_entry) and add the following lines right at the beginning (in Drupal 7.15 this is line 140):

      if (!mb_detect_encoding($log_entry['message'], 'UTF-8', true))
              $log_entry['message'] = mb_convert_encoding($log_entry['message'], 'UTF-8', 'ISO-8859-1');
      if (!mb_detect_encoding($log_entry['request_uri'], 'UTF-8', true))
              $log_entry['request_uri'] = mb_convert_encoding($log_entry['request_uri'], 'UTF-8', 'ISO-8859-1');
      if (!mb_detect_encoding($log_entry['referer'], 'UTF-8', true))
              $log_entry['referer'] = mb_convert_encoding($log_entry['referer'], 'UTF-8', 'ISO-8859-1');

    Then at file includes\bootstrap.inc, function check_plain($text) which at Drupal 7.15 is at line 1571 add the following lines right at the beginning again:

        if (!mb_detect_encoding($text, 'UTF-8', true))
            $text = mb_convert_encoding($text, 'UTF-8', 'ISO-8859-1');
    

    I really banged my head against a wall several times these months and I am very happy to have found a solution that works always. Ofcourse, Robert's help has been invaluable.

    Cheers!

    samedi 11 août 2012 17:27
  • Might be worth submitting back to Drupal? :)

    Cheers,

    Jonathan


    This posting is provided 'AS IS' with no warranties, and confers no rights.

    samedi 11 août 2012 23:36