none
Getting exception of E_RUNTIME_USER_STRINGTOOBIG - String size 1422375 exceeds the maximum allowed size of 131072

    Question

  • I'm getting the following error from my u-sql, running locally.  Here's the error:

    An unhandled
    exception of type
    'Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.ScopeDebugException'
    occurred in Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.dll
    
     
    
    Additional
    information:
    {"diagnosticCode":195887119,"severity":"Error","component":"RUNTIME","source":"User","errorId":"E_RUNTIME_USER_STRINGTOOBIG","message":"String
    size 1422375 exceeds the maximum allowed size of
    131072","description":"","resolution":"","helpLink":"","details":"String
    size 1422375 exceeds the maximum allowed size of 131072"

    And here is my u-sql:

    @rawLogRecords =
      EXTRACT line string
      FROM @"D:\New Logs\f5.log-201701{*}.gz"  
      USING Extractors.Text(encoding: System.Text.Encoding.UTF8, silent: true);
    
    @logRecordsWithFields =
       PROCESS @rawLogRecords
       PRODUCE
                Month string,
                Day string,
                Time string,
                DeviceName string,
                LogLevel string,
                ProcessName string,
                LogType string,
                Message string,
                Client string,
                WebUrl string
       USING new USQL_UDO.F5.LogRecordExtractor();
        
      
       @rs1 =
        SELECT Client, WebUrl
        FROM @logRecordsWithFields
    WHERE LogType LIKE "%HTTP_RESPONSE%" 
          AND 
          WebUrl LIKE "%.asmx%"
    GROUP BY WebUrl, Client;
    
    OUTPUT @rs1   
        TO @"D:\F5 Logs\output.txt"
        USING Outputters.Text();

    The error is being raised on the last OUTPUT command.   I'm new to U-SQL and I'm stumped on how to debug.   There must be a way to look at the @rs1 recordset to assist the debugging but haven't found any tips online.   Any suggestions?

    Here's the full exception:

    An unhandled exception of type 'Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.ScopeDebugException' occurred in Microsoft.Cosmos.ScopeStudio.BusinessObjects.Debugger.dll

    Additional information: {"diagnosticCode":195887119,"severity":"Error","component":"RUNTIME","source":"User","errorId":"E_RUNTIME_USER_STRINGTOOBIG","message":"String size 1422375 exceeds the maximum allowed size of 131072","description":"","resolution":"","helpLink":"","details":"String size 1422375 exceeds the maximum allowed size of 131072","internalDiagnostics":"     7fed3661607\tScopeEngine!?ToStringInternal@KeySampleCollection@SSLibV3@ScopeEngine@@AEAA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ + 11b7\t\n     7fed36071d1\tScopeEngine!??0ExceptionWithStack@ScopeEngine@@QEAA@W4ErrorNumber@1@AEBV?$initializer_list@VScopeErrorArg@ScopeCommon@@@std@@_N@Z + 121\t\n     7fed3607d8b\tScopeEngine!??0RuntimeException@ScopeEngine@@QEAA@W4ErrorNumber@1@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z + 8b\t\n     7fed55c562b\t__ScopeCodeGenEngine__!ScopeEngine::ElementTraits<char>::VerifyLimit + 15b\td:\\vstssource\\emasonsourcecode\\architecture\\noeldevsandbox\\f5logparsing\\f5logparser\\usqlapplication1\\bin\\debug\\f69e65693843fccf\\script.usql.debug_d2493d5083ae938d\\scopecontainers.h line:3048\n     7fed55c1ec8\t__ScopeCodeGenEngine__!ScopeEngine::FString::MoveFrom + c8\td:\\vstssource\\emasonsourcecode\\architecture\\noeldevsandbox\\f5logparsing\\f5logparser\\usqlapplication1\\bin\\debug\\f69e65693843fccf\\script.usql.debug_d2493d5083ae938d\\scopecontainers.h line:4086\n     7fed55afbff\t__ScopeCodeGenEngine__!ScopeEngine::TextInputStream<ScopeEngine::TextInputStreamTraitsConst<44,0,0,0,1,4,1,1,0,0,0>,ScopeEngine::GZipInput>::Read<ScopeEngine::FString> + a3f\td:\\vstssource\\emasonsourcecode\\architecture\\noeldevsandbox\\f5logparsing\\f5logparser\\usqlapplication1\\bin\\debug\\f69e65693843fccf\\script.usql.debug_d2493d5083ae938d\\scopeio.h line:5062\n     7fed55b8c50\t__ScopeCodeGenEngine__!ScopeEngine::TextExtractPolicy<Extract_0_Data0,1>::Deserialize + 60\td:\\vstssource\\emasonsourcecode\\architecture\\noeldevsandbox\\f5logparsing\\f5logparser\\usqlapplication1\\bin\\debug\\f69e65693843fccf\\script.usql.debug_d2493d5083ae938d\\__scopecodegenengine__.dll.cpp line:944\n     7fed55bc6fe\t__ScopeCodeGenEngine__!ScopeEngine::Extractor<Extract_0_Data0,ScopeEngine::TextExtractPolicy<Extract_0_Data0,1>,ScopeEngine::TextInputStream<ScopeEngine::TextInputStreamTraitsConst<44,0,0,0,1,4,1,1,0,0,0>,ScopeEngine::GZipInput> >::GetNextRowImpl + 5e\td:\\vstssource\\emasonsourcecode\\architecture\\noeldevsandbox\\f5logparsing\\f5logparser\\usqlapplication1\\bin\\debug\\f69e65693843fccf\\script.usql.debug_d2493d5083ae938d\\scopeoperators.h line:630\n     7fed55bce11\t__ScopeCodeGenEngine__!ScopeEngine::ParallelUnionAll<ScopeEngine::Extractor<Extract_0_Data0,ScopeEngine::TextExtractPolicy<Extract_0_Data0,1>,ScopeEngine::TextInputStream<ScopeEngine::TextInputStreamTraitsConst<44,0,0,0,1,4,1,1,0,0,0>,ScopeEngine::GZipInput> >,Extract_0_Data0,0>::GetNextRowImpl + 51\td:\\vstssource\\emasonsourcecode\\architecture\\noeldevsandbox\\f5logparsing\\f5logparser\\usqlapplication1\\bin\\debug\\f69e65693843fccf\\script.usql.debug_d2493d5083ae938d\\scopeoperators.h line:12929\n     7fe8f27ac04\t(no module)!(no name)\t\n"}


    Noel Austin

    Wednesday, February 15, 2017 6:33 PM

Answers

  • Found this issue.   One of the records in my input files was millions of chars.  Removing this offending record allowed the file to be processed.    If anyone from the ADLA dev team is reading this,  it would have made troubleshooting easier if the error had been raised from EXTRACT statement instead of the OUTPUT statement.

    Noel Austin

    • Marked as answer by noelaustin Monday, February 27, 2017 4:32 PM
    Monday, February 27, 2017 3:36 PM