none
Microsoft Teams office webhook stopped working!! RRS feed

  • Question

  • Hello,

    We are using Microsoft Teams channels to convey mission critical updates between teams. For that purpose we have utilized Jenkins to post MessageCard via web hook. It worked correctly in past but recently we are seeing the post request is failing randomly throwing the following error

    System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
     

    The most horrific part is the response code 200 which doesn't make any sence. I am posting a sample message here please check,

    {
        "@context": "https://schema.org/extensions",
        "@type": "MessageCard",
        "potentialAction": [
            {
                "@type": "OpenUri",
                "name": "View All Tweets",
                "targets": [
                    {
                        "os": "default",
                        "uri": "https://twitter.com/search?q=%23MicrosoftTeams"
                    }
                ]
            }
        ],
        "sections": [
            {
                "facts": [
                    {
                        "name": "Tweet:",
                        "value": "1. Numbered\r2. List\r"
                    }
                ],
                "text": "A tweet with #MicrosoftTeams has been posted:"
            }
        ],
        "summary": "Tweet Posted",
        "themeColor": "0072C6",
        "title": "Tweet Posted"
    }

    Tuesday, October 8, 2019 6:32 AM

All replies

  • same problem, only started happening in the past 30 mins

    prometheus-msteams_1  | time="2019-10-08T15:56:45Z" level=info msg="/prom-alerts-dev-leo received a request"
    prometheus-msteams_1  | time="2019-10-08T15:56:45Z" level=debug msg="Prometheus Alert: {\"receiver\":\"msteams-default-receiver\",\"status\":\"firing\",\"alerts\":[{\"status\":\"firing\",\"labels\":{\"alertname\":\"Watchdog\",\"severity\":\"none\"},\"annotations\":{\"description\":\"Watchdog\",\"message\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\nDeadMansSnitch integration in PagerDuty.\\n\",\"summary\":\"Watchdog\"},\"startsAt\":\"2019-10-08T15:56:40.689259224Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"http://1a653efc6574:9090/graph?g0.expr=vector%281%29\\u0026g0.tab=1\"}],\"groupLabels\":{\"alertname\":\"Watchdog\"},\"commonLabels\":{\"alertname\":\"Watchdog\",\"severity\":\"none\"},\"commonAnnotations\":{\"description\":\"Watchdog\",\"message\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\nThis alert is always firing, therefore it should always be firing in Alertmanager\\nand always fire against a receiver. There are integrations with various notification\\nmechanisms that send a notification when this alert is not firing. For example the\\nDeadMansSnitch integration in PagerDuty.\\n\",\"summary\":\"Watchdog\"},\"externalURL\":\"http://d89d49f9d850:9093\",\"version\":\"4\",\"groupKey\":\"{}/{}:{alertname=\\\"Watchdog\\\"}\"}"
    prometheus-msteams_1  | time="2019-10-08T15:56:45Z" level=debug msg="Alert rendered in template file: \n{\n  \"@type\": \"MessageCard\",\n  \"@context\": \"http://schema.org/extensions\",\n  \"themeColor\": \"808080\",\n  \"summary\": \"firing:Watchdog\",\n  \"title\": \"firing: Prometheus Alert\",\n  \"sections\": [ \n    {\n      \"activityTitle\": \"[Watchdog](http://d89d49f9d850:9093)\",\n      \"facts\": [\n        {\n          \"name\": \"description\",\n          \"value\": \"Watchdog\"\n        },\n        {\n          \"name\": \"message\",\n          \"value\": \"This is an alert meant to ensure that the entire alerting pipeline is functional.\\u000AThis alert is always firing, therefore it should always be firing in Alertmanager\\u000Aand always fire against a receiver. There are integrations with various notification\\u000Amechanisms that send a notification when this alert is not firing. For example the\\u000ADeadMansSnitch integration in PagerDuty.\\u000A\"\n        },\n        {\n          \"name\": \"summary\",\n          \"value\": \"Watchdog\"\n        },\n        {\n          \"name\": \"alertname\",\n          \"value\": \"Watchdog\"\n        },\n        {\n          \"name\": \"severity\",\n          \"value\": \"none\"\n        }\n      ],\n      \"markdown\": true\n    }\n  ],\n  \"potentialAction\": [\n    {\n        \"@context\": \"http://schema.org\",\n        \"@type\": \"ViewAction\",\n        \"name\": \"Runbook\",\n        \"target\": [\n            \"\"\n        ]\n    }\n  ]\n}\n"
    prometheus-msteams_1  | time="2019-10-08T15:56:45Z" level=debug msg="Size of message is 934 Bytes (~0 KB)"
    prometheus-msteams_1  | time="2019-10-08T15:56:45Z" level=info msg="Created a card for Microsoft Teams /prom-alerts-dev-leo"
    prometheus-msteams_1  | time="2019-10-08T15:56:45Z" level=debug msg="Teams message cards: [{\"@type\":\"MessageCard\",\"@context\":\"http://schema.org/extensions\",\"themeColor\":\"808080\",\"summary\":\"firing:Watchdog\",\"title\":\"firing: Prometheus Alert\",\"sections\":[{\"activityTitle\":\"[Watchdog](http://d89d49f9d850:9093)\",\"facts\":[{\"name\":\"description\",\"value\":\"Watchdog\"},{\"name\":\"message\",\"value\":\"This is an alert meant to ensure that the entire alerting pipeline is functional.\\u000AThis alert is always firing, therefore it should always be firing in Alertmanager\\u000Aand always fire against a receiver. There are integrations with various notification\\u000Amechanisms that send a notification when this alert is not firing. For example the\\u000ADeadMansSnitch integration in PagerDuty.\\u000A\"},{\"name\":\"summary\",\"value\":\"Watchdog\"},{\"name\":\"alertname\",\"value\":\"Watchdog\"},{\"name\":\"severity\",\"value\":\"none\"}],\"markdown\":true}],\"potentialAction\":[{\"@context\":\"http://schema.org\",\"@type\":\"ViewAction\",\"name\":\"Runbook\",\"target\":[\"\"]}]}]"
    prometheus-msteams_1  | time="2019-10-08T15:56:46Z" level=info msg="Microsoft Teams response text: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information."
    prometheus-msteams_1  | time="2019-10-08T15:56:46Z" level=info msg="A card was successfully sent to Microsoft Teams Channel. Got http status: 200 OK"



    • Edited by wonderous Tuesday, October 8, 2019 3:59 PM
    Tuesday, October 8, 2019 3:51 PM
  • issue appers to have been resolved in the past 10 minutes.

    Returns 200, with content body '1'.

    $ curl -v -X POST --header "Content-Type: application/json" --data "$( cat - <<'EOF'
    > {
    >     "@type": "MessageCard",
    >     "@context": "http://schema.org/extensions",
    >     "themeColor": "2EB886",
    >     "summary": "some-summary",
    >     "title": "some-title",
    >     "sections": [
    >         {
    >             "title": "some-title",
    >             "text": "this is some test text\n\nthis is another line of test text\n\n`some code`\n\n    some code\n\n    some more code"
    >         }
    >     ]
    > }
    > EOF
    > )" https://outlook.office.com/webhook/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/IncomingWebhook/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    Note: Unnecessary use of -X or --request, POST is already inferred.
    *   Trying 13.107.18.11...
    * TCP_NODELAY set
    * Connected to outlook.office.com (13.107.18.11) port 443 (#0)
    * ALPN, offering h2
    * ALPN, offering http/1.1
    * successfully set certificate verify locations:
    *   CAfile: /etc/ssl/certs/ca-certificates.crt
      CApath: /etc/ssl/certs
    * TLSv1.2 (OUT), TLS handshake, Client hello (1):
    * TLSv1.2 (IN), TLS handshake, Server hello (2):
    * TLSv1.2 (IN), TLS handshake, Certificate (11):
    * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
    * TLSv1.2 (IN), TLS handshake, Server finished (14):
    * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
    * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
    * TLSv1.2 (OUT), TLS handshake, Finished (20):
    * TLSv1.2 (IN), TLS handshake, Finished (20):
    * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
    * ALPN, server accepted to use h2
    * Server certificate:
    *  subject: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; CN=Outlook.office.com
    *  start date: Jul 12 00:00:00 2019 GMT
    *  expire date: Jul 12 12:00:00 2021 GMT
    *  subjectAltName: host "outlook.office.com" matched cert's "Outlook.office.com"
    *  issuer: C=US; O=DigiCert Inc; CN=DigiCert Cloud Services CA-1
    *  SSL certificate verify ok.
    * Using HTTP2, server supports multi-use
    * Connection state changed (HTTP/2 confirmed)
    * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
    * Using Stream ID: 1 (easy handle 0x560cc36d28e0)
    > POST /webhook/d3994817-0134-41b8-94fd-becac151a393@4e4f53c4-6f09-4818-9c77-020eeb613480/IncomingWebhook/2d22e5c35c8e463583b2e1acb3bae0cf/81689d20-a890-4091-9b64-31da5ca8d98e HTTP/2
    > Host: outlook.office.com
    > User-Agent: curl/7.58.0
    > Accept: */*
    > Content-Type: application/json
    > Content-Length: 380
    >
    * Connection state changed (MAX_CONCURRENT_STREAMS updated)!
    * We are completely uploaded and fine
    < HTTP/2 200
    < cache-control: no-cache
    < pragma: no-cache
    < content-length: 1
    < content-type: text/plain; charset=utf-8
    < expires: -1
    < request-id: 47e7070c-d62d-4b7d-acb3-b0fe4e581b6a
    < x-calculatedbetarget: SL2PR01MB3098.apcprd01.prod.exchangelabs.com
    < x-backendhttpstatus: 200
    < x-aspnet-version: 4.0.30319
    < x-cafeserver: SLXP216CA0026.KORP216.PROD.OUTLOOK.COM
    < x-beserver: SL2PR01MB3098
    < x-proxy-backendserverstatus: 200
    < x-powered-by: ASP.NET
    < x-feserver: SLXP216CA0026
    < x-msedge-ref: Ref A: B2F703A3900A4A5A85BEDA4A0073D6D1 Ref B: SG2EDGE0912 Ref C: 2019-10-08T17:36:17Z
    < date: Tue, 08 Oct 2019 17:36:17 GMT
    <
    * Connection #0 to host outlook.office.com left intact


    Bass and treble are inseparable. Music sounds bad with either absent. The same applies to us The Oh Brothers.

    Tuesday, October 8, 2019 5:39 PM