failed to flush chunk

• 8 September 2023

The Promtail configuration contains a __path__ entry to a directory that Promtail cannot find. [2022/03/24 04:19:52] [debug] [http_client] not using http_proxy for header I can see majority of the logs coming in, however I've noticed the following errors from fluent-bit pods running on each kubern. Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [outputes.0] task_id=10 assigned to thread #1 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available For debugging you could use tcpdump: sudo tcpdump -i eth0 tcp port 24224 -X -s 0 -nn. Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [task] created task=0x7ff2f183a840 id=13 OK [2022/03/22 03:57:49] [ warn] [engine] failed to flush chunk '1-1647920670.171006292.flb', retry in 1657 seconds: task_id=477, input=tail.0 > output=es.0 (out_id=0) Helm chart configuration. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [ warn] [engine] failed to flush chunk '1-1648192120.74298017.flb', retry in 9 seconds: task_id=14, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [upstream] KA connection #36 to 10.3.4.84:9200 is now available [2022/03/24 04:21:08] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Name es Once set the username/password using hardcode. Fluentd does not handle a large number of chunks well when starting up, so that can be a problem as well. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"JOMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:19:59] [debug] [outputes.0] task_id=2 assigned to thread #1 [2022/03/24 04:20:51] [debug] [out coro] cb_destroy coro_id=6 Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [retry] new retry created for task_id=3 attempts=1 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"IOMoun8BI6SaBP9lIP7t","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [http_client] not using http_proxy for header "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"deMnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. next_retry=2019-01-27 19:00:14 -0500 error_class="ArgumentError" error="Data too big (189382 bytes), would create more than 128 chunks!" plugin_id="object:3fee25617fbc" Because of this cache memory increases and td-agent fails to send messages to graylog Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [input chunk] update output instances with new chunk size diff=697 [2022/03/24 04:19:47] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:40 pm | [2022/03/25 07:08:40] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [retry] new retry created for task_id=10 attempts=1 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Host 10.3.4.84 Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=104226845 events: IN_ATTRIB Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [outputes.0] HTTP Status=200 URI=/_bulk #Write_Operation upsert Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [outputes.0] HTTP Status=200 URI=/_bulk [2022/03/24 04:19:24] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) I'm trying to configure Loki to use Apache Cassandra both for index and chunk storage. Fri, Mar 25 2022 3:08:23 pm | [2022/03/25 07:08:23] [debug] [out coro] cb_destroy coro_id=2 * Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [input chunk] update output instances with new chunk size diff=1083 Logstash_Format On $ sudo kubectl logs -n rtf -l app=external-log-forwarder [2021/03/01 12:55:57] [ warn . Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. {"took":2217,"errors":true,"items":[{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"yeMnun8BI6SaBP9lo-jn","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Logs not being flushed after x amount of time. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=69479190 removing file name /var/log/containers/hello-world-dsfcz_argo_main-13bb1b2c7e9d3e70003814aa3900bb9aef645cf5e3270e3ee4db0988240b9eff.log hi @yangtian9999 Fri, Mar 25 2022 3:08:47 pm | [2022/03/25 07:08:47] [debug] [outputes.0] HTTP Status=200 URI=/_bulk I am seeing this in fluentd logs in kubernetes. Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [ warn] [engine] failed to flush chunk '1-1648192130.216865179.flb', retry in 6 seconds: task_id=20, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [input chunk] update output instances with new chunk size diff=695 Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [outputes.0] task_id=20 assigned to thread #1 [2022/04/17 14:48:10] [ warn] [engine] failed to flush chunk '1-1650206880.316011791.flb', retry in 16 seconds: task_id=4, input=tail.0 > output=es.0 (out_id=0 . [2022/03/24 04:21:08] [debug] [retry] re-using retry for task_id=1 attempts=5 Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Results are the same. [2022/03/24 04:20:36] [ warn] [http_client] cannot increase buffer: current=512000 requested=544768 max=512000 Fri, Mar 25 2022 3:08:22 pm | [2022/03/25 07:08:22] [debug] [retry] new retry created for task_id=4 attempts=1 Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=69464185 file has been deleted: /var/log/containers/hello-world-ctlp5_argo_main-276b9a264b409e931e48ca768d7a3f304b89c6673be86a8cc1e957538e9dd7ce.log Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [debug] [http_client] not using http_proxy for header TLS error: unexpected EOF Issue #6165 fluent/fluent-bit Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ffffhello-world-dcqbx_argo_wait-6b82c7411c8433b5e5f14c56f4b810dc3e25a2e7cfb9e9b107b9b1d50658f5e2.log, inode 67891711 [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ffffhello-world-dcqbx_argo_wait-6b82c7411c8433b5e5f14c56f4b810dc3e25a2e7cfb9e9b107b9b1d50658f5e2.log, inode 67891711 edited Jan 15, 2020 at 19:20. Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [input chunk] update output instances with new chunk size diff=634 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [retry] re-using retry for task_id=0 attempts=3 Fri, Mar 25 2022 3:08:48 pm | [2022/03/25 07:08:48] [debug] [retry] new retry created for task_id=18 attempts=1 Fri, Mar 25 2022 3:08:27 pm | [2022/03/25 07:08:27] [debug] [input chunk] update output instances with new chunk size diff=862 Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [input chunk] update output instances with new chunk size diff=661 "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"MuMmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:32 pm | [2022/03/25 07:08:32] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 is now available Fri, Mar 25 2022 3:08:30 pm | [2022/03/25 07:08:30] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:50 pm | [2022/03/25 07:08:50] [debug] [retry] re-using retry for task_id=13 attempts=2 Fri, Mar 25 2022 3:08:36 pm | [2022/03/25 07:08:36] [ warn] [engine] failed to flush chunk '1-1648192099.641327100.flb', retry in 11 seconds: task_id=2, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:44 pm | [2022/03/25 07:08:44] [debug] [task] created task=0x7ff2f183afc0 id=17 OK Fluent-bit is taking long time to uninstall in kubernetes #2411 - Github Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [retry] new retry created for task_id=15 attempts=1 [2022/03/24 04:20:20] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/hello-world-g74nr_argo_wait-227a0fdb4663e03fecebe61f7b6bfb6fdd2867292cacfe692dc15d50a73f29ff.log, inode 1885001 Fri, Mar 25 2022 3:08:39 pm | [2022/03/25 07:08:39] [ warn] [engine] failed to flush chunk '1-1648192107.811048259.flb', retry in 20 seconds: task_id=6, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [upstream] KA connection #118 to 10.3.4.84:9200 has been assigned (recycled) [2022/03/24 04:21:06] [debug] [outputes.0] task_id=1 assigned to thread #0 Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [outputes.0] HTTP Status=200 URI=/_bulk "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"f-Mnun8BI6SaBP9l3vN-","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamicall The most common reason i've seen for "failed to flush chunk" is that the batch upload queue on the ES cluster is full. privacy statement. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. failed to flush chunk Fri, Mar 25 2022 3:08:46 pm | [2022/03/25 07:08:46] [debug] [retry] re-using retry for task_id=12 attempts=2 Fri, Mar 25 2022 3:08:49 pm | [2022/03/25 07:08:49] [debug] [http_client] not using http_proxy for header Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [upstream] KA connection #35 to 10.3.4.84:9200 has been assigned (recycled) Fri, Mar 25 2022 3:08:28 pm | [2022/03/25 07:08:28] [debug] [retry] re-using retry for task_id=1 attempts=2 failed to flush the buffer fluentd. Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [outputes.0] HTTP Status=200 URI=/_bulk Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [outputes.0] task_id=9 assigned to thread #1 Fri, Mar 25 2022 3:08:41 pm | [2022/03/25 07:08:41] [debug] [input:tail:tail.0] inode=69179617 events: IN_MODIFY Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [debug] [input:tail:tail.0] inode=69336502 removing file name /var/log/containers/hello-world-bjfnf_argo_wait-8f0faa126a1c36d4e0d76e1dc75485a39ecc2d43a4efc46ae7306f4b86ea9964.log Fri, Mar 25 2022 3:08:42 pm | [2022/03/25 07:08:42] [debug] [task] created task=0x7ff2f183ade0 id=16 OK "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"N-Mmun8BI6SaBP9luqVq","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. Fri, Mar 25 2022 3:08:33 pm | [2022/03/25 07:08:33] [debug] [outputes.0] task_id=11 assigned to thread #1 [2022/03/24 04:21:08] [debug] [out coro] cb_destroy coro_id=7 Fri, Mar 25 2022 3:08:29 pm | [2022/03/25 07:08:29] [debug] [input chunk] update output instances with new chunk size diff=695 Fri, Mar 25 2022 3:08:38 pm | [2022/03/25 07:08:38] [debug] [outputes.0] HTTP Status=200 URI=/_bulk "}}},{"create":{"_index":"logstash-2022.03.24","_type":"_doc","_id":"I-Mmun8BI6SaBP9lh4vZ","status":400,"error":{"type":"mapper_parsing_exception","reason":"Could not dynamically add mapping for field [app.kubernetes.io/instance]. [2022/03/24 04:20:20] [debug] [input:tail:tail.0] inode=1756313 removing file name /var/log/containers/hello-world-7mwzw_argo_wait-970c00b906c36cb89ed77fe3fa3cd1abc2702078fee737da0062d3b25680bf9c.log Using Cassandra to store both indexes and chunks #1503 - Github Fri, Mar 25 2022 3:08:21 pm | [2022/03/25 07:08:21] [ warn] [engine] failed to flush chunk '1-1648192100.653122953.flb', retry in 11 seconds: task_id=3, input=tail.0 > output=es.0 (out_id=0) Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [http_client] not using http_proxy for header Existing mapping for [kubernetes.labels.app] must be of type object but found [text]. Fri, Mar 25 2022 3:08:51 pm | [2022/03/25 07:08:51] [debug] [input chunk] update output instances with new chunk size diff=693 Fluentbit failed to send logs to elasticsearch ( Failed to flush chunk [2022/03/24 04:21:20] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/hello-world-89knq_argo_wait-a7f77229883282b7aebce253b8c371dd28e0606575ded307669b43b272d9a2f4.log, inode 1772851 [2022/03/24 04:19:49] [ warn] [engine] failed to flush chunk '1-1648095560.254537600.flb', retry in 15 seconds: task_id=1, input=tail.0 > output=es.0 (out_id=0)

Florida Man December 18, 2003, Parrs Wood High School Term Dates 2020 2021, Thrasymachus' Definition Of Justice, Articles F