How to detect the issue of CPU spike for Tableau cluster on Linux

-

 Hi!

Today I would like to share how I am detecting the problem of CPU spikes.

This morning we had a few CPU spikes. So let’s check

uptime

10:19:23 up 116 days, 12:13, 1 user, load average: 0.36, 0.81, 2.57

dmesg --human | tail -n 50

[Dec30 16:15] audit: type=1327 audit(1703952955.024:85698311): proctitle=”/var/opt/tableau/tableau_server/data/tabsvc/services/backgrounder_0.20231.23.0806.1229/bin/run-backgrounder”

[ +0.631996] audit: type=1327 audit(1703952955.024:85698311): proctitle=”/var/opt/tableau/tableau_server/data/tabsvc/services/backgrounder_0.20231.23.0806.1229/bin/run-backgrounder”

[ +0.002027] audit: type=1300 audit(1703952955.029:85698312): arch=c000003e syscall=42 success=yes exit=0 a0=47 a1=7f5bb5d4b5c0 a2=10 a3=6f8da items=0 ppid=91808 pid=67952 auid=4294967295 uid=991 gid=1087000453 euid=991 suid=991 fsuid=991 egid=1087000453 sgid=1087000453 fsgid=1087000453 tty=(none) ses=4294967295 comm=”pool-29-thread-” exe=”/opt/tableau/tableau_server/packages/bin.20231.23.0806.1229/run-backgrounder” key=”network_connect_4"

[ +0.003330] audit: type=1306 audit(1703952955.029:85698312): saddr=020001850A8102DE0000000000000000

[ +0.001054] audit: type=1327 audit(1703952955.029:85698312): proctitle=”/var/opt/tableau/tableau_server/data/tabsvc/services/backgrounder_0.20231.23.0806.1229/bin/run-backgrounder”

[ +0.001971] audit: type=1300 audit(1703952955.037:85698313): arch=c000003e syscall=42 success=yes exit=0 a0=51 a1=7f5bb5d4b5c0 a2=10 a3=6f8da items=0 ppid=91808 pid=67952 auid=4294967295 uid=991 gid=1087000453 euid=991 suid=991 fsuid=991 egid=1087000453 sgid=1087000453 fsgid=1087000453 tty=(none) ses=4294967295 comm=”pool-29-thread-” exe=”/opt/tableau/tableau_server/packages/bin.20231.23.0806.1229/run-backgrounder” key=”network_connect_4"

[ +0.003145] audit: type=1306 audit(1703952955.037:85698313): saddr=020001850A8102DE0000000000000000

[ +0.000914] audit: type=1327 audit(1703952955.037:85698313): proctitle=”/var/opt/tableau/tableau_server/data/tabsvc/services/backgrounder_0.20231.23.0806.1229/bin/run-backgrounder”

[ +0.001850] audit: type=1300 audit(1703952955.042:85698314): arch=c000003e syscall=42 success=yes exit=0 a0=4b a1=7fb19785c840 a2=10 a3=cdace items=0 ppid=91808 pid=110383 auid=4294967295 uid=991 gid=1087000453 euid=991 suid=991 fsuid=991 egid=1087000453 sgid=1087000453 fsgid=1087000453 tty=(none) ses=4294967295 comm=”pool-29-thread-” exe=”/opt/tableau/tableau_server/packages/bin.20231.23.0806.1229/run-backgrounder” key=”network_connect_4"

[ +0.002892] audit: type=1306 audit(1703952955.042:85698314): saddr=020001850A8102DE0000000000000000

[Jan 2 15:15] kauditd_printk_skb: 250 callbacks suppressed

[ +0.000002] audit: type=1327 audit(1704208542.451:87759651): proctitle=”/var/opt/tableau/tableau_server/data/tabsvc/services/backgrounder_1.20231.23.0806.1229/bin/run-backgrounder”

cd /var/opt/tableau/tableau_server/data/tabsvc/services/backgrounder_0.20231.23.0806.1229

cd crashdumps/stacktraces

cat _opt_tableau_tableau_server_packages_hyper.20231.23.0806.1229_hyperd.50889.6.1704225028.txt | grep -i terminated

Program terminated with signal 6, Aborted.

Signal 6 (SIGABRT) = SIGABRT is commonly used by libc and other libraries to abort the program in case of critical errors. For example, glibc sends an SIGABRT in case of a detected double-free or other heap corruptions.

https://faculty.cs.niu.edu/~hutchins/csci480/signals.htm

Therefore, we identified the source of signal 6 to be an OOM issue during the operation of hyperd parking. Consequently, I strongly advise examining the volume of data you are receiving. The process of fetching data through the backgrounder and storing it in hyperd could potentially pose a problem.

Comments

Post a Comment

Popular posts from this blog

Overview and practical use cases with open source tracing tool for Java Apps. Glowroot. Installation

-
Image
In that small article, you can find how to install the agent of Glowroot. Installation of the application is quite simple on agents, and this article shows a scheme for working with a built-in collector, but for industrial installations we use it with a bundle with elasticsearch. The installation can be both for individual nodes and so for the central collector. In this case, consider for a separate node. Prerequisites: {jira_home} - is location of Jira home directory. {jira_installation_directory} is the location of Jira installation directory.   Steps: Downloading the installer wget -c https://github.com/glowroot/glowroot/releases/download/v0.13.6/glowroot-0.13.6-dist.zip To check, create a directory via command   mkdir -p /{jira_home}/glowroot/tmp Let the Jira process write and set ownership for simplicity, but it is recommended that you just give the write ability to the owner of the Jira process. chown -R jira: /{jira_home}/glowroot We additionally set the argument to setenv.sh in
Read more

Atlassian Community, let's collaborate and provide stats to vendors about our SQL index usage

-
  Hi!  Gonchik is in touch. Today I would like to get a little bit of help from you related SQL indexes on your Jira installation (Data Center, Server edition does not matter).    I would like to share a story about  PostgreSQL  and  Jira 8.13.3  ( JSD 4.13.3 )  release.  In that post I will be happy if you answer and comment about your instance indexes, as those info can help to improve the existing situation of heavy indexes.  Typically, it starts from improvement of configuration (e.g. using  https://postgresqltuner.pl/ ),  then query analyzing, adding indexes, after removing as heavy one (shortly  https://wiki.postgresql.org/wiki/Index_Maintenance ) :)    I use like this query in PostgreSQL: SELECT s.schemaname,        s.relname AS tablename,        s.indexrelname AS indexname,        pg_size_pretty(pg_relation_size(s.indexrelid::regclass)) AS index_size,        idx_tup_read,         idx_tup_fetch,         idx_scan FROM pg_catalog.pg_stat_user_indexes s JOIN pg_catalog.pg_index i O
Read more

How only 2 parameters of PostgreSQL reduced anomaly of Jira Data Center nodes

-
Image
  Hello community, Gonchik Tsymzhitov is in touch. Today, I would like to share with you a short story about postgresqltuner.pl in usage.  So from previous articles I hope you configured the PostgreSQL server and postgresqltuner.pl.  After executing command, ./postgresqltuner.pl --ssd for ssd disks server. You can see like this output: postgresqltuner.pl version 1.0.1 [OK]      I can invoke executables Connecting to /var/run/postgresql:5432 database template1 as user 'postgres'... [OK]      The user account used by me for reporting has superuser rights on this PostgreSQL instance =====  OS information  ===== [INFO]    OS: linux Version: 5.4.17-2036.102.0.2.el8uek.x86_64 Arch: x86_64-linux-thread-multi [INFO]    OS total memory: 41.03 GB [OK]      vm.overcommit_memory is adequate: no memory overcommitment [INFO]    Running under a vmware hypervisor [INFO]    Currently used I/O scheduler(s): mq-deadline =====  General instance informations  ===== -----  PostgreSQL version  -----
Read more