Segmentation fault in decode_varbyte (src/rumtsquery.c:236)

[alisnic]

Hi, this is a followup from #62. We upgraded our production systems, and while the rate of segmentation faults decreased, they are still occurring. Please advise on the next steps

System info

root@8669f1edccb6:/var/lib/systemd/coredump# apt-cache policy postgresql-10
postgresql-10:
  Installed: 10.9-1.pgdg90+1
  Candidate: 10.9-1.pgdg90+1
  Version table:
 *** 10.9-1.pgdg90+1 500
        500 http://apt.postgresql.org/pub/repos/apt stretch-pgdg/main amd64 Packages
        100 /var/lib/dpkg/status
root@8669f1edccb6:/var/lib/systemd/coredump# apt-cache policy postgresql-10-rum
postgresql-10-rum:
  Installed: 1.3.6-1.pgdg90+1
  Candidate: 1.3.6-1.pgdg90+1
  Version table:
 *** 1.3.6-1.pgdg90+1 500
        500 http://apt.postgresql.org/pub/repos/apt stretch-pgdg/main amd64 Packages
        100 /var/lib/dpkg/status

GDB backtrace

#0  decode_varbyte (ptr=ptr@entry=0x7ffc4c457fd8) at src/rumtsquery.c:236
#1  0x00007f6d2ee2a408 in ruminv_tsvector_consistent (fcinfo=<optimized out>) at src/rumtsquery.c:509
#2  0xfffffffe2ee39f92 in ?? ()
#3  0x000055dbada64fa0 in ?? ()
#4  0x00000000fffffffe in ?? ()
#5  0xfffffffe00000000 in ?? ()
#6  0x000a550000000064 in ?? ()
#7  0x000055dbfffffffe in ?? ()
#8  0xfffffffe00000001 in ?? ()
#9  0x000055dbad166f48 in ?? ()
#10 0x00000000fffffffe in ?? ()
#11 0xfffffffe00000000 in ?? ()
#12 0x000055dbac80bdb9 in ?? ()
#13 0x000055dbfffffffe in ?? ()
#14 0xfffffffeac80bcf0 in ?? ()
#15 0x000055dbac80be70 in ?? ()
#16 0x000055dbfffffffe in ?? ()
#17 0xfffffffe4c45903c in ?? ()
#18 0x000055dbabeb5733 in hash_search_with_hash_value (hashp=0xfffffffe, keyPtr=<error reading variable: Cannot access memory at address 0x7ffcffffffbe>,
    hashvalue=<optimized out>, action=<error reading variable: Cannot access memory at address 0x7ffcffffffa6>,
    foundPtr=<error reading variable: Cannot access memory at address 0x7ffcffffffae>) at ./build/../src/backend/utils/hash/dynahash.c:983

GDB full backtrace

#0  decode_varbyte (ptr=ptr@entry=0x7ffc4c457fd8) at src/rumtsquery.c:236
        val = 4294967295
        p = 0x550100000000 <error: Cannot access memory at address 0x550100000000>
        c = <optimized out>
#1  0x00007f6d2ee2a408 in ruminv_tsvector_consistent (fcinfo=<optimized out>) at src/rumtsquery.c:509
        sumVal = <optimized out>
        sum = <optimized out>
        not = <optimized out>
        num = <optimized out>
        index = <optimized out>
        ptr = 0x5500ffffffff <error: Cannot access memory at address 0x5500ffffffff>
        ptrEnd = 0x55dbac876383 ""
        size = <optimized out>
        child = 0x7ffc4c457fd4
        check = 0x55dbac80be58 "\001"
        nkeys = 2
        recheck = <optimized out>
        addInfo = 0x55dbac80be70
        addInfoIsNull = 0x55dbac80be90 ""
        res = 0 '\000'
        allFalse = 0 '\000'
        i = 0
        lastIndex = 588
        nodes = {{sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2,
            not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0,
            parent = -2, not = 0 '\000'}, {sum = 1279624208, parent = -2, not = -100 '\234'}, {sum = 21979, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2,
            not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 65, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = -4 '\374'}, {
            sum = -1410660158, parent = -2, not = -96 '\240'}, {sum = 32764, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 152832,
            parent = -2, not = 0 '\000'}, {sum = 65, parent = -2, not = -32 '\340'}, {sum = 32764, parent = -2, not = -37 '\333'}, {sum = 1279623648, parent = -2,
            not = 67 'C'}, {sum = 0, parent = -2, not = -4 '\374'}, {sum = -1410663012, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0,
            parent = -2, not = 0 '\000'}, {sum = 65536, parent = -2, not = 0 '\000'}, {sum = -1410520674, parent = -2, not = -32 '\340'}, {sum = 32764, parent = -2,
            not = -37 '\333'}, {sum = 1279624512, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 65 'A'}, {sum = 0,
            parent = -2, not = 0 '\000'}, {sum = 1279623616, parent = -2, not = -99 '\235'}, {sum = 32621, parent = -2, not = -4 '\374'}, {sum = 1279624760,
            parent = -2, not = -112 '\220'}, {sum = 21979, parent = -2, not = -37 '\333'}, {sum = -1400418632, parent = -2, not = -32 '\340'}, {sum = 32764,
            parent = -2, not = -4 '\374'}, {sum = 1, parent = -2, not = 56 '8'}, {sum = 32764, parent = -2, not = -37 '\333'}, {sum = 1279624688, parent = -2,
            not = -12 '\364'}, {sum = 21979, parent = -2, not = 0 '\000'}, {sum = -1410640077, parent = -2, not = -48 '\320'}, {sum = 32621, parent = -2,
            not = 7 '\a'}, {sum = 1, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = -37 '\333'}, {sum = 0, parent = -2, not = -32 '\340'}, {sum = 32764,
            parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 100 'd'}, {sum = 480512, parent = -2, not = -37 '\333'}, {sum = 1279624760, parent = -2,
            not = 1 '\001'}, {sum = 0, parent = -2, not = -4 '\374'}, {sum = 1279624776, parent = -2, not = 80 'P'}, {sum = 32764, parent = -2, not = -4 '\374'}, {
            sum = 11, parent = -2, not = 44 ','}, {sum = 0, parent = -2, not = -37 '\333'}, {sum = -1400414584, parent = -2, not = 1 '\001'}, {sum = 0, parent = -2,
            not = -4 '\374'}, {sum = -1410663012, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {
            sum = 87296, parent = -2, not = 0 '\000'}, {sum = 1279624080, parent = -2, not = -112 '\220'}, {sum = 32764, parent = -2, not = -37 '\333'}, {
            sum = 1279624944, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 96 '`'}, {sum = 0, parent = -2,
            not = 0 '\000'}, {sum = -1400438784, parent = -2, not = 96 '`'}, {sum = 32764, parent = -2, not = -4 '\374'}, {sum = 1, parent = -2, not = 0 '\000'}, {
            sum = 32764, parent = -2, not = -37 '\333'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2,
            not = 96 '`'}, {sum = 0, parent = -2, not = 113 'q'}, {sum = 1279625008, parent = -2, not = -62 '\302'}, {sum = 21979, parent = -2, not = -4 '\374'}, {
            sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 96, parent = -2, not = 96 '`'}, {sum = 0, parent = -2,
            not = -4 '\374'}, {sum = -1401588392, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {
            sum = -1360691111, parent = -2, not = -37 '\333'}, {sum = -1844972346, parent = -2, not = 32 ' '}, {sum = 32764, parent = -2, not = 89 'Y'}, {sum = 16966,
            parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = -4 '\374'}, {sum = -1410786894, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2,
            not = 113 'q'}, {sum = 1279624352, parent = -2, not = -40 '\330'}, {sum = 21979, parent = -2, not = -4 '\374'}, {sum = -1410781857, parent = -2,
            not = 52 '4'}, {sum = 57, parent = -2, not = -4 '\374'}, {sum = 1279624336, parent = -2, not = -32 '\340'}, {sum = 32625, parent = -2, not = 6 '\006'}, {
            sum = -1401581864, parent = -2, not = 32 ' '}, {sum = 21979, parent = -2, not = 2 '\002'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2,
            not = -37 '\333'}, {sum = 184, parent = -2, not = 2 '\002'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = -1401753736, parent = -2, not = 0 '\000'}, {
            sum = 0, parent = -2, not = 0 '\000'} <repeats 11 times>, {sum = 0, parent = -2, not = 48 '0'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 128,
            parent = -2, not = 1 '\001'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 1279624608, parent = -2, not = -126 '\202'}, {sum = 21979, parent = -2,
            not = 0 '\000'}, {sum = 1375059936, parent = -2, not = -32 '\340'}, {sum = 32625, parent = -2, not = -37 '\333'}, {sum = 1279624656, parent = -2,
            not = 118 'v'}, {sum = 21979, parent = -2, not = 113 'q'}, {sum = 16384, parent = -2, not = -16 '\360'}, {sum = 32764, parent = -2, not = -37 '\333'}, {
            sum = 1, parent = -2, not = -128 '\200'}, {sum = 0, parent = -2, not = -4 '\374'}, {sum = -1414052728, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2,
            not = -37 '\333'}, {sum = 1, parent = -2, not = -88 '\250'}, {sum = 0, parent = -2, not = -4 '\374'}, {sum = 2068021760, parent = -2, not = 65 'A'}, {
            sum = 0, parent = -2, not = -37 '\333'}, {sum = 1279625072, parent = -2, not = 24 '\030'}, {sum = 21979, parent = -2, not = 0 '\000'}, {sum = 1024,
            parent = -2, not = 56 '8'}, {sum = 21979, parent = -2, not = -37 '\333'}, {sum = -1391040664, parent = -2, not = 120 'x'}, {sum = 21979, parent = -2,
            not = -4 '\374'}, {sum = -1410520674, parent = -2, not = 112 'p'}, {sum = 32764, parent = -2, not = -37 '\333'}, {sum = 1279624896, parent = -2,
            not = -111 '\221'}, {sum = 21979, parent = -2, not = -37 '\333'}, {sum = 1279625072, parent = -2, not = 48 '0'}, {sum = 32764, parent = -2,
            not = -37 '\333'}, {sum = 1279625888, parent = -2, not = -100 '\234'}, {sum = 21979, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {
            sum = 0, parent = -2, not = 0 '\000'}, {sum = 1258, parent = -2, not = -48 '\320'}, {sum = 21979, parent = -2, not = -4 '\374'}, {sum = -1410660158,
            parent = -2, not = 48 '0'}, {sum = 32764, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 152832, parent = -2,
            not = 0 '\000'}, {sum = 1258, parent = -2, not = -54 '\312'}, {sum = 32764, parent = -2, not = 0 '\000'}, {sum = -1391040664, parent = -2, not = 0 '\000'},
          {sum = 0, parent = -2, not = 0 '\000'}, {sum = -1391039936, parent = -2, not = 61 '='}, {sum = 21979, parent = -2, not = -37 '\333'}, {sum = -1391039720,
            parent = -2, not = -44 '\324'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = -1409375255, parent = -2, not = -64 '\300'}, {sum = 32764, parent = -2,
            not = -37 '\333'}, {sum = 1279625184, parent = -2, not = 64 '@'}, {sum = 21979, parent = -2, not = -37 '\333'}, {sum = 16, parent = -2, not = 16 '\020'}, {
            sum = 0, parent = -2, not = -4 '\374'}, {sum = 1279625232, parent = -2, not = -25 '\347'}, {sum = 32764, parent = -2, not = 0 '\000'}, {sum = 0,
            parent = -2, not = 48 '0'}, {sum = 32764, parent = -2, not = -37 '\333'}, {sum = -1391037496, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2,
            not = -4 '\374'}, {sum = -1412797389, parent = -2, not = -24 '\350'}, {sum = 21979, parent = -2, not = -37 '\333'}, {sum = 0, parent = -2, not = 0 '\000'},
          {sum = 0, parent = -2, not = -4 '\374'}, {sum = -1400415864, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = -37 '\333'}, {sum = 0, parent = -2,
            not = 9 '\t'}, {sum = 21979, parent = -2, not = -37 '\333'}, {sum = -1393797512, parent = -2, not = 112 'p'}, {sum = 32764, parent = -2, not = -37 '\333'},
          {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2, not = -22 '\352'}, {sum = 0, parent = -2,
            not = 89 'Y'}, {sum = 1279626400, parent = -2, not = -62 '\302'}, {sum = 21979, parent = -2, not = -4 '\374'}, {sum = 0, parent = -2, not = 0 '\000'}, {
            sum = 0, parent = -2, not = 0 '\000'}, {sum = 1258, parent = -2, not = -22 '\352'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 0, parent = -2,
            not = 0 '\000'}, {sum = 0, parent = -2, not = 0 '\000'}, {sum = 65, parent = -2, not = 7 '\a'}...}
        __func__ = "ruminv_tsvector_consistent"
#2  0xfffffffe2ee39f92 in ?? ()
No symbol table info available.
#3  0x000055dbada64fa0 in ?? ()
No symbol table info available.
#4  0x00000000fffffffe in ?? ()
No symbol table info available.
#5  0xfffffffe00000000 in ?? ()

[za-arthur]

Hello Andrei,
Thank you for the issue! Can you describe your test case? I guess you can't show full queries.

[alisnic]

hi @za-arthur !

So this time unfortunately I don't have a test scenario which reproduces the issue. It occurs seemingly randomly and I was unable to pin down the root cause. All I can provide is a redacted query that was executed before the crash:

2019-08-06 14:45:41 UTC DETAIL:  Failed process was running:       SELECT *
          FROM tablename
          WHERE
            (int_column1 = 188 OR int_column1 IS NULL) AND
            int_column2 = 2 AND
            sring_column1 IS NOT NULL AND
            to_tsvector('simple', 'redacted search string') @@ tsquery_column1 AND
            'redacted search string' ~* ('(^|A| +)' || string_column2 || '( +|Z|$)')
          ORDER BY length(string_column2) DESC LIMIT 1;

Note here that tsquery_column1 is a column of type tsquery which we store inside the table. So our use-case is a bit reverse. The search text is received as input, while the query that matches this text is inside our DB

[alisnic]

Hi @za-arthur! Is there any progress on this? Are there any ways I can help to speed up the process?

[za-arthur]

Hello!
Unfortunately I couldn't reproduce it yet. And I have another urgent project and it is hard to say when I'll fix the issue.

[alisnic]

Ok, then meanwhile I will try to come up with repro steps

…

On Sat, Aug 17, 2019 at 12:37 Artur ***@***.***> wrote: Hello! Unfortunately I couldn't reproduce it yet. And I have another urgent project and it is hard to say when I'll fix the issue. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#63?email_source=notifications&email_token=AANQG32V6JTOLJFQSTQZ5LLQE7BGLA5CNFSM4IJ6M3M2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4QHTVY#issuecomment-522222039>, or mute the thread <https://github.com/notification

[alisnic]

Hi @za-arthur, I have new insight regarding this bug. We found out that the server that was running the postgres instance with the extension had periods of time during the day when it was low on RAM due to other processes running on the same server. After we fixed the issue with free RAM, the issue was no longer reproducing. So I guess for us the problem is solved. Though I'm not sure that segfault is a proper behavior of a program when it is starving on memory

[alisnic]

the culprit was a malformed index. rebuilding it via REINDEX INDEX fixed the problem.