Segmentation fault in "decode_varbyte" (src/rumtsquery.c:228)

[alisnic]

Hi all, I am encountering segfaults with your plugin in our production environment. We were able to reproduce it on a development machine. The repro case involves updating randomly rows in a table and while doing rum queries on that table.

Please let us know how we can fix it. I will be more than happy to open up a PR with a fix, but I need some guidance from you. Please let me know if there's any other information I can provide to help debug the issue

Environment

alexandrt@alexandrt-HP-OMEN:~$ uname -a
Linux alexandrt-HP-OMEN 4.18.0-21-generic #22~18.04.1-Ubuntu SMP Thu May 16 15:07:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

alexandrt@alexandrt-HP-OMEN:~$ psql --version
psql (PostgreSQL) 10.8 (Ubuntu 10.8-0ubuntu0.18.04.1)

select * from pg_extension;
  extname  | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition 
-----------+----------+--------------+----------------+------------+-----------+--------------
 plpgsql   |       10 |           11 | f              | 1.0        |           | 
 btree_gin |    16384 |         2200 | t              | 1.2        |           | 
 pg_trgm   |    16384 |         2200 | t              | 1.3        |           | 
 rum       |    16384 |         2200 | t              | 1.2        |           | 
(4 rows)

gdb backtrace:

gdb) bt
#0  decode_varbyte (ptr=ptr@entry=0x7ffe32bab1f8) at src/rumtsquery.c:228
#1  0x00007fe9b5e93a28 in ruminv_tsvector_consistent (fcinfo=<optimized out>) at src/rumtsquery.c:540
#2  0x00007fe9b5ea2ff8 in FunctionCall10Coll (flinfo=<optimized out>, collation=<optimized out>, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=<optimized out>, 
    arg5=0, arg6=94294333292185, arg7=94294333291792, arg8=94294333291872, arg9=94294333292416, arg10=94294333292496) at src/rumutil.c:986
#3  0x00007fe9b5e9a24b in callConsistentFn (rumstate=rumstate@entry=0x55c29c78bbf8, key=key@entry=0x55c29c3d7e08) at src/rumget.c:165
#4  0x00007fe9b5e9dca2 in keyGetItem (key=0x55c29c3d7e08, tempCtx=0x55c29c37d8e8, rumstate=<optimized out>) at src/rumget.c:1408
#5  scanGetItemRegular (scan=0x55c29c3d7e08, scan=0x55c29c3d7e08, recheck=0x7ffe32bac33f "", item=0x7ffe32bac340, advancePast=0x55c29c78bbe8) at src/rumget.c:1487
#6  scanGetItem (scan=scan@entry=0x55c29c78b3d8, advancePast=advancePast@entry=0x7ffe32bac340, item=item@entry=0x7ffe32bac340, recheck=recheck@entry=0x7ffe32bac33f "") at src/rumget.c:2084
#7  0x00007fe9b5e9e501 in rumgetbitmap (scan=0x55c29c78b3d8, tbm=0x55c29c80cce0) at src/rumget.c:2122
#8  0x000055c29a8a3c92 in index_getbitmap (scan=scan@entry=0x55c29c78b3d8, bitmap=bitmap@entry=0x55c29c80cce0) at ./build/../src/backend/access/index/indexam.c:726
#9  0x000055c29a9f5b56 in MultiExecBitmapIndexScan (node=0x55c29c78b1e8) at ./build/../src/backend/executor/nodeBitmapIndexscan.c:104
#10 0x000055c29a9e853e in MultiExecProcNode (node=<optimized out>) at ./build/../src/backend/executor/execProcnode.c:490
#11 0x000055c29a9f52b9 in BitmapHeapNext (node=node@entry=0x55c29c785ee8) at ./build/../src/backend/executor/nodeBitmapHeapscan.c:117
#12 0x000055c29a9e9607 in ExecScanFetch (recheckMtd=0x55c29a9f47d0 <BitmapHeapRecheck>, accessMtd=0x55c29a9f4860 <BitmapHeapNext>, node=0x55c29c785ee8)
    at ./build/../src/backend/executor/execScan.c:97
#13 ExecScan (node=0x55c29c785ee8, accessMtd=0x55c29a9f4860 <BitmapHeapNext>, recheckMtd=0x55c29a9f47d0 <BitmapHeapRecheck>) at ./build/../src/backend/executor/execScan.c:164
#14 0x000055c29aa065e5 in ExecProcNode (node=0x55c29c785ee8) at ./build/../src/include/executor/executor.h:250
#15 CteScanNext (node=0x55c29c78b5c8) at ./build/../src/backend/executor/nodeCtescan.c:103
#16 0x000055c29a9e9607 in ExecScanFetch (recheckMtd=0x55c29aa064b0 <CteScanRecheck>, accessMtd=0x55c29aa064e0 <CteScanNext>, node=0x55c29c78b5c8)
    at ./build/../src/backend/executor/execScan.c:97
#17 ExecScan (node=0x55c29c78b5c8, accessMtd=0x55c29aa064e0 <CteScanNext>, recheckMtd=0x55c29aa064b0 <CteScanRecheck>) at ./build/../src/backend/executor/execScan.c:164
#18 0x000055c29aa065e5 in ExecProcNode (node=0x55c29c78b5c8) at ./build/../src/include/executor/executor.h:250
#19 CteScanNext (node=0x55c29c797118) at ./build/../src/backend/executor/nodeCtescan.c:103
#20 0x000055c29a9e9607 in ExecScanFetch (recheckMtd=0x55c29aa064b0 <CteScanRecheck>, accessMtd=0x55c29aa064e0 <CteScanNext>, node=0x55c29c797118)
    at ./build/../src/backend/executor/execScan.c:97
#21 ExecScan (node=0x55c29c797118, accessMtd=0x55c29aa064e0 <CteScanNext>, recheckMtd=0x55c29aa064b0 <CteScanRecheck>) at ./build/../src/backend/executor/execScan.c:164
#22 0x000055c29aa065e5 in ExecProcNode (node=0x55c29c797118) at ./build/../src/include/executor/executor.h:250
#23 CteScanNext (node=0x55c29c7a5738) at ./build/../src/backend/executor/nodeCtescan.c:103
#24 0x000055c29a9e9607 in ExecScanFetch (recheckMtd=0x55c29aa064b0 <CteScanRecheck>, accessMtd=0x55c29aa064e0 <CteScanNext>, node=0x55c29c7a5738)
    at ./build/../src/backend/executor/execScan.c:97
#25 ExecScan (node=0x55c29c7a5738, accessMtd=0x55c29aa064e0 <CteScanNext>, recheckMtd=0x55c29aa064b0 <CteScanRecheck>) at ./build/../src/backend/executor/execScan.c:164
#26 0x000055c29a9e9607 in ExecScanFetch (recheckMtd=0x55c29aa08c20 <SubqueryRecheck>, accessMtd=0x55c29aa08c50 <SubqueryNext>, node=0x55c29c7a5418)
    at ./build/../src/backend/executor/execScan.c:97
#27 ExecScan (node=0x55c29c7a5418, accessMtd=0x55c29aa08c50 <SubqueryNext>, recheckMtd=0x55c29aa08c20 <SubqueryRecheck>) at ./build/../src/backend/executor/execScan.c:164
#28 0x000055c29a9ef3ca in ExecProcNode (node=0x55c29c7a5418) at ./build/../src/include/executor/executor.h:250
#29 ExecAppend (pstate=0x55c29c7a5238) at ./build/../src/backend/executor/nodeAppend.c:222
#30 0x000055c29aa05787 in ExecProcNode (node=0x55c29c7a5238) at ./build/../src/include/executor/executor.h:250
#31 setop_fill_hash_table (setopstate=0x55c29c7a5078) at ./build/../src/backend/executor/nodeSetOp.c:365
#32 ExecSetOp (pstate=0x55c29c7a5078) at ./build/../src/backend/executor/nodeSetOp.c:210
#33 0x000055c29aa0445d in ExecProcNode (node=0x55c29c7a5078) at ./build/../src/include/executor/executor.h:250
#34 ExecResult (pstate=0x55c29c7a4e08) at ./build/../src/backend/executor/nodeResult.c:115
#35 0x000055c29a9ef3ca in ExecProcNode (node=0x55c29c7a4e08) at ./build/../src/include/executor/executor.h:250
#36 ExecAppend (pstate=0x55c29c7a4c28) at ./build/../src/backend/executor/nodeAppend.c:222
#37 0x000055c29aa05787 in ExecProcNode (node=0x55c29c7a4c28) at ./build/../src/include/executor/executor.h:250
#38 setop_fill_hash_table (setopstate=0x55c29c7a4a68) at ./build/../src/backend/executor/nodeSetOp.c:365
#39 ExecSetOp (pstate=0x55c29c7a4a68) at ./build/../src/backend/executor/nodeSetOp.c:210
#40 0x000055c29a9e9607 in ExecScanFetch (recheckMtd=0x55c29aa08c20 <SubqueryRecheck>, accessMtd=0x55c29aa08c50 <SubqueryNext>, node=0x55c29c7a4748)
    at ./build/../src/backend/executor/execScan.c:97
#41 ExecScan (node=0x55c29c7a4748, accessMtd=0x55c29aa08c50 <SubqueryNext>, recheckMtd=0x55c29aa08c20 <SubqueryRecheck>) at ./build/../src/backend/executor/execScan.c:164
---Type <return> to continue, or q <return> to quit---
#42 0x000055c29aa065e5 in ExecProcNode (node=0x55c29c7a4748) at ./build/../src/include/executor/executor.h:250
#43 CteScanNext (node=0x55c29c7f1c08) at ./build/../src/backend/executor/nodeCtescan.c:103
#44 0x000055c29a9e95aa in ExecScanFetch (recheckMtd=0x55c29aa064b0 <CteScanRecheck>, accessMtd=0x55c29aa064e0 <CteScanNext>, node=0x55c29c7f1c08)
    at ./build/../src/backend/executor/execScan.c:97
#45 ExecScan (node=0x55c29c7f1c08, accessMtd=0x55c29aa064e0 <CteScanNext>, recheckMtd=0x55c29aa064b0 <CteScanRecheck>) at ./build/../src/backend/executor/execScan.c:147
#46 0x000055c29a9ef3ca in ExecProcNode (node=0x55c29c7f1c08) at ./build/../src/include/executor/executor.h:250
#47 ExecAppend (pstate=0x55c29c7f19b8) at ./build/../src/backend/executor/nodeAppend.c:222
#48 0x000055c29a9ef74c in ExecProcNode (node=0x55c29c7f19b8) at ./build/../src/include/executor/executor.h:250
#49 fetch_input_tuple (aggstate=aggstate@entry=0x55c29c79c238) at ./build/../src/backend/executor/nodeAgg.c:695
#50 0x000055c29a9f1aa3 in agg_fill_hash_table (aggstate=0x55c29c79c238) at ./build/../src/backend/executor/nodeAgg.c:2539
#51 ExecAgg (pstate=0x55c29c79c238) at ./build/../src/backend/executor/nodeAgg.c:2151
#52 0x000055c29a9e9607 in ExecScanFetch (recheckMtd=0x55c29aa08c20 <SubqueryRecheck>, accessMtd=0x55c29aa08c50 <SubqueryNext>, node=0x55c29c7f11c8)
    at ./build/../src/backend/executor/execScan.c:97
#53 ExecScan (node=0x55c29c7f11c8, accessMtd=0x55c29aa08c50 <SubqueryNext>, recheckMtd=0x55c29aa08c20 <SubqueryRecheck>) at ./build/../src/backend/executor/execScan.c:164
#54 0x000055c29aa05c86 in ExecProcNode (node=0x55c29c7f11c8) at ./build/../src/include/executor/executor.h:250
#55 ExecSort (pstate=0x55c29c7f0f58) at ./build/../src/backend/executor/nodeSort.c:106
#56 0x000055c29a9e35d5 in ExecProcNode (node=0x55c29c7f0f58) at ./build/../src/include/executor/executor.h:250
#57 ExecutePlan (execute_once=<optimized out>, dest=0x55c29c33d558, direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>, operation=CMD_SELECT, 
    use_parallel_mode=<optimized out>, planstate=0x55c29c7f0f58, estate=0x55c29c785bc8) at ./build/../src/backend/executor/execMain.c:1723
#58 standard_ExecutorRun (queryDesc=0x55c29c424c58, direction=<optimized out>, count=0, execute_once=<optimized out>) at ./build/../src/backend/executor/execMain.c:364
#59 0x000055c29ab1a0c6 in PortalRunSelect (portal=portal@entry=0x55c29c2a2758, forward=forward@entry=1 '\001', count=0, count@entry=9223372036854775807, dest=dest@entry=0x55c29c33d558)
    at ./build/../src/backend/tcop/pquery.c:932
#60 0x000055c29ab1b6b0 in PortalRun (portal=portal@entry=0x55c29c2a2758, count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=1 '\001', run_once=<optimized out>, 
    dest=dest@entry=0x55c29c33d558, altdest=altdest@entry=0x55c29c33d558, completionTag=0x7ffe32bad090 "") at ./build/../src/backend/tcop/pquery.c:773
#61 0x000055c29ab18cf4 in exec_execute_message (max_rows=9223372036854775807, portal_name=0x55c29c33d148 "") at ./build/../src/backend/tcop/postgres.c:2007
#62 PostgresMain (argc=<optimized out>, argv=argv@entry=0x55c29c2d7aa8, dbname=<optimized out>, username=<optimized out>) at ./build/../src/backend/tcop/postgres.c:4180
#63 0x000055c29a85868c in BackendRun (port=0x55c29c2d16a0) at ./build/../src/backend/postmaster/postmaster.c:4402
#64 BackendStartup (port=0x55c29c2d16a0) at ./build/../src/backend/postmaster/postmaster.c:4074
#65 ServerLoop () at ./build/../src/backend/postmaster/postmaster.c:1756
#66 0x000055c29aaa8781 in PostmasterMain (argc=5, argv=0x55c29c283060) at ./build/../src/backend/postmaster/postmaster.c:1364
#67 0x000055c29a85a37b in main (argc=5, argv=0x55c29c283060) at ./build/../src/backend/main/main.c:228

[za-arthur]

Hello Andrei,
There was similar issue #30. It was fixed by 2cdd6d2
Do you install RUM from sources? Can you try the fix? Your installed version doesn't contain it.
Also you can try to update to the latest version 1.3.3.

[alisnic]

The build fails when we try to compile version 1.3.3:

alexandrt@alexandrt-HP-OMEN:~/projects/rum$ make USE_PGXS=1
gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -g -g -O2 -fdebug-prefix-map=/build/postgresql-10-SoIDbZ/postgresql-10-10.8=. -fstack-protector-strong -Wformat -Werror=format-security -fno-omit-frame-pointer -fPIC -I. -I./ -I/usr/include/postgresql/10/server -I/usr/include/postgresql/internal  -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2  -I/usr/include/mit-krb5  -c -o src/rumsort.o src/rumsort.c
In file included from src/rumsort.c:142:0:
src/rum.h:792:2: error: unknown type name ‘RBTNode’
  RBTNode  rbnode;
  ^~~~~~~
<builtin>: recipe for target 'src/rumsort.o' failed
make: *** [src/rumsort.o] Error 1

alexandrt@alexandrt-HP-OMEN:~/projects/rum$ git status
HEAD detached at 1.3.3
nothing to commit, working tree clean

We made sure to have the PG headers installed, but it didn't help:

alexandrt@alexandrt-HP-OMEN:~/projects/rum$ sudo apt-get install postgresql-server-dev-10
Reading package lists... Done
Building dependency tree       
Reading state information... Done
postgresql-server-dev-10 is already the newest version (10.8-0ubuntu0.18.04.1).
postgresql-server-dev-10 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 37 not upgraded.
alexandrt@alexandrt-HP-OMEN:~/projects/rum$ sudo apt-get install postgresql-server-dev-all
Reading package lists... Done
Building dependency tree       
Reading state information... Done
postgresql-server-dev-all is already the newest version (190).
0 upgraded, 0 newly installed, 0 to remove and 37 not upgraded.

[za-arthur]

It is strange. Starting from PGA 10.7 RBNode was renamed to RBTNode, see:
https://github.com/postgres/postgres/blob/REL_10_8/src/include/lib/rbtree.h

Can you check rbtree.h content in your system? It is possible that you have older headers.

[alisnic]

The version number in sources on the machine (located at /usr/include/postgresql/pg_config.h):

#define PG_VERSION_STR "PostgreSQL 10.8 (Ubuntu 10.8-0ubuntu0.18.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0, 64-bit"

This is the content of the file (located at /usr/include/postgresql/10/server):

/*-------------------------------------------------------------------------
 *
 * rbtree.h
 *      interface for PostgreSQL generic Red-Black binary tree package
 *
 * Copyright (c) 2009-2017, PostgreSQL Global Development Group
 *
 * IDENTIFICATION
 *        src/include/lib/rbtree.h
 *
 *-------------------------------------------------------------------------
 */
#ifndef RBTREE_H
#define RBTREE_H

/*
 * RBNode is intended to be used as the first field of a larger struct,
 * whose additional fields carry whatever payload data the caller needs
 * for a tree entry.  (The total size of that larger struct is passed to
 * rb_create.)    RBNode is declared here to support this usage, but
 * callers must treat it as an opaque struct.
 */
typedef struct RBNode
{
    char color;                    /* node's current color, red or black */
    struct RBNode *left;        /* left child, or RBNIL if none */
    struct RBNode *right;        /* right child, or RBNIL if none */
    struct RBNode *parent;        /* parent, or NULL (not RBNIL!) if none */
} RBNode;

/* Opaque struct representing a whole tree */
typedef struct RBTree RBTree;

/* Available tree iteration orderings */
typedef enum RBOrderControl
{
    LeftRightWalk,                /* inorder: left child, node, right child */
    RightLeftWalk,                /* reverse inorder: right, node, left */
    DirectWalk,                    /* preorder: node, left child, right child */
    InvertedWalk                /* postorder: left child, right child, node */
} RBOrderControl;

/*
 * RBTreeIterator holds state while traversing a tree.  This is declared
 * here so that callers can stack-allocate this, but must otherwise be
 * treated as an opaque struct.
 */
typedef struct RBTreeIterator RBTreeIterator;

struct RBTreeIterator
{
    RBTree       *rb;
    RBNode       *(*iterate) (RBTreeIterator *iter);
    RBNode       *last_visited;
    char        next_step;
    bool        is_over;
};

/* Support functions to be provided by caller */
typedef int (*rb_comparator) (const RBNode *a, const RBNode *b, void *arg);
typedef void (*rb_combiner) (RBNode *existing, const RBNode *newdata, void *arg);
typedef RBNode *(*rb_allocfunc) (void *arg);
typedef void (*rb_freefunc) (RBNode *x, void *arg);

extern RBTree *rb_create(Size node_size,
          rb_comparator comparator,
          rb_combiner combiner,
          rb_allocfunc allocfunc,
          rb_freefunc freefunc,
          void *arg);

extern RBNode *rb_find(RBTree *rb, const RBNode *data);
extern RBNode *rb_leftmost(RBTree *rb);

extern RBNode *rb_insert(RBTree *rb, const RBNode *data, bool *isNew);
extern void rb_delete(RBTree *rb, RBNode *node);

extern void rb_begin_iterate(RBTree *rb, RBOrderControl ctrl,
                 RBTreeIterator *iter);
extern RBNode *rb_iterate(RBTreeIterator *iter);

#endif                            /* RBTREE_H */

[alisnic]

Update: we used docker to build the extension (version 1.3.3), and we were able to crash the server.

Dockerfile:

FROM postgres:10.8

ADD rum /rum
WORKDIR /rum
RUN apt-get update
RUN apt-get install -y build-essential postgresql-server-dev-10
RUN apt-get install -y systemtap-sdt-dev
RUN make USE_PGXS=1
RUN make USE_PGXS=1 install

2019-06-14 13:54:59.318 UTC [1] LOG:  server process (PID 121) was terminated by signal 11: Segmentation fault
2019-06-14 13:54:59.318 UTC [1] DETAIL:  Failed process was running:       WITH by_name AS (
                SELECT <REDACTED>
                FROM <REDACTED>
                        WHERE
                  to_tsvector('simple', 'some redacted words') @@ query_simple AND
                  POSITION(lower(redacted_column_name) IN lower('some redacted words')) > 0
              )
2019-06-14 13:54:59.332 UTC [1] LOG:  terminating any other active server processes

We will try to fetch the core dump from container and see where the stack trace ends

[za-arthur]

Thank you! I'll try it on Monday.

[alisnic]

Hi @za-arthur, we were able to reproduce the problem again in 1.3.3 and capture a core dump:

backtrace:

Core was generated by `postgres:'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  decode_varbyte (ptr=ptr@entry=0x7fff7378b878) at src/rumtsquery.c:236
236             c = *(p++);
(gdb) bt
#0  decode_varbyte (ptr=ptr@entry=0x7fff7378b878) at src/rumtsquery.c:236
#1  0x00007f9ecfc8d448 in ruminv_tsvector_consistent (fcinfo=<optimized out>)
    at src/rumtsquery.c:509
#2  0x00007f9ecfc9ce52 in FunctionCall10Coll (flinfo=<optimized out>, 
    collation=<optimized out>, arg1=<optimized out>, arg2=<optimized out>, 
    arg3=<optimized out>, arg4=<optimized out>, arg5=0, arg6=94070445379769, 
    arg7=94070445379376, arg8=94070445379456, arg9=94070445380000, arg10=94070445380080)
    at src/rumutil.c:1086
#3  0x00007f9ecfc93bd6 in callConsistentFn (rumstate=rumstate@entry=0x558e7bb782f8, 
    key=key@entry=0x558e7b7b1028) at src/rumget.c:168
#4  0x00007f9ecfc97710 in keyGetItem (key=0x558e7b7b1028, tempCtx=0x558e7b75b2a8, 
    rumstate=0x558e7bb782f8) at src/rumget.c:1422
#5  scanGetItemRegular (scan=0x0, scan=0x0, recheck=0x7fff7378c9cf "", item=0x7fff7378c9d0, 
    advancePast=0x558e7bb782e8) at src/rumget.c:1501
#6  scanGetItem (scan=scan@entry=0x558e7bb77ad8, 
    advancePast=advancePast@entry=0x7fff7378c9d0, item=item@entry=0x7fff7378c9d0, 
    recheck=recheck@entry=0x7fff7378c9cf "") at src/rumget.c:2117
#7  0x00007f9ecfc980ca in rumgetbitmap (scan=0x558e7bb77ad8, tbm=0x558e7bbf93e0)
    at src/rumget.c:2155
#8  0x0000558e7acfd752 in index_getbitmap ()
#9  0x0000558e7ae5529a in MultiExecBitmapIndexScan ()
#10 0x0000558e7ae54af6 in ?? ()
#11 0x0000558e7ae48dc7 in ExecScan ()
#12 0x0000558e7ae65eb5 in ?? ()
#13 0x0000558e7ae48dc7 in ExecScan ()
#14 0x0000558e7ae65eb5 in ?? ()
#15 0x0000558e7ae48dc7 in ExecScan ()
#16 0x0000558e7ae48dc7 in ExecScan ()
#17 0x0000558e7ae4eb85 in ?? ()
#18 0x0000558e7ae65044 in ?? ()
#19 0x0000558e7ae63d2d in ?? ()
#20 0x0000558e7ae4eb85 in ?? ()
#21 0x0000558e7ae65044 in ?? ()
#22 0x0000558e7ae48dc7 in ExecScan ()
#23 0x0000558e7ae65eb5 in ?? ()
#24 0x0000558e7ae48ec9 in ExecScan ()
#25 0x0000558e7ae4eb85 in ?? ()
#26 0x0000558e7ae4eedc in ?? ()
#27 0x0000558e7ae51243 in ?? ()
#28 0x0000558e7ae48dc7 in ExecScan ()
#29 0x0000558e7ae65566 in ?? ()
#30 0x0000558e7ae42c83 in standard_ExecutorRun ()
#31 0x0000558e7af7e50b in ?? ()
#32 0x0000558e7af7fa50 in PortalRun ()

Dockerfile:

FROM postgres:10.8

ADD rum /rum
WORKDIR /rum
RUN apt-get update
RUN apt-get install -y build-essential postgresql-server-dev-10
RUN apt-get install -y systemtap-sdt-dev
RUN make USE_PGXS=1
RUN make USE_PGXS=1 install

RUN ulimit -c unlimited
RUN apt-get install -y gdb
RUN echo '* soft core unlimited' >> /etc/security/limits.conf

DBNAME=# select * from pg_extension;
  extname  | extowner | extnamespace | extrelocatable | extversion | extconfig | extcondition 
-----------+----------+--------------+----------------+------------+-----------+--------------
 plpgsql   |       10 |           11 | f              | 1.0        |           | 
 btree_gin |       10 |         2200 | t              | 1.2        |           | 
 pg_trgm   |       10 |         2200 | t              | 1.3        |           | 
 rum       |       10 |         2200 | t              | 1.3        |           | 
(4 rows)

[za-arthur]

Unfortunately, I cannot reproduce it with installcheck. I tried different versions of RUM and with Arch Linux, Ubuntu 16.04 and Docker.
As far I understand you run your query on your own database, not installcheck.
I'll try to take a look at your backtrace.

[za-arthur]

Can you share your table schema and how did you create an index?

[alisnic]

Can you share your table schema

I'm not sure about that, will have to talk with our security team :(

how did you create an index?

We have several indexes using rum (including several ones with conditions), we create them using Rails migrations (CREATE INDEX CONCURRENTLY IF NOT EXISTS):

"idx_1" rum (query_simple) WHERE redacted_integer_column1 > 0
"idx_2" rum (query_simple) WHERE redacted_string_column1::text = 'constant'::text
"idx_3" rum (query_simple) WHERE redacted_integer_column2 = 2
"idx_4" rum (query_simple)

all 4 indexes are on the same column - query_simple. The value for column is generated on insert as plainto_tsquery('simple', original_string_column)

Our test which reproduces the segfault has 2 processes:

1. randomly update redacted_integer_column1 (1000 records at a time)
2. select on the table with the following query

 SELECT <REDACTED>
                FROM <REDACTED>
                        WHERE
                  to_tsvector('simple', 'some redacted words') @@ query_simple AND
                  POSITION(lower(original_string_column) IN lower('some redacted words')) > 0

Note: that might explain why you can't reproduce using installcheck, it may be that this is a race condition bug. I'm saying that because our DB is segfaulting only when both of these 2 conditions are met - updating and select from 2 different processes

We will try to create a minimal case which reproduces the problem that we can share with you

[za-arthur]

Oh, it might be fixed in master. Can you try it?
There was commit 793829d, which fixes undefined behavior with parallel updates. This fix will be part of 1.3.4.

[alisnic]

We did 20 runs with 500 requests each, and were unable to trigger the segfault (previously that happen around 3rd or 4th run). It seems that the commit that you mentioned fixes the issue. When do you plan to release 1.3.4?

[za-arthur]

It is good.
I plan to release it when I fix #61 issue and before PostgreSQL 12 release.