[Bug] addInfo is (Datum) 0, which is expected to be not, coredump in func DatumGetByteaP

[markrui3]

My PG server crashed after rum search.
PG version: 12.4
rum version: 1.3.3, 1.3.8, and master

code: https://github.com/postgrespro/rum/blob/master/src/rum_ts_utils.c#L289

codedump
(gdb) bt
#0  0x0000000000aa4c83 in pg_detoast_datum (datum=0x0) at fmgr.c:1764
#1  0x00007f259e33d182 in checkcondition_rum (checkval=0x7ffecb597cc0, val=0x1fe2e28, data=0x0) at src/rum_ts_utils.c:289
#2  0x00007f259e33dd2b in rum_TS_execute (curitem=0x1fe2e28, arg=0x7ffecb597cc0, flags=1, chkcond=0x7f259e33d083 <checkcondi
tion_rum>) at src/rum_ts_utils.c:786
#3  0x00007f259e33e06a in rum_tsquery_consistent (fcinfo=0x7ffecb597d50) at src/rum_ts_utils.c:907
#4  0x00007f259e35dc55 in FunctionCall10Coll (flinfo=0x2016328, collation=100, arg1=33665552, arg2=1, arg3=33435168, arg4=1,
 arg5=33665112, arg6=33665385, arg7=33665048, arg8=33665208, arg9=33665584, arg10=33665616) at src/rumutil.c:1104
#5  0x00007f259e34fc2f in callConsistentFn (rumstate=0x2014690, key=0x201b0d8) at src/rumget.c:167
#6  0x00007f259e35554f in scanGetItemFast (scan=0x2010890, advancePast=0x7ffecb597f90, item=0x7ffecb597f90, recheck=0x7ffecb
597f7f) at src/rumget.c:2001
#7  0x00007f259e355a3b in scanGetItem (scan=0x2010890, advancePast=0x7ffecb597f90, item=0x7ffecb597f90, recheck=0x7ffecb597f
7f) at src/rumget.c:2117
#8  0x00007f259e355b48 in rumgetbitmap (scan=0x2010890, tbm=0x2029530) at src/rumget.c:2159
#9  0x00000000005134f5 in index_getbitmap (scan=0x2010890, bitmap=0x2029530) at indexam.c:665
#10 0x0000000000728b5e in MultiExecBitmapIndexScan (node=0x2010b20) at nodeBitmapIndexscan.c:105
#11 0x00000000007110c5 in MultiExecProcNode (node=0x2010b20) at execProcnode.c:505
#12 0x0000000000727187 in BitmapHeapNext (node=0x200a0a0) at nodeBitmapHeapscan.c:145
#13 0x0000000000713236 in ExecScanFetch (node=0x200a0a0, accessMtd=0x726fae <BitmapHeapNext>, recheckMtd=0x727d0b <BitmapHea
pRecheck>) at execScan.c:133
#14 0x00000000007132d7 in ExecScan (node=0x200a0a0, accessMtd=0x726fae <BitmapHeapNext>, recheckMtd=0x727d0b <BitmapHeapRech
eck>) at execScan.c:200
#15 0x0000000000727d89 in ExecBitmapHeapScan (pstate=0x200a0a0) at nodeBitmapHeapscan.c:626
#16 0x0000000000710fc8 in ExecProcNodeFirst (node=0x200a0a0) at execProcnode.c:445
#17 0x000000000071ece1 in ExecProcNode (node=0x200a0a0) at ../../../src/include/executor/executor.h:239
#18 0x000000000071f197 in ExecAppend (pstate=0x2009748) at nodeAppend.c:292
#19 0x0000000000710fc8 in ExecProcNodeFirst (node=0x2009748) at execProcnode.c:445
#20 0x0000000000744158 in ExecProcNode (node=0x2009748) at ../../../src/include/executor/executor.h:239
#21 0x00000000007443a3 in ExecResult (pstate=0x20094f0) at nodeResult.c:115
#22 0x0000000000710fc8 in ExecProcNodeFirst (node=0x20094f0) at execProcnode.c:445
#23 0x000000000074686d in ExecProcNode (node=0x20094f0) at ../../../src/include/executor/executor.h:239
#24 0x00000000007469c0 in ExecSort (pstate=0x20088d0) at nodeSort.c:107
#25 0x0000000000710fc8 in ExecProcNodeFirst (node=0x20088d0) at execProcnode.c:445
#26 0x0000000000744158 in ExecProcNode (node=0x20088d0) at ../../../src/include/executor/executor.h:239
#27 0x00000000007443a3 in ExecResult (pstate=0x20092c0) at nodeResult.c:115
#28 0x0000000000710fc8 in ExecProcNodeFirst (node=0x20092c0) at execProcnode.c:445
#29 0x00000000007061e2 in ExecProcNode (node=0x20092c0) at ../../../src/include/executor/executor.h:239
#30 0x00000000007089b9 in ExecutePlan (estate=0x2008628, planstate=0x20092c0, use_parallel_mode=false, operation=CMD_SELECT,
 sendTuples=true, numberTuples=4, direction=ForwardScanDirection, dest=0x1f94288, execute_once=true) at execMain.c:1648
#31 0x0000000000706811 in standard_ExecutorRun (queryDesc=0x1ff4ad0, direction=ForwardScanDirection, count=4, execute_once=t
rue) at execMain.c:366
#32 0x00007f25a63d7922 in pgss_ExecutorRun (queryDesc=0x1ff4ad0, direction=ForwardScanDirection, count=4, execute_once=true)
 at pg_stat_statements.c:903
#33 0x00007f25a5fcf999 in explain_ExecutorRun (queryDesc=0x1ff4ad0, direction=ForwardScanDirection, count=4, execute_once=tr
ue) at auto_explain.c:320
#34 0x00007f25a59b1f4b in pgss_ExecutorRun (queryDesc=0x1ff4ad0, direction=ForwardScanDirection, count=4, execute_once=true)
 at sql_firewall.c:972
#35 0x000000000070662e in ExecutorRun (queryDesc=0x1ff4ad0, direction=ForwardScanDirection, count=4, execute_once=true) at e
xecMain.c:308
#36 0x000000000070d2ad in ParallelQueryMain (seg=0x1e8d3f8, toc=0x7f25a4db9000) at execParallel.c:1399
#37 0x000000000055a32f in ParallelWorkerMain (main_arg=1878290192) at parallel.c:1431
#38 0x00000000008502f6 in StartBackgroundWorker () at bgworker.c:834
#39 0x00000000008652ec in do_start_bgworker (rw=0x1f0ffb0) at postmaster.c:5806
#40 0x00000000008656bb in maybe_start_bgworkers () at postmaster.c:6032
#41 0x00000000008645e8 in sigusr1_handler (postgres_signal_arg=10) at postmaster.c:5203
#42 <signal handler called>
#43 0x00007f25a9795fc3 in __select_nocancel () from /lib64/libc.so.6
#44 0x000000000085f9fc in ServerLoop () at postmaster.c:1670
#45 0x000000000085f3bf in PostmasterMain (argc=3, argv=0x1e8bc70) at postmaster.c:1379
#46 0x000000000077d293 in main (argc=3, argv=0x1e8bc70) at main.c:228

I think the reason is that gcv->addInfoIsNull[j] is false, so we believe gcv->addInfo[j] must not (Datum) 0, but actually gcv->addInfo[j] is (Datum) 0.

There are two solutions can avoid this:

1. vacuum the table, and all is ok.
2. add some code when construct gcv->addInfo in https://github.com/postgrespro/rum/blob/master/src/rumget.c#L1986

if (entry->isFinished == false &&
	entry->curItem.addInfo != (Datum) 0 &&
	rumCompareItemPointers(&entry->curItem.iptr,
&so->sortedEntries[so->totalentries - 1]->curItem.iptr) == 0 &&
(!entry->curItem.addInfoIsNull && entry->curItem.addInfo != (Datum) 0))
{
	key->entryRes[j] = true;
	key->addInfo[j] = entry->curItem.addInfo;
	key->addInfoIsNull[j] = entry->curItem.addInfoIsNull;
}
else
{
	key->entryRes[j] = false;
	key->addInfo[j] = (Datum) 0;
	key->addInfoIsNull[j] = true;
}

The problem may be caused by insert and delete frequently.
If we should believe addInfo and addInfoIsNull are in the same meaning? When addInfoIsNull is false, addInfo must not (Datum) 0?

[pashkinelfe]

Do you use index with "USING rum (a rum_tsvector_ops); " or something else: rum_tsvector_addon_ops, rum_tsvector_hash_ops... ?

Thanks for the report.

[markrui3]

Do you use index with "USING rum (a rum_tsvector_ops); " or something else: rum_tsvector_addon_ops, rum_tsvector_hash_ops... ?

Thanks for the report.

"rum_individual_news_idx" rum (search_vector), which search_vector is tsvector

[pashkinelfe]

When using a default indexing option (or specifying rum_tsvector_ops, which is the same) addinfo contain word positions that should be > 0 when addInfoIsNull == false. addInfoIsNull == true basically means that lexeme is not included in any of the table rows. Therefore varbyte encoding should not be read as it can contain only garbage. But this is only for the case of the index with default opclass (rum_tsvector_ops)

With rum_tsvector_addon_ops addinfo contains data from the attached column which can be legit 0 and we shouldn't confuse it with NULL. In this case addInfoIsNull == false when addinfo column contains 0 (or any other value). If addInfoIsNull == true then addinfo column contains NULL and the addinfo column doesn't need and isn't going to be read (very much like a NULL attribute in a table row, which is not stored in the tuple, just have nullmask bit set to true),

It is not easy to fix the former and don't break the latter. Your fix will probably suit your case, but if you apply it you should not create RUM index with options other than the default. Otherwise, you can silently get wrong select results due to confusion between 0 and NULL in the attached column.

[pashkinelfe]

After further contemplation, I guess that having NULL in lexemes positions list is quite improbable. On the other hand, there is a possibility that the backtrace provided is due to a broken stack. I'd like to be sure the problem is related to RUM query, not something else below in the stack. Could you please turn on query logging and give me several queries just before the crash occurs?

If it is indeed due to RUM query, it then could be useful to add some debug checks in the code to check (addInfo[i] != NULL || addInfoIsNull[i] == true) for standard text search opclass on insert etc. Do you have possibility to build custom RUM builds from the source and try to repeat a crash on it? If so, I will create you appropriate debug branch.

[pashkinelfe]

@markrui3 I've discovered a similar backtrace in a master rum build when run under valgrind. The issue is fixed and I'd appreciate it if you could check whether you can still see the bug you initially reported in this issue on a fresh master RUM build.

[markrui3]

@markrui3 I've discovered a similar backtrace in a master rum build when run under valgrind. The issue is fixed and I'd appreciate it if you could check whether you can still see the bug you initially reported in this issue on a fresh master RUM build.

Is this commit 1605f9eebe70278a60dac53476312dc6d66e39d0? I will try if it works.
Thanks.

[pashkinelfe]

@markrui3 there were several fixes. Please try current master branch state (655f5b3) not just one commit. It is fully backward compatible, you should find no problems when checking with old indexes.

[pashkinelfe]

I'll close the issue as per my comment #99 (comment)
Should not exist anymore.
If you can detect it again please do not hesitate to reopen the issue!