Changes for ARM

[yanchick]

Hi. I try build gragemq for ARM(QNAP NAS) and get fail at work. I try made some changes for running code at ARM. All changes here. What you recommend improve in commit for pull request?

[valinurovam]

Hi.
Do you have more details? What kind of error did you get at build or runtime? Why it was failed?

valinurovam@Valinurovs-MacBook-Pro ~/g/s/g/v/garagemq> env GO111MODULE=on GOOS=linux GOARCH=arm go build -a -o bin/garagemq main.go

I have no any error on arm build.
Also one comment into your commit

[yanchick]

Well, I understand my code not good. Main problem is using AddUint64 at ARM arch. I will make fork for myself with using AddUint32 in code. I not good in Golang, so I can't now create code with 64 and 32 support.

[valinurovam]

@yanchick can you check that branch https://github.com/valinurovam/garagemq/tree/fix-arm-uin64-issues?

[yanchick]

Hi. Thanks for changing. Yes this commit is help. But in runtime I have a crash. If I correctly understand issue golang/go#23345 the main problem is uint64 in code. Especially AddUint64. I make changes for myself, i replace all uint64 variables to uint32. I don't understand all effects of these changes. But these changes help me. I use garagemq for testing amqp in my apps, not for prods. And i not sure for these modifications in main repository.

[valinurovam]

Can you give any stacktrace of error and what model is your ARM? arm7?

[yanchick]

Hi. I build your code and haven't crash at tests. So, I think issue is solved. But I will take more tests at week I use ARMv5.

[andhe]

Here's a full build log for 32bit arm, same problem occurs also on i386 so seems to be a generic 32bit alignment issue:

https://buildd.debian.org/status/fetch.php?pkg=garagemq&arch=armhf&ver=0.0%7Egit20191230.94d8b87%2Bds-1&stamp=1580751181&raw=0

Here's the full i386 build log:
https://buildd.debian.org/status/fetch.php?pkg=garagemq&arch=i386&ver=0.0%7Egit20191230.94d8b87%2Bds-1&stamp=1580750974&raw=0

[andhe]

Did some experiments....

The problem in the above (armhf + i386) build log can be "fixed" by moving the connSeq member of the Server struct to be the first member (apparently beginning of struct is guaranteed to be aligned in memory).

The next problem that occurs can be seen in this (armel) build log (which I reproduced on i386 locally):

https://buildd.debian.org/status/fetch.php?pkg=garagemq&arch=armel&ver=0.0%7Egit20191230.94d8b87%2Bds-1&stamp=1580789446&raw=0

This problem might in fact be self-inflicted. Badger doesn't build on 32bit archs either so I'm using a patched version which is 2.0.1 + fixes that makes badger pass it's own testsuite. The patches for badger can be found here:

https://salsa.debian.org/go-team/packages/badger/tree/master/debian/patches

The Mmap failure originally was also hit by badgers test-suite and was "fixed" by lowering the default size of the value log at : https://salsa.debian.org/go-team/packages/badger/blob/master/debian/patches/fix-32bit.patch#L9

Apparently allocations made by garagemq before the mmap makes the default value no longer be mmap:able (because it exhausts the memory space of the process). As a test we could of course just lower the setting e.g. by doing opts.ValueLogFileSize = (1<<25 - 1). This gets us a bit further, but soon we'll instead hit the next hurdle where a standard memory allocation of 134MB fails (because of exhausted memory space):

--- PASS: Test_BasicQos_Global_Success (0.07s)
=== RUN   Test_BasicQos_Check_Global_Success
runtime: out of memory: cannot allocate 134217728-byte block (3148939264 in use)
fatal error: out of memory

runtime stack:
runtime.throw(0x856a09d, 0xd)
	/usr/lib/go-1.13/src/runtime/panic.go:774 +0x6a
runtime.largeAlloc(0x8000000, 0x8090101, 0xc4d2e000)
	/usr/lib/go-1.13/src/runtime/malloc.go:1140 +0x108
runtime.mallocgc.func1()
	/usr/lib/go-1.13/src/runtime/malloc.go:1033 +0x39
runtime.systemstack(0xc94a0c00)
	/usr/lib/go-1.13/src/runtime/asm_386.s:399 +0x53
runtime.mstart()
	/usr/lib/go-1.13/src/runtime/proc.go:1146

goroutine 53 [running]:
runtime.systemstack_switch()
	/usr/lib/go-1.13/src/runtime/asm_386.s:360 fp=0xa8e93d8 sp=0xa8e93d4 pc=0x809f7f0
runtime.mallocgc(0x8000000, 0x84db040, 0x3d582801, 0x6d5890a0)
	/usr/lib/go-1.13/src/runtime/malloc.go:1032 +0x6a8 fp=0xa8e942c sp=0xa8e93d8 pc=0x8054b58
runtime.makeslice(0x84db040, 0x1000000, 0x1000000, 0x27f5d8c0)
	/usr/lib/go-1.13/src/runtime/slice.go:49 +0x4f fp=0xa8e9440 sp=0xa8e942c pc=0x808aaff
runtime.makeslice64(0x84db040, 0x1000000, 0x0, 0x1000000, 0x0, 0x0)
	/usr/lib/go-1.13/src/runtime/slice.go:63 +0x4b fp=0xa8e9454 sp=0xa8e9440 pc=0x808abbb
github.com/dgraph-io/ristretto/z.(*Bloom).Size(...)
...

We can ofcourse now continue tweaking ValueLogFileSize to a lower value in hope that we can squeeze things in, but something feels really fundamentally wrong when we exhaust the entire memory space on 32bit systems....