Android NDK r23 build problem

Development
1

ref. Android NDK r22 build problem - #4 by tomasz86

2

Iā€™ve set up test branch with NDK r23 but Iā€™m running into the compile error @tomasz86 mentionend in the other thread as well.

ld: error: cannot open output file $WORK\b083\_cgo_.o: Function not implemented
clang: error: linker command failed with exit code 1 (use -v to see invocation)
exit status 2
exit status 1

Also tried to changed the CC_FLAGS environment variable but I didnā€™t find a working solution to compile it.

Branch: https://github.com/Catfriend1/syncthing-android/tree/android-ndk-r23

I hope anyone knows anything on how to fix or workaround this CGO problem. Iā€™m not so into it, so I lack understanding why we need cgo. Turned it off for one try, Syncthing compiled well but I didnā€™t get the linker to output the libsyncthingnative.so executable file.

Adding CC_FLAGS="-v" outputs more info, but nothing that helped me when Googlā€™ing it.

3

NDK r23 ChangeLog from NDK Revision History Ā |Ā  Android NDK Ā |Ā  Android Developers

* GNU binutils, excluding the GNU Assembler (GAS), has been removed. GAS will be removed in the next release. If you are building with `-fno-integrated-as`, file bugs if anything is preventing you from removing that flag.
* Support for GDB has ended. GDB will be removed from the next release. Use LLDB instead. Note that `ndk-gdb` uses LLDB by default.
* NDK r23 is the last release that will support non-Neon. Beginning with NDK r24, the armeabi-v7a libraries in the sysroot will be built with Neon. A very small number of very old devices do not support Neon so most apps will not notice aside from the performance improvement.
* Jelly Bean (APIs 16, 17, and 18) will not be supported in the next NDK release. The minimum OS supported by the NDK for r24 will be KitKat (API level 19).

How can we build without GNU binutils? Or is there a replacement we could drop in?

4

Hereā€™s another project struggling with the removed GNU binutils on Android NDK r23: https://github.com/sqlcipher/android-database-sqlcipher/pull/564

5

And this one has migrated to LLVM. Can we do this, too? (How?)

6

The change is documented here: Build System Maintainers Guide

Binutils
LLVM's binutils tools are installed to the NDK at <NDK>/toolchains/llvm/prebuilt/<host-tag>/bin/llvm-<tool>. These include but are not limited to:

llvm-ar
llvm-objcopy
llvm-objdump
llvm-readelf
llvm-strip
Note that llvm-as is not an equivalent of GNU as, but rather a tool for assembling LLVM IR. If you are currently using as directly, you will need to migrate to using clang as a driver for building assembly. See Clang Migration Notes for advice on fixing assembly to be LLVM compatible.

GNU Binutils remains available up to and including r23, but has been removed in r24. In r23 or earlier, GNU binutils tools are installed to <NDK>/toolchains/llvm/prebuilt/<host-tag>/bin/<triple>-<tool> and <NDK>/toolchains/llvm/prebuilt/<host-tag>/<triple>/bin/<tool>.

Note that by default /usr/bin/as is used by Clang if the -fno-integrated-as argument is used, which is almost certainly not what you want!

I still donā€™t understand what Iā€™ve got to do :-(.

7

Sounds like binutils are still there and you could adjust the PATH to have them at the front.

1 Like
8

Itā€™s interesting I can put ā€œCGO_ENABLED=0ā€ and then only build the arm64 variant successfully. x86 and armeabi-v7a wonā€™t build without cgo. Why is 32-Bit Syncthing dependent of CGO - for my deeper understanding.

[INFO] Checking prerequisites ...
[INFO] Performing cleanup ...
[INFO] Building submodule syncthing_v1.18.2 ...
git_bin='C:\Program Files\GitHubDesktop\app-2.5.3\resources\app\git\cmd\git.exe'
Warning: go is not available on the PATH.
Downloaded prebuilt-go to: C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\go_1.17.1.zip
SHA-256: C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\go_1.17.1.zip 2f2d0a5d7c59fb38fcacaf1e272cf701bb8c050300ba8b609fc30d2c5800f02e
[ok] Checksum of C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\go_1.17.1.zip matches expected value.
Adding to PATH: C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\go_1.17.1\bin
go_bin='C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\go_1.17.1\bin\go.exe'
Warning: ANDROID_NDK_HOME environment variable not defined.
Downloaded NDK to: C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\ndk_r23.zip
SHA-1: C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\ndk_r23.zip 14af52e23af9f7a9e7576a17e1814701192745be
[ok] Checksum of C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\ndk_r23.zip matches expected value.
Adding ANDROID_NDK_HOME='C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\android-ndk-r23'
ANDROID_NDK_HOME='C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\android-ndk-r23'
Invoking git fetch ...
Invoking git describe ...
Cleaning go-build cache
Building syncthing version v1.18.2

*** Building for arm64
Notice: Next generation GUI will not be built; see --with-next-gen-gui.
internal/unsafeheader
internal/goexperiment
internal/abi
internal/cpu
runtime/internal/sys
internal/race
runtime/internal/math
sync/atomic
unicode
unicode/utf8
internal/bytealg
internal/itoa
runtime/internal/atomic
container/list
math/bits
crypto/internal/subtle
crypto/subtle
unicode/utf16
vendor/golang.org/x/crypto/cryptobyte/asn1
math
internal/nettrace
vendor/golang.org/x/crypto/internal/subtle
encoding
golang.org/x/sys/internal/unsafeheader
google.golang.org/protobuf/internal/flags
google.golang.org/protobuf/internal/set
runtime
golang.org/x/net/internal/iana
golang.org/x/crypto/internal/subtle
github.com/gobwas/glob/util/runes
github.com/cheekybits/genny/generic
golang.org/x/crypto/cryptobyte/asn1
image/color
github.com/greatroar/blobloom
internal/reflectlite
sync
internal/singleflight
internal/testlog
math/rand
internal/sysinfo
google.golang.org/protobuf/internal/pragma
errors
sort
internal/oserror
io
strconv
crypto/elliptic/internal/fiat
syscall
bytes
strings
hash
crypto
reflect
path
bufio
crypto/internal/randutil
crypto/hmac
crypto/rc4
time
internal/syscall/unix
internal/syscall/execenv
vendor/golang.org/x/net/dns/dnsmessage
vendor/golang.org/x/crypto/hkdf
regexp/syntax
container/heap
context
encoding/binary
internal/poll
internal/fmtsort
io/fs
regexp
crypto/cipher
crypto/sha512
os
crypto/ed25519/internal/edwards25519/field
crypto/aes
crypto/des
crypto/ed25519/internal/edwards25519
crypto/md5
fmt
crypto/sha1
crypto/sha256
encoding/base64
encoding/pem
path/filepath
net
encoding/hex
math/big
net/url
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/crypto/poly1305
vendor/golang.org/x/crypto/curve25519
vendor/golang.org/x/crypto/chacha20poly1305
encoding/json
github.com/pkg/errors
go/token
crypto/elliptic
encoding/asn1
crypto/rand
crypto/dsa
crypto/ed25519
crypto/rsa
go/scanner
internal/lazyregexp
vendor/golang.org/x/crypto/cryptobyte
crypto/x509/pkix
text/template/parse
go/ast
io/ioutil
crypto/ecdsa
os/user
flag
crypto/x509
html
text/template
text/tabwriter
github.com/russross/blackfriday/v2
github.com/flynn-archive/go-shlex
log
crypto/tls
go/doc
github.com/gogo/protobuf/proto
github.com/cpuguy83/go-md2man/v2/md2man
github.com/alecthomas/kong
github.com/urfave/cli
github.com/AudriusButkevicius/recli
compress/flate
hash/crc32
github.com/hashicorp/golang-lru/simplelru
compress/gzip
os/exec
github.com/hashicorp/golang-lru
github.com/gogo/protobuf/protoc-gen-gogo/descriptor
github.com/syncthing/syncthing/lib/logger
golang.org/x/sys/unix
github.com/shirou/gopsutil/v3/internal/common
github.com/klauspost/cpuid/v2
github.com/petermattis/goid
github.com/gogo/protobuf/gogoproto
github.com/minio/sha256-simd
github.com/sasha-s/go-deadlock
golang.org/x/text/transform
github.com/syncthing/syncthing/lib/sync
github.com/syncthing/syncthing/lib/sha256
golang.org/x/text/unicode/norm
hash/fnv
github.com/shirou/gopsutil/v3/disk
github.com/syncthing/notify
runtime/debug
runtime/trace
encoding/xml
github.com/gogo/protobuf/sortkeys
google.golang.org/protobuf/internal/detrand
testing
google.golang.org/protobuf/internal/errors
github.com/gogo/protobuf/types
google.golang.org/protobuf/encoding/protowire
google.golang.org/protobuf/reflect/protoreflect
google.golang.org/protobuf/internal/version
github.com/syncthing/syncthing/lib/fs
github.com/thejerf/suture/v4
google.golang.org/protobuf/internal/descfmt
google.golang.org/protobuf/internal/descopts
google.golang.org/protobuf/internal/strs
google.golang.org/protobuf/internal/encoding/messageset
google.golang.org/protobuf/internal/encoding/text
google.golang.org/protobuf/internal/genid
github.com/syncthing/syncthing/lib/locations
google.golang.org/protobuf/internal/order
github.com/syncthing/syncthing/cmd/syncthing/cmdutil
google.golang.org/protobuf/reflect/protoregistry
google.golang.org/protobuf/runtime/protoiface
google.golang.org/protobuf/internal/encoding/defval
github.com/syncthing/syncthing/lib/events
github.com/syncthing/syncthing/lib/connections/registry
github.com/syncthing/syncthing/lib/util
golang.org/x/net/bpf
google.golang.org/protobuf/proto
golang.org/x/net/internal/socket
golang.org/x/net/internal/socks
vendor/golang.org/x/text/transform
golang.org/x/net/proxy
golang.org/x/net/ipv4
google.golang.org/protobuf/internal/filedesc
google.golang.org/protobuf/encoding/prototext
golang.org/x/net/ipv6
vendor/golang.org/x/text/unicode/bidi
vendor/golang.org/x/text/unicode/norm
net/textproto
vendor/golang.org/x/text/secure/bidirule
google.golang.org/protobuf/internal/encoding/tag
vendor/golang.org/x/net/http2/hpack
mime
google.golang.org/protobuf/internal/impl
vendor/golang.org/x/net/idna
mime/quotedprintable
net/http/httptrace
mime/multipart
vendor/golang.org/x/net/http/httpguts
vendor/golang.org/x/net/http/httpproxy
net/http/internal
net/http/internal/ascii
encoding/base32
github.com/bkaradzic/go-lz4
net/http
github.com/miscreant/miscreant.go/block
github.com/syncthing/syncthing/lib/rand
github.com/miscreant/miscreant.go/cmac
github.com/miscreant/miscreant.go/pmac
github.com/syncthing/syncthing/proto/ext
github.com/miscreant/miscreant.go
golang.org/x/crypto/chacha20
golang.org/x/crypto/poly1305
golang.org/x/crypto/hkdf
golang.org/x/crypto/chacha20poly1305
golang.org/x/crypto/pbkdf2
github.com/syncthing/syncthing/lib/upgrade
golang.org/x/crypto/scrypt
github.com/dchest/siphash
github.com/syncthing/syncthing/lib/protocol
google.golang.org/protobuf/internal/filetype
github.com/syndtr/goleveldb/leveldb/util
google.golang.org/protobuf/runtime/protoimpl
github.com/syndtr/goleveldb/leveldb/cache
google.golang.org/protobuf/types/known/timestamppb
github.com/syndtr/goleveldb/leveldb/comparer
github.com/golang/protobuf/ptypes/timestamp
github.com/syndtr/goleveldb/leveldb/storage
github.com/syndtr/goleveldb/leveldb/filter
github.com/syndtr/goleveldb/leveldb/opt
github.com/golang/snappy
github.com/syncthing/syncthing/lib/svcutil
github.com/syndtr/goleveldb/leveldb/errors
log/syslog
runtime/pprof
github.com/syndtr/goleveldb/leveldb/iterator
github.com/syncthing/syncthing/lib/dialer
github.com/syndtr/goleveldb/leveldb/journal
github.com/syndtr/goleveldb/leveldb/memdb
github.com/syndtr/goleveldb/leveldb/table
github.com/syncthing/syncthing/lib/osutil
github.com/rcrowley/go-metrics
github.com/gobwas/glob/util/strings
github.com/gobwas/glob/match
github.com/syncthing/syncthing/lib/config
github.com/syndtr/goleveldb/leveldb
github.com/gobwas/glob/syntax/lexer
hash/adler32
github.com/gobwas/glob/syntax/ast
github.com/syncthing/syncthing/lib/build
github.com/gobwas/glob/compiler
github.com/gobwas/glob/syntax
archive/zip
github.com/gobwas/glob
github.com/syncthing/syncthing/lib/db/backend
github.com/syncthing/syncthing/lib/ignore
golang.org/x/crypto/md4
github.com/Azure/go-ntlmssp
github.com/syncthing/syncthing/lib/db
github.com/syncthing/syncthing/lib/scanner
github.com/go-asn1-ber/asn1-ber
github.com/julienschmidt/httprouter
github.com/syncthing/syncthing/cmd/syncthing/decrypt
github.com/go-ldap/ldap/v3
github.com/syncthing/syncthing/lib/assets
github.com/lucas-clemente/quic-go/internal/protocol
github.com/syncthing/syncthing/lib/api/auto
github.com/lucas-clemente/quic-go/internal/utils
golang.org/x/crypto/cryptobyte
golang.org/x/crypto/curve25519/internal/field
golang.org/x/sys/cpu
github.com/lucas-clemente/quic-go/quicvarint
golang.org/x/crypto/curve25519
github.com/syncthing/syncthing/lib/beacon
github.com/syncthing/syncthing/lib/nat
github.com/syncthing/syncthing/cmd/syncthing/cli
github.com/marten-seemann/qtls-go1-17
github.com/syncthing/syncthing/lib/discover
github.com/jackpal/gateway
github.com/jackpal/go-nat-pmp
github.com/calmh/xdr
github.com/syncthing/syncthing/lib/pmp
github.com/syncthing/syncthing/lib/stats
github.com/syncthing/syncthing/lib/relay/protocol
github.com/ccding/go-stun/stun
github.com/syncthing/syncthing/lib/upnp
github.com/syncthing/syncthing/lib/relay/client
golang.org/x/time/rate
database/sql/driver
github.com/kballard/go-shellquote
github.com/syncthing/syncthing/lib/watchaggregator
github.com/syncthing/syncthing/lib/ur/contract
github.com/syncthing/syncthing/lib/versioner
github.com/chmduquesne/rollinghash
github.com/syncthing/syncthing/lib/tlsutil
github.com/chmduquesne/rollinghash/adler32
github.com/vitrun/qart/gf256
github.com/syncthing/syncthing/lib/weakhash
image
github.com/vitrun/qart/coding
golang.org/x/crypto/blowfish
github.com/lucas-clemente/quic-go/internal/qtls
golang.org/x/crypto/bcrypt
golang.org/x/text/runes
github.com/lucas-clemente/quic-go/internal/qerr
internal/profile
github.com/vitrun/qart/qr
github.com/lucas-clemente/quic-go/internal/flowcontrol
github.com/lucas-clemente/quic-go/internal/wire
os/signal
github.com/lucas-clemente/quic-go/logging
net/http/pprof
github.com/lucas-clemente/quic-go/internal/logutils
github.com/lucas-clemente/quic-go/internal/congestion
github.com/lucas-clemente/quic-go/internal/handshake
github.com/lucas-clemente/quic-go/internal/ackhandler
github.com/lucas-clemente/quic-go
github.com/AudriusButkevicius/pfilter
github.com/syncthing/syncthing/lib/stun
github.com/syncthing/syncthing/lib/connections
github.com/syncthing/syncthing/lib/ur
github.com/syncthing/syncthing/lib/model
github.com/syncthing/syncthing/lib/api
github.com/syncthing/syncthing/lib/syncthing
github.com/syncthing/syncthing/cmd/syncthing
*** Finished build for arm64
All builds finished
[INFO] Reverting "go.mod", "go.sum" to checkout state ...
[INFO] Checking if SyncthingNative was built successfully ...
9

@calmh Iā€™ve compared Android NDK r22 vs. Android NDK r23. They ā€œjustā€ have removed the GNU binutils, only LLVM is left. I can see this when comparing the directory structures and searching for the removed files.

image

image

10

It sounds like they should still be there under the llvm dir somewhere:

I have no idea why the arm64 build passes; maybe it uses a different linking mechanism or something.

1 Like
11

With cgo_enabled=0, it doesnā€™t need a linker. It ends up a standard go binary I guess. I know you used to need to build with cgo before to have dns working, but perhaps that is no longer the case.

Never the less, we should still built with cgo and the ndk.

1 Like
12

Can we switch to lld?

Looks like the linker can be changed:

export CC="/path/to/clang"
export GO_LDFLAGS="-extld ${CC}"
1 Like
13

Thanks @bt90 . Syncthingā€™s build.go contains:

func ldflags(tags []string) string {
	b := new(strings.Builder)
	b.WriteString("-w")
	fmt.Fprintf(b, " -X github.com/syncthing/syncthing/lib/build.Version=%s", version)
	fmt.Fprintf(b, " -X github.com/syncthing/syncthing/lib/build.Stamp=%d", buildStamp())
	fmt.Fprintf(b, " -X github.com/syncthing/syncthing/lib/build.User=%s", buildUser())
	fmt.Fprintf(b, " -X github.com/syncthing/syncthing/lib/build.Host=%s", buildHost())
	fmt.Fprintf(b, " -X github.com/syncthing/syncthing/lib/build.Tags=%s", strings.Join(tags, ","))
	if v := os.Getenv("EXTRA_LDFLAGS"); v != "" {
		fmt.Fprintf(b, " %s", v)
	}
	return b.String()
}

I assume I need to put EXTRA_LDFLAGS env var to Android workspaceā€™s /syncthing/build-syncthing.py so it says:

environ = os.environ.copy()
environ.update({
    'GOPATH': module_dir,
    'GO111MODULE': 'on',
    'CGO_ENABLED': '1',
    'EXTRA_LDFLAGS': '-extld ' + target['clang'],
})

Is that correct?

14

@bt90

  • EXTRA_LDFLAGS: It seems the build process doesnā€™t respect this flag.
  • GO_LDFLAGS: It seems the build process doesnā€™t respect this flag.
  • CGO_LDFLAGS: Is respected.
'CGO_LDFLAGS': '-extld ' + ndk_clang_fullfn,

lead me to another error:

# runtime/cgo
ld: error: C:\Temp\git\syncthing-android\syncthing\..\..\syncthing-android-prereq\android-ndk-r23\toolchains\llvm\prebuilt\windows-x86_64\bin\armv7a-linux-androideabi16-clang:2: unknown directive: if
>>> if [ "$1" != "-cc1" ]; then
>>> ^
clang: error: linker command failed with exit code 1 (use -v to see invocation)

2nd try:

'CGO_LDFLAGS': '-extld C:\\Temp\\git\\syncthing-android-prereq\\android-ndk-r23\\toolchains\\llvm\\prebuilt\\windows-x86_64\\bin\\clang.exe',

# runtime/cgo
ld: error: C:\Temp\git\syncthing-android-prereq\android-ndk-r23\toolchains\llvm\prebuilt\windows-x86_64\bin\clang.exe: unknown file type
clang: error: linker command failed with exit code 1 (use -v to see invocation)
15

I think clang is the compiler, not the linker.

1 Like
16

The llvm dir looks like this.

image

17

Iā€™ve tried ld.exe and ld.lld.exe but the error is still the same:

# runtime/cgo
ld: error: C:\Temp\git\syncthing-android-prereq\android-ndk-r23\toolchains\llvm\prebuilt\windows-x86_64\bin\ld.lld.exe: unknown file type
clang: error: linker command failed with exit code 1 (use -v to see invocation)
18

Sorry, Iā€™m trying EVERYTHING but Iā€™m unable to build with CGO.

19

You should set CC_FLAGS to have -v to see how it invokes the linker.

1 Like
20

Also you can set BUILDDEBUG=1 to see what the build script does, and hack it to add -x to the go build command line to get extensive output from go itself about what it invokes.

1 Like