Looking at /var/log/syslog I see
Jul 11 15:02:32 mostro syncthing[5123]: [D4VPV] INFO: My name is "mostro"
Jul 11 15:02:32 mostro syncthing[5123]: [D4VPV] INFO: Device DGIGZHY-43MUYIF-3FFDU6N-5HQ7AGK-5O27CEE-ZCIDI5L-SHNN2WN-ZJYIWAD is "nas-fisso" at [tcp4://192.168.68.20:22000]
Jul 11 15:02:32 mostro syncthing[5123]: [D4VPV] INFO: Device JOF5Y27-OGDEFBY-TE5NURP-RQREZ65-NI3FIC5-SQM2VSR-ZIERYC7-4INYIQC is "enrico-n56vz" at [dynamic]
Jul 11 15:02:32 mostro syncthing[5123]: [D4VPV] INFO: Device PRZRF3G-7NY2XKA-Y73IDY2-32YFBPY-V6M3NZK-UBOD7IR-FVFYBKK-7HQEAAQ is "c37-mirto" at [tcp4://192.168.68.2:22000]
Jul 11 15:02:32 mostro syncthing[5123]: [D4VPV] INFO: Device 2AZGNLU-3EA2MKT-PBVZA2X-4EZX4TI-6CTSCIW-V4UTK7I-Q3ORHV2-DCKZQAG is "vm-mirto" at [tcp4://192.168.102.232:22000]
Jul 11 15:02:32 mostro syncthing[5123]: [D4VPV] INFO: Established secure connection to PRZRF3G-7NY2XKA-Y73IDY2-32YFBPY-V6M3NZK-UBOD7IR-FVFYBKK-7HQEAAQ at 192.168.68.10:22000-192.168.68.2:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
Jul 11 15:02:32 mostro syncthing[5123]: [D4VPV] INFO: Device PRZRF3G-7NY2XKA-Y73IDY2-32YFBPY-V6M3NZK-UBOD7IR-FVFYBKK-7HQEAAQ client is "syncthing v1.18.0" named "C37-mirto" at 192.168.68.10:22000-192.168.68.2:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
Jul 11 15:02:33 mostro syncthing[5123]: [D4VPV] INFO: Completed initial scan of sendreceive folder "federica-sync" (4dodo-ewz76)
Jul 11 15:02:37 mostro syncthing[5123]: [D4VPV] INFO: Completed initial scan of sendreceive folder "mirto-dati" (csfr9-c3gdz)
Jul 11 15:02:37 mostro syncthing[5123]: [D4VPV] WARNING: Fatal error: rqjyz-fieaz Update(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4, [7]): leveldb/table: corruption on data-block (pos=903927): checksum mismatch, want=0x669dca55 got=0x498c79eb [file=021154.ldb]
Jul 11 15:02:37 mostro syncthing[5123]: panic: leveldb/table: corruption on data-block (pos=903927): checksum mismatch, want=0x669dca55 got=0x498c79eb [file=021154.ldb]
Jul 11 15:02:37 mostro syncthing[5123]: [monitor] WARNING: Panic detected, writing to "/home/mirto/.config/syncthing/panic-20210711-150237.log"
Jul 11 15:02:37 mostro syncthing[5123]: [monitor] WARNING:
Jul 11 15:02:37 mostro syncthing[5123]: *********************************************************************************
Jul 11 15:02:37 mostro syncthing[5123]: * Crash due to corrupt database. *
Jul 11 15:02:37 mostro syncthing[5123]: * *
Jul 11 15:02:37 mostro syncthing[5123]: * This crash usually occurs due to one of the following reasons: *
Jul 11 15:02:37 mostro syncthing[5123]: * - Syncthing being stopped abruptly (killed/loss of power) *
Jul 11 15:02:37 mostro syncthing[5123]: * - Bad hardware (memory/disk issues) *
Jul 11 15:02:37 mostro syncthing[5123]: * - Software that affects disk writes (SSD caching software and simillar) *
Jul 11 15:02:37 mostro syncthing[5123]: * *
Jul 11 15:02:37 mostro syncthing[5123]: * Please see the following URL for instructions on how to recover: *
Jul 11 15:02:37 mostro syncthing[5123]: * https://docs.syncthing.net/users/faq.html#my-syncthing-database-is-corrupt *
Jul 11 15:02:37 mostro syncthing[5123]: *********************************************************************************
Jul 11 15:02:38 mostro syncthing[5123]: [monitor] INFO: Reporting crash found in panic-20210711-150237.log (report ID b21341bc) ...
I have to delere the index db with the command
rm -Rf /home/mirto/.config/syncthing/index-v0.14.0.db
Then syncthing restarts correctly and works for eigth / ten hours
Syncthing version: v1.18.0, Linux (64-bit Intel/AMD) Operating system of all involved machines Kubuntu 20.04.2 (updated daily) one machine with Ubuntu Studio 21.04
Every 8 / 10 hours syncthing crashes saying that the index db is corrupted
syncthing works forever as it worked until two or three weeks ago (the crashing machine worked continuously for months the suddenly some weeks ago started crashing)
to reproduce start syncthing an wait the crash
When I clear the database and restart syncthing I see that the itemes to sync are more than one million.
Out of Sync Items 1.026.147 items, ~198 GiB
There is some limit for Syncthing?
In /etc/sysctl.conf I have
fs.inotify.max_user_watches=204800
net.core.rmem_max=2500000
This error happens in the last weeks; before everything worked fine.
What can I do?
imsodin
July 11, 2021, 6:27pm
2
Check your disk (e.g. with smartmontools) and ram (e.g. memtest), it’s likely faulty. Or well first check system logs for anything unusual.
With dmesg I see
[26187.143589] BUG: Bad page state in process syncthing pfn:366d6b
[26187.143592] page:ffffc5d28d9b5ac0 refcount:11141120 mapcount:0 mapping:0000000000000000 index:0x1
[26187.143593] flags: 0x17ffffc0000000()
[26187.143594] raw: 0017ffffc0000000 dead000000000100 dead000000000122 0000000000000000
[26187.143595] raw: 0000000000000001 0000000000000000 00aa0000ffffffff 0007000000000000
[26187.143595] page dumped because: page still charged to cgroup
[26187.143595] page->mem_cgroup:0007000000000000
[26187.143596] Modules linked in: xt_recent xt_addrtype xt_hashlimit xt_mark xt_nat xt_CT iptable_raw nfnetlink_log xt_NFLOG nf_log_ipv4 nf_log_common xt_LOG nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_pptp nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp rfcomm xt_MASQUERADE xt_conntrack xt_CHECKSUM ipt_REJECT nf_reject_ipv4 xt_tcpudp ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter bpfilter bridge stp llc cmac algif_hash algif_skcipher af_alg bnep iwlmvm mac80211 libarc4 iwlwifi nls_iso8859_1 snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi uvcvideo snd_usb_audio snd_hda_intel edac_mce_amd btusb snd_intel_dspcfg
[26187.143613] btrtl videobuf2_vmalloc snd_usbmidi_lib videobuf2_memops kvm_amd snd_hda_codec videobuf2_v4l2 snd_seq_midi btbcm ccp snd_seq_midi_event videobuf2_common btintel snd_rawmidi videodev snd_hda_core kvm snd_seq mc joydev snd_hwdep input_leds bluetooth snd_pcm mxm_wmi wmi_bmof ecdh_generic efi_pstore ecc k10temp snd_seq_device snd_timer cfg80211 snd soundcore nvidia_uvm(OE) mac_hid nfsd sch_fq_codel auth_rpcgss nfs_acl lockd parport_pc grace ppdev lp parport sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul cec crc32_pclmul ghash_clmulni_intel rc_core aesni_intel crypto_simd cryptd drm glue_helper igb r8169 i2c_piix4 ahci nvme xhci_pci libahci dca realtek xhci_pci_renesas nvme_core i2c_algo_bit wmi
[26187.143639] CPU: 6 PID: 6729 Comm: syncthing Tainted: P OE 5.8.0-61-generic #68~20.04.1-Ubuntu
[26187.143639] Hardware name: Micro-Star International Co., Ltd. MS-7C35/MEG X570 ACE (MS-7C35), BIOS 1.D0 01/22/2021
[26187.143640] Call Trace:
[26187.143644] dump_stack+0x74/0x92
[26187.143646] bad_page.cold+0x63/0x94
[26187.143648] check_new_page_bad+0x6d/0x80
[26187.143649] rmqueue+0xb58/0xd30
[26187.143651] ? page_counter_try_charge+0x34/0xb0
[26187.143652] get_page_from_freelist+0x197/0x370
[26187.143653] __alloc_pages_nodemask+0x161/0x2f0
[26187.143654] alloc_pages_current+0x87/0xe0
[26187.143656] __page_cache_alloc+0x72/0x90
[26187.143657] page_cache_readahead_unbounded+0x93/0x200
[26187.143658] __do_page_cache_readahead+0x35/0x40
[26187.143658] ondemand_readahead+0x148/0x2a0
[26187.143659] page_cache_async_readahead+0xb5/0xe0
[26187.143660] generic_file_buffered_read+0x178/0xc50
[26187.143661] ? __fpu__restore_sig+0x281/0x630
[26187.143662] generic_file_read_iter+0xdc/0x140
[26187.143664] ? dput+0x39/0x310
[26187.143666] ext4_file_read_iter+0x58/0x180
[26187.143668] new_sync_read+0x10c/0x1a0
[26187.143669] vfs_read+0x161/0x190
[26187.143670] ksys_read+0x67/0xe0
[26187.143671] __x64_sys_read+0x1a/0x20
[26187.143672] do_syscall_64+0x49/0xc0
[26187.143674] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[26187.143676] RIP: 0033:0x4c523b
[26187.143677] Code: fa ff eb bd e8 46 af fa ff e9 61 ff ff ff cc e8 1b 7b fa ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[26187.143677] RSP: 002b:000000c00bf2b350 EFLAGS: 00000206 ORIG_RAX: 0000000000000000
[26187.143678] RAX: ffffffffffffffda RBX: 000000c000043000 RCX: 00000000004c523b
[26187.143678] RDX: 0000000000008000 RSI: 000000c0413c6000 RDI: 0000000000000014
[26187.143679] RBP: 000000c00bf2b3a0 R08: 0000000000000000 R09: 0000000000000002
[26187.143679] R10: 0000000054480f7d R11: 0000000000000206 R12: 0000000000000089
[26187.143679] R13: 0000000000000003 R14: 000000000000000a R15: ffffffffffffffff
Now I tri to verify memory
Is the database on a drive with some unusual filesystem?
Not sure why you think NFS is relevant in this case.
The SSD file system is a normal ext4
Still trying to test memories
scumola
July 14, 2021, 4:14pm
6
syncthing had a kernel oops according to your logs. restart syncthing or just reboot to clean any residual stuff up and try again maybe?
Resterted and rebooted.
The day after the problem disappeared.
Thanks
system
August 14, 2021, 4:36pm
8
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.