App has been syncing for a long time after multiple edits, many image attachments

Issues
1

Bug report

New user setting up inkdrop, and have manually added a number of notes with attachments - totalling ~50mb of images - into the app.

  • Initially it was syncing OK, however after recent edits the app seems to be stuck in “Syncing” mode.
  • The changes are not showing up on other copies of inkdrop or in the mobile app (iOS).
  • Inspecting the console there are no errors or warnings.
  • I would estimate the app has now been syncing for around 30 minutes or more.

I have performed both a full export and a backup to confirm the size and number of notes. Does it just take a long time to sync at first with a large number of changes?

Info

  • Platform: Fedora 29 (5.0.7-200.fc29.x86_64)
  • Platform version: Linux Desktop.
  • App Version: 3.25.4

Reproduce

  • Manually import a number of notes.
  • Paste images into the notes as required.
  • Syncing has failed without any obvious indicators in the developer console as to errors or failures.

Thank you!

2

Hi zed_g,

Thank you for reporting.
That’s weird.
I’m seeing the server load is increasing and that might be caused by your data.
Could you please stop running the app for now?
I’ll look into it.

3

Ok I’ve shut it down just now! Please let me know how it goes, I will leave it offline until I hear back from you.

4

Thanks!
I found socket hang up error is happening while syncing your image data but the app keeps retrying.
I guess sending data from the client is taking too long time and the connection timed out.

5

Seems like the app is still running. Could you check if the process is running on other computers?

6

Huh, that’s strange - they should all be offline, everything else is switched off and I have closed it on my main workstation. Is it still the case now?

The largest image file in the encrypted backups is 11mb if that helps frame the problem.

edit - I’ve shut down everything and uninstalled the app just in case!

7

Looks like they stopped. Thanks.
And your image size looks fine. The server is not reporting any other problem. Hmm

8

So the client is hanging up and retrying repeatedly? Nothing unusual to report about internet speeds here, they seem typical. Are there any client side logs I can send? I went looking for some but couldn’t seem to find them.

9

Yes. What I can know at the moment is that the connection hangs up while syncing.
So I would like to know if it works: Changing batch size of sync.
I’ll post the config for that.

10

Ok awesome! Apologies but it is late here, I will test the config in the morning and report back if that is OK.

11

Sure! It’s also late in Japan here. See you tomorrow!

12

Please add sync options in ~/.config/inkdrop/config.cson like so:

"*":
  core:
    db:
      sync:
        options:
          push:
            batch_size: 2
            batches_limit: 2
          pull:
            batch_size: 20
            batches_limit: 2

And launch the app.
It will send only 2 documents per request so the connection should not hang due to timeout.
Since you have many notes and images, it will take for a while until the sync finishes.
I’ll check the server log tomorrow morning if the same error happens.

Thank you for your cooperation!

13

Ok, will test and report! Are there any other settings you would like tested if these don’t work as expected?

Also, any limit you would like on the time I let it try and sync to avoid server disruptions again?

Thank you for the help!

14

Not sure for now, but if a particular file/note causes this issue, it will be hard.
You will have to remove local database and have to add notes slowly again while checking the sync is working fine.

I will be working from 9:00 to 18:00 (JST) :slight_smile:

15

Ok thanks for the guidance! Sleep well, I will kick off the sync again in the morning and try to step-test the notes if the issue still happens with the new settings. We are close in time-zones so should be able to give feedback during better hours :slight_smile:

16

Got it, thank you! Good night~

17

Ok, I have the sync settings in place as described and have started the sync process now, please let me know how it looks at your end, or if I need to turn it off again!

18

Ok, I think it is a weird note somewhere? I have some testing and observations:

  • I logged out, deleted all local data, logged back in.
  • My sync was up to date with around 60% of the notes I added yesterday.
  • I deleted all notes, then imported a backup made yesterday before the issue started.
  • I noticed after this process, some notes were missing from the backup.
  • However, all notes are present in the markdown export created by “Export -> All notes”.

So I wonder if there is content in one of the notes that is causing the sync or encryption to have issues?

19

Hi. I found that the same error occurs again. But it stopped happening since 20 minutes ago.

20

Yeah I suspect that some notes causing the problem.
You can run the app from terminal so you can see error logs from the main process. It might be helpful.

1 Like