Silent sync failure behind corporate proxy, data loss scenario

Issues
1

Bug report

Observed this in the following conditions:

  • launch inkdrop without any proxy configuration in place (e.g, do not configure .cson file).
  • connect to a network that uses a corporate proxy for web access.
  • create a new note and edit it, then save.
  • sync indicator at bottom left indicates a recent sync, however note does not show up on other clients.
  • Requesting a manual sync completes OK without error, but not does not turn up on other clients.

I have not seen any issues reported under the developer tools. Would it be expected to create a synchronisation error in this case?

Switching from the corporate network to an open network then allows the sync to complete OK.

I have noticed that this situation can be used to create data loss though, e.g., by doing the following:

Using two clients, for example, “Client A” on normal internet, and “Client B” switching between a corporate proxy and open internet:

  • Edit note on “Client A” and save changes.
  • Note synchronises to “Client B”.
  • Switch “Client B” to a proxy connection (without configuring proxy information in .cson).
  • Update note on “Client A” and then “Client B”
  • Switch “Client B” from proxy to open connection.
  • When “Client B” syncs, it destroys the changes on “Client A” instead of creating a conflicted note.

Info

  • Platform: Windows 10 Enterprise
  • Platform version: 1909
  • App Version: 5.1.2

Reproduce

As above, for the silent sync failure:

  • create content while behind a corporate proxy or restricted internet connection, sync does not indicate any failure, but content is not synced.

For data loss scenario:

  • Update the same note across two different clients while one has restricted internet with a silent sync failure. When a normal connection is restored, data is lost from one update.
2

Hi Zed,

The app can’t sync if you are behind a corporate proxy.
You have to configure it:

3

Yes definitely! But I was confused about the false-positive around the sync completing, is that expected? I thought it might be better if it indicated that it couldn’t sync when it was blocked by a proxy.

4

Yeah, that makes sense.
For not only web proxy, but network errors also happen for various reasons, like disconnected environment, unstable network, Kerberos authentication, etc.
It is annoying to get noticed that it couldn’t sync due to those reasons if you are aware of them.
Besides, it is hard to distinguish if it is blocked by a web proxy because the error code/message is not always the same.

5

Yeah, is it appropriate or useful to do something similar to apple where they try and get a simple remote file (like inkdrop.app/test.txt) as a confirmation of external access? When it fails it gives you a general “bad network state” under a range of different disruptions.

I was curious though, how come the sync process still thinks it is OK when it cannot contact the users remote DB endpoint? Like in this case “https://store.inkdrop.app/user-xxxxxx” is likely to be getting a 403 auth error when it tries to sync?

Apologies, I meant to say: not a totally serious issue on my end, inkdrop is a very incredible experience for making notes and documentation, thank you! =) just got caught-out by the silent sync failure here.

6

You can know if there is a bad network state but I’m saying that it’s hard to distinguish if it’s by a web proxy or not.
PouchDB does not report errors for live sync but it just pauses syncing and retries automatically:
https://pouchdb.com/guides/replication.html#live–replication

Inkdrop checks the account state via the account API, not via the sync API. The auth session will be expired if it keeps failing.

Thanks for letting me your thoughts!

7

No, thank you for entertaining them. I feel like I’m taking up your time for little reason!

That makes sense with PouchDB, so by design of it’s resiliency it accepts intermittent network failures with the assumption that it will sync at a later point when the network resumes, so this kind of scenario does not look that unusual to it.

Is there any way to avoid the note conflict that can occur in that circumstance as per the second point in the original report?

1 Like
8

Inkdrop does not support auto-merging, so it is not possible at the moment.
But you can restore the old revisions:

1 Like
9

Great, thank you for your patience!

1 Like
10

My plain text backup plugin may have helped a little here:
https://my.inkdrop.app/plugins/plain_text_backups

11

Hi @craftzdog! Apologies for resurrecting this thread, but I encountered the scenario again today and realised I can’t restore from revisions as they are destroyed by the merge issue. I lost data from a number of notes as a result of the pattern.

I’ve tried to illustrate the sequence as best as possible as follows:

Screenshot from 2021-01-27 20-19-08
Screenshot from 2021-01-27 20-19-08990×718 39.1 KB

I understand, when using most recent timestamp, Update (B) is more recent than update (A), but as it does not already contain update (A), update (A) is lost forever. Looking in the revision history of Client A, I could not find the local changes for update (A).

I had some thoughts / questions in relation to this:

  • Is there a reasonable way of ensuring clients with diverged updates (due to connection issues or due to timestamp issues) do not destroy changes from the other? I know auto-merging isn’t yet supported but it would be good if the client could warn if this situation happened, or prompt with the patch info for the revision?.
  • Is there a way of warning when step (3) occurs on a client? e.g., can the user be warned immediately in the sync notification if the local DB cannot be synced to the remote server so they have a chance to prevent data loss?

Thank you!

12

Hi zed,

Thank you for illustrating the issue. That’s clear to understand.
Didn’t you find the update (A) in the client A instead of B?
Revision histories are only stored in local.
I mean, if you make a change on client A, the change is only retained in the revision history on client A.

Is there a reasonable way of ensuring clients with diverged updates

No, since it is designed to be basically used by a person, not a team.

Is there a way of warning when step (3) occurs on a client?

No. Because there is no standard error code to tell clients that the error was due to a web proxy.
Clients can fail for various reasons not only web proxy.

13

I mean, if you make a change on client A, the change is only retained in the revision history on client A.

Edit - to confirm, on testing again this is working as you have described. The local revision is present, but is sometimes several entries deep from the previous revision, so it looks like I’ve missed it during previous sync issues.

Additionally, after further sync attempts on and off the proxy from the client, it generated a [conflicted] entry for the note which is great.

Apologies! Also, the patch view is very cool.

No, since it is designed to be basically used by a person, not a team.

This is individual use - for example, my home workstation has Inkdrop installed, as does my laptop which I take to different locations, some proxied.

No. Because there is no standard error code to tell clients that the error was due to a web proxy.
Clients can fail for various reasons not only web proxy.

But in this case the failure is being falsely reported as “Synced”, instead of “Failed”?

  1. If I completely disconnect the internet it will remain stuck in “Syncing” state, which is expected.
  2. Behind a proxy it will confirm a recent sync (e.g., "Synced at <now>) when it has not actually completed a successful sync.

I was wondering if there is any way for (2) to present an error instead of a false-positive?

14

If it’s useful, I captured the logs when the app is reporting Synced but failing due to not being able to navigate past the proxy.

It’s indicating a relevant 40x error via System.AggregateException: One or more errors occurred. ---> System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required., and bugsnag is also failing as a result of the loss of comms.

Would there be benefit in using a simplified connectivity check, like a captive-portal URL? Apple uses http://captive.apple.com/hotspot-detect.html to detect if a connection is proxied or blocked, as does Firefox (http://detectportal.firefox.com/success.txt). This can provide a “ground truth” access test without having to deal with a range of different error codes and simplify your network validation.

Extracted log below:

PS C:\Users\myUser\AppData\Local\inkdrop\app-5.2.1> .\Inkdrop.exe --enable-logging
PS C:\Users\myUser\AppData\Local\inkdrop\app-5.2.1>
[bugsnag] Loaded!
2021-01-28T10:42:25.721Z app:info Initializing inkdrop app..
2021-01-28T10:42:25.725Z app:info Arguments: C:\Users\myUser\AppData\Local\inkdrop\app-5.2.1\Inkdrop.exe
2021-01-28T10:42:25.726Z app:info Arguments: --enable-logging
2021-01-28T10:42:25.777Z app:info Starting inkdrop app..
2021-01-28T10:42:25.864Z app:info axios default config: { proxy: false, httpAgent: undefined, httpsAgent: undefined }
[25144:0128/204226.689:INFO:CONSOLE(51270)] "[bugsnag] Loaded!", source: <embedded> (51270)
<< LINES REMOVED >>
2021-01-28T10:42:39.705Z app:error error during auto update: Error: Command failed: 4294967295
System.AggregateException: One or more errors occurred. ---> System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required. ---> System.ComponentModel.Win32Exception: The system cannot contact a domain controller to service the authentication request. Please try again later
   at System.Net.NTAuthentication.GetOutgoingBlob(Byte[] incomingBlob, Boolean throwOnError, SecurityStatus& statusCode)
   at System.Net.NTAuthentication.GetOutgoingBlob(String incomingBlob)
   at System.Net.NegotiateClient.DoAuthenticate(String challenge, WebRequest webRequest, ICredentials credentials, Boolean preAuthenticate)
   at System.Net.NegotiateClient.Authenticate(String challenge, WebRequest webRequest, ICredentials credentials)
   at System.Net.AuthenticationManagerDefault.Authenticate(String challenge, WebRequest request, ICredentials credentials)
   at System.Net.AuthenticationState.AttemptAuthenticate(HttpWebRequest httpWebRequest, ICredentials authInfo)
   at System.Net.HttpWebRequest.CheckResubmitForAuth()
   at System.Net.HttpWebRequest.CheckResubmit(Exception& e, Boolean& disableUpload)
   --- End of inner exception stack trace ---
<< LINES REMOVED >>
---> (Inner Exception #0) System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required. ---> System.ComponentModel.Win32Exception: The system cannot contact a domain controller to service the authentication request. Please try again later
<< LINES REMOVED >>
[bugsnag] Report failed to send…
Error: unable to get local issuer certificate

 [Error: unable to get local issuer certificate

] {
  code: 'UNABLE_TO_GET_ISSUER_CERT_LOCALLY'
}
1 Like
15

Additionally, after further sync attempts on and off the proxy from the client, it generated a [conflicted] entry for the note which is great.

It happens while you are editing the same note which is not saved yet but a new revision was synced from another client.
If it was already saved, the revision with a newer timestamp will be a winner.

This is individual use

Right. I mean, I assume that local revision histories are enough for the issue because you are only the owner of your notes.

But in this case the failure is being falsely reported as “Synced”, instead of “Failed”?

Yes, it is error-tolerant because it automatically retries syncing.
As it fails for various reasons, it’d be annoying if the app notifies every network failure.
It often happens in mobile networks.
In terms of implementation, the app uses PouchDB to accomplish the live replication and it does not report errors as it automatically retries:

https://pouchdb.com/guides/replication.html#live–replication

So, I have to say it is an intended behavior by design.

It’s indicating a relevant 40x error via System.AggregateException: One or more errors occurred. ---> System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required., and bugsnag is also failing as a result of the loss of comms.

Oh, it looks useful.
This error message seems to be output during auto update which is a built-in function of Electron.
So, it appears that we are able to know if there is an issue caused by a web proxy by seeing the 407 HTTP response code. I don’t know other web proxies also work as well though.

Overall, I think the point is:

  1. Whether the app should always make a [conflicted] note when detected a conflict
  2. Whether the app should report the 407 proxy error to prevent users from making conflicts

I’m currently working on improving the mobile version for the current roadmap.
Let me consider them after finishing it.

16

First thanks for replying here, I feel like I’m asking too much effort over an occasional issue. I was thinking it was good to capture after I caught myself out and made a range of edits across different notes and had to work back through them all to fix it.

Yes, I think this could be painful: we tried dealing with this for an internal app syncing content on a mobile device that, as an example, would move between Corporate Network (Proxied) -> Public network (Sometimes Captive) -> Offline -> Public network (VPN + Proxy) and so on.

Not all proxies or portals will return the 407 auth failure, they may redirect to an auth page (captive), drop the connection, successfully return the wrong content (e.g., a “access blocked” page), etc.

State validation became pointlessly complex so we opted for a “captive style” check of get https://app.privateEnterpriseServer.org/online.txt -> if "200 OK" && online.txt contains <hash within file> then { online } else { offline } , as we don’t really care about anything other than confirming we are connected, and the expected content in online.txt isn’t being replaced (like can happen with MITM proxies and captive portals).

Thats what got me thinking something like “https://store.inkdrop.app/success.txt” would be useful to avoid parsing the Electron behaviours for different network states.

I’ve used other apps that default to [Conflicted], it can be OK as it gives you a chance to do your own external merge. Could it also suggest a patch if there’s mismatched changes coming in from another client?

I’m not sure, as a personal preference I don’t mind a non-modal warning to indicate sync is offline, e.g., inkdrop uses the non-modal confirmation of Synced at the bottom left, would something subtle (such as a low-tone orange “Sync Offline”) be too intrusive in the design?

Thank you again for the time in your replies!

1 Like
17

Update:

The app will check network reachability when pausing the live sync.
If it failed to retrieve a static file on www.inkdrop.app, it sets the status as “Offline” like so:

Screen Shot 2021-04-30 at 17.07.17

It will be rolled out in the next release.

Also, I’m making it able to review conflicted revisions manually:

1200×1012 84.3 KB

18

Fixed in v5.3.0🎉 Thanks again for reporting. Credited here: