Determining why connections fail on web CGI

[tickerguy]

I've got an interesting issue here with the Amcrest API's that I cannot appear to run down.

I have an external application that uses the API to fetch snapshots. It does so using https and a user and password that has access to the correct permissions. A good percentage of the time it works, but sometimes the execution fails and I get this back (verbose logging turned on):

resolving server address: 192.168.4.211:443
SSL options: 83004bff
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES256-GCM-SHA384
Certificate subject: /CN=192.168.4.211/C=US/ST=none/L=none/O=none/OU=none
Certificate issuer: /CN=Product Root CA/C=US/ST=Taxas/L=Houston/O=Amcrest Techno
logies LLC/OU=Amcrest
requesting https://192.168.4.211/cgi-bin/snapshot.cgi?channel=0
server requires authorization
resolving server address: 192.168.4.211:443
SSL options: 83004bff
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES256-GCM-SHA384
Certificate subject: /CN=192.168.4.211/C=US/ST=none/L=none/O=none/OU=none
Certificate issuer: /CN=Product Root CA/C=US/ST=Taxas/L=Houston/O=Amcrest Techno
logies LLC/OU=Amcrest
requesting https://192.168.4.211/cgi-bin/snapshot.cgi?channel=0
fetch: https://user:[email protected]/cgi ... ?channel=0: Internal Server Error

instead of the snapshot.

It appears the TLS Connection is fine but the requested snapshot comes back with nothing. Turning off https and using a straight http connection results in the same problem.

WHEN this occurs the erroring-out command never returns immediately; it's always a few seconds (~5ish) after the command is issued. An immediate retry USUALLY succeeds but that's a problem because when I want the snapshot I want it *now* and the ~5 second delay means whatever I wanted the picture of might not be in the frame anymore.

The usual practice I have is to issue a PTZ command first to point the camera and wait until that returns success, then issue the snapshot request. Of note the PTZ command never fails with the above return; out of thousands of request I've never had a PTZ request log a failure. Whether the snap command fails or succeeds is also not linked to whether I issue the PTZ command first; if I change the "grab" to not make sure the camera is pointing in the right place first (in other words no PTZ command first; just shoot wherever it's pointing) it still fails the same percentage (more or less) of the time. Occasionally an immediate retry will fail as well.

Unfortunately the camera's log tells me nothing; it logs the successful accesses, but not the failures. It LOOKS like what's happening is that the camera accepts the connection and then the CGI process errors out internally (e.g. "segfaults"?) and when it blows up the connection is closed without sending anything (e.g. no headers come back, etc) as I never get the expected size of the transmission which comes with the HTTP headers.

I see this most-often on a 1080p camera although my 2k cam sometimes does the same, both with firmware:
Software Version2.520.AC00.18.R, Build Date: 2017-06-29
WEB Version3.2.1.453504

This MAY be linked to my use of motion notification -- that is, I have the camera issuing SMTP notifications on motion to my controller, which is then turning around and asking for the snapshots as soon as the request comes in. If I sit at my terminal and HAMMER the requests one after another I don't get failures. If I have them issued when I get notifies a fairly-decent percentage of the requests error out.

Any ideas?

[tickerguy]

amc.png (58.33 KiB) Viewed 1030 times

Updating on this if I turn on motion detection with the above screen to "send an email" WITHOUT an attachment, and WITHOUT taking a snapshot, I get the failures a good percentage of the time.

If I trigger the snapshot by placing a motion detector right next to the camera (but not using the camera) it NEVER fails.

It appears that when motion triggers via the on-camera detector (and sends an email to a target, which succeeds) an attempt to fetch a snapshot using the API within the next few seconds WILL fail a good percentage of the time even though you did NOT tell the camera to take a picture on motion detection (one or more.)

[tickerguy]

OK, so here's an update on this....

It appears if I set the dither time to "0" (so I get immediate notification, and handle delays if I desire in my application -- say, if I get a bunch of notifications in a row before I can respond to them all) I get failures. If I set the dither to "1" or any other number I do not.

Is this a bug or expected behavior (in other words, should I not be using zero -- ever?)

[tickerguy]

Setting dither to "1" (or any other number) REDUCES but does NOT eliminate the failures....

No reply at all from the Amcrest people on this?!