Latest revision of xPico wifi modules breaking existing applications

[RoyCole]

I've been having an issue with the latest revision of XPW100100B-01 (Rev:E11) causing lots of strange issues over the serial port. The modem is taking longer to respond to things like CLI directory changes which is breaking existing applications that I am using in production. These devices are coming with 1.5.0.3R3 and I have been downgrading to 1.5.0.0R43 to get around the issue. I don't think the firmware is entirely the issue - I have tested E11's that were downgraded and then upgraded back to 1.5.0.3R3 and the issues have disappeared.

• E11 takes 11 seconds to boot to CLI whereas D11 would only take 4 seconds
  • I had to increase the timeout in the program that sets up the devices in production
• D11 would respond to CLI directory changes (such as navigating to "Device" from "Status") in less than 0.1 seconds whereas the E11 periodically takes longer than 1 second

In the software my company uses to program the xPico chips, I have had to increase these transition delays by several seconds to prevent them from timing out. I haven't even bothered testing the xPico with the actual production software because it uses CLI navigation and it runs a high risk of breaking transition delays. I don't trust the latest firmware and/or hardware revision of the xPico devices and I cannot run the risk of these changes breaking production software.

Can someone at Lantronix please look into this?

[mariano]

We'll need more information, I'll send you an email with what we need.

[RoyCole]

I wasn't able to look into this until now. It seemed like it was working but now I'm having the same issue that others are having where the modem stops responding to serial commands after a period of hours. I verified this issue with a serial sniffer to check communication and the modems are not responding to any commands.

The modems are E13 that are getting downgraded to 1.5.0.0R43 are eventually not responding after a period of hours, needing to be power cycled before working again and failing. I am running a reliabilty test of 16 of our machines and all 16 machines have their xPico modem stop responding.

Currently, I am programming the E13 with the latest firmware (1.5.0.3R3) and am going to install these modems in all 16 machines to see if the problem is resolved. I will update this thread with more information after that test.

[RoyCole]

EDIT: This problem seems to transcend modem/firmware revision leading me to believe something changed in the environment causing it to fail when doing network scans. See the most recent post for the specific issue being related to scanning wifi networks. There still seems like an issue with the modems as it shouldn't be failing like this when doing simple network scans.

Follow up with testing 16 xPico modems with revision E13 and 1.5.0.3R3:

All 16 modems failed to respond to commands within 18 hours. Additionally, these modems were having periodic failures where the modem would not respond to any commands for minutes. This would occur roughly several times per hour.

(Edit: The following test was done in a different location, doing it where the problem was originally occuring is causing problems on all recent revisions of xPico modems/firmware. I have updated the text to reflect the situation). Doing our due diligence, I repeated test with an older xPico modem (D11 with 1.5.0.0R43) and there was still periodic interruptions and hard failures. This test was done at an offsite location and there was no failures detected (Note: this test needs to be repeated as it was too casual of a test to rely on for hard evidence).

Based on other posts by users on this forum, it appears that this is affecting others as well and is not an isolated problem.

It looks like the FAE team does not respond to posts on this forum as they used to in the past so I will be following up with Lantronix customer service this week to determine what the next steps will be.

[RoyCole]

I have narrowed the issue down to running "config>wlan scan" when connected to a specific network. The modem freezes for 2 minutes until the scan times out. In the mean time it disconnects and attempts to reconnect. In the tlog below, the scan starts at 20:28. I've noticed that the tlog doesn't always display the verbosity below, sometimes the tlog will only print "SCAN TIMED OUT!!!" without the preceding text about the link going down as shown in the tlog example below.

Sometimes the system prevents further scans that show "scan.c 135 Scan request failed" in the tlog. In this state, the modem is still connected to the AP (Link up with valid IP address); however, it looks like internet connectivity is lost as ping tests fail when trying to hit google.com (status>Diagnostics>Ping ping google.com). Attempting to connect to a server using ATD server_name.com:PORT fails with "NO ANSWER". Continuing to run "config>wlan scan" after this point will eventually cause it to lock up permanently, which must be the root cause of the hard failure I have seen with the modem eventually not responding.

For this test setup I have two serial terminals running on xPico evaluation board to access the modem like normal with serial port 1 (the DB9 port) and using J5 USB->UART to connect to serial port 2 for accessing tlog while the modem is running. This test was done on D11 with FW 1.4.0.0R28.

>tlog
# S Filename Line Description Time
---- - ---------- ----- ---------------------------------------------- --------
1 I ltrx_platf 66 Power-on reset 00:10:47
2 I ltrx_platf 512 Resetting backup domain 00:10:47
3 I ltrx_platf 465 Configuring default RTC time 00:10:47
4 I spi_flash. 217 Macronix SPI flash 14 sec sz: 4K total: 256 00:00:00
5 I lffs.c 1373 Bank A: 2-96 Bank B: 97-191 00:00:00
6 I lffs.c 811 Using bank B 00061000-000BFFFF 00:00:00
7 I clock.c 185 Current Local Time: 2007-01-01 00:00:00 00:00:00
8 I clock.c 190 UTC offset: 0 minutes 00:00:00
9 I cfgvar_lib 572 Config struct sized at 1110 bytes. 00:00:00
10 I main.c 196 xPicoWifi v1.4.0.0R28 Oct 22 2015 (18:55:25) 00:00:01
11 I main.c 212 MAC Address 00:80:A3:94:BC:85 00:00:01
12 I cli.c 238 Command Line started on Line 2 00:00:01
13 I setup.c 219 Modem Emulation started on Line 1 00:00:01
14 I ltrx_wlan. 443 WLAN Region: United States. 00:00:02
15 I network_co 2911 Taking wlan0 up... 00:00:03
16 I ltrx_wlan. 607 Attempting association with NETWORK_SSID. 00:00:03
17 I ltrx_wlan. 639 Associated with NETWORK_SSID. 00:00:06
18 I ltrx_wlan. 648 BSSID: 86:15:44:a8:6b:26 00:00:06
19 I network_co 3015 wlan0 up. 00:00:06
20 I ltrx_ip.c 285 wlan0 v4 down. 00:00:06
21 I qp_server. 186 Started wlan0 Query Port service. 00:00:16
22 I ltrx_ip.c 285 wlan0 v4 up. 00:00:16
23 I ltrx_ip.c 305 wlan0 IP Address = 169.254.0.1 00:00:16
24 I dns_client 77 wlan0 DNS PRI = 192.168.40.3 00:01:10
25 I dns_client 77 wlan0 DNS SEC = 192.168.40.6 00:01:10
26 I ltrx_ip.c 285 wlan0 v4 up. 00:01:10
27 I ltrx_ip.c 305 wlan0 IP Address = 192.168.45.34 00:01:10
28 I network_co 3054 wlan0 going down... 00:20:28
29 I ltrx_ip.c 285 wlan0 v4 down. 00:20:28
30 I network_co 2911 Taking wlan0 up... 00:20:30
31 I ltrx_wlan. 607 Attempting association with NETWORK_SSID. 00:20:30
32 I ltrx_wlan. 629 Join failed, code 6. 00:20:37
33 E network_co 2565 Associate profile failed. 00:20:37
34 I network_co 2951 Association with 0 failed, try later. 00:20:37
35 I network_co 2911 Taking wlan0 up... 00:20:39
36 I ltrx_wlan. 607 Attempting association with NETWORK_SSID. 00:20:39
37 I ltrx_wlan. 639 Associated with NETWORK_SSID. 00:20:42
38 I ltrx_wlan. 648 BSSID: 86:15:44:a8:6b:26 00:20:42
39 I network_co 3015 wlan0 up. 00:20:42
40 I ltrx_ip.c 285 wlan0 v4 down. 00:20:42
41 I ltrx_ip.c 285 wlan0 v4 up. 00:20:52
42 I ltrx_ip.c 305 wlan0 IP Address = 169.254.0.1 00:20:52
43 I dns_client 77 wlan0 DNS PRI = 192.168.40.3 00:20:56
44 I dns_client 77 wlan0 DNS SEC = 192.168.40.6 00:20:56
45 I ltrx_ip.c 285 wlan0 v4 up. 00:20:56
46 I ltrx_ip.c 305 wlan0 IP Address = 192.168.45.34 00:20:56
47 I ltrx_wlan. 410 SCAN TIMED OUT!!! 00:22:28