We are seeing widespread connectivity issues with PowerDash's AT&T connections. We have escalated this issue with our cellular services provider. We will provide more updates here. We appreciate your patience with this issue.
The AT&T platform issue was resolved Thursday 4/11/2019 at 11:30am EDT. PowerDash's processing of gateway data queued during the communication outage is now complete. As soon as we receive the Root Cause Analysis from our provider, we will post it here for your reference.
Update 8:47am Eastern Tue 4/09/2019: We just received word that the AT&T service "is currently experiencing a data services degradation. Engineers are engaged and working on a resolution." We believe the issue may be constrained to AT&T connections.
Update 11:40am Eastern Tue 4/09/2019: We have compiled additional information for our provider about the affected AT&T connections. We will post a more detailed update about their cause analysis and resolution timetable as soon as we have that information.
Update 2:15pm Eastern Tue 4/09/2019: We continue to track this issue with our service provider. We just confirmed that the issue is affecting many customer fleets, not just PowerDash, and it has been escalated within AT&T multiple times.
Update 5:10pm Eastern Tue 4/09/2019: The root cause of this issue appears to have been DNS server maintenance being done by AT&T. AT&T is currently working on resolution for all affected devices, and we expect to have an update within 6 hours on the resolution.
Update 10:40pm Eastern Tue 4/09/2019: We have heard a basic update that resolution efforts continue. Our service provider states: "We are engaged with the Carrier [AT&T] as we continue to investigate the issue."
Update 8:55am Eastern Wed 4/10/2019: Our provider has given this further detail about the root cause: This issue was due to a carrier-side configuration change to move traffic from one Gateway GPRS Support Node (GGSN) to another; that transfer failed at the carrier level. Our provider began investigating the issue yesterday (Tuesday) as a netflow traffic issue internally and engaged AT&T to have them investigate on their end. Once our provider exhausted their investigation, their continued follow-up with AT&T yielded initial information on the GGSN issue at around 4:45pm EDT yesterday (Tuesday).
Update 5:00pm Eastern Wed 4/10/2019: Despite several reports today from AT&T through our provider that the issue was nearing resolution, the issue is not yet resolved. PowerDash has escalated this issue to top Customer Support and Engineering management at our provider. We will continue to engage on this issue into the evening tonight as we press for resolution.
Update 11:15am Eastern Thu 4/11/2019: Troubleshooting efforts last night and this morning by AT&T and our provider are yielding some progress in understanding the core issue, but we don't have resolution yet. Traces are being run by our provider on affected devices to determine where TCP network traffic is being blocked.
Update 1:00pm Eastern Thu 4/11/2019: Our provider has made significant progress toward resolution of the issue. A significant portion of our AT&T gateway fleet has come back online. However, processing of data queued during the outage period may take some time. All Digi data processing may be delayed during this higher-than-normal data ingestion time.
Update 1:30pm Eastern Fri 4/12/2019: PowerDash's review of the affected gateways indicates that there are no remaining issues with connectivity beyond known, preexisting issues. PowerDash's processing of gateway data queued during the communication outage is now complete.