Renzo Downey
Most state government websites and links were back online Tuesday afternoon as state officials working with vendors labored to overcome hardware issues that first crashed the system last Friday and reportedly continue to cause problems.
The Governor’s Office said Tuesday that the vast majority of the affected state government computer servers were working and online.
Meanwhile, sources close to the project say the system continues to crash and that the Governor’s Office is overly optimistic.
The original outage affected approximately 1,100 servers and took down pages like Gov. Ron DeSantis‘ homepage, FLGov.com, and the state portal, MyFlorida.com.
There was a setback at 3 a.m. Monday when some recovery efforts went offline. But as of 3 p.m. Tuesday, a Governor’s Office spokesperson said only 86 of the servers were still offline and all site links were expected to return by the end of the day.
“(The Florida Digital Service) along with the manufacturer continue to take all necessary actions to restore the remaining servers and respond to changing circumstances,” DeSantis Communications Director Taryn Fenske said.
As Florida Digital Service and the servers’ manufacturer Dell EMC worked to bring everything back on line, the root cause of the failure was not yet apparent Tuesday.
The breadth of the outage and recovery efforts, and the impact on public and intergovernmental services, also were not entirely clear Tuesday.
One agency critical to providing services to the public, The Agency for Health Care Administration, sent Florida Politics a statement Tuesday afternoon that said all “services to the public and public facing Agency IT systems are fully operational.”
The agency did not say in its statement whether its internal systems also were functioning properly.
Then, the agency recalled the statement one minute later and offered no further explanation.
AHCA is the agency charged with regulating health care providers ranging from abortion clinics to hospitals to nursing homes. It also conducts criminal background checks and administers the state’s Medicaid program.
Another agency, the Department of Management Services, did not respond to Florida Politics’ request for information regarding the number of servers at the Department of Children and Families that remain offline. DCF is the state agency that processes applications for public assistance programs such as Medicaid.
Fenske’s statement that 86 servers were offline Tuesday differed from what Chief Information Officer Jamie Grant wrote in 2:48 p.m. and 2:55 p.m. emails Tuesday. In them, he said 110 servers were offline. However, Fenske’s statement noted that 110 servers were down “at the last update.”
In his email, sent to the chief information officers at all the state agencies, Grant said the Florida Digital Service was entering the home stretch and had deployed “an all-hands-on-deck approach.”
“Though we had a little bit of a setback yesterday morning when some of the recovery efforts went offline, we continue making good progress toward restoration because of your partnership and teamwork,” Grant wrote in an email obtained by Florida Politics.
Sources close to the project say the system continues to crash and that, in turn, “undoes the progress that has been made getting the servers back online.”
Prior to the early Monday morning “setback” Florida Politics reported the system crashed at least once Saturday morning as it was being worked on.
Fenske’s update noted that links were expected to return by the end of the day. But there are contrarians.
“I don’t think it’s going to happen,” the source said.
The Florida Digital Service, an agency under DMS, began responding to the hardware failure Friday, DMS spokeswoman Rose Hebert told Florida Politics on Monday.
“The backup processor is designed to take over when the main processor fails, but that routine function did not instantly occur as architected,” Hebert said. “FLDS continues to work alongside significant resources deployed by the machine’s manufacturer as a part of the maintenance contract to restore functionality and mitigate the impact.”
Lawmakers created the Florida Digital Service in 2020 to maintain state data, set up testing environments to demo state software before it’s rolled out, and facilitate data sharing between government agencies.
Grant, a former Republican state representative who was instrumental in creating the Digital Services, heads the agency. Emails obtained by Florida Politics showed Grant explaining the root cause of the problem could wait till after the team finishes its priority of restoring operations and functionality.
However, according to one source familiar with the issues plaguing the state system, that “sh*t happens in computing but what CIO Jamie Grant has done since is a head scratcher.”
Because of delays in supply chains, getting new hardware quickly is extremely difficult. In lieu of using the cloud to host and run the applications a decision was made to try and fix the same Dell hardware that initially failed the state early Friday, which resulted in another crash Saturday morning.
Grant took to Twitter Monday evening after Florida Politics reported that the state websites were down.
“Your source is more than welcome to quit scratching their head long enough to explain to the dedicated technical resources who have been & still are working around the clock that there’s an instantaneous, automagical solution to solve for the hardware failures. We’re all ears …” he tweeted.
Your source is more than welcome to quit scratching their head long enough to explain to the dedicated technical resources who have been & still are working around the clock that there’s an instantaneous, automagical solution to solve for the hardware failures.
We’re all ears… https://t.co/A6HzaNJ2A9
— James Grant (@JamesGrantFL) November 2, 2021
___
Florida Politics reporter Christine Sexton contributed to this story.
