Past Incidents

Wednesday 19th August 2020

Recruit: Recruit Restart - langerj

Operational

Start Date: 2020-08-19 09:50:00
Updates & Comments

The Recruit environment was restarted to resolve an email delivery issue within the application.

Impact: apply.marshall.edu, recruitment.marshall.edu were unavailable for approximately two minutes from 9:50am-9:52am.

Affected Applications: Ellucian Recruit

Monday 17th August 2020

Phone Service: Teams Call Queue Dropping Calls - brumfield77

Operational

Start Date: 2020-08-17 19:59:00
Updates & Comments

Microsoft Teams service alert

Title: Some call queue agent users may experience intermittently dropped calls when answering a call queue call ID: TM220755

Impact: Major

Affected Applications: Teams Call Queues

Sunday 16th August 2020

MU Remote Desktop Connection Terminal Server: MURemote Offline for Storage Migration - moore267

Operational

Start Date: 2020-08-16 04:00:00
Updates & Comments

The MURemote RDP cluster is offline for a migration of its storage. During this migration the service will be unavailable

Impact: LOW

Affected Applications: muremote

Saturday 15th August 2020

V-Drive (Deprecated): MUPFC Offline for Planned Maintenance - moore267

Operational

Start Date: 2020-08-15 23:00:00
Updates & Comments

MUPFC is offline to migrate the server to a new storage platform. During this move V drive services, and personal webpages served via mupfc.marshall.edu, users.marshall.edu, and webpages.marshall.edu will be unavailable. Reference CR-141

Impact: LOW

Affected Applications: V Drive, mupfc,webpages.marshall.edu

Friday 14th August 2020

Internet Access: Partial Internet Outage - johnson146

Operational

Start Date: 2020-08-14 07:30:00
Updates & Comments

We are experiencing a partial internet outage. At this time we do not know the cause or the scope.

Impact: Access to some external pages. External access to internal resources.

Affected Applications: Internet access.

Sunday 9th August 2020

Exchange Mail: Mail Flow Issues - brooksjas

Operational

Start Date: 2020-08-09 16:00:00
Updates & Comments

Around 4PM, the Systems group was added to a ticket where a user wasn't getting emails from Microsoft. Upon checking the Quarantine in Office 365, the messages were located, but the public IP addresses of the servers were not. This was not limited to one node on Exchange, but affected all of them. Additionally, the ListServ was also noted to have the same issue.

The issue is the NAT through the Firewall. ITIC and ITIC are both working to identify other possibly affected systems and fix the issues.

Impact: Currently, all mail services are affected by this. Email is being delayed, not lost.

Affected Applications: -----

Network (Wired Access): Internet2 Web Traffic - johnson146

Operational

Start Date: 2020-08-09 07:00:00
Updates & Comments

A routing issue to Internet2 created a partial internet outage that affected all users enterprise wide.

Impact: Web traffic to Internet2 websites.

Affected Applications: Box.net, Dropbox.com, genome.jp and many .edu websites

Wednesday 29th July 2020

Exchange Mail: Exchange authentication issues - burgess67

Operational

Start Date: 2020-07-29 13:30:00
Updates & Comments

Due to a temporary network outage, 3 of our Exchange servers terminated a few key services due to not being able to communicate with the DAG witness server. Once the issue was discovered, the services were restarted and authentication events began to pass as expected.

Impact: continuous authentication requests, inability to communicate with on-prem mailboxes and calendars.

Affected Applications: all mail applications

Internet Access: Complete Network Outage - maynar12

Operational

Start Date: 2020-07-29 11:00:00
Updates & Comments

Late this morning, the IT Infrastructure Communications Group (ITIC) was alerted to a calamitous network outage that effected the complete data and voice traffic. Quickly responding to the issue, ITIC started troubleshooting various network devices to find the primary core router causing the outage. A reboot of said core router corrected the data and voice traffic back to routing properly. Special core members of ITIC are actively working to comb through logs in determining, ultimately, where the failure lies.

Impact: ALL DATA AND VOICE TRAFFIC

Affected Applications: Primary Core Router

Phone Service: Telephone outages - mccoyb

Operational

Start Date: 2020-07-29 11:00:00
Updates & Comments

There were several calling scenarios that were failing due to the telephone systems not fully converging after the network outage. Each issue was identified and addressed by 4:00 pm on July 29.

Impact: Major

Affected Applications: Avaya, Skype for Business, and Teams