Infrastructure News Type: Degraded

Affected Infrastructure: bridges2-ocean.psc.access-ci.org

Start Date: July 19, 2024, 11:15 p.m.

End Date: July 22, 2024, 3:00 p.m.

We are aware of some intermittent issues with the Bridges-2 Ocean Filesystem. You might notice that your jobs pause and take slightly longer to run than usual, but most should continue to complete correctly. Our staff is working to resolve the issue.

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: kerberos.access-ci.org

Start Date: July 18, 2024, 9:12 p.m.

End Date: July 20, 2024, 10:00 a.m.

Due to a cooling issue at NCSA, it is possible that the replica KDC hosted on-site could become unresponsive. Kerberos services should still operate in the event the replica goes down but their may be a delay as things fail over to another replica server.

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: registry.access-ci.org

Start Date: July 18, 2024, 11:00 a.m.

End Date: July 18, 2024, 11:30 a.m.

The COmanage Registry software which powers https://registry.access-ci.org/ will be upgraded from v4.1.x to v4.3.x . This process will take about 15 minutes and will invalidate existing web sessions. In particular, new user enrollments may be affected. Users should contact support@access-ci.atlassian.net (mailto:support@access-ci.atlassian.net) to report any issues that occur during user registration.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: identity.access-ci.org

Start Date: July 15, 2024, 3:07 p.m.

End Date: July 15, 2024, 5:28 p.m.

Web login to ACCESS portals and other services is failing for some users due to a problem in the CILogon service. Technical details: The AWS EFS mount points used by the LDAP servers became unavailable. We are in the process of removing/replacing EFS mount points and restarting the LDAP servers. The process will take several minutes.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: July 15, 2024, 1:30 p.m.

End Date: July 15, 2024, 11:00 p.m.

Bridges-2 will be unavailable on Monday, July 15 in order to upgrade the Jet filesystem. The vendor believes that this patch will fix an issue that has been causing Jet filesystem hangs and crashes over the past few weeks. Thank you for your patience. If you have any questions or problems, please submit a ticket to help@psc.edu.

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: tickets.access-ci.org

Start Date: July 3, 2024, 11:42 p.m.

End Date: July 4, 2024, 1:43 a.m.

Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced issue creation and project for Jira Service Managements. The issue has been resolved and the service is operating normally. https://jira-service-management.status.atlassian.com/incidents/ctpbgx33bd7r

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: July 2, 2024, 3:08 a.m.

End Date: July 2, 2024, 10:08 p.m.

Update: Bridges-2 has returned to normal service! The Bridges-2 Filesystem continues to experience some issues. The vendor has been notified and the PSC team continues to work to restore complete service.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: July 1, 2024, 1:00 p.m.

End Date: July 1, 2024, 9:00 p.m.

Bridges-2 has returned early from scheduled maintenance. Over the past few weeks, the Bridges-2 Ocean filesystem has encountered multiple disruptions to service. The vendor has a fix in place, but we will need to take an emergency maintenance period to implement it. As such, Bridges-2, including all VMs and filesystems, will be unavailable from 8:00AM Eastern time on Monday, July 1 through Noon Eastern time on Tuesday, July 2. We apologize for the short notice, but this is a fix that must be implemented as quickly as possible to prevent ongoing disruptions. Please contact help@psc.edu (mailto:help@psc.edu) if you have any questions or problems.

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: tickets.access-ci.org

Start Date: June 29, 2024, 6:19 p.m.

End Date: July 1, 2024, 5:00 a.m.

Due to recent licensing changes implemented by Atlassian, we have had to re-configure how automated ticket actions are performed. As a result, the status changes based on the comments added to tickets won’t be happening anymore. Additionally, some minor automatic corrections will no longer happen instantaneously but will happen eventually. These changes are being implemented over the next few days, so please bear with us if you encounter any discrepancies. We expect these changes to be completed by July 1, 2024.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: June 29, 2024, 4:16 p.m.

End Date: July 1, 2024, 1:00 p.m.

It appears that the Bridges-2 Ocean filesystem is trying to get a head start on it's scheduled maintenance for Monday. We are experiencing some odd failures. While some of the system remains available, you may see some degraded service this weekend. The maintenance beginning at 8AM Eastern Time on Monday should address these issues.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: tickets.access-ci.org

Start Date: June 21, 2024, 4:00 a.m.

End Date: June 21, 2024, 6:01 a.m.

Hi Everyone, We have reached the limit of automation rule executions on our ticketing system. As a result, you may miss notifications, and auto-generated fields will not be populated. However, ticket creation and management (queues, etc.) through the portal are functioning as usual. Please bear with us as we work to resolve this issue as soon as possible. Thank you for your understanding. Regards, Dinuka (On behalf of ACCESS Operations)

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: June 17, 2024, 10:00 p.m.

End Date: June 18, 2024, 1:00 a.m.

Due to a power disruption, Bridges-2 scheduling has been stopped so that our teams can restore all necessary services.

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: bridges2-ocean.psc.access-ci.org

Start Date: June 11, 2024, 3:45 a.m.

End Date: June 11, 2024, 8:00 p.m.

The Bridges-2 team is working to restore an issue affecting file transfers via the Globus GridFTP services.

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: support.access-ci.org

Start Date: June 7, 2024, 5:00 p.m.

End Date: June 7, 2024, 5:00 p.m.

This is a test.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: June 7, 2024, 6:01 a.m.

End Date: June 10, 2024, 2:30 p.m.

Starting around midnight Eastern time on June 7, Bridges-2 became unavailable due to a power and cooling issue. The team is working on the issue and will return the machine to service as soon as it is safe to do so.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: aces.tamu.access-ci.org

Start Date: May 29, 2024, 2:00 p.m.

End Date: May 31, 2024, 1:00 a.m.

ACES will be partially available during May 29-30 as we repair additional hardware issues for the PCIe composability fabrics.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: xdmod.access-ci.org

Start Date: May 23, 2024, 1:00 p.m.

End Date: May 24, 2024, 1:00 p.m.

Update 05/24/24: ACCESS XDMoD is fully functional now. All services are available at this time. There will be a full outage for ACCESS XDMoD today from approximately 09:00 EDT on 5/23 until 09:00 EDT on 5/24. The web service and API access through the Data Analytics Framework may be unavailable at points during this time. An update will be posted when the service is fully restored.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: May 21, 2024, 1:00 p.m.

End Date: May 24, 2024, 1:00 p.m.

Bridges-2 login nodes, production nodes, all filesystems, VMs and anything else associated with Bridges-2 will be unavailable from Tuesday, May 21 at 8AM through Friday, May 24 at 8AM Eastern time for extensive file system maintenance. If you have any questions or concerns, please send an email to help@psc.edu (mailto:help@psc.edu) and one of our team members will be happy to assist you.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: May 20, 2024, 3:00 p.m.

End Date: May 21, 2024, 3:00 a.m.

Dear Expanse User, We will have a maintenance period on Expanse 8AM-8PM (PT), May 20, 2024. During this maintenance, we will be performing kernel updates and also work on the Ceph filesystem. We have a reservation in place to prevent jobs from running during this period. The "squeue" output will show "ReqNodeNotAvail, Reserved for maintenance" for jobs that do not fit in the time period before the maintenance begins. These jobs will run after we release the maintenance reservation. Thanks SDSC User Support Staff

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: May 16, 2024, 11:30 a.m.

End Date: May 17, 2024, 8:00 p.m.

2024-05-17 09:00 - Progress was made overnight with the Delta high-speed network. Work remains in integrating storage resources into the Delta high-speed network. It is anticipated the system will be returned to service sometime in the afternoon. 2024-05-16 18:00 - An unexpected issue was encountered during the reconfiguration of Delta's high speed network. NCSA staff are working with the vendors to resolve this issue. This will delay Delta's Return to Service. An update will be provided tomorrow (Friday) morning. Other maintenance items have been completed. Delta Resource Users: A maintenance outage for the Delta system is planned for Thursday, May 16th from 6:30AM to 6:30PM. This will be a full outage with login nodes, home and scratch file systems, Open OnDemand and Globus endpoints being unavailable during the maintenance outage. Please note that data on the home and scratch file systems will not be accessible during the maintenance outage. During this maintenance the following actions will take place: 1. The home and scratch file system server software will be updated. 2. Software on the high speed network management nodes and switch firmware will be updated. 3. Reconfiguration of the high speed network for eventual deployment of DeltaAI and associated file systems. 4. OS update with minor changes: apptainer 1.3.0 -> 1.3.1, and some other packages. 5. Slurm update with cleanup of Slurm job database. No rebuilding of software is anticipated. There will be no access to data on the home and scratch file systems on Delta during this maintenance outage. Access to data on the /projects file system will be available via the NCSA Taiga Collection Endpoint (https://app.globus.org/file-manager?origin_id=127afd1b-4c9b-4e24-ac8d-0139c6d6b4ef&origin_path=%2Ftaiga%2Fnsf%2Fdelta%2F) via the path: /taiga/nsf/delta/ and then your 4 letter project code. Please send comments or questions to help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu) and be sure to mention Delta somewhere in the Subject line. -The Delta Project

Posted: March 20, 2026