Infrastructure News Type: Outage Full

Affected Infrastructure: globus.org

Start Date: November 2, 2024, 2:00 p.m.

End Date: November 2, 2024, 3:30 p.m.

Globus services will be unavailable for up to 1.5 hours beginning at 9:00 AM US Central on Saturday, November 2, 2024 due to planned maintenance that includes database upgrades. More details available here (https://www.globus.org/blog/maintenance-downtime-for-database-upgrades-saturday-november-2-2024).

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: delta-storage.ncsa.access-ci.org

Start Date: October 30, 2024, 11:00 p.m.

End Date: October 31, 2024, 1:00 p.m.

The /work file system emergency maintenance will continue overnight. A small portion of the /work file system remains unavailable. The job scheduler will remain paused until the /work file system is whole again. The NCSA Help Portal (http://help.ncsa.illinois.edu) is the preferred method to contact us. If you run into problems using the portal, you can still email help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu)for support. - The Delta and DeltaAI Project ________________________________________ The /work file system is experiencing issues at this time. We have contacted the vendor to help resolve the issue. This may impact new login sessions if you load a conda base from /work/hdd (formerly known as /scratch). Try to control-c once your login session makes it past the Duo prompt. An update will be sent when more information is available.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: October 30, 2024, 1:00 p.m.

End Date: November 1, 2024, 10:00 p.m.

Dear Bridges-2 User, We believe that all Bridges-2 Ocean issues have been resolved at this point. This is a reminder that Bridges-2 will be unavailable beginning at 8:00AM Eastern time on Wednesday, October 30 in order to install and test new hardware to add capacity to both the Bridges-2 Ocean and Jet filesystems. Maintenance is scheduled to end at Noon on Friday, November 1 but we hope to complete the upgrade and return the system earlier than that if possible. During the outage, Bridges-2, including all VMs and filesystems, will be unavailable. Please contact help@psc.edu with any problems or concerns.

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: duo.access-ci.org

Start Date: October 28, 2024, 1:00 p.m.

End Date: October 28, 2024, 1:10 p.m.

On Monday, October 28, 2024, Duo Mobile Passcodes (https://guide.duo.com/universal-prompt#duo-mobile-passcode) for ACCESS will be reconfigured to use TOTP (https://en.wikipedia.org/wiki/Time-based_one-time_password) instead of HOTP (https://en.wikipedia.org/wiki/HMAC-based_one-time_password). The TOTP configuration will be pushed to users with the Duo security app (https://duo.com/product/multi-factor-authentication-mfa/duo-mobile-app#download) v4.49.0 or later. Users with older versions of the Duo security app will continue to use HOTP. For more information, please see TOTP vs HOTP (https://duo.com/blog/totp-vs-hotp).

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: October 22, 2024, 11:00 a.m.

End Date: October 24, 2024, 1:00 a.m.

Delta Resource Users: This is a reminder of the scheduled maintenance event on Delta and DeltaAI on Tuesday October 22nd from 6AM to 8PM. There are some IMPORTANT updates to bring to your attention. 1. To address recent GPU XID errors, the Delta GPU nodes will move to a newer NVIDIA driver: moving from the 535 driver family to the 550 driver family. Initial testing did not indicate any need for recompilation or linking. Please check your GPU jobs as they run tomorrow for any unexpected behavior. 2. The maintenance will require the /scratch and /work/hdd file systems to be offline and unavailable during the maintenance. 3. The login nodes will need to be rebooted to unmount /scratch and /ime. 4. Please note that the /ime cache front-end to /scratch is currently being flushed and jobs that use the ime constraint will not be allowed to run before the maintenance. 5. The home and projects file system will be available, and jobs not using /scratch or /work/hdd will be allowed to run if submitted to one of the following reservations: On Delta, to run jobs that DO NOT require /scratch, please submit jobs using the no_scratch_requirements reservation by adding: --reservation=no_scratch_requirements to your sbatch, srun and Open OnDemand scripts. On DeltaAI, to run jobs that DO NOT require /work/hdd, please submit jobs using the no_workhdd_requirements reservation by adding: --reservation=no_workhdd_requirements to your sbatch, srun and Open OnDemand scripts. A final sync'ing of data on /scratch to /work/hdd will be done during the maintenance period. Upon maintenance completion, the Delta /scratch file system will be replaced by /work/hdd, which will also be accessible from DeltaAI, and the new NVME portion of /work called /work/nvme (currently available on DeltaAI) will also be accessible from Delta. On Delta a symbolic link will be set to link /scratch/hdd/abcd to /work/hdd/abcd to maintain functionality of existing scripts.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: October 21, 2024, 11:00 a.m.

End Date: October 22, 2024, 1:00 a.m.

UPDATE: October 21, 2024 09:00 PM EDT As of 9:00PM EDT, the maintenance work on Anvil has been completed and job scheduling has been resumed. If you encounter any issues post maintenance, please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/help-ticket. Original Post: The Anvil system will be unavailable from Monday, October 21st at 7:00am to Tuesday, October 22nd at 6:00pm EDT, 2024 for scheduled maintenance. During the maintenance, we will perform cooling work for the system to integrate a new NSF funded AI resource into Anvil. Any Slurm jobs which request a walltime which would take them past Monday, October 21st at 7:00am EDT will not start and will remain in the queue until after the maintenance is completed. Anvil will return to full production by Tuesday, October 22nd at 6:00pm, EDT 2024. Please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/help-ticket if you have any questions.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: faster.tamu.access-ci.org

Start Date: October 15, 2024, 2:00 p.m.

End Date: October 16, 2024, 1:00 a.m.

The FASTER cluster will be unavailable during maintenance from 9am to 8pm CDT on Tuesday October 15. A reservation is in place to prevent jobs from running past the start time of the maintenance period.

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: October 15, 2024, 1:00 p.m.

End Date: October 21, 2024, 4:00 p.m.

The Bridges-2 issues are persisting in spite of dedicated efforts from both the PSC team and the hardware vendor. We are working towards a long term solution but the issue has been more complicated than we had expected. We understand your concerns and appreciate your patience while we work through the issue. This is of the highest priority for our team. We will let you know when we have a resolution and/or we will update you early next week with any developments. PSC User Services Team

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: registry.access-ci.org

Start Date: October 14, 2024, 1:00 p.m.

End Date: October 14, 2024, 1:30 p.m.

On October 14, 2024 at 8:00am CDT, the ACCESS User Registry at https://registry.access-ci.org/ will be upgraded to COmanage Registry v4.3.5 (https://spaces.at.internet2.edu/display/COmanage/Registry+4.3.5+Release+Announcement). This is a maintenance release containing various bug fixes. No downtime is expected for this upgrade. The upgrade has been deployed to the TEST ACCESS User Registry instance at https://registry-test.access-ci.org/ . ACCESS OIDC Client Administrators can test their client by logging on to https://registry-test.access-ci.org/ , creating a new OIDC Client (or using an existing one), and reconfiguring their client with the OIDC client_id/client_secret associated with https://registry-test.access-ci.org/ . For questions or concerns with this update, please contact help@cilogon.org (mailto:help@cilogon.org) or open an ACCESS Help Ticket (https://access-ci.atlassian.net/servicedesk/customer/portal/2/create/30).

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: October 13, 2024, 1:00 p.m.

End Date: October 13, 2024, 9:00 p.m.

Update: As of 5:00PM EST Oct 13, 2024, most compute nodes have been brought back online and Anvil has been back to normal operations. If you had jobs running around 09:00AM when the power interruption occurred, those jobs were terminated, so you may need to check the state of running jobs that were dependent upon work that was interrupted. Original Post: Shortly before 9:00AM EST Oct 13, 2024, Anvil has experienced a power interruption due to a campus power outage. It has caused compute nodes down and scheduling paused. Our engineers have arrived on campus and found additional impacts from today's power interruption. At of 12:18PM EST, scheduling has been resumed and we are still working on bringing remaining nodes back. We are at reduced capacity now but users should see stable systems at the moment. Additional efforts to restore full service are undertaken. Will post updates before the end of today. Please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/help-ticket if you have any questions.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: aces.tamu.access-ci.org

Start Date: October 9, 2024, 2:00 p.m.

End Date: October 10, 2024, 11:00 p.m.

There will be upcoming maintenance on some GPU components for the ACES cluster. The maintenance period is 9am on Wednesday October 9 to 6pm on Thursday October 10. Other parts of ACES will remain available. For specific details, please visit the HPRC website at https://hprc.tamu.edu/.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: October 8, 2024, 1:00 p.m.

End Date: October 9, 2024, 1:00 p.m.

Thank you for your patience, Bridges-2 has returned early from it's scheduled maintenance! Bridges-2 Team ========================================================================================================================================= Dear Bridges-2 users, Bridges-2, including all VMs and filesystems, will be unavailable due to scheduled maintenance starting on Tuesday October 8, 2024 at 8am Eastern Time. We anticipate that the system will return by Thursday October 10 at 8am Eastern Time. During this time, you will be unable to access the system. The slurm queue will be preserved and queued jobs will begin running once the machine has returned to service. This maintenance is for an operating system upgrade to all compute nodes, login nodes and data transfer nodes(DTNs). Please direct any questions to help@psc.edu (mailto:help@psc.edu) and our team will be happy to assist you. Thank you, Bridges-2 Team

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: system-news.access-ci.org

Start Date: October 7, 2024, 11:00 a.m.

End Date: October 31, 2024, 11:00 p.m.

ACCESS introduces the ability for members of the community to signup to receive emails about ACCESS central/online services outages and other news. To signup for these emails please JOIN the ACCESS System Status News affinity group (https://support.access-ci.org/affinity-groups/access-system-status-news).

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: support.access-ci.org

Start Date: October 1, 2024, 6:00 a.m.

End Date: July 13, 2024, 6:00 a.m.

This is a test.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: xdmod.access-ci.org

Start Date: September 30, 2024, 12:00 p.m.

End Date: October 3, 2024, 5:00 p.m.

Update 10/01/2024: The main web service has successfully been updated, however there was an unrelated power outage that is causing a hardware issue for one of the databases. The following services are only partially available at this time: - Efficiency tab - Job Viewer tab for the SUPReMM realm Another update will be posted when the service is fully restored in the coming days. We apologize for any inconveniences. End of Update There is a scheduled downtime for ACCESS XDMoD on Monday, September 30th from approximately 07:00 EDT until 07:00 EDT the following day. The service will be updated from XDMoD 10.5 to 11.0 during this time. The web service and API access through the Data Analytics Framework will be unavailable during the outage. If you wish to use the xdmod-data Python package with ACCESS XDMoD after the upgrade, you will need to upgrade the package to the latest 1.0.1 version using pip install --upgrade xdmod-data. An update will be posted when the service is fully restored. We apologize for any inconvenience this may cause.

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: registry.access-ci.org

Start Date: September 10, 2024, 1:00 p.m.

End Date: September 10, 2024, 1:05 p.m.

On September 10, 2024 at 8:00am CDT, the ACCESS User Registry at https://registry.access-ci.org will be reconfigured to use DynamoDB for user attribute storage instead of LDAP. The switch to DynamoDB affects only OAuth2/OIDC clients using the ACCESS CI "Named Configuration" which causes ACCESSID@access-ci.org to be returned as the sub claim in the id_token. DynamoDB (https://aws.amazon.com/dynamodb/) is an AWS-managed serverless NoSQL service with high performance and a 99.999% SLA. Switching to DynamoDB is a one-line configuration change which can easily be reverted if any issues arise. No downtime is expected. For any questions or concerns, please contact help@cilogon.org (mailto:help@cilogon.org) or open an ACCESS Help Ticket (https://access-ci.atlassian.net/servicedesk/customer/portal/2/create/30).

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: allocations.access-ci.org, identity.access-ci.org, registry.access-ci.org, access-ci.org

Start Date: September 7, 2024, 9:00 a.m.

End Date: September 7, 2024, 3:00 p.m.

DUO Security has announced a maintenance period for Saturday September 7 starting at 05:00am EDT, for an estimated period of 6 hours, which may cause disruption to ACCESS DUO authentication attempts: Begin forwarded message: From: support-noreply@status.duosecurity.com Subject: Duo Maintenance - Multiple Deployments: Scheduled Maintenance - 7 September 2024 Date: August 30, 2024 at 4:45:20 PM EDT Multiple Deployments: Scheduled Maintenance Upcoming scheduled maintenance noticeThe Duo Site Reliability Engineering (SRE) team is scheduling regular maintenance on the following deployments, beginning with DUO70: DUO70 DUO1 DUO55 DUO62 DUO63 DUO73 DUO65 DUO79 DUO77 DUO80 DUO78 The goal of this maintenance is to better balance authentication traffic across all Duo deployments in our continued efforts to provide a high performance and resilient service. We expect minimum user impact since each specific migration window will occur during non-peak times for your organization. Do I need to take action? If your organization has strict IP filtering/firewall rules in place, you should ensure that all Duo IP ranges listed here https://help.duo.com/s/article/1337 and DNS names exist in any filtering rules if firewalls, SSL inspection, or other proxy rules whitelist communication to specific Duo IP ranges. In the event of an outage or failure, Duo’s service could automatically failover to any IP in those ranges. Otherwise, no action is required. How does this migration affect you? During the migration, we expect minimum intermittent authentication failures when users attempt to log in to Duo protected applications. Users may need to retry the authentication or potentially wait a few minutes before attempting another login. After the migration Duo Administrators with the Owner and Administrator role will have their notification settings for the Duo Status Page https://status.duo.com/ adjusted to reflect the new deployment within 24 hours. Duo Administrators with the other roles will need to update their Duo Status Page preferences manually using the steps in https://help.duo.com/s/article/2060. Resources • Documentation: How do I find my StatusPage deployment ID in the Duo Admin Panel and sign up for updates? - https://help.duo.com/s/article/2060 * Article: What are Duo's IP ranges and data residency areas by deployment? - https://help.duo.com/s/article/1337 * Duo Status Page - https://status.duo.com/ For any questions or concerns please email support@duo.com (mailto:support@duo.com). Thank you for being a Cisco Duo customer! Duo Site Reliability Engineering Start time Sep 7, 05:00 EDT Estimated duration 6 hours Components affected DUO1 - Core Authentication Service DUO1 - Admin Panel DUO1 - Push Delivery DUO1 - Phone Call Delivery DUO1 - SMS Message Delivery ...and 72 more components. View full scheduled maintenance details (https://stspg.io/dx573gm8pvdm)

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: identity.access-ci.org

Start Date: August 19, 2024, 2:00 p.m.

End Date: August 19, 2024, 2:30 p.m.

On Monday August 19, 2024 at 9:00am CDT (14:00 UTC), the Shibboleth IdP (https://shibboleth.atlassian.net/wiki/spaces/IDP5) software used by the ACCESS CI Identity Provider (https://idp.access-ci.org/idp/) will be upgraded from v4.3.3 to v5.1.2. No downtime is expected.

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: bridges2-ocean.psc.access-ci.org

Start Date: August 9, 2024, 3:30 p.m.

End Date: August 11, 2024, 2:30 p.m.

We are aware of the ongoing Bridges-2 Filesystem issues and are working with the vendor to address them. We will let you know when all of the issues have been resolved. We thank you for your continued patience.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: July 24, 2024, 12:00 p.m.

End Date: July 25, 2024, 12:00 p.m.

SDSC Expanse will be under maintenance from 5AM (PT) 07/24/2024 to 5AM (PT) 07/25/2024. During this time we will be working on the direct liquid cooling (DLC) infrastructure. A reservation has been put in place to prevent jobs from running during the maintenance period. The "squeue" command output will show "ReqNodeNotAvail, Reserved for maintenance" for jobs that do not fit in the time period before the maintenance begins. These jobs will run after we release the maintenance reservation.

Posted: March 20, 2026