Infrastructure News Type: Outage Partial

Affected Infrastructure: jetstream2.indiana.access-ci.org, jetstream2-gpu.indiana.access-ci.org, jetstream2-lm.indiana.access-ci.org, jetstream2-storage.indiana.access-ci.org

Start Date: August 8, 2023, 1:30 p.m.

End Date: August 8, 2023, 7:56 p.m.

8/8/2023 3:56pm ET: UPDATE 2: The system has been restored to service. All nodes are operational again. All instances that were active prior to the outage should be active again. You may wish to reboot instances if you see any odd behavior prior to opening a ticket. If there are lingering issues, please open a ticket via support.access-ci.org/open-a-ticket 8/8/2023 1:52pm ET: UPDATE 1 - CPU and Large Memory nodes have been restored to service. We are still working to bring up GPU nodes. Exosphere is operational again. Non-GPU instances that were active prior to the outage should be active again. You may wish to reboot instances if you see any odd behavior prior to opening a ticket. If there are lingering issues, please open a ticket via support.access-ci.org/open-a-ticket 8/8/2023 9:30am ET: Jetstream2 is currently experiencing an unexpected outage related to power maintenance in the IU Bloomington data center. Engineers are currently investigating; please refrain from submitting support tickets for inaccessible instances at this time. This status item will be updated and further notices sent out when all systems are back online. Thank you for your patience and understanding as we work through these issues. -Jetstream2 Support

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: registry.access-ci.org

Start Date: July 1, 2023, 11:00 a.m.

End Date: July 1, 2023, 12:00 p.m.

The ACCESS User Identity and OAuth Client Registry (https://registry.access-ci.org) will be down for upgrades early Saturday, July 1, 2023.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: June 30, 2023, 12:00 a.m.

End Date: June 30, 2023, 1:03 a.m.

The Delta resource has has been returned to service. All compute nodes and file systems remained up during the outage. Please check you running jobs. --Ealier Post-- The NCSA Delta resource experienced a 2nd power interruption causing an outage. Login nodes and file systems are currently accessible but the resource scheduler is unavailable. Please expect 4 hours for the system to recover and be returned to service. This posting will be updated when additional information is available.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: jetstream2.indiana.access-ci.org, jetstream2-gpu.indiana.access-ci.org, jetstream2-lm.indiana.access-ci.org, jetstream2-storage.indiana.access-ci.org

Start Date: June 29, 2023, 7:55 p.m.

End Date: June 30, 2023, 3:14 a.m.

Due to an ongoing adverse weather event at the Indiana University Bloomington data center, Jetstream2 is experiencing an unplanned outage. All services appear to be affected. Engineers are currently investigating the extent of the outage and next steps, though a timeline is still unknown. We appreciate your patience as we work through these technical difficulties. At this time, we ask that you please avoid submitting support tickets regarding this outage. UPDATE - 5:20p ET - Engineers are currently working to bring all affected nodes back online. We are presently waiting for all network services to be restored. Once we are cleared by operations and networks, we will begin powering up Jetstream2 infrastructure. There is no ETA at this time for return to service. We will update as soon as we have additional information. UPDATE 2 - 9:57p ET - Existing instances should be online within the next hour and available for use. New instances are not getting full connectivity. Engineers are still looking into that issue. We appreciate your patience. UPDATE 3 - 11:12p ET - The system has been restored to service. VMs that were active prior to the outage should be active again. You may wish to reboot instances if you see any odd behavior prior to opening a ticket. If there are lingering issues, please open a ticket via https://support.access-ci.org/open-a-ticket Thanks, Jetstream2 Support

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: June 29, 2023, 6:30 p.m.

End Date: June 29, 2023, 8:45 p.m.

The NCSA Delta resource experienced a power outage during a severe thunderstorm event. Please expect 4 - 8 hours for the system to recover and be returned to service. This posting will be updated when additional information is available.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: June 28, 2023, 12:00 p.m.

End Date: June 29, 2023, 3:00 p.m.

As of 10:14 AM EDT, engineers have completed maintenance and have returned the Anvil system back to normal service. All queues have been enabled and jobs have resumed scheduling. Please see detailed list of changes (https://www.rcac.purdue.edu/news/5921) in a separate article. Please report any issues through ACCESS Help Desk at https://support.access-ci.org/open-a-ticket ————————————————————— We have extended the maintenance until 10 AM EDT today. ______________________________________________________________ The Anvil system will be unavailable Wednesday, June 28th, 2023 from 8:00am - Thursday, June 29th at 8:00am EDT for scheduled maintenance. Any Slurm jobs which request a walltime which would take them past Wednesday, June 28th, 2023 at 8:00am EDT will not start and will remain in the queue until after the maintenance is completed. **How does this maintenance impact you?** * This is a standard maintenance * OS and Slurm will be updated to newest. stable release * Storage system configuration updates by the vendor * No user impact is expected once Anvil is returned to service** Anvil will return to full production by Thursday, June 29th, 2023 at 8:00am EDT. Please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/open-a-ticket (00am EDT will not start and will remain in the queue until after the maintenance is completed. **How does this maintenance impact you?** * This is a standard maintenance * OS and Slurm will be updated to newest. stable release * Storage system configuration updates by the vendor * No user impact is expected once Anvil is returned to service** Anvil will return to full production by Thursday, June 29th, 2023 at 8:00am EDT. Please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/open-a-ticket) if you have any questions.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: tickets.access-ci.org

Start Date: May 19, 2023, 11:00 a.m.

End Date: May 19, 2023, 4:25 p.m.

5-19-2022 11:30 AM Central Update Ticketing system functionality is restored. Original post: The Jira Service Management ticketing system is unavailable. We are working to resolve the issue and expect ticketing capability to be restored soon.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: May 16, 2023, 3:30 p.m.

End Date: May 16, 2023, 7:45 p.m.

The issues have been resolved and affected services are now functioning again. _________________ Anvil storage began experiencing networking issues around 11:30 AM this morning. The issues have affected access to ondemand.anvil. thinlinc and ssh keys. Engineers are diagnosing the issues now. We will post an update by 5:00PM EDT

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: May 8, 2023, 3:00 p.m.

End Date: May 10, 2023, 6:59 a.m.

We will have maintenance scheduled on Expanse starting 8AM (PT) May 8 through May 9, 2023. During this maintenance several upgrades will be performed. Details of the upgrades and the process are below: - The operating system will be upgraded from Rocky Linux 8.5 to Rocky Linux 8.7. The OFED version will be updated to 5.8.1.1. In addition, the firmware on all nodes will be updated. - The drivers on the GPU nodes will be updated to version 515.65.01 (CUDA 11.7). - The Lustre client version will be updated to version 2.15.2. - A new software stack build using Spack 0.17.3 will become available and will be the default (available as modules cpu/0.17.3b or gpu/0.17.3b). - The original software stack will continue to be available and will work with the new OS and OFED versions. The old software stack can be accessed by using "module load cpu/0.15.4" or "module load gpu/0.15.4". - We will be making the new software environment available in advance of the maintenance and will send out an update once it is available. - Jobs submitted before the upgrade without specifying the version of cpu (0.15.4) or gpu (0.15.4) must be cancelled and resubmitted after the maintenance to ensure that the right environment is picked up for their jobs (since the defaults will change to 0.17.3b). - We are reserving nodes for multiple days given the large number of updates involved. Jobs that will not fit in the time window before the maintenance will be pending and held before the upgrade. - After our service nodes are upgraded and the Lustre update is completed, we will look to put fully upgraded nodes back into the queue as possible to help mitigate the long downtime. Please submit a ticket via ACCESS Support (https://support.access-ci.org (https://urldefense.com/v3/__https://support.access-ci.org/__;!!Mih3wA!HahR9U4iAF_EKq7SJwqOxyu08ng7j6AyntEFwNvdhS50YdBqK4fndYVUklk9YLomcjxQK6kJjjYviBVeeeVl$)) or SDSC's local ticketing system (consult@sdsc.edu (mailto:consult@sdsc.edu)) if you have any questions. Thanks, SDSC User Services Team

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: bridges2-em.psc.access-ci.org, bridges2-gpu.psc.access-ci.org, bridges2-rm.psc.access-ci.org, bridges2-ocean.psc.access-ci.org

Start Date: April 26, 2023, 1:00 p.m.

End Date: April 26, 2023, 10:00 p.m.

Bridges-2, including all VMs and filesystems, will be unavailable due to scheduled maintenance Wednesday, April 26 8:00AM-5:00PM Eastern Time. The slurm queue will be preserved and queued jobs will begin running once the machine has returned to service. Please direct any questions to help@psc.edu (mailto:help@psc.edu) and our team will be happy to assist you. Thank you, Bridges-2 Team

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: April 26, 2023, 6:00 a.m.

End Date: April 27, 2023, 11:30 p.m.

Update 04/27/2023: We have restored the overlay functionality on all Expanse nodes. Original message: In response to a security advisory, we have disabled the overlay kernel module on all Expanse nodes. This will prevent the use of some Singularity containers and possibly CVMFS mounts on Expanse. We are working on a mitigation approach and will update once this functionality is restored. Please send a ticket via ACCESS support (https://support.access-ci.org (https://urldefense.com/v3/__https://support.access-ci.org__;!!Mih3wA!DwrHmYkjt_poeQCAi67dZKlq73yxtpBB5_DWxOIFYI2TQKhrssmOLNDvj573Rwg2DVkkiWov67YTxMaz1S2M$)) or SDSC's support email (consult@sdsc.edu) if you have any questions. Thanks, SDSC User Services Staff

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: delta-cpu.ncsa.access-ci.org, delta-gpu.ncsa.access-ci.org

Start Date: April 19, 2023, 1:00 p.m.

End Date: April 20, 2023, 1:00 a.m.

The Delta resource will undergo maintenance from 8:00AM to 8:00PM CDT on Wednesday April 19th, 2023. Additional notices will be sent if changes to the plan occur. During the maintenance period the following changes will be made: - Operating system (OS) on the Delta login and compute nodes will be updated from RHEL 8.4 to 8.6 with no expected impact to the existing software stack. - Software for the Delta high-speed network (HSN) switches and fabric manager will be upgraded. - Lustre file system client software will be updated to address known issues. - NVIDIA GPU driver will be upgraded to CUDA 12, while the default CUDA software module will remain at CUDA 11.6.1 until the next planned maintenance. - The hostname command will provide the public-facing hostname and IP address rather than the internal hostname and IP address. - Workflows that use the hostname --ip command need to check that references to 172.x are changed to the matching 141.x addresses. All Delta resources will be unavailable during the maintenance period including: - Delta login nodes - unavailable - Delta compute nodes - unavailable - Delta services - Open OnDemand - unavailable - Delta Globus Online endpoint - unavailable All running jobs will have been drained in advance of the maintenance. Queued jobs will persist and be eligible to run once maintenance is complete. A follow-up message will be sent once maintenance is complete. Please send questions to help@ncsa.illinois.edu (mailto:help@ncsa.illinois.edu) and be sure to mention Delta in the subject.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: tickets.access-ci.org

Start Date: April 3, 2023, 10:00 a.m.

End Date: April 3, 2023, 2:45 p.m.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: March 30, 2023, 10:45 p.m.

End Date: March 31, 2023, 2:02 a.m.

UPDATE: March 31, 2023 10:02pm EDT As of 10:02pm EDT, the Anvil system has been returned to normal service. Job queues have been enabled and job scheduling has been resumed. We apologize for the disruption of service. Please report any issues through ACCESS Help Desk at https://support.access-ci.org/open-a-ticket ORIGINAL: March 31, 2023 6:45pm EDT The Anvil cluster began experiencing issues with its scratch filesystem around 6:45pm EDT. Access to scratch directories may be slow or hang. Engineers are currently diagnosing the issue and are working to identify a fix. Job scheduling has been paused while this issue is being addressed. We will provide an update by 10pm Eastern time, or sooner as we have more information. This outage is being tracked at https://www.rcac.purdue.edu/news/5704

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: ranch.tacc.access-ci.org

Start Date: March 26, 2023, 1:00 p.m.

End Date: March 26, 2023, 7:00 p.m.

TACC network infrastructure will not be available from 8:00 AM to 2:00 PM (CDT) on Sunday, 26 March 2023. Network maintenance will be performed during this time. Please submit any questions you may have via the TACC User Portal. https://portal.tacc.utexas.edu/tacc-consulting

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: tickets.access-ci.org

Start Date: March 24, 2023, 2:00 p.m.

End Date: May 1, 2023, 4:59 a.m.

On March 24, 2023, ACCESS introduced a new ticket system based on Jira Service Management (JSM), replacing the previous Resource Tracker (RT) system. ACCESS users can submit tickets following the links on the ACCESS Support site at: - https://support.access-ci.org As additional documention is available it will be listed here: - General Ticket System FAQ (https://operations.access-ci.org/faqs) -- ACCESS Support and Operations

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: March 21, 2023, 12:00 p.m.

End Date: March 22, 2023, 12:00 a.m.

As of 8:27pm EDT, engineers have completed maintenance and have returned the Anvil system back to normal service. All queues have been enabled and jobs have resumed scheduling. Please see detailed list of changes (https://www.rcac.purdue.edu/news/5689) in a separate article. Please report any issues through ACCESS Help Desk at https://support.access-ci.org/open-a-ticket ————————————————————— We have extended the maintenance until 8PM ______________________________________________________________ The Anvil system will be unavailable Tuesday, March 21, 2023 from 8:00am - 6:00pm EDT for scheduled maintenance. Any Slurm jobs which request a walltime which would take them past Tuesday, March 21st, 2023 at 8:00am EDT will not start and will remain in the queue until after the maintenance is completed. How does this maintenance impact you? - The default queue will be changed from wholenode to shared. - NVIDIA GPU driver will be updated and CUDA 12.0.0 will be added as a module. Anvil will return to full production by Tuesday, March 21st, 2023 at 6:00pm EDT. Please submit a ticket through ACCESS Help Desk at https://support.access-ci.org/open-a-ticket if you have any questions.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: February 24, 2023, 3:00 p.m.

End Date: February 26, 2023, 4:50 a.m.

Anvil scratch has been returned to a fully operational state , and scheduling has been resumed. ** Update 5:40 PM 02/25/23" Our engineers continue to collaborate with vendors to return Anvil scratch to service. Work will continue throughout the evening and possibly overnight. We will send an update by 10 AM EST 02/26/23 _______________________________________________________________________________________________ ** Update 9:30AM 02/25/23" Logins to Anvil via Open OnDemand gateway (https://gateway.anvil.rcac.purdue.edu/) are affected by the outage. Other login methods (direct SSH (https://www.rcac.purdue.edu/knowledge/anvil/access/login) and Thinlinc Remote Desktop (https://www.rcac.purdue.edu/knowledge/anvil/access/login/thinlinc)) continue to function as expected provided you have previously enabled (https://www.rcac.purdue.edu/knowledge/anvil/access/login/sshkeys) SSH keys-based authentication for your account. Storage engineers continue to work with the vendor to return scratch and project filesystems to full functionality. Scheduling remains paused while we address the problem. We will provide an update by 8:00PM EST 02/25/23 **Update 8PM 02/24/23** Storage engineers continue to work with the vendor to return scratch to full functionality. Scheduling remains paused. We will provide an update by 10AM EST 02/25/23 Around 10AM EST on Friday, February 24th, Anvil scratch began experiencing performance issues.. Storage engineers are working with the storage vendor for a resolution. Scheduling has been paused until scratch is returned to service

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: lyric.uky.access-ci.org

Start Date: February 24, 2023, 9:00 a.m.

End Date: February 25, 2023, 12:00 a.m.

The KyRIC system will be offline on Friday, 2/24/2023, 3AM-6PM ET due to maintenance of the storage system. All nodes and data operations will not be available in that period. We aim to bring the system back on the same day. KyRIC Staff

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: February 21, 2023, 4:00 p.m.

End Date: February 22, 2023, 1:00 a.m.

We will have a maintenance period on Expanse 8AM-5PM (PT), Feb 21, 2023. During this maintenance, we will work on the NFS home filesystems. We have a reservation in place to prevent jobs from running during this period. The "squeue" output will show "ReqNodeNotAvail, Reserved for maintenance" for jobs that do not fit in the time period before the maintenance begins. These jobs will run after we release the maintenance reservation. Thanks SDSC User Support Staff

Posted: March 20, 2026