Repository navigation
7.4.6/7 storage servers locking up #102
Replies: 1 comment 1 reply
|
I'm not aware of any known issues related to the storage servers locking up in 7.4.6, so I wouldn't expect updating to 7.4.7 or 8.3 to resolve the issue (though it wouldn't hurt, there are other fixes in those releases). Is it always the same two storage nodes? What provides the underlying storage for their targets? My first suspicion would be an issue with the underlying storage, for example a failing disk and access to certain LBAs hanging, causing the storage service in turn to appear locked up. A restart would appear to shake things loose temporarily (and failing disks tend to behave unpredictably). I would check Note: This discussion forum is primarily community-driven. Maintainers may occasionally comment to clarify behavior or help determine whether something could be a bug, but support here is provided on a best-effort basis. If you require guaranteed assistance, please review our support policy. |
Uh oh!
There was an error while loading. Please reload this page.
Hi folks, we have an issue with a previously flawlessly working Beegfs 7.4.6 setup.
We have 30 storage nodes providing about 6 PB of storage and a single metadata/management server, with an infiniband backbone.
In the last week or so two out of 30 of the storage nodes simultaneously started to exhibit the same problem. The storage service just stops working -- it doesn't die or report any errors but it becomes unresponsive to clients and the metadata server. beegfs-df will hang on clients when it gets to these nodes. Sometimes beegfs-check-servers will report that the servers are still there, if left long enough that also fails.
Eventually the metadata server also locks up trying to talk to these two storage servers and the whole file system becomes unusable.
systemctl restart beegfs-storage solves the problem for a few hours, then one or the other server will go down again and we have to repeat the restart.
I have tried upgrading these two storage nodes to 7.4.7 (as they are basically down anyway) and have also done a full yum update and reboot (we are running Rocky 9), to no avail.
Other than downtime to upgrade fully to 7.4.7 or 8.3, does anyone have any suggestions?
All reactions