Lustre IO500 Test - Client soft lock up analysis when destroy the test environment and test files

Description

When we remove the test file and stop the service in io500, we can observe the software lockup in the client node.

While usually the software lockup will last for about less than 25s, and it will be back finally.

Upstream bug: https://jira.whamcloud.com/browse/LU-12832

Activity

Xinliang Liu

April 8, 2024 at 3:53 AM

Kevin Zhao

October 12, 2022 at 3:18 AM

(edited)

The client concurrency will affect this. Set the parameters with:

osc..max_pages_per_rpc=1048576
osc..max_rpcs_in_flight=256

The test env is : 6 clients 3 servers

Kevin Zhao

October 12, 2022 at 3:17 AM

In the io500.sh, we can setup some parameters for client like this:

Lustre Client Setting

$PDSH -w ${LUSTRE_CLIENT} lctl set_param
osc..max_pages_per_rpc=1048576
osc..max_rpcs_in_flight=256
osc..max_dirty_mb=2000
osc..checksums=0
osc..short_io_bytes=65536
llite..max_read_ahead_mb=2048
llite..max_read_ahead_per_file_mb=16
llite..max_cached_mb=8192
ldlm.namespaces..lru_size=0
ldlm.namespaces..lru_max_age=5000
mdc..max_rpcs_in_flight=128
mdc..max_mod_rpcs_in_flight=8

Then the software lockup will disappear.

Resize work item view side panel

Done

Details

Assignee

Chunsong Feng

Reporter

Kevin Zhao

Parent

STOR-67 Lustre Performance Analysis and Optimization on Arm64

Original estimate

Time tracking

No time logged2w remaining

Sprint

Add sprint

Priority

Undecided

Checklist

Sentry

Created October 7, 2022 at 12:45 PM

Updated April 9, 2024 at 3:02 AM

Resolved October 12, 2022 at 3:17 AM