Following are COOL read workload use cases, as suggested by Richard Hawkings and summarized by David Front:
- DCS data is updated once every 5-10 minutes, hence 'DCS IOV duration' is 5-10 minutes.
- The amount of DCS data in each IOV duration is 10-100 MB.
Use case 1: 'Monitoring':
Goal: combined reading rate should be as fast as insertion rate.
Test: A few (10) COOL reading clients, each reading a subset of the folders, read all data of the latest IOV once every IOV duration, for all channelIds.
Use case 2: 'Reconstruction':
Goal: (1000) Reconstruction processes, one starting every 5 seconds, all running on 100 machines, may read the whole DCS data of the latest IOV, within a non-growing time period (of a few minutes).
(Since a process starts every 5 seconds, and IOV duration is 5-10 minutes, 60-120 processes read the same COOL data).
Test: Start a process every 5 seconds. Read X amount of data, where X starts from 1MB.
Measure COOL read time. Grow X by 1 MB, until reading time exceeds a predefined limit (for example: 10 minutes).
Once an optimal X is found, continuously repeat test with this value for a few hours.
- Both tests should continuously run for a few hours, to check for memory leaks, slowing down etc.
- Explore the use of the bulk retrieve over multiple channel IDs when it is available in COOL1-2-3.
This should make this sort of query significantly more efficient.
- For both tests, client and server machine and DB statistics should be collected to performance detect bottlenecks
- Related item: 'overview: Scalability tests for COOL data retrieval performance.' https://savannah.cern.ch/task/?func=detailitem&item_id=2398