fix: use /proc/partitions instead of $data_dir/partitions

[YangchenYe323]

It looks wrong xtop is looking for partitions file in $data_dir/partitions. I got this error when I try to group by devname which brings in the partitions source:

2025-11-25 17:56:52,253 - DEBUG - ================================================================================
2025-11-25 17:56:52,389 - ERROR - Query execution failed: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
Traceback (most recent call last):
File "/codemill/yeya/git/vendor/0xtools/xtop/./xtop", line 1182, in execute_current_query
result = self.query_engine.execute_with_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 244, in execute_with_params
return self.execute(query, params, debug, debug_profile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 178, in execute
result = conn.execute(query)
^^^^^^^^^^^^^^^^^^^
_duckdb.IOException: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
2025-11-25 17:56:52,393 - INFO - samples: using csv for 1 hour(s)

[yarel79]

It looks like the proposed fix breaks one of xtop’s main features: the ability to work with multiple data directories.

On a single machine you can have multiple work directories from different cases/hosts, and in that situation the partitions file may differ per data directory.

If needed, copy /proc/partitions into <data-dir> and use xtop with -d switch. e.g. xtop -d <data-dir>.

[tanelpoder]

Yes, I should have documented that for now you'd need to copy /proc/partitions into data-dir or run ln -s /proc/partitions ./partitions in data-dir. I'll add this to documentation for now.

The reason for this limitation is that the device major:minor number to device name mapping can change over time, especially over reboots. Different maj:min numbers can be discovered in a different order across reboot (and devices can disappear or re-appear when they're added on the fly in a running OS).

So, the current approach is a limited workaround for now, but for correctness, I need to improve xcapture so that it would read /proc/partitions periodically itself and if it sees a change, will dump a new timestamped version of partition mapping to an xcapture_partitions.csv file. That way this file would act as a slowly changing dimension in the data model. I might be able to work on this over the weekend (or if not, the holidays).

In longer term, xcapture would run in many hosts and all the files can be consolidated into some central storage (or DuckLake!), so we do need to start copying/shipping the latest partition mapping info to the performance warehouse. If anyone wants to implement this change, please open an issue and first propose how you would do this. This could even be a separate shell/python script to keep xcapture code simpler (or perhaps better to add it in xcapture userspace itself to reduce the number of services running)

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

[tanelpoder]

Sorry for missing this in the documentation, can you add the ln -s /proc/partitions DATA_DIR/partitions setup command into the readme for now?

[YangchenYe323]

Thank you for the clarification! Yeah it makes a lot of sense, closing this PR now.

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

Wouldn't there be a need to join a historical record with "the matching partition at that point in time"? We could discuss in a separate issue though

[YangchenYe323]

It looks wrong xtop is looking for partitions file in $data_dir/partitions. I got this error when I try to group by devname which brings in the partitions source:

2025-11-25 17:56:52,253 - DEBUG - ================================================================================
2025-11-25 17:56:52,389 - ERROR - Query execution failed: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
Traceback (most recent call last):
File "/codemill/yeya/git/vendor/0xtools/xtop/./xtop", line 1182, in execute_current_query
result = self.query_engine.execute_with_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 244, in execute_with_params
return self.execute(query, params, debug, debug_profile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 178, in execute
result = conn.execute(query)
^^^^^^^^^^^^^^^^^^^
_duckdb.IOException: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
2025-11-25 17:56:52,393 - INFO - samples: using csv for 1 hour(s)

[yarel79]

It looks like the proposed fix breaks one of xtop’s main features: the ability to work with multiple data directories.

On a single machine you can have multiple work directories from different cases/hosts, and in that situation the partitions file may differ per data directory.

If needed, copy /proc/partitions into <data-dir> and use xtop with -d switch. e.g. xtop -d <data-dir>.

[tanelpoder]

Yes, I should have documented that for now you'd need to copy /proc/partitions into data-dir or run ln -s /proc/partitions ./partitions in data-dir. I'll add this to documentation for now.

The reason for this limitation is that the device major:minor number to device name mapping can change over time, especially over reboots. Different maj:min numbers can be discovered in a different order across reboot (and devices can disappear or re-appear when they're added on the fly in a running OS).

So, the current approach is a limited workaround for now, but for correctness, I need to improve xcapture so that it would read /proc/partitions periodically itself and if it sees a change, will dump a new timestamped version of partition mapping to an xcapture_partitions.csv file. That way this file would act as a slowly changing dimension in the data model. I might be able to work on this over the weekend (or if not, the holidays).

In longer term, xcapture would run in many hosts and all the files can be consolidated into some central storage (or DuckLake!), so we do need to start copying/shipping the latest partition mapping info to the performance warehouse. If anyone wants to implement this change, please open an issue and first propose how you would do this. This could even be a separate shell/python script to keep xcapture code simpler (or perhaps better to add it in xcapture userspace itself to reduce the number of services running)

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

[tanelpoder]

Sorry for missing this in the documentation, can you add the ln -s /proc/partitions DATA_DIR/partitions setup command into the readme for now?

[YangchenYe323]

Thank you for the clarification! Yeah it makes a lot of sense, closing this PR now.

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

Wouldn't there be a need to join a historical record with "the matching partition at that point in time"? We could discuss in a separate issue though

[YangchenYe323]

It looks wrong xtop is looking for partitions file in $data_dir/partitions. I got this error when I try to group by devname which brings in the partitions source:

2025-11-25 17:56:52,253 - DEBUG - ================================================================================
2025-11-25 17:56:52,389 - ERROR - Query execution failed: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
Traceback (most recent call last):
File "/codemill/yeya/git/vendor/0xtools/xtop/./xtop", line 1182, in execute_current_query
result = self.query_engine.execute_with_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 244, in execute_with_params
return self.execute(query, params, debug, debug_profile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 178, in execute
result = conn.execute(query)
^^^^^^^^^^^^^^^^^^^
_duckdb.IOException: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
2025-11-25 17:56:52,393 - INFO - samples: using csv for 1 hour(s)

[yarel79]

It looks like the proposed fix breaks one of xtop’s main features: the ability to work with multiple data directories.

On a single machine you can have multiple work directories from different cases/hosts, and in that situation the partitions file may differ per data directory.

If needed, copy /proc/partitions into <data-dir> and use xtop with -d switch. e.g. xtop -d <data-dir>.

[tanelpoder]

Yes, I should have documented that for now you'd need to copy /proc/partitions into data-dir or run ln -s /proc/partitions ./partitions in data-dir. I'll add this to documentation for now.

The reason for this limitation is that the device major:minor number to device name mapping can change over time, especially over reboots. Different maj:min numbers can be discovered in a different order across reboot (and devices can disappear or re-appear when they're added on the fly in a running OS).

So, the current approach is a limited workaround for now, but for correctness, I need to improve xcapture so that it would read /proc/partitions periodically itself and if it sees a change, will dump a new timestamped version of partition mapping to an xcapture_partitions.csv file. That way this file would act as a slowly changing dimension in the data model. I might be able to work on this over the weekend (or if not, the holidays).

In longer term, xcapture would run in many hosts and all the files can be consolidated into some central storage (or DuckLake!), so we do need to start copying/shipping the latest partition mapping info to the performance warehouse. If anyone wants to implement this change, please open an issue and first propose how you would do this. This could even be a separate shell/python script to keep xcapture code simpler (or perhaps better to add it in xcapture userspace itself to reduce the number of services running)

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

[tanelpoder]

Sorry for missing this in the documentation, can you add the ln -s /proc/partitions DATA_DIR/partitions setup command into the readme for now?

[YangchenYe323]

Thank you for the clarification! Yeah it makes a lot of sense, closing this PR now.

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

Wouldn't there be a need to join a historical record with "the matching partition at that point in time"? We could discuss in a separate issue though

[YangchenYe323]

It looks wrong xtop is looking for partitions file in $data_dir/partitions. I got this error when I try to group by devname which brings in the partitions source:

2025-11-25 17:56:52,253 - DEBUG - ================================================================================
2025-11-25 17:56:52,389 - ERROR - Query execution failed: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
Traceback (most recent call last):
File "/codemill/yeya/git/vendor/0xtools/xtop/./xtop", line 1182, in execute_current_query
result = self.query_engine.execute_with_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 244, in execute_with_params
return self.execute(query, params, debug, debug_profile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 178, in execute
result = conn.execute(query)
^^^^^^^^^^^^^^^^^^^
_duckdb.IOException: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
2025-11-25 17:56:52,393 - INFO - samples: using csv for 1 hour(s)

[yarel79]

It looks like the proposed fix breaks one of xtop’s main features: the ability to work with multiple data directories.

On a single machine you can have multiple work directories from different cases/hosts, and in that situation the partitions file may differ per data directory.

If needed, copy /proc/partitions into <data-dir> and use xtop with -d switch. e.g. xtop -d <data-dir>.

[tanelpoder]

Yes, I should have documented that for now you'd need to copy /proc/partitions into data-dir or run ln -s /proc/partitions ./partitions in data-dir. I'll add this to documentation for now.

The reason for this limitation is that the device major:minor number to device name mapping can change over time, especially over reboots. Different maj:min numbers can be discovered in a different order across reboot (and devices can disappear or re-appear when they're added on the fly in a running OS).

So, the current approach is a limited workaround for now, but for correctness, I need to improve xcapture so that it would read /proc/partitions periodically itself and if it sees a change, will dump a new timestamped version of partition mapping to an xcapture_partitions.csv file. That way this file would act as a slowly changing dimension in the data model. I might be able to work on this over the weekend (or if not, the holidays).

In longer term, xcapture would run in many hosts and all the files can be consolidated into some central storage (or DuckLake!), so we do need to start copying/shipping the latest partition mapping info to the performance warehouse. If anyone wants to implement this change, please open an issue and first propose how you would do this. This could even be a separate shell/python script to keep xcapture code simpler (or perhaps better to add it in xcapture userspace itself to reduce the number of services running)

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

[tanelpoder]

Sorry for missing this in the documentation, can you add the ln -s /proc/partitions DATA_DIR/partitions setup command into the readme for now?

[YangchenYe323]

Thank you for the clarification! Yeah it makes a lot of sense, closing this PR now.

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

Wouldn't there be a need to join a historical record with "the matching partition at that point in time"? We could discuss in a separate issue though

[YangchenYe323]

It looks wrong xtop is looking for partitions file in $data_dir/partitions. I got this error when I try to group by devname which brings in the partitions source:

2025-11-25 17:56:52,253 - DEBUG - ================================================================================
2025-11-25 17:56:52,389 - ERROR - Query execution failed: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
Traceback (most recent call last):
File "/codemill/yeya/git/vendor/0xtools/xtop/./xtop", line 1182, in execute_current_query
result = self.query_engine.execute_with_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 244, in execute_with_params
return self.execute(query, params, debug, debug_profile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 178, in execute
result = conn.execute(query)
^^^^^^^^^^^^^^^^^^^
_duckdb.IOException: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
2025-11-25 17:56:52,393 - INFO - samples: using csv for 1 hour(s)

[yarel79]

It looks like the proposed fix breaks one of xtop’s main features: the ability to work with multiple data directories.

On a single machine you can have multiple work directories from different cases/hosts, and in that situation the partitions file may differ per data directory.

If needed, copy /proc/partitions into <data-dir> and use xtop with -d switch. e.g. xtop -d <data-dir>.

[tanelpoder]

Yes, I should have documented that for now you'd need to copy /proc/partitions into data-dir or run ln -s /proc/partitions ./partitions in data-dir. I'll add this to documentation for now.

The reason for this limitation is that the device major:minor number to device name mapping can change over time, especially over reboots. Different maj:min numbers can be discovered in a different order across reboot (and devices can disappear or re-appear when they're added on the fly in a running OS).

So, the current approach is a limited workaround for now, but for correctness, I need to improve xcapture so that it would read /proc/partitions periodically itself and if it sees a change, will dump a new timestamped version of partition mapping to an xcapture_partitions.csv file. That way this file would act as a slowly changing dimension in the data model. I might be able to work on this over the weekend (or if not, the holidays).

In longer term, xcapture would run in many hosts and all the files can be consolidated into some central storage (or DuckLake!), so we do need to start copying/shipping the latest partition mapping info to the performance warehouse. If anyone wants to implement this change, please open an issue and first propose how you would do this. This could even be a separate shell/python script to keep xcapture code simpler (or perhaps better to add it in xcapture userspace itself to reduce the number of services running)

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

[tanelpoder]

Sorry for missing this in the documentation, can you add the ln -s /proc/partitions DATA_DIR/partitions setup command into the readme for now?

[YangchenYe323]

Thank you for the clarification! Yeah it makes a lot of sense, closing this PR now.

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

Wouldn't there be a need to join a historical record with "the matching partition at that point in time"? We could discuss in a separate issue though

[YangchenYe323]

It looks wrong xtop is looking for partitions file in $data_dir/partitions. I got this error when I try to group by devname which brings in the partitions source:

2025-11-25 17:56:52,253 - DEBUG - ================================================================================
2025-11-25 17:56:52,389 - ERROR - Query execution failed: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
Traceback (most recent call last):
File "/codemill/yeya/git/vendor/0xtools/xtop/./xtop", line 1182, in execute_current_query
result = self.query_engine.execute_with_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 244, in execute_with_params
return self.execute(query, params, debug, debug_profile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 178, in execute
result = conn.execute(query)
^^^^^^^^^^^^^^^^^^^
_duckdb.IOException: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
2025-11-25 17:56:52,393 - INFO - samples: using csv for 1 hour(s)

[yarel79]

It looks like the proposed fix breaks one of xtop’s main features: the ability to work with multiple data directories.

On a single machine you can have multiple work directories from different cases/hosts, and in that situation the partitions file may differ per data directory.

If needed, copy /proc/partitions into <data-dir> and use xtop with -d switch. e.g. xtop -d <data-dir>.

[tanelpoder]

Yes, I should have documented that for now you'd need to copy /proc/partitions into data-dir or run ln -s /proc/partitions ./partitions in data-dir. I'll add this to documentation for now.

The reason for this limitation is that the device major:minor number to device name mapping can change over time, especially over reboots. Different maj:min numbers can be discovered in a different order across reboot (and devices can disappear or re-appear when they're added on the fly in a running OS).

So, the current approach is a limited workaround for now, but for correctness, I need to improve xcapture so that it would read /proc/partitions periodically itself and if it sees a change, will dump a new timestamped version of partition mapping to an xcapture_partitions.csv file. That way this file would act as a slowly changing dimension in the data model. I might be able to work on this over the weekend (or if not, the holidays).

In longer term, xcapture would run in many hosts and all the files can be consolidated into some central storage (or DuckLake!), so we do need to start copying/shipping the latest partition mapping info to the performance warehouse. If anyone wants to implement this change, please open an issue and first propose how you would do this. This could even be a separate shell/python script to keep xcapture code simpler (or perhaps better to add it in xcapture userspace itself to reduce the number of services running)

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

[tanelpoder]

Sorry for missing this in the documentation, can you add the ln -s /proc/partitions DATA_DIR/partitions setup command into the readme for now?

[YangchenYe323]

Thank you for the clarification! Yeah it makes a lot of sense, closing this PR now.

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

Wouldn't there be a need to join a historical record with "the matching partition at that point in time"? We could discuss in a separate issue though

[YangchenYe323]

It looks wrong xtop is looking for partitions file in $data_dir/partitions. I got this error when I try to group by devname which brings in the partitions source:

2025-11-25 17:56:52,253 - DEBUG - ================================================================================
2025-11-25 17:56:52,389 - ERROR - Query execution failed: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
Traceback (most recent call last):
File "/codemill/yeya/git/vendor/0xtools/xtop/./xtop", line 1182, in execute_current_query
result = self.query_engine.execute_with_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 244, in execute_with_params
return self.execute(query, params, debug, debug_profile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 178, in execute
result = conn.execute(query)
^^^^^^^^^^^^^^^^^^^
_duckdb.IOException: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
2025-11-25 17:56:52,393 - INFO - samples: using csv for 1 hour(s)

[yarel79]

It looks like the proposed fix breaks one of xtop’s main features: the ability to work with multiple data directories.

On a single machine you can have multiple work directories from different cases/hosts, and in that situation the partitions file may differ per data directory.

If needed, copy /proc/partitions into <data-dir> and use xtop with -d switch. e.g. xtop -d <data-dir>.

[tanelpoder]

Yes, I should have documented that for now you'd need to copy /proc/partitions into data-dir or run ln -s /proc/partitions ./partitions in data-dir. I'll add this to documentation for now.

The reason for this limitation is that the device major:minor number to device name mapping can change over time, especially over reboots. Different maj:min numbers can be discovered in a different order across reboot (and devices can disappear or re-appear when they're added on the fly in a running OS).

So, the current approach is a limited workaround for now, but for correctness, I need to improve xcapture so that it would read /proc/partitions periodically itself and if it sees a change, will dump a new timestamped version of partition mapping to an xcapture_partitions.csv file. That way this file would act as a slowly changing dimension in the data model. I might be able to work on this over the weekend (or if not, the holidays).

In longer term, xcapture would run in many hosts and all the files can be consolidated into some central storage (or DuckLake!), so we do need to start copying/shipping the latest partition mapping info to the performance warehouse. If anyone wants to implement this change, please open an issue and first propose how you would do this. This could even be a separate shell/python script to keep xcapture code simpler (or perhaps better to add it in xcapture userspace itself to reduce the number of services running)

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

[tanelpoder]

Sorry for missing this in the documentation, can you add the ln -s /proc/partitions DATA_DIR/partitions setup command into the readme for now?

[YangchenYe323]

Thank you for the clarification! Yeah it makes a lot of sense, closing this PR now.

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

Wouldn't there be a need to join a historical record with "the matching partition at that point in time"? We could discuss in a separate issue though

[YangchenYe323]

It looks wrong xtop is looking for partitions file in $data_dir/partitions. I got this error when I try to group by devname which brings in the partitions source:

2025-11-25 17:56:52,253 - DEBUG - ================================================================================
2025-11-25 17:56:52,389 - ERROR - Query execution failed: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
Traceback (most recent call last):
File "/codemill/yeya/git/vendor/0xtools/xtop/./xtop", line 1182, in execute_current_query
result = self.query_engine.execute_with_params(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 244, in execute_with_params
return self.execute(query, params, debug, debug_profile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/codemill/yeya/git/vendor/0xtools/xtop/core/query_engine.py", line 178, in execute
result = conn.execute(query)
^^^^^^^^^^^^^^^^^^^
_duckdb.IOException: IO Error: No files found that match the pattern "/var/lib/xcapture/partitions"
LINE 146: read_csv('/var/lib/xcapture/partitions', skip=1, header...
^
2025-11-25 17:56:52,393 - INFO - samples: using csv for 1 hour(s)

[yarel79]

It looks like the proposed fix breaks one of xtop’s main features: the ability to work with multiple data directories.

On a single machine you can have multiple work directories from different cases/hosts, and in that situation the partitions file may differ per data directory.

If needed, copy /proc/partitions into <data-dir> and use xtop with -d switch. e.g. xtop -d <data-dir>.

[tanelpoder]

Yes, I should have documented that for now you'd need to copy /proc/partitions into data-dir or run ln -s /proc/partitions ./partitions in data-dir. I'll add this to documentation for now.

The reason for this limitation is that the device major:minor number to device name mapping can change over time, especially over reboots. Different maj:min numbers can be discovered in a different order across reboot (and devices can disappear or re-appear when they're added on the fly in a running OS).

So, the current approach is a limited workaround for now, but for correctness, I need to improve xcapture so that it would read /proc/partitions periodically itself and if it sees a change, will dump a new timestamped version of partition mapping to an xcapture_partitions.csv file. That way this file would act as a slowly changing dimension in the data model. I might be able to work on this over the weekend (or if not, the holidays).

In longer term, xcapture would run in many hosts and all the files can be consolidated into some central storage (or DuckLake!), so we do need to start copying/shipping the latest partition mapping info to the performance warehouse. If anyone wants to implement this change, please open an issue and first propose how you would do this. This could even be a separate shell/python script to keep xcapture code simpler (or perhaps better to add it in xcapture userspace itself to reduce the number of services running)

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

[tanelpoder]

Sorry for missing this in the documentation, can you add the ln -s /proc/partitions DATA_DIR/partitions setup command into the readme for now?

[YangchenYe323]

Thank you for the clarification! Yeah it makes a lot of sense, closing this PR now.

Edit: I realized that it might be simpler to just (re)read the /proc/partitions in xcapture userspace code and just output the current matching device name as a column in xcapture itself. That way there's no need to join to the /proc/partitions in post-processing at all (and no need to deal with the slowly changing dimensions).

Wouldn't there be a need to join a historical record with "the matching partition at that point in time"? We could discuss in a separate issue though