feat: Add bigframes.execution_history API to track BigQuery jobs

[shuoweil]

This PR promotes execution_history() to the top-level bigframes namespace and upgrades it to track rich metadata for every BigQuery job executed during your session.

Key User Benefits:

• Easier Access: Call bigframes.execution_history() directly instead of digging into sub-namespaces.

• Rich Metadata Tracking: Captures structured statistics for both Query Jobs and Load Jobs including:

  • job_id and a direct Google Cloud Console URL for easy debugging.
  • Performance metrics: total_bytes_processed, duration_seconds, and slot_millis.
  • Query details (truncated preview of the SQL ran).

• Clean, Focused Logs: Automatically filters out internal library overhead (like schema validations and index uniqueness checks) so your history only shows the data processing steps you actually care about.

  Usage Example:

 1 import bigframes.pandas as bpd
2 import pandas as pd
3 import bigframes
4
5 # ... run some bigframes operations ...
6 df = bpd.read_gbq("SELECT 1")
7
8 # Upload some local data (triggers a Load Job)
9 bpd.read_pandas(pd.DataFrame({'a': [1, 2, 3]}))
10
11 # Get a DataFrame of all BQ jobs run in this session
12 history = bigframes.execution_history()
13
14 # Inspect recent queries, their costs, and durations
15 print(history[['job_id', 'job_type', 'total_bytes_processed', 'duration_seconds', 'query']])

verified at:
vs code notebook: screen/8u2yhaRV9iHbDbF
colab notebook: screen/9L8VrP5y9DXhnZz

Fixes #<481840739> 🦕

[review-notebook-app]

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[TrevorBergeron]

can we add a static factory method to build this from an sdk query job object?

[shuoweil]

Done! Added a from_job classmethod (and from_row_iterator) to handle building the metadata object directly from the jobs.

[TrevorBergeron]

I do worry that at a certain point, storing all query test generated by the session might clog up memory?

[shuoweil]

Good point! To prevent memory bloat during long sessions, I have added truncation so we cap the stored query text strings at a maximum of 1024 characters.

[sycai]

I have the concern of placing job_history under the bigframes.pandas package. We may consider bigframes or session instances as the residing places, mainly because functionalities under bigframes.ml and bigframes.bigquery can also trigger jobs but they do not belong to bigframes.pandas.

[chalmerlowe]

Migration Notice: This library is moving to the google-cloud-python monorepo soon.

We closed this PR due to inactivity to ensure a clean migration. Please re-open this work in the new monorepo once the migration is complete!

[shuoweil]

I have the concern of placing job_history under the bigframes.pandas package. We may consider bigframes or session instances as the residing places, mainly because functionalities under bigframes.ml and bigframes.bigquery can also trigger jobs but they do not belong to bigframes.pandas.

I agree with you. I have fully moved it out of bf.pandas. The API is now renamed to execution_history() to better reflect the broadened abstraction and is directly exposed via the root module (bigframes.execution_history()) and on the Session instance.

[tswast]

Closing. Let's move this over to https://github.com/googleapis/google-cloud-python once googleapis/google-cloud-python#16493 merges.