Skip to content

refactor: Parseable as the main interface - #1143

Merged
nitisht merged 48 commits into
parseablehq:mainfrom
de-sh:ingest
Feb 12, 2025
Merged

refactor: Parseable as the main interface#1143
nitisht merged 48 commits into
parseablehq:mainfrom
de-sh:ingest

Conversation

@de-sh

@de-shde-sh commented Jan 29, 2025

Copy link
Copy Markdown
Contributor

Fixes #XXXX.

Description

Refactors code to share a Parseable object that is the interface to perform all operations in parseable, instead of having distinct interfaces such as CONFIG, StreamInfo, etc.

This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

@de-sh
de-sh marked this pull request as draft January 29, 2025 10:08
@de-sh
de-sh marked this pull request as ready for review February 1, 2025 16:57
nitisht
nitisht reviewed Feb 1, 2025
View reviewed changes

@nitishtnitisht left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some comments

Comment threadsrc/parseable.rs Outdated
Comment threadsrc/parseable.rs Outdated
Comment threadsrc/parseable.rs Outdated
Comment threadsrc/parseable.rs Outdated
Comment threadsrc/analytics.rs Outdated
@de-sh
de-sh requested a review from nitishtFebruary 1, 2025 18:35
nitisht
nitisht reviewed Feb 2, 2025
View reviewed changes
Comment threadsrc/parseable.rs Outdated
nitisht
nitisht reviewed Feb 2, 2025
View reviewed changes

@nitishtnitisht left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small comment otherwise LGTM

@de-sh
de-sh marked this pull request as draft February 3, 2025 08:15
nikhilsinhaparseable
nikhilsinhaparseable reviewed Feb 6, 2025
View reviewed changes
Comment threadsrc/parseable/streams.rs Outdated
nikhilsinhaparseable
nikhilsinhaparseable reviewed Feb 6, 2025
View reviewed changes
Comment threadsrc/parseable/streams.rs Outdated
nikhilsinhaparseable
nikhilsinhaparseable reviewed Feb 6, 2025
View reviewed changes
Comment threadsrc/storage/object_storage.rs Outdated
nikhilsinhaparseable
nikhilsinhaparseable reviewed Feb 6, 2025
View reviewed changes
Comment threadsrc/handlers/http/modal/ingest_server.rs Outdated
nikhilsinhaparseable
nikhilsinhaparseable reviewed Feb 6, 2025
View reviewed changes
Comment threadsrc/handlers/http/modal/ingest_server.rs Outdated
let path = meta.file_path();

// we are considering that we can always get from object store
if let Some(mut store_data) = storage_ingestor_metadata {

@nikhilsinhaparseablenikhilsinhaparseableFeb 6, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test initialising multiple ingestors setup / delete ingestor for distributed

nikhilsinhaparseable
nikhilsinhaparseable reviewed Feb 6, 2025
View reviewed changes
Comment threadsrc/lib.rs Outdated
Devdutt Shenoiand others added 14 commits February 7, 2025 01:35
Devdutt Shenoi
refactor: Parseable.store_ingestor_metadata()
e489692
Devdutt Shenoi
doc: flush_all_streams
612f104
Devdutt Shenoi
refactor: merge if-blocks
71cd7cc
Devdutt Shenoi
delete_stream -> streams.delete
9b11e78
Devdutt Shenoi
schema() -> get_schema()
cee7222
Devdutt Shenoi
stream_type -> get_stream_type
e644335
Devdutt Shenoi
rm unused method
1ebc843
Devdutt Shenoi
refactor: get stream handle first and then metadata
49d2308
Devdutt Shenoi
remove unsused method that leaked in during merge
ef53501
Devdutt Shenoi@nikhilsinhaparseable
get rid of stream_exists
4cd49b7
Co-authored-by: Nikhil Sinha <131262146+nikhilsinhaparseable@users.noreply.github.com>
Signed-off-by: Devdutt Shenoi <devdutt@outlook.in>
Devdutt Shenoi@nikhilsinhaparseable
undo merge change that breaks expectation
bca8f4a
Co-authored-by: Nikhil Sinha <131262146+nikhilsinhaparseable@users.noreply.github.com>
Signed-off-by: Devdutt Shenoi <devdutt@outlook.in>
Devdutt Shenoi
refactor: move set_metadata
12a9ad5
Devdutt Shenoi
associate create_internal_stream_if_not_exists with Parseable
06c0009
Devdutt Shenoi
Merge branch 'main' into ingest
614af32
nikhilsinhaparseable
nikhilsinhaparseable previously approved these changes Feb 10, 2025
View reviewed changes

@nikhilsinhaparseablenikhilsinhaparseable left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@nitishtnitisht closed this Feb 12, 2025
@nitishtnitisht reopened this Feb 12, 2025
@nitisht
nitisht merged commit 1cd0da6 into parseablehq:mainFeb 12, 2025
@de-sh
de-sh deleted the ingest branch February 12, 2025 10:58
nitisht pushed a commit that referenced this pull request Feb 14, 2025
Devdutt Shenoi
fix: bugs introduced in#1143(#1185)
523ecc7
parmesant pushed a commit to parmesant/parseable that referenced this pull request Feb 19, 2025
Devdutt Shenoi@parmesant
fix: bugs introduced inparseablehq#1143(parseablehq#1185)
d7a5627
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nitishtnitishtnitisht left review comments
@nikhilsinhaparseablenikhilsinhaparseablenikhilsinhaparseable left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@de-sh@nitisht@nikhilsinhaparseable