Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(cdp): remove legacy kafka based consumer #27760

Merged
merged 8 commits into from
Jan 23, 2025
Merged

feat(cdp): remove legacy kafka based consumer #27760

merged 8 commits into from
Jan 23, 2025

Conversation

meikelmosby
Copy link
Contributor

@meikelmosby meikelmosby commented Jan 22, 2025 •
edited
Loading

Problem

  1. Cyclotron has been in use for ages now. Would be good to remove the kafka queue option
  2. we need a docker compose setup to run cyclotron (db, fetch, janitor) for hobby deploys

Changes

  1. Removed Kafka-based approach completely and only use Cyclotron
  2. Updated tests to verify Cyclotron job creation instead of Kafka messages
  3. added docker compose setup

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

  • adjusted the tests
  • old tests still pass
  • e2e test running docker compose locally with twilio destination

@meikelmosby meikelmosby requested review from benjackwhite and a team January 22, 2025 08:39
benjackwhite
benjackwhite requested changes Jan 22, 2025
View reviewed changes
Copy link
Contributor

@benjackwhite benjackwhite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks legit. The only thing I think we need before merging is adding cyclotron to the docker compose stack so that hobby deploys and such will still work, similar to how we have capture running there

@meikelmosby meikelmosby mentioned this pull request Jan 22, 2025
Merged
@meikelmosby meikelmosby force-pushed the feat/cdp/remove-legacy-kafka-based-consumer branch from fe02d6a to 2a57e12 Compare January 22, 2025 14:42
benjackwhite and others added 3 commits January 23, 2025 13:15
@benjackwhite
Merge branch 'master' into feat/cdp/remove-legacy-kafka-based-consumer
29da3bf 
# Conflicts:
#	plugin-server/src/capabilities.ts
#	plugin-server/tests/cdp/consumers/cdp-events-consumer.test.ts
@github-actions
Update query snapshots
2a21a1e
@github-actions
Update query snapshots
1391b09
@meikelmosby meikelmosby merged commit d4dbac3 into master Jan 23, 2025
93 checks passed
@meikelmosby meikelmosby deleted the feat/cdp/remove-legacy-kafka-based-consumer branch January 23, 2025 13:09
timgl pushed a commit that referenced this pull request Jan 28, 2025
@meikelmosby @benjackwhite
@github-actions @timgl
feat(cdp): remove legacy kafka based consumer (#27760)
10e8aec 
Co-authored-by: Ben White <ben@posthog.com>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@benjackwhite benjackwhite

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@meikelmosby @benjackwhite