[bounty] nextjs loom replacement pipe

[louis030195]

/bounty 100

definition of done:

• at the end of the day i can go to screenpipe and ask for a loom of a specific time range
• it should be a nextjs pipe
• minimum version is taking a time range and stitching together mp4 files (video and audio) and giving me back the video with audio

bonus:

• highlights, good UX like loom (somehow)

check this nextjs pipe https://github.com/mediar-ai/screenpipe/tree/main/examples/typescript/pipe-simple-nextjs

also be aware nextjs does have some issue with deno sometimes - for example openai streaming failed for me once idk why (we're thinking in switching to bun soon)

related #582

more context:

screenpipe/screenpipe-server/src/server.rs

Line 967 in cf2b374

.route("/experimental/frames/merge", post(merge_frames_handler))

we have endpoint to merge video but might need some changes, e.g. this task might involve a little bit of rust

my use case:

PLEASE say the UX you think of and how you're going to implement this before starting the task plz

[linear]

MED-238 [bounty] nextjs loom replacement pipe

[algora-pbc]

💎 $100 bounty • Screenpi.pe

Steps to solve:

1. Start working: Comment /attempt #583 with your implementation plan
2. Submit work: Create a pull request including /claim #583 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to mediar-ai/screenpipe!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @tribhuwan-kumar	Oct 25, 2024, 9:43:29 PM	#619
🟢 @volfcan	Oct 28, 2024, 9:50:57 AM	WIP

[tribhuwan-kumar]

/attempt #583

Algora profile	Completed bounties	Tech	Active attempts	Options
@tribhuwan-kumar	4 mediar-ai bounties	Rust, JavaScript, Lua & more		Cancel attempt

[volfcan]

/attempt #583

Options

• Cancel my attempt

[tribhuwan-kumar]

@louis030195
i'm thinking to implement something like this, there will a time & date option, user can select a time interval for loom, then it'll merge those specific time ranged videos and audio in one video and send back to the user. there will also an option for quick share, so user can directly share that video on whatsapp, x, etc. i might take some inspiration from loom

(extra feature in this pipe could be a quick summary of that time interval and summary of that video, also it'd be great if user can interact with this pipe)

[volfcan]

pretty much the same way with the @tribhuwan-kumar.

there will be a time & date component and when users select the time range for a loom video it'll create the video and send back to the user.

[tribhuwan-kumar]

pretty much the same way with the @tribhuwan-kumar.

there will be a time & date component and when users select the time range for a loom video it'll create the video and send back to the user.

the searching and merging are done, i'm currently working on ux, if you have any suggestion for ux we can collaborate!

[volfcan]

pretty much the same way with the @tribhuwan-kumar.
there will be a time & date component and when users select the time range for a loom video it'll create the video and send back to the user.

the searching and merging are done, i'm currently working on ux, if you have any suggestion for ux we can collaborate!

check your twitter dms

[tribhuwan-kumar]

check your twitter dms

there's nothing

[louis030195]

sounds good

@louis030195 i'm thinking to implement something like this, there will a time & date option, user can select a time interval for loom, then it'll merge those specific time ranged videos and audio in one video and send back to the user. there will also an option for quick share, so user can directly share that video on whatsapp, x, etc. i might take some inspiration from loom

(extra feature in this pipe could be a quick summary of that time interval and summary of that video, also it'd be great if user can interact with this pipe)

[tribhuwan-kumar]

@louis030195
is this ui is okay ?
it will open a dialog of merged video and audio

[tribhuwan-kumar]

there was a 500 internal problem with merge frame route, i fixed that. but i noticed some strange behavior while merging videos, imagine this situation, i kill the screenpipe while processing some videos. after killing the screenpipe the video remains uncompleted. and these uncompleted videos are causing errors while merging large amount of videos. maybe i need to add some checks before merging those videos.

i got a corrupt video

PS C:\Users\eirae\Downloads\app\screenpipe> C:\Users\eirae\Downloads\app\screenpipe\screenpipe-app-tauri\src-tauri\ffmpeg\bin\x64\ffmpeg.exe -v error
 -i C:\\Users\\eirae\\.screenpipe\\data\\monitor_65537_2024-10-31_01-32-59.mp4 -f null -
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000022CB0AF1780] moov atom not found
[in#0 @ 0000022CB0ADA540] Error opening input: Invalid data found when processing input
Error opening input file C:\\Users\\eirae\\.screenpipe\\data\\monitor_65537_2024-10-31_01-32-59.mp4.
Error opening input files: Invalid data found when processing input

[louis030195]

@tribhuwan-kumar

nice

video should have audio too at least

i'm actually shipping an endpoint that allows you to stream frames through SSE at high speed for #343

https://github.com/mediar-ai/screenpipe/tree/timeline-ui

i think you might wait i merge this and i'm likely going to do some hacks to stream the audio too so it might solve your problem

[tribhuwan-kumar]

@tribhuwan-kumar

nice

video should have audio too at least

right now, i'm able to merge video to video and audio to audio, (merged video and merged audio are separate)
i tried to merge video to audio but the problem i'm getting is order correction, some audio have only beep sound, some have sound but how can i merge those audio with their corresponding video frame.

i'm actually shipping an endpoint that allows you to stream frames through SSE at high speed for #343

https://github.com/mediar-ai/screenpipe/tree/timeline-ui

i think you might wait i merge this and i'm likely going to do some hacks to stream the audio too so it might solve your problem

for now, i fixed those corrupt video error, by filtering out those video. it prevents corrupt videos getting in input_list.txt, this can be slow bcz it adds an extra check before merging them

[tribhuwan-kumar]

@louis030195
a working beta version:

a.mp4

[algora-pbc]

💡 @tribhuwan-kumar submitted a pull request that claims the bounty. You can visit your bounty board to reward.

[louis030195]

@louis030195 a working beta version:

a.mp4

thats cool

is there audio in it?

[tribhuwan-kumar]

@louis030195 a working beta version:

a.mp4

thats cool

is there audio in it?

no, that merged video doesn't have audio, it's just a series of video concatenated by ffmpeg, it doesn't have embedded audio. I tried to embed video with audio but I'm facing issues of order synchronizing :(

[louis030195]

@louis030195 a working beta version:

a.mp4

thats cool
is there audio in it?

no, that merged video doesn't have audio, it's just a series of video concatenated by ffmpeg, it doesn't have embedded audio. I tried to embed video with audio but I'm facing issues of order synchronizing :(

are you using database metadata?

you can fetch all the things and order by time

i guess it's possible to have 1-3 seconds of audio desync because we capture timestamp not right sync

also if you try to create a loom over too long range things that i expect to happen:

• say i closed my lid (screen capture stop but mic keeps going i think) -> will have weird desync with audio

although this should be only for short times

i guess a version 10.0 of this would be to feed this to a voice LLM and fix all the things automatically and comment on the video haha

[tribhuwan-kumar]

are you using database metadata?

you can fetch all the things and order by time

i guess it's possible to have 1-3 seconds of audio desync because we capture timestamp not right sync

also if you try to create a loom over too long range things that i expect to happen:

• say i closed my lid (screen capture stop but mic keeps going i think) -> will have weird desync with audio

although this should be only for short times

i guess a version 10.0 of this would be to feed this to a voice LLM and fix all the things automatically and comment on the video haha

then i've to create an another route for embedding audio in video bcz this can only concatenate the videos and audios. & thanks for the metadata it would be helpful for embedding :)