timeline ui

[louis030195]

#607

https://screenmemory.app/
https://www.rewind.ai/

#343

definition of done:

• simple scrollable timeline UI that shows up when pressing a shortcut (definable in settings like Feat/tray menu/shortcut recording #552 )
• should show the apps
• (optional) can search

additional details:

• i'd just use the search API and paginate as user scrolls, use the API to merge mp4 files into a single one either on the fly in a temporary dir or always have it ready somehow (to make things faster)
• should be able to show on top of existing app using the shortcut like we do currently, but implement another page for this timeline
• also i think we could do additional nice UX things since we have a local LLM but focus on the main things

[linear]

MED-253 [bounty] timeline ui

[louis030195]

i'm going to design and implement and release it today myself any feedback/idea appreciated

[louis030195]

the thing that excite me about this feature is to do AI features with the app embedded local LLM (llama3.2) to classify stuff etc

[louis030195]

ugly as hell v0 that works

[louis030195]

[louis030195]

@DmacMcgreg @craftyshaun @Glavin001

i did a simple & fast frame streaming API

you can test an very early version:

brew tap mediar-ai/screenpipe https://github.com/mediar-ai/screenpipe.git
brew install screenpipe
screenpipe

then

open screenpipe-server/examples/timeline_ui/index.html # or open manually in your browser

this will open a rewind.ai alike timeline powered by a single html file :) that you can tweak around!

working on the desktop version and also adding audio to it

Screen.Recording.2024-10-31.at.10.38.02.AM.mov

[Glavin001]

Awesome progress! 🎉 I'll test this weekend.

[louis030195]

brain dumping other ideas i have for next things:

• make it properly scrollable over days/months
• audio on top (related to [bounty] nextjs loom replacement pipe #583)
• show apps
• select part of the timeline and summarize it with LLM (whether audio or screen data) or chat with it (or add it to the normal chat)
• maybe/optional LLM logs/tags of your timeline (probably want to allow pipes developer to be able to add things on the timeline, including the AI inbox and IDK maybe creating agents that can interact with your timeline IDK)
• search (later imho)
• teams feature: easily share summaries / blurbs with your team
• automation: select part of the timeline and ask LLM to generate an automation to take over your computer for this specific task (think gumloop/zapier but on your computer)

[volfcan]

@louis030195 we can still attempt this bounty? it looks like you're making it yourself. why did you open a bounty for this then?

[DmacMcgreg]

@louis030195 i've been testing it out with some of the updates.
At least on my machine, the scroll feature is not working. I'm scrolling, and i see the top stats change, but not the frame itself.

[DmacMcgreg]

I'm also getting extremely high CPU usage, and also running into a few issues.

1. Sometimes it gets stuck on "Loading frames"
2. Sometimes I get client side application error.

[DmacMcgreg]

I did manage to test the drag-time function, and it honestly seems to work REALLY well. If we can fix some of those issues, this will be killer.

[louis030195]

@DmacMcgreg thx a lot for the feedback!

Screen.Recording.2024-11-04.at.1.02.41.PM.mov

i've been optimising a lot the performance but there is still room for it, also need to make sure it does not increase storage

basically i turn mp4 + database data into a timeseries with some caching on disk to reduce cpu usage

atm biggest issues:

• some weird scroll because of the way i process videos - can be fixed but might take few hours, work for now
• i did a cursor-like cmd+k to ask questions about the timeline or run custom "agents" the reason for agents is context limit is easily reached
• better UX in general
• still some fundamental issues with transcription [bounty] iphone & microphone transcription does not work on some macos #610 on some macos versions (working on it)
• anything else i missed?

[DmacMcgreg]

Seems to be working for you in your video. For me, the scroll simply is not working. Happy to provide logs if you let me know what you need.

Also whenever I hit reload frames button, it loads a random frame from any point during the day.

[louis030195]

Seems to be working for you in your video. For me, the scroll simply is not working. Happy to provide logs if you let me know what you need.

Also whenever I hit reload frames button, it loads a random frame from any point during the day.

did you scroll vertically?

maybe you can check this:

right click somewhere (not image) and click inspect then console logs

will work on CPU usage and the bug you said

PS: how many FPS do you have with screenmemory? they say one frames every 60 seconds on their website 🤔

[DmacMcgreg]

@louis030195

1. yes I scroll vertically, i can see the app stats along the top changing on every scroll, but the video frame does nothing
2. there are no console logs or network calls
3. once every 10 seconds

Would also be important to have a day selector so I can view previous days.

The app stats on the Timeline are also honestly pretty great, something to think about.

[louis030195]

didnt expect that it was so easy 😂, going to put it in the timeline

update 1

i guess should deal with this to many context switch

not sure how to make the good UX in this case, maybe magicui globe thing

Screen.Recording.2024-11-05.at.10.14.59.AM.mov

[louis030195]

still a a few things to finish up

[DmacMcgreg]

So some additional info that might help: I'm using an external 4k monitor.

Here you can see the call to frames is going super high. Its now sitting at over 51+GB.
I checked my data folder, and it seems it only is taking up 19GB, so this is weird. Possibly some data streaming going on a loop?

So thats perhaps why i'm not able to scroll...

Here you can see the error in the console when first starting up:

[louis030195]

thx - but you don't have the latest version, i have bunch of changes not released on main and also more locally still fixing things

[DmacMcgreg]

@louis030195 just updated and the scrolling is now working!

[louis030195]

added finally some colors & icons to main (had issues with performance), icons only works on macos #643 for now

other ideas with icons:

• hover = highlight
• click = it filters this app in "ask AI" so you can ask questions about this app in particular
• cmd+click multiple = filter multiple
• same for window (somehow)

still some improvement on icons positioning to do but want to add audio now

i'm wondering how audio could work

ideally user can go back in the day and re-listen conversations with captions, ask AI questions relative to it, etc.

not sure how to do the UX yet

[DmacMcgreg]

@louis030195
This is awesome and will really step this tool up to the next level. I'm here to help test if you need!

Should we have a place to mark down all the possible ideas for the timeline ui?

[louis030195]

@louis030195 This is awesome and will really step this tool up to the next level. I'm here to help test if you need!

Should we have a place to mark down all the possible ideas for the timeline ui?

just share anything here for now :)

[louis030195]

i love readwise app and their UX is quite good to read books with AI

i'm thinking in highlighting words as the audio is played (somehow, probably not possible atm, need to return these data from the AI models to the client i guess)

[DmacMcgreg]

I think this is where search is going to be killer. Allowing search across transcripts and screen data. Later, AI contextual search. So for example if youre really searching for the word "aims" or "goals" or "accomplishments" yet the document you worked on was actually called "achievements" then you can still find it.

Can't tell you how much time I've wasted searching for documents I had open months ago but forget the exact name of, but would be easily findable with some sort of contextual search. The AI can tokenize a bunch of data to get the "gist" or "summary" out of it, in the background, and then allow a RAG-type search across multiple days, weeks, months. That for me would be a killer feature I haven't seen anywhere before.

[DmacMcgreg]

Maybe you could sell licenses to schools so that students can prove they are actually doing an assignment themselves and not getting AI help. It would analyze all the websites they visited, the sources, then analyze the document they wrote and give a score to see if they cheated or not.

[craftyshaun]

I think this is where search is going to be killer. Allowing search across transcripts and screen data.

The search -> RAG -> AI is the most used feature of Reclaim I use. I work in Technical Management Consulting and one of my favourite 'Ask Reclaim' prompts is

"Yesterday I did some things in VS Code & Slack between 10:00 and 14:00. Summarise this into 5 sentences focusing on outcomes achieved"

This goes into my timesheets/billing. Building a pipe myself so I can hook it into my tooling and it 'drafts a timesheet' would be a nightmare.

[louis030195]

I think this is where search is going to be killer. Allowing search across transcripts and screen data.

The search -> RAG -> AI is the most used feature of Reclaim I use. I work in Technical Management Consulting and one of my favourite 'Ask Reclaim' prompts is

"Yesterday I did some things in VS Code & Slack between 10:00 and 14:00. Summarise this into 5 sentences focusing on outcomes achieved"

This goes into my timesheets/billing. Building a pipe myself so I can hook it into my tooling and it 'drafts a timesheet' would be a nightmare.

interesting

yeah we're trying to make a good low-code version of pipe with powerful abstractions so AI can write the pipes automatically, think of it like Zapier or Gumloop but based on your screen & mic (and does not require to share your very sensitive auth keys to them)

did a recursive summarization agent for long time range (here with llama3.2 that runs on my computer, embedded AI feature of screenpipe)

Screen.Recording.2024-11-07.at.9.52.45.AM.mov

i was thinking if we could allow people to create these agents/automation that works w the timeline or other parts of the app (both low code and no code) that'd be interesting

[craftyshaun]

did a recursive summarization agent for long time range (here with llama3.2 that runs on my computer, embedded AI feature of screenpipe)

This type of thing would be cool, I was going to play with some of the sources and build some pipes. Espically since now we have the time line to see what/why the AI said that.

i was thinking if we could allow people to create these agents/automation that works w the timeline or other parts of the app (both low code and no code) that'd be interesting.

Long term, that would be amazing and open your user base up!

In the short term, a decent SDK would be sweet. As a dev I'm happy to spend the time writing my tooling to do the above. Also means I can integrate with my own LLM etc.

I'm planning to play this weekend with the new timeline, etc., now that there is a nice, simple homebrew package.

Side note for development: have you seen Devbox. by Jetify?

It is akin to NPM/pip but for dev environments. You'd commit a devbox.json, then anyone who clones goes devbox shell, and all the rust/npm tooling is ready to rumble. Here Is there a rust example?

[louis030195]

@craftyshaun lmk how it went!

next things i want to do with the timeline:

• be able to scroll over 100% of your data
• see more clearly when there is audio on the timeline
• shortcut to show the UI on top of your app (and feels more native like rewind)
• probably other things

[craftyshaun]

@craftyshaun lmk how it went!

Will do, as I'm running the MacOS beta I was having issues compiling the crate for the audio SDK. I'll try again when I'm bored again.

[DmacMcgreg]

app display is currently a little off. Its only picking up Bartender and something else:

Meanwhile ScreenMemory is able to detect it correctly: