alt Hacker NewsHister: A private search engine for the pages you visit and the files you keep
Comments
Kind of related to this in that I built it to hoard knowledge from web pages I've visited along with implementing a Karpathy-style LLM Wiki, but the knowledge is collected automatically from sources I browse.
I have it up on GitHub, but I don't think anyone should use my implementation.
Loosely, what I built:
* On each of my machines I have a cron job running that looks at all my web browser history (usualy it's inspecting the brower's SQLlite across firefox and chrome). If it matches my rule list: hacker news stories, certain reddits, etc. it'll grab the page, convert to markdown and drop in my Obsidian Vault incoming.
* It has a whole de-duping architecture since I might open the same page on multiple machines. Uses the CloudFlare SQLITE D1 storage for tracking the processed links.
* it'll then trigger the LLM to do some Karpathy wiki style taxonomy assignment to the articles, organize them, create an index etc.
It's then available for my "bot" stuff to do writings for me.... I will probably write more about it at some point. I'm not certain it's totally useful and not just a yak-shave on hoarding knowledge.
Ai-drafted article on this [1]
Example AI-Drafted article based on some discussions the other day on Ollma vs LLama.cpp [2]
[1] https://taude.xyz/posts/how-archivore-turns-browsing-into-a-...
[2] https://taude.xyz/posts/skip-ollama-run-llama-cpp-directly-o...
Google Chrome did this in 2008. Full-text search over all visited pages, stored offline. It was very useful and I miss it.
Nobody seems to remember it, even though it was a headline feature. Was removed in 2013, I think due to technical constraints.
Will definitely try this.
I'd like to use it, but I'm hesitant to use anything that isn't a reviewed and approved package in my Linux distribution. Even if the chance is 1% that a program I download has malware or security problems that even the author doesn't know about (eg., due to libraries used), odds are that my system's going to be compromised if I run 50 such programs. This extends to browser add-ons, bookmarklets, and extensions too.
How do other people handle this dilemma?
Even solution I can think of involves are a great amount of extra work.
I'd love a extension setting to only send tabs that were visible for ~4+ seconds.
I built myself a little extension last year that tracks what information I was looking at, but focused on generating "new info" recaps for the day / week.
I realized that I open / quick view a lot of pages and close them, which is a strong signal that I don't care about that specific page, and it shouldn't be a source of "new insights" that I learnt that day (since I probably don't care about that topic).
I'd love to re-try a simpler version of that project that builds on Hister as a backend actually.
I use Hister as my default search engine. I’m quite happy with it. If I know I need to branch out, I use the double-bang before my query “!! Hacker News Fujitsu” and it kicks straight to the configured search engine (Kagi in my case).
I had the same problem for a very long time but it is largely solved now. I started to simply ask chatgpt "hey I read something about x, y month ago but can't find it now". There is a surprisingly high chance chatbot can just give the exact answer back to me, usually with extra interesting reading materials as a plus.
Roughly in the same space is Zotero <https://www.zotero.org/> which is intended as a way of organizing research materials but is also a great way of quickly archiving web pages you've visited, including quick tagging and full text search.
A recent discussion about this tool
https://bugzilla.mozilla.org/show_bug.cgi?id=342916
Can I add NOTES about pages? This might be a good spot to do that...? Maybe the interface can be in a web page instead of terminal?
Before Google took off there was a vibrant ecosystem of FOSS dev around search, all different little aspects of it. Then after Google people stopped fiddling with search, search became "solved" or maybe "must be coded by the big boys". Shame.
Thank you for this, looooong time coming
This is pretty useful, actually. I’ve been using it for the past 2–3 days, and I didn’t find it through YC Hacker News but through some other channel. It’s great to see that so many other people are already using it and praising it—congrats on that!
One of the most useful use cases for me, especially since I work in GTM, is keeping track of new ways to get replies from prospects, whether through cold outbound or things like SEO/GEO optimization. I read at least 2-3 articles a day on this, and it genuinely helps me figure out which ideas are worth trying because I can now keep track of everything I’ve read.
Been running hister on my work machine for a month or two now. It's one of those things I don't look at until I need to, but when I do need it, it solves the dreaded hunting through the last month of browser history.
I also have it index my Obsidian notes, which is another little bonus for global search.
I did need to build up quite a few exclusion rules early on, but it's been hands off since.
I am happy to see the idea of history search more. I am on my 3rd version of my own. the use of local LLMs has made it easier to support features like weekly summarized and recipe extraction.
I like the search ui. my projects become functional but never polished. https://github.com/sbeckeriv/memoir
Interesting, I actually very recently built a similar project [1]. I was unaware of this...If I were I probably wouldn't have bothered!
[1] https://git.brucewillis.sexy/~tombert/fs_index I promise, safe for work, despite the URL.
Ooh, very nice! I have my own tool I’ve been using for this [https://github.com/bradrn/full-history-search/], and it’s incredibly useful, but it’s also pretty primitive. This one looks a lot nicer.
> Your own search engine — Hister is a private search engine for the pages you visit and the files you keep.
Is there any site/project that works as a fully customizable personal front-end to all other SERPs?
When I search for something, I always want a link to the best Wikipedia result. This should always be in the same place and have a giant icon/picture.
Then there could be easily clickable links to the SERP pages for Google, DDG, etc. for that query.
A big link to route it to your favorite LLM.
Seems like you could have a really useful "homepage" for all searches that sat in front of all the other sites. It could be local only and would not require indexing the web. Also wouldn't be a files search thing, as Hister appears to be.
I attain this without involving an untrustworthy third party, with one simple trick: Print to PDF.
Every single web page I’ve found interesting, since the advent of the Web, I have printed to PDF and stored locally for my own personal reference.
Something like 80,000+ files - my own copy of my own Internet - indexable, searchable.
Available offline. Something to read when I am far out to sea.
There is no need to involve third parties in your Internet history - no matter how trustworthy they seem to want to appear.
Print to PDF, and you’ve got everything you need, safe and sound.
I can see using this to index the thousands of bookmarks I have and my every growing reading list and see if my local model and find a pattern and/or help surface anything interesting related to the papers I read
Opera browser has this feature baked in a long time ago, before the Chromium era.
Completely random but o remember reading back in the day that Nostradamus had predicted the rise of a Hitler like figure except he had named a Hister, not the actual.
https://www.history.co.uk/articles/did-nostradamus-really-pr...
I've been using this since the last time it came up on here. I don't have it index every page I visit, I use the browser plugin to tell it to index specific ones. It is useful for sure, but I think it will really shine once I've been using it long enough for it to build up a bigger index of things that are old enough that I've actually forgotten about them.
Is the plugin compatible with Brave?
The local file indexing sounds useful. Often forget where I saved that important PDF and Spotlight fails me.
I've been using Hister for over 2 weeks. So far, so good. I really like the interface, the search, everything. Thank you!
This is amazing and something I'd like to integrate with my personal AI assistant. Is there a way to make it also index pages visited from an Android phone?
Great product concept. I explored this some time ago and created a browser extension product. Check it out.
I've been trying this out the past few weeks. I was literally just in there searching for a link 5 minutes ago.
It's badly needed, and so far it's working well for me.
I was just thinking about a tool like this this morning.
Will definitely be giving this a try. The marketing website looks good. Did you have an LLM do the design?
Love this kind of system, so many useful mashup apps to build on top!
Wtf with everyone suggesting new names? Is this an LLM thing?
This is really cool. I like the idea of combining it with a offline Wikipedia cache.
Bonus point for having an IRC channel for community forum.
first line of the open-source readme: ~download the binary for windows~ -- paved with good intentions?
Saw on Discord that they have to change their name, since https://histre.com sent them a letter.
Had a double take as initially read it as Hitler...
Hi, was wondering if you would be alright with a github set private fork done by someone else if they got bored and wanted to add some private features? Thanks. Now back to my steak frites! :9
at first glance i thought it was called hitler
It's really great you've done all that work for importing from other apps - I use Karakeep and at first didnt know how much redundancy I have between the two, but now I am going to try this since I have so much of it already setup.
I thought that said Hitler at first.
Integrate it with linkwarden so it searches the bookmarked pages.
Imagine the security vulnerabilities
[flagged]
[dead]
Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach.
Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline result previews, so information remains searchable even when the original page changes or disappears. It supports full text and semantic search, can run entirely on your own machine, and includes a web interface, command line tools, and an MCP endpoint for assistant integrations.
Website: https://hister.org/
Tiny read-only demo: https://demo.hister.org/
Ps.: It looks like our name conflicts with a registered trademark in the US. The owner of the other project has asked us to change it, so we’ll probably need to comply sooner or later.
Name suggestions are welcome! Ideally, the new name should be relatively short, sound good, and have an available .org domain.
Thanks!