alt Hacker NewsI put my whole life into a single database
Comments
A simple back of the envelope calculation shows that Felix causes between 70 and 110 tonnes of CO2 emissions per year just from flying.
Paris accord says 1.5t per person per year, from all activities, Felix's flying alonre is ~10-15x current European yearly per person emissions and ~50-75x those compatible with +1.5C.
In my experience, tracking objective things like "nutrition" and "sleep hours" is immensely useful to reflect on what went wrong, and tracking subjective things like "mood" or "stress" is useless given hedonic adaptation or heavy swings that make problems obvious, and not need tracking.
What's key is be able to visualize metrics easily on the data and frictionless data entry, I've got a decent setup with iPhone Action + Obsidian + QuickAdd scripts on Obsidian Sync (mobile + laptop). for visualization I use Obsidian Bases and Obsidian notes that run Dataview code blocks and Chart.js, couldn't be happier.
I could track things that are not interesting to reflect on like vitamin D supplementation for accountability but I've never bothered, especially if it's taken ~daily.
Bravo. This is my dream and also my nightmare. I was super into the quantified self movement a while back, before Apple Watch, writing Withings Scale API wrappers in Ruby and Fitbit days.
Every time I try to seriously track metrics of my life, the excitement of the insight gets worn away by the friction of recording and managing. I expect LLMs can help reduce the cost of this by an order of magnitude but then, as you mention, the question is, what do you do / change / learn because of the data?
I recently started tracking nutrition macros with an iOS app MacroFactor which I really like. This is the first time taking my weight doesn't feel like a IDK SHRUG moment and I can actually map my food intake to my weight.
Finances is probably the other highly actionable data source that is such high friction to manage (downloading CSVs, OFXs, monthly...) that it has always been a false start for me. I finally wrote a service to talk to Plaid directly and I successfully used it to categorize my business expenses at tax time. I finally have programmatic access to my bank account data!
You conclusion is definitely a cautionary take: > the main conclusion is that it is not worth building your own solution, and investing this much time.
But, perhaps a subset of that data you find useful.
That last takeaway "it's not worth it" hits hard.
I went through a similar process and came to the same conclusion, although with a slightly different twist. The project became the point and I almost stopped noticing the data towards the end.
I get that everyone wants to be cynical about this, but you really can't deny that both the visualization and sheer scale of data is impressive. The way the "my life in weeks" is done is also very cool, I'll be stealing that for myself.
Why don’t you just query Palantir DB by your human ID? It shows your entire life data and much more.
I ended up building something similar but way smaller in scope -- just a cron job that pulls Apple Health exports, bank CSV dumps, and git commit stats into a single SQLite db. The queries that actually proved useful were embarrassingly simple, stuff like correlating sleep hours with commit frequency or spotting that my spending spikes every time I start a new side project. The real 80/20 of personal analytics is just getting your data into one place where you can join across sources. Everything beyond that is diminishing returns.
If anyone is interested in doing this sort of thing themselves, I make an iPhone app called Reflect meant for this exact purpose
https://apps.apple.com/us/app/reflect-track-anything/id64638...
As someone who dislikes being tracked, I find it disconcerting to have this level of data stored in 3rd party services, but joining up multiple services to give one coherent picture is cool. How cool is questionable, trying to correlate health data with location data is going to give a strange picture, and I question how relating health data to weather is useful. (Do I have lower blood pressure on rainy days?)
Forgive me for I am being sceptical. The might be some insight here I have not considered, but I'd feel a lot more comfortable if it was all self-hosted / self-collected data.
> Probably obvious for many, but I didn't realize ACs don't transport any air into the room, but just moves it around
I had the same epiphany as you days after acquiring a CO2 monitor. Most people notice poor indoor air quality from proxies such as humidity and temperature. AC (without ventilation) eliminates these and tricks our senses very effectively, giving us cool and fresh feeling indoor spaces full of CO2 and devoid of oxygen.
I can see the value of doing this but I think that this is a case of the 80-20 rule. Just the low effort 80, which could be keeping a journal where you reflect on how you felt a given day, why that might be, and what steps to take to avoid it from happening (or keep it happening) should get you pretty far.
Most of the things that really hinder my productiveness and happiness are the same things that everyone tells you about; sleep, diet, sunlight, not being stuck indoors all day, socializing. And most of these can be improved by making changes to my habits.
Other than that, is that I think that it's important to get into the habit of self reflection. Having a feedback loop on my own life has helped me find out what works and what doesn't. It's too easy to just go through the days slogging through and not making any changes because you don't even realize what's going on.
Tangent, I've always had notes and have this longing to store things but at the same time I like "moving on" too. For ex. I used to have windows phones and saw I still have an accessible One Note from 10 years ago... tempted to read through it but also I've changed as a person.
Another tangent, recently bought a paper shredder, started shredding through boxes of mail I have kept since they have personal info like cc statements but on the same idea of moving on/reducing stuff I'm hoarding whether it's data or physical.
This was far more interesting than I first thought it would be when clicking the link. In particular, the place/time and life events and such being presented this way told a story and was fun.
This looks like it requires a heavy amount of discipline to track everything consistently over time. How do you build that into your daily routine?
The hardest part of this kind of personal data system is retrieval not storage. At some point you have more data than fits in a prompt, so you need to decide what's relevant per query. Did you build any ranking or filtering logic, or do you query specific tables directly?
Flying stats dashboards always amuse me. I get it, for the non-pilot it's kind of like a status thing, "Well I traveled X times in 2025!". As a pilot though, I have gobs of stats I could put up there but from flying for 15 years I realize there's not really anything meaningful in there other than "Gee whiz, I flew a little more/less than last year." I know some other professional pilots do track some of their stats a little closer as they try to optimize for hours flown:hours paid but I've never cared to hyper optimize my schedule in that way.
And here I am, feeling slightly ashamed at taking 1 or 2 flights a year. These "commute" statistics are staggering.
An interesting experiment, I think I'm too uncomfortable leaking data I don't yet know why someone would curate to me free of charge until I knew. If there was a FOSS suite like Home Assistant that would do a few of those things I might try it out, especially the weather (I would add air quality) correlation to mood and other subjective states.
Tried the manual tracking a few times and it does not tend to stick. Passive capture is the approach that ever worked for me. Having different apps and devices with permissions to track you and good APIs that enable integrating the data seems to be the way to go.
Did you think of building some proactive AI tools to make use of all this centralized data?
The value rarely shows up where you expected it to.
I kept a rough log of my sleep and mood for about a year with no specific goal. Mostly forgot about it. Then I had a weirdly bad few months and went back to look — turns out there was a pretty clear pattern I would've never noticed in the moment.
Maybe the framing of "was it worth it" is the wrong question. It's less like an investment with a return and more like keeping receipts. Useless 99% of the time, then suddenly you really need one.
I thought you created a database from scratch, got me excited! (I’m a db guy)
I store my travel in YAML files in git.
https://edwardbetts.com/agenda/trip/past https://edwardbetts.com/agenda/trip/stats
Mmm... yes. It definitely looks like he's in the market for a new mattress.
> I walk more than twice as much when I'm in New York, compared to any other city
This is why I moved to Tokyo. Even if I want to avoid exercising I still take many steps
New Zealand almost made the world map.
"GitHub Open Source Contributions" aside, everything else falls into the consumption/intake/internal categories; thin data on production/output/external ones.
I wonder how much time you spend daily on tracking things / data entry
The step count in NYC stands out like One World Trade Center compared to the rest of the building when looking at the skyline ;-)
How do you sync the steps data from iOS to your own server automatically?
It's all nice and all, but I'm just sitting here thinking "How can one afford all this flying around"?
Taking “Know thyself” to a whole new level. I’d love to have these stats on me, if it could be done by inference, rather than conscious effort.
I'd like to make the same but with Owntracks instead of Swarm and ActivityWatch instead of RescueTime.
Interesting approach. One thing I noticed when people build systems like this is that the real challenge becomes decision-making rather than storage, deciding what actually deserves to become structured data. Curious how you decide what goes into the database vs what stays unstructured.
Wow, this sounds amazing! I am a statistics and chart lover, I love tracking various types of data, but this? I can't even imagine how much time it must have taken to input all the data. Huge respect to you! Keep going.
Related to this, I highly recommend anyone to install github.com/ActivityWatch/activitywatch, it's an amazing tool to keep track of your computer use completely locally. I think there are lots of possibilities with data analysis/AI aimed to improved one self's life
I had an idea similar to this where you could add information about yourself and answer daily questions and get paid by companies who access this data. This could be an ethical way to share information benefitting all parties.
No need to spy on him, he publishes everything.
This is fascinating. One question I always have with large personal datasets like this: at what point did you start getting genuinely surprising insights versus confirming things you already suspected?
380k datapoints sounds incredible but I imagine the real challenge is turning that into decisions that actually change behavior
Bro queried his entire database, added a LIMIT 10 and it only produced 2 rows. Just a little humor in good faith.
Hope they made backups :)
Dude. Backup that database.
[dead]
[dead]
So it didnt end up working too well seeing that the latest data is from 4 years ago.
This might sound harsh but for someone who is keen on investing time to track so many things, you should invest some time in learning how to make better visualizations. A few tweaks here and there would really improve what you have.
The takeaways at the very bottom of the page are valuable:
> Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
/edit: quantified, not qualified