I do this as well. Mine is to track actuals against my schedule. Helps knowing what I should be doing. It's just a habit now. It's all on paper though, no digital aspect.

This seems like the Holy Grail. Is it?

I use anki with org-noter and the anki-editor minor mode for digesting PDFs, a sort of hacked version of incremental reading. I just stuff entire definitions and proofs into it and use cloze deletions for almost everything, then I attach a bunch of screenshots from the source material in the "Extra" field for any context when/if I fail the card. That, combined with the load balancer plugin and a limit of 5 new cards per day keeps my daily reviews around 1 hour. I have 18,000 cards. I'm familiar with the 20 rules for formulating knowledge but I just don't care to break things down into atoms. So I cloze and forget.

There are DIY book scanners (http://diybookscanner.org) and products such as the Fujitsu ScanSnap SV600. The SV600 has decent features like page-detection and finger-removal (I recommend using a pencil's eraser tip). I have personally used it to scan dozens of books, with satisfactory results.

Just saw a father who had to do it fully manually for her blind daughter. I shall show your comment to him.

I track all my time for the week in 15 minute intervals on a piece of paper, spending ~15 minutes planning my week on Sunday evening or Monday morning. Funny enough I started by doing what you suggest: only tracking the time that is most important. I would suggest the same for anyone thinking of trying this out.