You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Jakub Valenta a2b1534d02 cli: Catch errors in providers 1 week ago
automatic_diary cli: Catch errors in providers 1 week ago
.ackrc ackrc: Add 10 months ago
.gitignore gitignore,Pipfile: Minor updates 9 months ago
COPYING Add COPYING and NOTICE 8 months ago
Makefile Don't change working dir when running from outside 5 months ago
NOTICE Add COPYING and NOTICE 8 months ago
Pipfile orgmodelist: Add provider 5 months ago
Pipfile.lock orgmodelist: Add provider 5 months ago
README.md orgmodelist: Add provider 5 months ago
automatic-diary Don't change working dir when running from outside 5 months ago
automatic-diary-visualize automatic-diary-visualize: Dont change working dir 4 months ago
automatic_diary_screenshot.png screenshot: Take it from Epiphany 8 months ago
config-sample.json README: Describe all providers 8 months ago
setup.cfg Rename to automatic-diary (from automatic-journal) 8 months ago
setup.py visualize: Add convenient script to run it 8 months ago
tox.ini tox,setup: Fix testrunners 8 months ago

README.md

Automatic Diary

Automatic Diary is a script that creates one timeline from various digital sources describing your life.

Automatic Diary screenshot

On the screenshot above, you can see: calendar events (blue), completed todo list items (yellow), software development work (gray), sport activity (green), and films watched (red).

Why

Automatic Diary can be useful to those who:

(a) have bad memory,\ (b) want to remember what they did,\ © but don’t have the time to write a real diary.

What is collected

  • [X] Calendar events (via CalDAV service and iCalendar file reading)
  • [X] Emails sent and received (via Maildir reading)
  • [X] Sport activity (via spreadsheet parsing)
  • [X] Facebook and Twitter posts (via Facebook/Twitter data archive parsing)
  • [X] Software development work (via Git repository log)
  • [X] Completed todo list items (via todo.txt parsing)
  • [X] Films watched (via ČSFD website parsing)
  • [X] Custom diary notes (via plain-text and Org-mode files parsing)

What is not collected

Automatic Diary could be extended to collect also the following data but right now it is considered low priority:

  • Custom notes and ideas
  • Articles and blog posts written
  • Phone calls and text messages (SMS)
  • Social media private messages and messaging apps
  • Browser history
  • Music listened (via Last.fm)
  • OpenStreetMap contributions
  • Money transfers
  • News headlines
  • Weather

Installation

Mac

$ brew install python
$ pip install pipenv
$ make setup

Arch Linux

# pacman -S pipenv libsecret
$ make setup

Other systems

Install these dependencies manually:

  • Python 3.7
  • pipenv
  • libsecret (Linux keyring) -- required by the caldav provider

Then run:

$ make setup

Configuration

Before you run Automatic Diary, you need to configure all the providers (sources from which the data for your timeline will be read). The following providers are supported:

All providers are configured using a single config.json file. Use config-sample.json as a template for your own configuration.

caldav

  • Input: CalDAV server

  • Output: Names and locations of calendar events

  • Configuration:

    {
        "url": "<server url>",
        "username": "<server authentication username>",
        "password_key": "<server authentication password -- libsecret key>",
        "password_val": "<server authentication password -- libsecret value>",
        "cache_dir": "<cache directory path>"
    }
    

csfd

  • Input: User profile on ČSFD (film database website, something like IMDB)

  • Output: Titles of films rated

  • Configuration:

    {
        "profile_url": "<csfd.cz profile url>",
        "cache_dir": "<cache directory path>"
    }
    

csv

  • Input: CSV spreadsheet (.csv) file

  • Output: Rows formatted using a template

  • Configuration

    {
        "path": "<csv file path>",
        "date_source": "{{<column name>}}",
        "date_format": "<strptime date format>",
        "text_source": "<template string in the Mustache format>"
    }
    

facebook

  • Input: Downloaded Facebook archive

  • Output: Texts of statuses

  • Configuration

    {
        "path": "<path to wall.htm or timeline.htm>",
        "username": "<facebook username>"
    }
    

git

  • Input: Directory with checked-out Git repositories and an author name

  • Output: Commit messages by the author from all repositories

  • Configuration

    "config": {
        "base_path": "<path to directory - will be searched recursively for git repos>",
        "author": "<author name>"
    }
    

icalendar

  • Input: Calendar events stored offline in iCalendar (.ics) files

  • Output: Names and locations of calendar events

  • Configuration:

    {
        "paths": [
            "<path to an .ics file>",
            ...
        ]
    }
    

    Not that events from all the listed .ics files will be merged -- duplicate events removed.

maildir

  • Input: Emails stored offline in the Maildir format

  • Output: Subjects of emails

  • Configuration:

    {
        "received_pathname": "<glob pathname of directories with received emails>",
        "sent_pathname": "<glob pathname of directories with sent emails>"
    }
    

orgmode

  • Input: Emacs Org-mode (.org) file in format:

    * <2019-01-17 Thu>
    
    Lorem ipsum
    foo.
    
    bar
    
    * <2019-01-18 Fri>
    
    spam spam
    ...
    
  • Output: Example:

    2019-01-17,Lorem ipsum foo.
    2019-01-17,bar
    2019-01-18,spam spam
    
  • Configuration:

    {
        "path": "<path to the .org file>"
    }
    

orgmodelist

  • Input: Emacs Org-mode (.org) file in format:

    - Lorem ipsum foo. <2019-01-17 Thu>
    - bar <2019-01-18 Fri 11:30>
    ...
    
  • Output: Example:

    2019-01-17,Lorem ipsum foo.
    2019-01-18T11:00:00+02:00,bar
    
  • Configuration:

    {
        "path": "<path to the .org file>"
    }
    

todotxt

  • Input: Todo.txt completed tasks file (done.txt)

  • Output: Texts of completed tasks

  • Configuration

    {
        "path": "<done.txt file path>"
    }
    

twitter

  • Input: Downloaded Twitter archive

  • Output: Texts of tweets

  • Configuration

    {
        "path": "<path to twitter archive directory>"
    }
    

txt

  • Input: Plain text (.txt) file in format:

    2015-12-02 St
        Some
        Text
        Lorem
            Ipsum
                Hierarchy
                Spam
    2015-12-03 Čt
        Foobar
        ...
    
  • Output: Example:

    2015-12-02,Some
    2015-12-02,Text
    2015-12-02,Lorem: Ipsum: Hierarchy
    2015-12-02,Lorem: Ipsum: Spam
    2015-12-03,Foobar
    
  • Configuration

    {
        "path": "<path to the .txt file>"
    }
    

Usage

Generating CSV

The basic output of Automatic Diary is a CSV file. Generate it by running:

./automatic-diary <config path> <output csv path>

Example:

./automatic-diary ~/.config/automatic-diary/config.json ~/Desktop/automatic_diary.csv

The CSV output is in format:

<datetime>,<provider>,<subprovider>,<text>

Example CSV output:

2019-01-23T15:31:54-08:00,git,human-activities,model: Log exceptions while scanning
2019-01-24T09:00:00+01:00,caldav,https://dav.mailbox.org/caldav/da39a3ee5e6b,DHL Packstation
2019-01-25,todotxt,done.txt,Opravit Ondrovi kolo

See the help for all command line options:

./automatic-diary --help

Visualization

The output CSV file can also be rendered as an HTML document which looks kind of like a calendar. See the screenshot above. Generate this HTML document by running:

./automatic-diary-visualize <csv path> <output html path>

Example:

./automatic-diary-visualize ~/Desktop/automatic_diary.csv ~/Desktop/automatic_diary.html

See the help for all command line options:

./automatic-diary-visualize --help

Development

Installation

make setup-dev

Testing and linting

make test
make lint

Help

make help