Merge lp://staging/~onboard/onboard/word-completion into lp://staging/~onboard/onboard/main

Some cleanup, added early support for multiple dictionaries.

Added auto-learning (always on) and dictionary saving (still too often, needs timed auto save)

Learn and Dict buttons are working now and hopefully properly connected to gconf.
Gconf schema has changed to add a new folder 'word_completion'.

updated makdicts to support options (see makdicts -h) and a list of languages on the command line.

Word completion keeps better track of recent input now and can restart at any point when backspacing.
Added auto-save for saving modified dictionaries, default is every 10min and on exit. New gconf key word_completion/auto_save_interval.

Only load dictionaries when the layout has use for them.
Removed depedency KeyCommon from WordPredictor.py.
Fixed spurious "U"s in auto punctuation
WC keeps track of additional editing keys: del, cursor left/right.
Toggling punctuation doesn't reset input line anymore. '

Added additional weighting of words based on their usage oder. A new gconf key 'frequency_time_ratio' controls the ratio between the old frequency based weighting and time of last use.

- fixed save on exit
- learn button discards input line when turning off, keeps it when turning on
- added Francesco's learning texts for french and italian
- makedicts defaults to "expand affixes", "don't include infrequent words" -> dictionary sizes around 200kB
- exclude Project Gutenberg license headers and footers from training data
- esc key clears input buffer
- don't learn words with more than 3 repeated characters

added ability to toggle word completion including its ui via new gconf key enable_word_completion

experimental detection of mouse clicks outside of onboard; reset word completion on every detected click.

- reworked the punctuation logic again to get key feedback; hopefully fixing the issues with ; and : in the process
- added Francescos fixes to gconf schemas and delete button
- fixed up experimental outside click detection

work around memory leak in pangocairo.CairoContext.create_layout() (gnome #599730)

- added input history with color highlighting (negotiable ;) blue: ignored, yellow/red: new word to learn
- added stealth button + new gconf key stealth_mode
- fixed wrong default values for auto_save_interval and frequency_time_ratio when gconf keys are missing
- replaced word_completion with word_prediction in gconf schemas and most everywhere every
- increased default frequency_time_ratio from 50 to 75
- another small update of the keyboard logic, potential for slightly less bugs

added word prediction test application, n-grams of arbitrary order, various smoothing algorithms incl. kneser-neyinterpolation

Merge from main branch

fixed crasher on start with unavailable dictionaries

- fixed updating problem with the color highlighting of the history line
- allow single letter words into the dictionaries for a, I,...

- learning with incremental parameter calculation for kneser-ney smoothing
- around 10 times speed-up of prediction queries

switched ngram-test from strings to indices, another 2x speed-up

switched ngram-test data structures to python trie as preparation for a C implementation

new (temporary) sub-project lm, python extension for a dynamically updatable n-gram language model

prediction:
- some clean up of the c++ code and improved code comments
- use python memory manager as often as possible
- added pool allocator for maybe 15% less memory usage
- added save/load to depth-first file format: too little improvement in loading speed, left the old arpa-like one in place
- added python tools split_corpus, entropy/perplexity, ksr (keystroke-savings-rati, see README)
- added a minimal d-bus prediction server + test-client

- initial word prediction support for onboard, prediction and learning through D-Bus calls
- prediction service loads, caches and saves language models
- added linear interpolation of language models
- reworked tokenization and moved it into the python extension in pypredict.py
- simplified and fixed all python tools: split_corpus, train, predict, entropy, ksr
- plenty of bug fixes, still more to do

- added log-linear interpolation and onboard's simple overlay-algo for merging lms
- comments and cleanups

- added two new smoothing options: Witten-Bell and Absolute Discounting
- reworked Kneser-Ney smoothing for more robust normalization
- changed default smoothing to Absolute Discounting
- added new tool analyze for plotting pretty entropy and ksr charts, needs matplotlib
- split_corpus supports additional parameters to influence the size of split texts
- added python unit tests for tokenization and language model normalization
- fixed word insertion bug in onboard; tokenization is fully done via D-Bus now
- fixed a crasher in PoolAllocator
- random fixes and comment updates throughout the code

fixed traceback at service startup when the models directory wasn't found'

- try multiple encodings in pypredict.read_corpus before giving up, default is [utf-8, latin-1]
- use timeout_add_seconds instead of timeout_add for the autosave timer in gpredict to allow for grouping wakeups

- reworked pypredict.split_sentences( )and prettyfied the results of sentence splitting
- fixed erroneous joining of sentences when using texts generated by the split_corpus tool

don't commit_input_line() when scrolling with the mouse wheel

wrapped DynamicModel and NGramTrie in templates to allow for alternative memory layouts, i.e. recency caching and no kneser-ney parameters if they aren't needed

experimental workaround for traceback at reset_clip on lucid

Added a new model type CachedDynamicModel for recency based ngram-caching with exponential fall-off over time.
The prediction now remembers recently used ngrams. The current parameters where found by trial and error, need to more thoroughly investigate what works best later.

Merge with onboard main

Added new D-Bus method lookup_text to get onboards input line display working again.

Removed all traces of dictionary auto saving from onboard. The D-Bus service has been saving language models for a while.

Use utf-8 coding to avoid problems with build_i18n

Added makemodels script to create language models for all available aspell dictionaries. Filter models based on the aspell vocabulary.

Experimented with loading models in a separate thread with mixed results, disabled again. Python's global interpreter lock complicates things.

- Added color feedback to the mouse click buttons
- Fixed old bug in Keyboard.iter_keys() that led to always returning to the main pane when pushing click buttons
- Set color of bright checked buttons to the same as buttons that are "on".

- Extended the analyze tool to investigate caching parameters
- Added an optimize tool that tries to find better caching parameters with simulated annealing
- Set new, marginally better caching parameters for recency caching
- Allow floats in addition to ints for recency_halflife property of CachedDynamicModel

Merge with onboard main

Fixed oversized key labels for small window heights. Fallout from last merge.

Merge from main. Needs additional work.

Merged with main, additional changes
- Converted to GTK3/gnome introspection
- Moved word prediction gconf keys to gsettings
- Kept mouse click polling for button updates and word learning
- Always convert key labels to unicode to avoid breaking calls to the word predictor
- Regression: input line display disabled because of broken get_char_extents, https://bugzilla.gnome.org/show_bug.cgi?id=654343

- Disabled more of the input line; Pango introspection is in bad shape
- Fixed merge mistake in classic layout

Merge from trunk. Word prediction technically still works, but the ui needs lots of polishing.

Unbreak auto-punctuation.

Partially bring back the input line. Introspection of pango attributes is still utterly broken, use parse_markup instead.