flow:
some conclusions:
--there is a mutation process mudel interacts Memstore with HLog,
1.MVCC insert 'memStoreWriter' number,then write back these edits to Memstore(s) with this number,and the clietns will not see them until the last step 4 below is done.
2.append this batch edits to LogSyncer(which wil checked the write queue periodly and sync to dfs if found) with key-HLogKey and value-WALEdit
sure,the WALEdit contains all the edits(KeyValues) of this transaction(entry),this is so called "transaction capacity in HBase on update".
3.manually sync to dfs with unflushedentries number (now this edits have been persisted in dfs)
4.advance memstore .
this wil update the memstoreRead version(number) so all the clients can see this mutations .
-use a last-effect-index to calculate the progress
when this index is not equals to the batch.length that means this batch is not complete success,so the caller will retry some more times.
-auto sync and manual sync are both present for 'double-check' the wal edits.
in the step 'appendNoSync() in doMiniBatchMutation() of HRegion' ,this will only add to the Hlog's pendingWrites (actually is LogSyncer's one),and LogSyncer will check periodly whether there are some unflushedentries ,if any ,it will run the syncer() to flush.
and in the 'sync to hdfs' opertaion,the condition 'DEFERRED_HLOG_FLUSH' will be checked,which may be set when creating tables,if it is 'true' then the sync operation will do nothging,the the logsyncer will take over 'logsyncer' s job. as to the interval for logsyncer to check,there is a defalut by '1000' ms ,but you can lightly change it to be larger with this property :
hbase.regionserver.optionallogflushinterval
this effects as a more safer level than without wal func:
(best perf)no wal < deferred-hlog-flush < wal(safest)
and will improve huge performace in heavy writings. TODO check
also there are many 'double-check' policies like some search engine e.g. Solr.
-within a minor revision changes,though,many differences occure there,e.g. 94.2 VS 94.20
ref: