Raphael Carvalho — Building state-of-art LSM-tree compaction in ScyllaDB
—
LSM-tree storage engines are known for providing very fast write throughput, as they allow new data to be flushed into immutable files called Sorted String Table, . SSTable. But there’s no such thing as a free lunch. This append-only nature of the LSM tree generates read amplification over time, because a given key can be redundantly stored across multiple immutable files. That’s the reason storage engines have to employ a background compaction process, to reduce both read and space amplification. Compaction itself is very complex and it plays an essential role in the system’s performance. There are multiple compaction policies, and each suits a particular workload better. For example, leveled policy is better for workloads performing overwrites that are very sensitive to read latency, but that comes at the cost of higher write amplification, meaning that the system cannot keep up with the write rate unless you throw faster disks or more instances at it. On the other hand, there’s the tiered policy that favors write performance but at the cost of higher read and space amplification. There can also be a policy which mixes both tiers and levels, to have something in between, efficiency wise. The RUM conjecture, which states that only two amplification factors can be optimized at the expense of a third, is used by engineers to come up with efficient policies for a variety of workloads. A good example is the ScyllaDB incremental compaction approach that breaks large files into fixed-size fragments, to fix the 100% space overhead that affects most implementations of tiered policy out there. ScyllaDB also leverages industrial controller theory to avoid the endless tuning cycle for compaction speed. If compaction is slower than it should be, the compaction backlog increases, resulting in bad read latency and potentially the system running out of disk space. If compaction is running too fast, then resources are being needlessly stolen from foreground requests, affecting both latency and throughput. So nothing is better than a closed-loop control system that measures backlog and adjusts compaction speed accordingly. That frees your DBAs to perform more important tasks.
Download slides:
9 views
1293
383
5 days ago 00:10:10 1
O IMPERIALISMO da Rússia: por que NÃO EXISTE ditadura no OCIDENTE? | Cortes do João Carvalho
1 week ago 00:03:53 1
EDU FALASCHI | Pegasus Fantasy ( Saint Seiya ) | The Ultimate Version
1 week ago 01:10:30 1
PRECISAMOS ACABAR com a Escala 6x1 URGENTEMENTE! feat. Ismeiow | João Carvalho
1 week ago 00:00:18 1
Real Madrid Vs CD Guadalajara July 20, 2011 🤔🔥 Where are they now? (Ronaldo, Benzema, Varane, Ozil)
2 weeks ago 00:03:08 1
SOFI TUKKER - Throw Some Ass (Official Music Video)
3 weeks ago 01:37:52 1
João Carvalho ANALISA o ATAQUE a RAFAH, Palestina: COMO fazer ISRAEL PARAR?
2 months ago 02:03:30 1
BRASIL PARASITA: Verdade Alternativa se chama MENTIRA! feat. Normose | João Carvalho reage
3 months ago 00:10:31 1
Premier Automne
3 months ago 00:02:14 1
Filipe Ret - Melhor Agora (Ao Vivo)
3 months ago 00:04:31 5
Angèle - Démons feat. Damso [ CLIP OFFICIEL ]
3 months ago 02:21:42 1
Do COLONIALISMO ao CAPITALISMO TARDIO: O AULÃO dos seus SONHOS! | João Carvalho
3 months ago 01:29:49 1
Curso sobre Materialismo Histórico Dialético - Professor João Carvalho
4 months ago 00:04:59 2
Maestrick - Ethereal (Official Video)
4 months ago 02:32:35 1
MAGDA BARREIRA “CIGANA“ - LARICA ROCK PODCAST | EP 36
4 months ago 01:54:48 1
Marx, Maquiavel & Lênin - Live ao vivão com Caixa de Ferramentas e Revista Opera
4 months ago 00:22:14 1
O NOVO PC DO RONALDO FENÔMENO | MACHINE MAKERS (FINAL)
4 months ago 01:02:44 1
DIA DA VITÓRIA! A Memória SOVIÉTICA e a Preservação HISTÓRICA ft. Troika Idiomas | João Carvalho
4 months ago 00:02:36 1
POCAH, Pabllo Vittar, DJ Biel do Furduncinho, Papatinho - BANDIDONA (Videoclipe Oficial)
4 months ago 00:02:55 1
AgroPlay Verão - Nosso Quadro @anacastelaoficial
4 months ago 00:04:43 1
Acid Arab - Leila Feat Sofiane Saidi (Official Music Video)
5 months ago 02:05:40 1
Paz, Pão e Terra: A História do País dos Sovietes - Introdução à História da União Soviética (URSS)
5 months ago 01:04:34 1
O PASTOR que CRIOU o PÂNICO moral e o MEDO do DIABO a CADA ESQUINA feat. @Fitoria | João Carvalho
6 months ago 01:36:12 1
Relações Internacionais e Guerras na visão de Marx e Engels feat. Caio Bugiato | João Carvalho
6 months ago 01:22:32 1
Porque AMAMOS FANON! feat. Douglas Barros e Jorgeval Borges | João Carvalho