将讨论Yandex对以下数据库的贡献。
- ClickHouse
- 奥德赛
- 时间点恢复（WAL-G）
- PostgreSQL（包括logerrors，Amcheck，heapcheck）
- 青梅
视频：
你好，世界！我叫Andrey Borodin。在Yandex.Cloud上，我为Yandex.Cloud和Yandex.Cloud客户的利益从事开放式关系数据库的开发。
, . ? - , , , . , .
. . , . , , - .
! ? , . , . .
?
- – . , , , open source, .
- . . . . . , up getting style - .
- – . , Google Summer of Code, Google , , .
- , , . Google, , , , . Google : « , . , ».
- open source – . , . , .
– ClickHouse. , , ..
open source , ( ) . , .
. ClickHouse Yandex Object Storage, . . .
? , , . , , SSD-, - . , . , . – .
? .
ClickHouse over MDS. MDS – . , S3, . . . , , .
S3 – , .
, ClickHouse , ., , ClickHouse. ClickHouse over S3, ClickHouse over MDS. .
:
https://github.com/ClickHouse/ClickHouse/pull/7946 "Filesystem abstraction layer”
https://github.com/ClickHouse/ClickHouse/pull/8011 "AWS SDK S3 integration”
https://github.com/ClickHouse/ClickHouse/pull/8649 "Base implementation of IDisk interafce for S3”
https://github.com/ClickHouse/ClickHouse/pull/8356 "Integration of log storage engines with IDisk interface"
https://github.com/ClickHouse/ClickHouse/pull/8862 "Log engine support for S3 and SeekableReadBuffer"
https://github.com/ClickHouse/ClickHouse/pull/9128 "Storage Stripe Log S3 support"
https://github.com/ClickHouse/ClickHouse/pull/9415 "Storage MergeTree initial support for S3"
https://github.com/ClickHouse/ClickHouse/pull/9646 "MergeTree full support for S3"
https://github.com/ClickHouse/ClickHouse/pull/10126 "Support ReplicatedMergeTree over S3"
https://github.com/ClickHouse/ClickHouse/pull/11134 "Add default credentials and custom headers for s3 storage"
https://github.com/ClickHouse/ClickHouse/pull/10576 "S3 with dynamic proxy configuration"
https://github.com/ClickHouse/ClickHouse/pull/10744 "S3 with proxy resolver"
pull request ClickHouse. pull requests.
:
https://github.com/ClickHouse/ClickHouse/pull/9760 "DiskS3 hardlinks optimal implementation"
https://github.com/ClickHouse/ClickHouse/pull/11522 "S3 HTTP client — Avoid copying response stream into memory"
https://github.com/ClickHouse/ClickHouse/pull/11561 "Avoid copying whole response stream into memory in S3 HTTP
client"
https://github.com/ClickHouse/ClickHouse/pull/13076 "Ability to cache mark and index files for S3 disk"
https://github.com/ClickHouse/ClickHouse/pull/13459 "Move parts from DiskLocal to DiskS3 in parallel"
. , , - , .
:
https://github.com/ClickHouse/ClickHouse/pull/12638 "Add SelectedRows and SelectedBytes events"
https://github.com/ClickHouse/ClickHouse/pull/12464 "Add profiling events from S3 request to system.events"
https://github.com/ClickHouse/ClickHouse/pull/13028 "Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQueryTimeMicroseconds"
, .
, , ClickHouse .
, OLTP , .
. .
, , , , Postgres.
Postgres – . , , .
, , . .
, – , , .
, . multiplexer . , connection pooler, , .
, postgres' . PgBouncer. PgBouncer. , PgBouncer, , , .
https://pgconf.ru/media/2017/04/03/20170316H1_V.Borodin.pdf
. , , , Bouncer, pull request upstream. .
Bouncers. Bouncers, Bouncers. , .
, , Odyssey. .
https://www.pgcon.org/2019/schedule/events/1312.en.html
2019- PgCon . 2 000 GitHub, . . , .
Postgres ., Odyssey, .
? – , , , , , , . .
PgBouncer .
. , pgagroal, Red Hat. , , , , pgagroal.
postgres’ – . , .
. Postgres .
, , . ? , - , . , , , . .
https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-faster-restores-for-postgres/
, CitusData WAL-G. , . CitusData – Microsoft. , , WAL-G, . .
https://github.com/wal-g/wal-g/graphs/contributors
, -10 WAL-G 6 . . , , , , production, , , , WAL-G.
, , , . , Postgres .
? : , .
? , , . , , , .
, , , - , , , .
. , , .
. . . Postgres. MySQL, MongoDB. FoundationDB. .
, , . , Postgres .
? , , , , . pull requests. , . .
, Postgres. Postgres . , Postgres .
, . . - .. , Postgres, , Postgres.
. , . - , .
, Postgres. , , . , . . . PgSQL hacker 40 . - , , .
Postgres – - . - . . .
. , . .
, , .
– - , , .
. , - , , . – - .
, , , . , . – , .
, , , . .
https://commitfest.postgresql.org/23/2171/
, , . , , . Postgres. , , , .
, , - , .
, , . , .
! – .
, Logerrors. , . , , , -.
, , CentOS. , . , open source.
https://www.postgresql.org/message-id/flat/DA9B33AC-53CB-4643-96D4-7A0BBC037FA1@yandex-team.ru
. Amcheck — , , .
, , . . .
https://www.postgresql.org/message-id/flat/59D0DA6B-1652-4D44-B0EF-A582D5824F83@yandex-team.ru
, GiST & GIT . . , .
https://commitfest.postgresql.org/29/2667/
, , , , follower . . . , .
, can… . Crunchy Data . B- Postgres , . . , , . . . , , Postgres, Linux, . , .
, heap, . . , . , .
, Heapcheck. . EnterpriseDB , Heapcheck. PgHeapcheck, Heapcheck. , , . - . open source.
, , , . – , , .
https://www.postgresql.org/message-id/flat/fe9b3722df94f7bdb08768f50ee8fe59%40postgrespro.ru
, . , 1. Postgres , , pg_dump .
. . , . , . , . Postgres.
. , , , , .
https://www.postgresql.org/message-id/flat/fe9b3722df94f7bdb08768f50ee8fe59%40postgrespro.ru
: «, , ».
. , , , , – . , . - , , . . , - . .
? : , . , , . . , , , .
- , , , pull request . , . .
– Greenplum. , Postgres, .
https://greenplum.org/greenplum-database-tables-compression/
Greenplum – append optimized . , . , .
, . . , , , , - .
: «, Postgres. . 20 . ». , , Postgres, 20 – .
https://github.com/greenplum-db/gpdb/commit/179feb77a034c2547021d675082aae0911be40f7
, 20 , . PgConf.Russia Heikki Linakangas Pivotal : « - ? append optimized ?». : « . , . ». : «, , ». : «, , ». , .
pull request, . pull request Pivotal . , .
https://github.com/greenplum-db/gpdb/issues/10150
, pull request Greenplum . , heap- . , . , . – , .
https://github.com/greenplum-db/gpdb/pull/10290
. pull request . .
https://github.com/greenplum-db/gpdb-postgres-merge/pull/53
, Greenplum PostgreSQL 12. . . 20 . , . .
https://github.com/greenplum-db/gpdb/pull/10565
. , .
. , Pivotal. . . , , .
, , . , ? : « , 10 ». ? , scale , , .
? , open source – , . - , - , - , - , - . Greenplum, Greenplum.
! – . . . , , . . – ! . ., , . , , , , .
. .
! . . , . open source. , . , , ., . : « , ?».
WAL-G Go. , . 3 . – . , C. Postgres 30 . C89. Postgres . , ClickHouse C++ . C C++.
, : « open source?».
. . ? , , - . . , 100 % , . , . production. . . ,
: « , ., , ?».
, , . , . , , , , . . , , , . - - , - , . . . .
: « , , , ?». , , ., .
. -, , - . . , . , , , . issues GitHub , DBA, , , , 30 % - , . , . , .
. , . ? Postgres Pro?
, . . , . . , postgres’ . , - .
, ClickHouse ?
, . ClickHouse – . , : « ?». .
: « Greenplum , junior’?». , , , , , , .
, , -. , . . , – . Greenplum .
junior’, . Postgres. , ?
: « ?». - . Postgres, , to do list. , , . . - , - , , . , , . Google Summer of code postgres’ , . , , . WAL-G , . Greenplum , Postgres, Greenplum pull request’ . Postgres – - , Greenplum , . , Greenplum . Greenplum , . , , .