Scaling connections with Ruby and MongoDB

By Michael de Hoog

Coinbase was launched 8 years ago as a Ruby on Rails app using MongoDB as its primary data store. Today, the primary paved-road language at Coinbase is Golang, but we continue to run and maintain the original Rails monolith, deployed at large scale with data stored across many MongoDB clusters.

This blog post outlines some scaling issues connecting from a Rails app to MongoDB, and how a recent change to our database connection management solved some of these issues.

Global VM Lock

At Coinbase we run our Ruby applications using CRuby (aka Ruby MRI). CRuby uses a Global VM Lock (GVL) to synchronize threads so that only a single thread can execute at once. This means a single Ruby process can only ever use a single CPU core at once, whether it runs a single thread or 100 threads.

Coinbase.com runs on machines with a large number of CPU cores. To fully utilize these cores, we spin up many CRuby processes, using a load-balancing parent process that allocates work across these child processes. In the application layer, it’s hard to share database connections between these processes, so instead each process has its own MongoDB connection pool that is shared by that process’ threads. This means each machine has 10–20K of outgoing connections to our MongoDB clusters.

Blue-green deploys

Maintaining product velocity is essential at Coinbase. We deploy to production hundreds of times a day across our fleet. In the general case we use blue-green deployments, spinning up a new set of instances for each deploy, waiting for these instances to report healthy, before shutting down the instances from the previous deploy.

This blue-green deploy approach means we have 2x the count of server instances during these deploys. It also means 2x the count of connections to MongoDB.

Connection storms

The large count of connections from each instance, combined with the amount of instances being created during deploys, leads to our application opening tens of thousands of connections to each MongoDB cluster. Deploying during high traffic periods, when our application is auto-scaled up to handle incoming traffic, we would see spikes of almost 60K connections in a single minute, or 1K per second.

Hoping to reduce some of this connection load on the database, in March we modified our deployment topology, introducing a routing layer designed to transfer this load from the `mongod` core database process to a `mongos` shard router process. Unfortunately the connections were similarly affecting the `mongos` process and didn’t resolve the problem.

We experienced various failure modes from these connection counts, including an unfortunate interaction where the Ruby driver could cause a connection storm on an already degraded database (this has since been fixed). This was seen during a prolonged incident in April, as described in this Post Mortem, where we saw connection attempts above MongoDB’s 128K maximum to a single host.

MongoDB connection attempts to a single cluster, grouped by replica set member

Proxying connections

The vast amount of connections from our Rails application is the root problem; we had to focus on reducing these. Analyzing the total time spent querying MongoDB demonstrated these connections went mostly unused; the application could serve the same amount of traffic with 5% of the current connection count. The obvious solution was some form of external connection pooling, similar to PgBouncer for PostgreSQL. While there was prior art, there was no currently supported solution for connection pooling for MongoDB.

We decided to prototype our own MongoDB connection proxy, which we call `mongobetween`. The requirements were simple: small + fast, with minimal complexity and state management. We wanted to avoid having to introduce a new layer in Rails, and didn’t want to reimplement MongoDB’s wire protocol.

`mongobetween` is written in Golang, and is designed to run as a sidecar alongside any application having trouble managing its own MongoDB connection count. It multiplexes the connections from the application across a small connection pool managed by the Golang MongoDB driver. It manages a small amount of state: a MongoDB cursorID -> server map, which it stores in an in-memory LRU cache.

Results

Since rolling out the connection proxy, we’ve dramatically reduced the overall count of outgoing connections to MongoDB, by around 20x. Deploy connection spikes which used to hit 30K now hit 1.5K connections. The application steady state, which used to require 10K connections per MongoDB router, now only needs 200–300 connections total:

MongoDB connections drop significantly May 21st after deploying the proxy

Open source

Today we’re announcing that we are open-sourcing the MongoDB connection proxy at github.com/coinbase/mongobetween. We would love to hear from you if you are experiencing similar MongoDB connection storm issues and would like to chat about our solution. If you’re interested in working on challenging availability problems and building the future of the cryptoeconomy, come join us.

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

All images provided herein are by Coinbase. All trademarks are property of their respective owners.


Scaling connections with Ruby and MongoDB was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Continue reading

Compound (COMP) is launching on Coinbase Pro

On Tuesday, June 23, COMP began trading on Coinbase Pro.

On Monday, June 22, we began accepting inbound transfers of COMP to Coinbase Pro. Trading began the following day, June 23.

Once sufficient supply of COMP is established on the platform, trading on our COMP-USD and COMP-BTC order books will start in phases, beginning with post-only mode and proceeding to full trading should our metrics for a healthy market be met.

Compound (COMP) is an Ethereum token that governs the Compound protocol. The protocol allows anyone to supply or borrow Ethereum tokens through a decentralized market. Suppliers earn interest on the crypto they supply to the protocol and borrowers pay interest to borrow it.

Please note that COMP is not yet available on Coinbase.com or via our Consumer mobile apps. We will make a separate announcement if and when this support is added.

The Stages of the COMP Launch

There will be four stages to the launch as outlined below. We will follow each of these stages independently for each new order book. If at any point one of the new order books does not meet our assessment for a healthy and orderly market, we may keep the book in one state for a longer period of time or suspend trading as per our Trading Rules.

We will publish tweets from our Coinbase Pro Twitter account as each order book moves through the following phases:

  1. Transfer-only. Starting on Monday, June 22, customers will be able to transfer COMP into their Coinbase Pro account. Customers will not yet be able to place orders and no orders will be filled on these order books. Trading will begin on or after 9AM Pacific Time (PT) the following day, if liquidity conditions are met.
  2. Post-only. In the second stage, customers can post limit orders but there will be no matches (completed orders). Order books will be in post-only mode for a minimum of one minute.
  3. Limit-only. In the third stage, limit orders will start matching, but customers are unable to submit market orders. Order books will be in limit-only mode for a minimum of ten minutes.
  4. Full trading. In the final stage, full trading services will be available, including limit, market, and stop orders.

One of the most common requests we receive from customers is to be able to trade more assets on our platform. Per the terms of our listing process, we anticipate supporting more assets that meet our standards over time.

You can sign up for a Coinbase Pro account here to start trading. For more information on trading COMP on Coinbase Pro, visit our support page.

Please note Coinbase owns COMP tokens as a result of a 2018 investment in Compound. Coinbase intends to maintain its investment in Compound for the foreseeable future and maintains internal policies that address the timing of permissible disposition of its digital assets, including COMP tokens.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

All images provided herein are by Coinbase.


Compound (COMP) is launching on Coinbase Pro was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Continue reading

Introducing Rosetta: Build once. Integrate your blockchain everywhere.

By Surojit Chatterjee, Chief Product Officer

At Coinbase, we believe a thriving and open ecosystem is necessary for the promise of crypto to be realized, ultimately leading to our vision of more economic freedom for the world. As early as 2016, the Coinbase Secret Master Plan stated our commitment to the open source communities around crypto protocols. And last year, we outlined our strategy of connecting and growing the cryptoeconomy.

Today we’re launching Rosetta, an open-source specification and set of tools that makes integrating with blockchains simpler, faster, and more reliable. This is a step toward our commitment to support an open ecosystem.

Coinbase initially developed Rosetta as the middleware used to integrate blockchains into its platform securely and painlessly, and today we’re sharing it with the world in the hopes it can ease development and integration time for both developers and crypto platforms alike.

The number of blockchains has grown dramatically in recent years, and with it has come a proliferation of different node and wallet APIs that can be challenging to navigate. The goal of Rosetta is to standardize how to interact with blockchains, making it easy for anyone to build on top of a blockchain for a variety of different use cases.

For developers of new blockchain projects, the Rosetta interface makes it easier to ensure compatibility with exchanges that use Rosetta, and can dramatically speed up the time it takes exchanges to integrate with new blockchains and protect customer funds by ensuring specific security conditions are met.

For the broader community of crypto developers, Rosetta makes it easier to build cross-blockchain applications such as block explorers, wallets and dapps. Instead of writing custom parsing for every supported blockchain, applications can use a blockchain project’s Rosetta implementation to read on-chain data and construct transactions in a standard format; minimizing code and simplifying maintenance.

As open source software, Rosetta is easy for anyone to contribute to, and some blockchain teams have already started providing feedback and support. These teams include Filecoin, Celo, Near, Oasis, Coda, Ontology, Kadena, Handshake, Blockstack, and Sia.

A guide to Rosetta, including full documentation, can be found here. The website also includes common development issues that incorporate lessons learned from our years of conversations with blockchain developers and customers. We hope these lessons will be uniquely valuable for new blockchains that want to attract developers, get listed on major exchanges, and build great user experiences.

Over time, our goal is to support a thriving ecosystem of Rosetta interfaces for many more blockchains including Bitcoin & Ethereum, along with even more tools and community resources. Check out our open source guidelines or get started here. If you’d like to learn more, join our Discourse.

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

Implementing the Rosetta tools and/or guidance found on the Rosetta website does not guarantee an asset to be listed on Coinbase. Coinbase evaluates prospective assets against our Digital Asset Framework to assess factors like security, compliance, and the project’s alignment with our mission of creating an open financial system for the world. To apply for listing, fill out an application here.

Coinbase does not endorse, promote, or specify the listing or integration requirements for any third party projects, exchanges, or cryptocurrencies mentioned in this blogpost or otherwise. Any descriptions of functionality and services provided are for information only. Coinbase is not responsible for any loss of funds or other damages caused as a result of using the projects described above.

Unless otherwise noted, all images provided herein are by Coinbase.


Introducing Rosetta: Build once. Integrate your blockchain everywhere. was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Continue reading

Recognizing Recent Efforts By Volunteer Contributors on the Translation Team

Through continued donations and support from the community, we’re thankful to
have recently been able to send out over $15,000 USD worth of bitcoin as a
gesture of gratitude to people who volunteer to help translate bitcoin.org and
spread the word about Bitcoin. Each month, contributors are helping localize
bitcoin.org so that more people around the world can easily get started and
learn more about Bitcoin in their own languages.

Thank you

  • 22296107f24ef76cb1766bc35fd2b3d6
  • Aliak
  • Balaxi
  • Btc4Arab
  • ChenPoWei
  • CoinColors
  • Deadlyweapon
  • GDP
  • GIANNAT
  • Henray0607
  • JurgenH
  • Komodorpudel
  • Rasakila
  • Sreysros
  • Trofo
  • UBS
  • Vinifire
  • YummyPT
  • alanst
  • amore111
  • andrew1992
  • annetypt
  • bitcoinstein
  • cocoklogi
  • cyrilblondel
  • dalovar
  • dende93
  • echo4py
  • elybon
  • fariascl
  • gwb3
  • hakka
  • hantolegionosug_gtc_YmM1MT
  • hoangton
  • iluvbitcoins
  • jodaki84
  • josefelip
  • klicman
  • lomik
  • mareo
  • mateusnds
  • meatfreak
  • miswo
  • nedved
  • nejlika
  • ordtrogen
  • pansvetadielu456
  • phung237337
  • pryds
  • quellobiondo
  • raindogdance
  • rudygodoy
  • shimhyemin
  • sitthykun
  • telstar
  • thisistolis
  • tramyargabam
  • triplay
  • tsb
  • vit05
  • xendez
  • zshilor
  • æ‘é•·

How to get involved

Anyone who is fluent in a language that isn’t English can help translate the
site. You can get started by following a few basic steps:

  1. Create a free Transifex account.

  2. Browse to the bitcoin.org translation
    project
    , find the language
    you’re fluent in, and join the translation team associated with it.

  3. Once you’re on the team you can start translating. Go to the
    “Dashboard” on the top of the page, then to “Languages” and select your
    language. You will see a lot of different resources and their progress. Each
    resource consists of a number of strings. A string is a “string” of text on
    bitcoin.org. The first resource (“bitcoin.org”) contains all strings for the
    main site. You can start there.

  4. Join the Telegram group, feel free to
    introduce yourself and let people know if you have questions. 🙂

Special acknowledgments

In addition to the many volunteers, many of the advances and recent progress in
the translation project wouldn’t be possible without the help of Simon
Hinterreiter and Koichi Hendrawan, who help manage and organize the translation
project, as well as the teams.

A special thanks is also owed to Transifex, for providing us with special access
to their platform.

About bitcoin.org

Bitcoin.org was originally registered and owned by Satoshi Nakamoto and Martti
Malmi. When Satoshi left the project, he gave ownership of the domain to
additional people, separate from the Bitcoin developers, to spread
responsibility and prevent any one person or group from easily gaining control
over the Bitcoin project. Since then, the site has been developed and
maintained by different members of the Bitcoin community.

Despite being a privately owned site, its code is open-source and there have
been thousands of commits from hundreds of contributors from all over the
world. In addition to this, over a thousand translators have helped to make the
site display natively to visitors in their own languages — now 25 different
languages and growing.

Bitcoin.org receives millions of visitors a year from people all over the world
who want to get started with and learn more about Bitcoin.

Continue reading

A New Design for Wallet Pages

A new, more user-friendly and simple set of pages designed to help people find
an ideal bitcoin wallet is now available. It includes a step-by-step wizard to
help people become more familiar with wallets, ratings to compare how they stack
up alongside other wallets, as well as explanations of features they provide
in order to help people make their own informed decisions. Aside from the
wizard, a completely new comparative table and selector is available so people
can see how wallets fare against one another. This is designed to help
people quickly find a wallet to meet their needs.

Check out the new wallet pages and curate your own list of
wallets.

The Old Design

While the old wallet pages presented an assortment of wallets people could
choose from, the experience of doing so was cumbersome and tedious. In order to
see how wallets were rated one would need to navigate to each individual wallet
and then browse back to the overview page to select another wallet to see how
the two might compare. It was not possible to see these comparisons side by
side. One would need to use multiple tabs or browser windows and toggle back and
forth, or a single window, navigating backward and forward.

In addition to the comparative difficulties, millions of people visit
bitcoin.org, many of whom are new to Bitcoin, and have little to no familiarity
with how it works. This is further complicated when a person needs to choose a
bitcoin wallet and has no idea what makes one an optimal choice, what the
features are and what they do, or what they need.

The new design resolves these issues by allowing people to easily compare
wallets, see how they’re rated and subsequently generate a list of wallets based
on available features – in addition to explaining things each step along the
way.

Wallet Ratings

Wallets are given one of four ratings – good, acceptable, caution or neutral.
These ratings are applied across six categories:

  • Control: Some wallets give you full control over your bitcoin. This means
    no third party can freeze or take away your funds. You are still responsible,
    however, for securing and backing up your wallet.
  • Validation: Some wallets have the ability to operate as a full node. This
    means no trust in a third party is required when processing transactions. Full
    nodes provide a high level of security, but they require a large amount of
    memory.
  • Transparency: Some wallets are open-source and can be built
    deterministically, a process of compiling software which ensures the resulting
    code can be reproduced to help ensure it hasn’t been tampered with.
  • Environment: Some wallets can be loaded on computers which are vulnerable
    to malware. Securing your computer, using a strong passphrase, moving most of
    your funds to cold store or enabling 2FA or multifactor authentication can help
    you protect your bitcoin.
  • Privacy: Some wallets make it harder to spy on your transactions by
    rotating addresses. They do not disclose information to peers on the network.
    They can also optionally let you setup and use Tor as a proxy to prevent others
    from associating transactions with your IP address.
  • Fees: Some wallets give you full control over setting the fee paid to the
    bitcoin network before making a transaction, or modifying it afterward, to
    ensure that your transactions are confirmed in a timely manner without paying
    more than you have to.

These ratings are available to review both on the overview page that includes
all wallets, as well as the individual landing pages for each wallet.

Wallet Features

There are nine features people can choose from to sort wallets by. These
features are:

  • 2FA: Two-factor authentication (2FA) is a way to add additional security
    to your wallet. The first ‘factor’ is your password for your wallet. The
    second ‘factor’ is a verification code retrieved via text message or from an app
    on a mobile device. 2FA is conceptually similar to a security token device that
    banks in some countries require for online banking. It likely requires relying
    on the availability of a third party to provide the service.
  • Bech32: Bech32 is a special address format made possible by SegWit (see
    the feature description for SegWit for more info). This address format is also
    known as ‘bc1 addresses’. Some bitcoin wallets and services do not yet support
    sending and/or receiving to or from Bech32 addresses.
  • Full Node: Some wallets fully validate transactions and blocks. Almost all
    full nodes help the network by accepting transactions and blocks from other
    full nodes, validating those transactions and blocks, and then relaying them to
    further full nodes.
  • Hardware Wallet Compatibility: Some wallets can pair and connect to a
    hardware wallet in addition to being able to send to them. While sending to a
    hardware wallet is something most all wallets can do, being able to pair with
    one is a unique feature. This feature enables you to be able to send and receive
    directly to and from a hardware wallet.
  • Legacy Addresses: Most wallets have the ability to send and receive legacy
    bitcoin addresses. Legacy addresses start with 1 or 3 (as opposed to starting
    with bc1). Without legacy address support you may not be able to receive bitcoin
    from older wallets or exchanges.
  • Lightning: Some wallets support transactions on the Lightning Network. The
    Lightning Network is new and somewhat experimental. It supports transferring
    bitcoin without having to record each transaction on the blockchain, resulting
    in faster transactions and lower fees.
  • Multisig: Some wallets have the ability to require more than one key to
    authorize a transaction. This can be used to divide responsibility and control
    over multiple parties.
  • SegWit: Some wallets support SegWit, which uses block chain space more
    efficiently. This helps reduce fees paid by helping the Bitcoin network scale
    and sets the foundation for second layer solutions such as the Lightning
    Network.

People can select features that are important to them alongside the ratings
described above, based on their operating system and/or environment.

Acknowledgments

The new wallet page improvements wouldn’t have been possible without donations
from the community, as well as community feedback that we received as various
milestones were passed and presented along the way. A special thanks is also
due to several people who spent a significant amount of their personal time on
this project – Craig Watkins, Cøbra, Natalia Kirejczyk, Alex Cherman, and
Maxwell Mons. Lastly, we appreciate the efforts of many contributors that spent
time reporting issues on GitHub with regard to both the old and new design, that
we were able to resolve as part of this work:

Adding a Wallet

For people who would like to submit a wallet that isn’t listed on the site for
potential inclusion, documentation is
available

for review.

Feedback

If you have any feedback on the new wallet pages, ideas on how they can be made
better, or if you’ve encountered a problem, please let us know by opening an
issue on GitHub
.

About bitcoin.org

Bitcoin.org was originally registered and owned by Satoshi Nakamoto and Martti
Malmi. When Satoshi left the project, he gave ownership of the domain to
additional people, separate from the Bitcoin developers, to spread
responsibility and prevent any one person or group from easily gaining control
over the Bitcoin project. Since then, the site has been developed and maintained
by different members of the Bitcoin community.

Despite being a privately owned site, its code is open-source and there have
been thousands of commits from hundreds of contributors from all over the world.
In addition to this, over a thousand translators have helped to make the site
display natively to visitors in their own languages — now 25 different languages
and growing.

Bitcoin.org receives millions of visitors a year from people all over the world
who want to get started with and learn more about Bitcoin.

Continue reading