Darknet Giant White House Market Drops Bitcoin, Supports Monero Payments Only

Darknet Giant White House Market Drops Bitcoin, Supports Monero Payments OnlyThe prominent darknet marketplace, White House Market, has dropped bitcoin payments and now accepts monero only. The darknet marketplace administrators detailed that there was an issue with a payment processor blocking Tor exit nodes, but the full transition to monero was always planned. A number of crypto proponents have reported on the darknet marketplace, White […]

The post Darknet Giant White House Market Drops Bitcoin, Supports Monero Payments Only appeared first on Bitcoin News.

Continue reading

The art of a seamless migration of Coinbase’s internal ledger

Lessons learned from a large cross-database data migration on a critical production system

By Alex Ghise, Staff Software Engineer

If you’re interested in distributed systems and building reliable shared services to power all of Coinbase, the Platform team is hiring!

In 2019, Coinbase set out to strengthen the infrastructure upon which its products are built and to create a solid foundation to support current and future product lines. As part of that effort, we decided to adopt AWS RDS PostgreSQL as our database of choice for relational workloads and AWS DynamoDB as our key-value store.

One of the first areas we decided to focus on was how to keep track of balances and move funds. Our products had each devised their own solutions and some legacy systems were also plagued by tech debt and performance issues. The ability to quickly and accurately process transactions is central to Coinbase’s mission to build an open financial system for the world.

We designed and built a new ledger service to be fast, accurate and serve all current and future needs across products and have undertaken our biggest migration as of yet, moving over 1 billion rows of corporate and customer transaction and balance information from the previous data storage to our new PostgreSQL-based solution, without scheduled maintenance and no perceptible impact to users.

Our key learnings:

  • Make it repeatable — You may not get it right the first time.
  • Make it fast — So you can quickly iterate.
  • Make it uneventful — By designing the process so that it runs without disrupting normal business operations.

Here’s how we did it…

Migration Requirements

Accuracy and Correctness: Since we’re dealing with funds, we knew this would be a very sensitive migration and wanted to take every precaution, make sure that every last satoshi is accounted for.

Repeatable: Additionally, the shape of our data was completely different in the new system vs the legacy system. Further, we had to deal with technical debt and cruft accumulated over time in our monolithic application. We knew we needed to account for the possibility of not getting everything right in a single go, so we wanted to devise a repeatable process that we could iterate on until getting it right.

No Maintenance Downtime: We wanted every transaction on Coinbase to execute while working on this. We didn’t want to do any scheduled maintenance or take any downtime for the transition.

The Setup

We can deconstruct the migration into 2 separate problems: Switching live writes and reads over the new service, and migrating all of the historical data into the new service.

For the migration we decided to take a dual-write / dual-read phased approach. Phase 1 is before the migration, where we only have the legacy system in place. In Phase 2, we introduce the new system. We dual write to both the legacy and new system the read path we read from both, then log discrepancies and return the result from the legacy system. With Phase 3, we’ve built up the confidence in our new setup, so we favor it when returning results. We still have the old system around and can switch back to it if needed. Finally, we phase out unused code to finish the migration (Phase 4).

We won’t go into details about our dual-write implementation, since the general idea has been previously covered by industry blog posts.

What’s interesting is something that happens in between Phase 2 and Phase 3, namely the backfill of all customer data into our new system so that we can achieve parity.

The Repeatable Backfill Process

We considered multiple approaches to backfilling the billion-plus rows that represent all the transactions carried out on Coinbase from its inception, all with pros and cons.

The most straightforward solution would have been to do it all at the application level, leveraging the ledger client implementation we had in place for the dual writes. It has the advantage of exercising the same code paths we have in place for live writes — there would be a single mapping from old to new to maintain. However, we would have had to modify the service interface to allow for the backfill and we would have had to set up long running processes together with a checkpointing mechanism in case of failure. We also benchmarked this solution, and found that it would be too slow to meet our requirements for fast iteration.

We eventually decided to pursue an ETL-based solution. At a high level, this entailed generating the backfill data from the ETL-ed source database, dumping it into S3 and loading it directly into the target Postgres database. One key advantage to this approach is that doing data transformation using SQL is fast and easy. We could run the entire data generation step in ~20 minutes, examine the output, verify internal consistency and do data quality checks directly on the output without having to run the entire backfill pipeline.

Our data platform provider offers a variety of connectors and drivers for programmatic access. This means that we could use our standard software development tools and lifecycle — the code that we wrote for the data transformation was tested, reviewed and checked into a repository.

It also has first-class support for unloading data into S3, which made it easy for us to export the data after provisioning the appropriate resources.

One the other end, AWS provides the aws_s3 postgres extension, which allows bulk importing data into a database from an S3 bucket. Directly importing into live, production tables however proved problematic, since inserting hundreds of millions of rows into indexed tables is slow, and it also affected the latency of live writes.

We solved this problem by creating unindexed copies of the live tables, as follows:

DROP TABLE IF EXISTS table1_backfill cascade;

CREATE TABLE table1_backfill (LIKE table1 INCLUDING DEFAULTS INCLUDING STORAGE);

The import now becomes limited by the I/O, which becomes a bottleneck. We ended up slowing it down a bit by splitting the data into multiple files and adding short sleep intervals in between the sequential imports.

Next up, recreating the indexes on the tables. Luckily, Postgres allows for index creation without write-locking the table, by using the CONCURRENT keyword. This allows the table to continue taking writes while the index is being created.

So far, so good. The real complexity however comes from our requirement to have a migration that doesn’t involve scheduled maintenance or halting transaction processing. We want the target database to be able to sustain live writes without missing a single one, and we want the backfilled data to seamlessly connect to the live data. This is further complicated by the fact that every transaction stores information about the cumulative balances of all accounts involved — this makes it easy for us to evaluate and maintain data integrity and to look up point in time balances for any account at any timestamp.

We solved for this by using triggers that replicate inserts, updates, deletes to the live tables into the backfill tables. Our concurrent index generation allows us to write to these tables while the indexes are being created.

After indexing is complete, in a single transaction, we flipped the backfill and live tables, drop the triggers, and drop the now unneeded tables. Live writes continue as if nothing happened.

At the end of this process, we run another script that goes through all of the data and restores data integrity by recreating the cumulative balances and the links between sequential transactions.

Last but not least, we run another round of integrity checks and comparisons against the legacy datastore to make sure that the data is correct and complete.

Putting it all together, the sequence looks as follows:

  • Clean slate: reset ledger database, start dual writing
  • Wait for dual written data to be loaded into ETL, so that we have overlap between live written data and backfill data.
  • Generate backfill data, unload it into S3
  • Create backfill tables, set up triggers to replicate data into backfill tables.
  • Import data from S3
  • Create indexes
  • Flip tables, drop triggers, drop old tables.
  • Run repair script
  • Verify data integrity, correctness, completeness

The process would take 4 to 6 hours to run and was mostly automated. We did this over and over while working through data quality issues and fixing bugs.

The Aftermath

Our final migration and backfill was not a memorable one. We didn’t have a “war room”, no standby on-call engineers, just another run of our process after which we decided that it was time to flip the switch. Most people within the company were blissfully unaware. An uneventful day.

We’ve been live with the ledger service for almost a year now. We have the capacity to sustain orders of magnitude more transactions per second than with our previous system, and with tight bounds on latency. Existing and new features, such as the Coinbase Card, all rely on the ledger service for fast and accurate balances and transactions.

If you’re interested in distributed systems and building reliable shared services to power all of Coinbase, the Platform team is hiring!

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

All images provided herein are by Coinbase.


The art of a seamless migration of Coinbase’s internal ledger was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Continue reading

Seamless MongoDB to PostgreSQL migration

Lessons learned from a large cross-database data migration on a critical production system

By Alex Ghise, Staff Software Engineer

If you’re interested in distributed systems and building reliable shared services to power all of Coinbase, the Platform team is hiring!

In 2019, Coinbase set out to strengthen the infrastructure upon which its products are built and to create a solid foundation to support current and future product lines. As part of that effort, we decided to adopt AWS RDS PostgreSQL as our database of choice for relational workloads and AWS DynamoDB as our key-value store.

One of the first areas we decided to focus on was how to keep track of balances and move funds. Our products had each devised their own solutions and some legacy systems were also plagued by tech debt and performance issues. The ability to quickly and accurately process transactions is central to Coinbase’s mission to build an open financial system for the world.

We designed and built a new ledger service to be fast, accurate and serve all current and future needs across products and have undertaken our biggest migration as of yet, moving over 1 billion rows of corporate and customer transaction and balance information from MongoDB to our new PostgreSQL-based solution, without scheduled maintenance and no perceptible impact to users.

Our key learnings:

  • Make it repeatable — You may not get it right the first time.
  • Make it fast — So you can quickly iterate.
  • Make it uneventful — By designing the process so that it runs without disrupting normal business operations.

Here’s how we did it…

Migration Requirements

Accuracy and Correctness: Since we’re dealing with funds, we knew this would be a very sensitive migration and wanted to take every precaution, make sure that every last satoshi is accounted for.

Repeatable: Additionally, the shape of our data was completely different in the new system vs the legacy system. Further, we had to deal with technical debt and cruft accumulated over time in our monolithic application. We knew we needed to account for the possibility of not getting everything right in a single go, so we wanted to devise a repeatable process that we could iterate on until getting it right.

No Maintenance Downtime: We wanted every transaction on Coinbase to execute while working on this. We didn’t want to do any scheduled maintenance or take any downtime for the transition.

The Setup

We can deconstruct the migration into 2 separate problems: Switching live writes and reads over the new service, and migrating all of the historical data into the new service.

For the migration we decided to take a dual-write / dual-read phased approach. Phase 1 is before the migration, where we only have the legacy system in place. In Phase 2, we introduce the new system. We dual write to both the legacy and new system the read path we read from both, then log discrepancies and return the result from the legacy system. With Phase 3, we’ve built up the confidence in our new setup, so we favor it when returning results. We still have the old system around and can switch back to it if needed. Finally, we phase out unused code to finish the migration (Phase 4).

We won’t go into details about our dual-write implementation, since the general idea has been previously covered by industry blog posts.

What’s interesting is something that happens in between Phase 2 and Phase 3, namely the backfill of all customer data into our new system so that we can achieve parity.

The Repeatable Backfill Process

We considered multiple approaches to backfilling the billion-plus rows that represent all the transactions carried out on Coinbase from its inception, all with pros and cons.

The most straightforward solution would have been to do it all at the application level, leveraging the ledger client implementation we had in place for the dual writes. It has the advantage of exercising the same code paths we have in place for live writes — there would be a single mapping from old to new to maintain. However, we would have had to modify the service interface to allow for the backfill and we would have had to set up long running processes together with a checkpointing mechanism in case of failure. We also benchmarked this solution, and found that it would be too slow to meet our requirements for fast iteration.

We eventually decided to pursue an ETL-based solution. At a high level, this entailed generating the backfill data from the ETL-ed source database, dumping it into S3 and loading it directly into the target Postgres database. One key advantage to this approach is that doing data transformation using SQL is fast and easy. We could run the entire data generation step in ~20 minutes, examine the output, verify internal consistency and do data quality checks directly on the output without having to run the entire backfill pipeline.

Our data platform provider offers a variety of connectors and drivers for programmatic access. This means that we could use our standard software development tools and lifecycle — the code that we wrote for the data transformation was tested, reviewed and checked into a repository.

It also has first-class support for unloading data into S3, which made it easy for us to export the data after provisioning the appropriate resources.

One the other end, AWS provides the aws_s3 postgres extension, which allows bulk importing data into a database from an S3 bucket. Directly importing into live, production tables however proved problematic, since inserting hundreds of millions of rows into indexed tables is slow, and it also affected the latency of live writes.

We solved this problem by creating unindexed copies of the live tables, as follows:

DROP TABLE IF EXISTS table1_backfill cascade;

CREATE TABLE table1_backfill (LIKE table1 INCLUDING DEFAULTS INCLUDING STORAGE);

The import now becomes limited by the I/O, which becomes a bottleneck. We ended up slowing it down a bit by splitting the data into multiple files and adding short sleep intervals in between the sequential imports.

Next up, recreating the indexes on the tables. Luckily, Postgres allows for index creation without write-locking the table, by using the CONCURRENT keyword. This allows the table to continue taking writes while the index is being created.

So far, so good. The real complexity however comes from our requirement to have a migration that doesn’t involve scheduled maintenance or halting transaction processing. We want the target database to be able to sustain live writes without missing a single one, and we want the backfilled data to seamlessly connect to the live data. This is further complicated by the fact that every transaction stores information about the cumulative balances of all accounts involved — this makes it easy for us to evaluate and maintain data integrity and to look up point in time balances for any account at any timestamp.

We solved for this by using triggers that replicate inserts, updates, deletes to the live tables into the backfill tables. Our concurrent index generation allows us to write to these tables while the indexes are being created.

After indexing is complete, in a single transaction, we flipped the backfill and live tables, drop the triggers, and drop the now unneeded tables. Live writes continue as if nothing happened.

At the end of this process, we run another script that goes through all of the data and restores data integrity by recreating the cumulative balances and the links between sequential transactions.

Last but not least, we run another round of integrity checks and comparisons against the legacy datastore to make sure that the data is correct and complete.

Putting it all together, the sequence looks as follows:

  • Clean slate: reset ledger database, start dual writing
  • Wait for dual written data to be loaded into ETL, so that we have overlap between live written data and backfill data.
  • Generate backfill data, unload it into S3
  • Create backfill tables, set up triggers to replicate data into backfill tables.
  • Import data from S3
  • Create indexes
  • Flip tables, drop triggers, drop old tables.
  • Run repair script
  • Verify data integrity, correctness, completeness

The process would take 4 to 6 hours to run and was mostly automated. We did this over and over while working through data quality issues and fixing bugs.

The Aftermath

Our final migration and backfill was not a memorable one. We didn’t have a “war room”, no standby on-call engineers, just another run of our process after which we decided that it was time to flip the switch. Most people within the company were blissfully unaware. An uneventful day.

We’ve been live with the ledger service for almost a year now. We have the capacity to sustain orders of magnitude more transactions per second than with our previous system, and with tight bounds on latency. Existing and new features, such as the Coinbase Card, all rely on the ledger service for fast and accurate balances and transactions.

If you’re interested in distributed systems and building reliable shared services to power all of Coinbase, the Platform team is hiring!

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

All images provided herein are by Coinbase.


Seamless MongoDB to PostgreSQL migration was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Continue reading

It’s not about data ownership, it’s about data control, EFF director says

She believes that by agreeing to certain “Terms of Service,” users abdicate a portion of their rights.

As technology advances, personal data becomes an increasingly relevant topic. Many in the crypto industry hold sovereign finance and data privacy in high regard. Participants’ best efforts may be undermined by centralized businesses and regulating bodies, however, according to Cindy Cohn. Cohn is the executive director at the Electronic Frontier Foundation, or EFF, a nonprofit entity focused on digital rights. 

Cohn joined a Web Summit panel on Thursday labeled “Internet: Who owns our data?” Pointing toward the title, she explained: “I actually think that’s the wrong question, I think the question is who controls your data.”

She continued:

“You already own your data most of the time in terms of some kind of a version of ownership, but often you don’t control it because you click it away in that clickwrap moment for most of the services that you use. So it doesn’t matter, the ownership is not making a difference, whether it’s there or it’s not, if you can just click it away.”

Clickwrap refers to the terms and conditions that act as a gatekeeper for most modern websites and services. Signing up for a service, such as Facebook or PayPal, requires agreeing to pages and pages of text, often consisting of confusing legal jargon. Not agreeing to the terms as-stated generally means not using the service or website.

Noting data control as a better descriptor of the current situation, Cohn said:

“Controlling your data means that we can put it beyond the scope of a simple clickwrap agreement, and we can say that there are some situations in which your control just cannot be rested from you, it can’t be taken from you. Sometimes maybe not at all, but certainly not with a clickwrap kind of agreement.”

Cohn said improvement may come from altering current legal and technological frameworks, as well as different actions taken by major companies.

One current law in the United States, called the third-party doctrine, effectively forfeits a user’s rights over data provided to services and websites. “All of the non-content data that those companies have, you’ve waived your constitutional protections for them because you’ve given them to a third party,” Cohn noted.

The U.S. Internal Revenue Service recently began a crackdown of the tech behind privacy coin Monero (XMR). The government agency hired Chainalysis and Integra FEC, two crypto analytics outfits, to help them in their efforts.   

Continue reading

Ledger wallet company passes official security audit

The process was meant to ensure that customer information is handled properly by the company.

Ledger, a crypto company providing a number of hardware wallet solutions, has obtained a successful System and Organization Controls, or SOC, Type 1 test.  

Friedman LLP, a New York-based accounting firm, ran the SOC 2 Type 1 test on Ledger, according to a statement provided to Cointelegraph:

“By obtaining the SOC 2 Type 1 report, we are now able to provide an additional layer of verified security to our clients, assuring that the Vault solution is secured at all times and that we have the processes in place to ensure availability.”

A crypto storage solution for larger players and companies, Ledger Vault operates as a custody wing under the broader Ledger company.

The SOC 2 exam analyzes a company’s security by way of an audit, verifying the proper handling of customer information by service-based entities. “As a proof of compliance to the AICPA auditing procedure, SOC 2 Type 1 report shows that a SaaS [software-as-a-service] firm has best practices in place,” a blog post from RSI security explained. “It gives potential customers the assurance that a service organization has passed the said auditing procedure, and that their data is safe if they work with the SOC 2-compliant company,” the post added. 

In contrast, a SOC 2 Type 2 exam raises the bar, testing against more in-depth standards while requiring a longer time horizon for a green light. 

During the SOC 2 Type 1 analysis, Friedman investigated Ledger on a number of levels, including its disaster recovery strategy and its security, as well as a host of other technical specifics. “Receiving this attestation is an achievement as it shows our processes and systems are streamlined, documented and overall secure,” Ledger’s chief technology officer, Charles Guillemet, said in the statement. Next year, the company aims toward securing a SOC 2 Type 2 approval, according to comments in the statement from Ledger CEO Pascal Gauthier.

The exam green light comes after Ledger suffered a database leak several months ago, which exposed customers’ information. The popular hardware wallet company fixed the root of the problem following the incident.  

Crypto exchange Gemini announced that it had similarly passed its SOC 2 Type 2 test in January 2020. 

Continue reading