Coinbase Giving: Half Year in Review

In May 2021, we announced the launch of Coinbase Giving. Coinbase Giving is the operational embodiment of our commitment to Pledge 1%: our promise to dedicate 1% of Coinbase profits, equity, and employee time toward charitable activities that leverage the power of crypto to increase economic freedom in the world.

Where we are

In the past 7 months we’re proud of what we’ve accomplished:

  • Coinbase Giving is the largest charitable organization focused on Crypto (>$500M endowed at the time of post)
  • GiveCrypto, founded in 2018, has been brought officially under the umbrella of Coinbase Giving to enable scale and maximize social impact
  • We have launched 7 programs, supported 410 organizations, deployed ~$2M, and are impacting 2K+ lives

We are only at the beginning of this journey. 2021 has been a test and iteration year — our impact is still modest. The purpose of this post is to share progress and lessons learned.

Areas of focus

In 2021 our focus has been to support endeavors that align to three broad categories:

  1. Increasing education around, and access to, crypto
  2. Accelerating the development of crypto protocols that underpin the cryptoeconomy
  3. Fostering the next generation of crypto talent, no matter where they are located

Highlights from 2021

GiveCrypto is our nonprofit that enables crypto based direct giving to some of the most needy people in the world. In doing so we also educate them about crypto and give them an alternative to their broken financial systems.

GiveCrypto has been one of Coinbase Giving’s largest investment areas in-house. GiveCrypto is a two-sided market. First, donations are driven by donors who are inspired to give charitably. Second, from these donations, crypto is put directly in the hands of people in need through our on-the-ground Ambassador program.

Bringing GiveCrypto directly within Coinbase has accelerated our ability to integrate donation collection within the consumer experience. As such, 2021 has been a critical year of rebuilding and integration. As the #1 crypto exchange in the US, we see this connection as a huge step forward in enabling public donations using crypto and Coinbase products. We have taken this chance to revamp our Ambassador program — our system to distribute funds directly into the hands of the most needy. In 2021, we were purely testing our distribution system and only reached 246 recipients. However, with those tests now under our belts, we feel ready to scale distributions dramatically in 2022.

Already we are seeing how crypto possesses unique characteristics that make it suitable as a cross-border donations mechanism. For example, it allows donors to avoid disintermediation by bad actors, who often take a cut at the expense of the recipient. Crypto adoption is growing, particularly in Latin American countries where hyperinflation can be devastating. Our independently validated randomized control test shows that direct giving continues to demonstrate high real-life impact, whether in enabling individuals to pay for food, education, or other day-to-day needs.

Developer Grants 2.0 to accelerate the development of crypto protocols.

In August, we launched the second iteration of our Crypto Community Fund whose goal is to make the cryptoeconomy safer and easier to access for everyone. We selected 6 developers to fund across 3 thematic areas: (1) Scalability; (2) Privacy, identity and zero knowledge research; and (3) Protocol security and other foundational infrastructure.

Grant terms are typically 1-year and we will share progress on their journeys throughout 2022.

Base 11 partnership to foster the next generation of global and diverse crypto talent.

Earlier this year Coinbase announced our three-year partnership with Base 11. Together, our goal is to develop the next generation of diverse crypto talent. In early October, Base11 hosted the Next Frontier Conference & Expo, which featured a fireside chat with Alesia Haas, CFO at Coinbase and Landon Taylor, Founder & Chairman, Base 11.

At the conclusion of the conference, the Coinbase Giving team launched our first ever Open Innovation Challenge, giving away over $10,000 in prizes and an exclusive NFT. The Open Innovation Challenge called for participants to focus on four key pillars: environmental sustainability, healthcare, education, and financial inclusion. The call to action has drawn hundreds of participants from all over the world and dozens of quality submissions that address the key pillars described above. We’re excited to enter the judging phase and will announce the winner in Q1 2022.

Learnings

  1. The crypto-forward charitable ecosystem — like the crypto ecosystem — is still nascent. That said, there are a growing number of novel partnerships between established non-profits (who deeply understand the social sector) and technical partners (who get crypto/blockchain) trying to better solve old problems with new solutions. We’ve learned this is Coinbase Giving’s sweet spot for grant-making.
  2. As more people hold crypto, more people want to give in crypto. “45% of crypto holders donated $1,000 or more to charities in 2020 compared to 33% of all investors”, according to a new report by Fidelity. We have engaged in many conversations with potential partners exploring the role Coinbase Giving could play in connecting nonprofits and endowments with the infrastructure they need to accept and hold crypto as an endowment (only ~2% have the ability to do so currently).
  3. Without education, there is no access. As the crypto market has heated up in 2021, so has the call for Coinbase Giving to provide more educational resources and support. Non-profits, NGOs and education leaders increasingly see crypto literacy as a next-generation skill required for the future success of their communities and constituencies. Coinbase’s investment in Crypto Learn and other initiatives distinctly well position Coinbase Giving to address this educational need. We look forward to exploring more in 2022.
  4. Accelerating and building infrastructure is more than open source code. There are many barriers to the simple and secure use of crypto and blockchain that are just as much about infrastructure development as having the right protocols. There are also many more applications for blockchain to solve social problems than have yet been envisioned or realized. This is all infrastructure. We can do better to support a flourishing ecosystem.

Next Up

With 2021 as a foundation-building (half) year, we are looking forward to incorporating what we’ve learned and to drive the social impact of crypto in 2022. We’ll keep the community up to date on what’s happening with a quarterly blog. Stay tuned!


Coinbase Giving: Half Year in Review was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Continue reading

Incident Post Mortem: November 23, 2021

Summary

Between 4:00 pm and approximately 5:36 pm PT on Tuesday, November 23rd, we experienced an outage across most Coinbase production systems. During this outage, users were unable to access Coinbase using our websites and apps, and therefore were unable to use our products. This post is intended to describe what occurred and the causes, and to discuss how we plan to avoid such problems in the future.

The Incident

On November 23rd, 2021, at 4:00pm PT (Nov 24, 2021 00:00 UTC) an SSL certificate for an internal hostname in one of our Amazon Web Services (AWS) accounts expired. The expired SSL certificate was used by many of our internal load balancers which caused a majority of inter-service communications to fail. Due to the fact that our API routing layer connects to backend services via subdomains of this internal hostname, about 90% of incoming API traffic returned errors.

Error rates returned to normal once we were able to migrate all load balancers to a valid certificate.

Chart depicting overall 90% error rate at our API routing layer for duration of incident.

Context: Certificates at Coinbase

It’s helpful to provide some background information about how we manage SSL certificates at Coinbase. For the most part, certificates for public hostnames like coinbase.com are managed and provisioned by Cloudflare. For certificates for internal hostnames used to route traffic between backend services, we historically leveraged AWS IAM Server Certificates.

One of the downsides of IAM Server Certificates is that certificates must be generated outside of AWS and uploaded via an API call. So last year, our infrastructure team migrated from IAM Server Certificates to AWS Certificate Manager (ACM). ACM solves the security problem because AWS generates both the public and private components of the certificate within ACM and stores the encrypted version in IAM for us. Only connected services like Cloudfront and Elastic Load Balancers will get access to the certificates. Denying the acm:ExportCertificate permission to all AWS IAM Roles ensures that they can’t be exported.

In addition to the added security benefits, ACM also automatically renews certificates before expiration. Given that ACM certificates are supposed to renew and we did a migration, how did this happen?

Root Cause Analysis

Incident responders quickly noticed that the expired certificate was an IAM Server Certificate. This was unexpected because the aforementioned ACM migration had been widely publicized in engineering communication channels at the time; thus we had been operating under the assumption that we were running exclusively on ACM certificates.

As we later discovered, one of the certificate migrations didn’t go as planned; the group of engineers working on the migration uploaded a new IAM certificate and postponed the rest of the migration. Unfortunately, the delay was not as widely communicated as it should have been and changes to team structure and personnel resulted in the project being incorrectly assumed complete.

Migration status aside, you may ask the same question we asked ourselves: “Why weren’t we alerted to this expiring certificate?” The answer is: we were. Alerts were being sent to an email distribution group that we discovered only consisted of two individuals. This group was originally larger, but shrank with the departure of team members and was never sufficiently repopulated as new folks joined the team.

In short, the critical certificate was allowed to expire due all of three factors:

  1. The IAM to ACM migration was incomplete.
  2. Expiration alerts were only being sent via email and were filtered or ignored.
  3. Only two individuals were on the email distribution list.

Resolution & Improvements

In order to resolve the incident we migrated all of the load balancers that were using the expired IAM cert to the existing auto-renewing ACM cert that had been provisioned as part of the original migration plan. This took longer than desired due to the number of load balancers involved and our cautiousness in defining, testing, and applying the required infrastructure changes.

In order to ensure we don’t run into an issue like this again, we’ve taken the following steps to address the factors mentioned in the RCA section above:

  1. We’ve completed the migration to ACM, are no longer using IAM Server Certificates and are deleting any legacy certificates to reduce noise.
  2. We’re adding automated monitoring that is connected to our alerting and paging system to augment the email alerts. These will page on impending expiration as well as when ACM certificates drop out of auto-renewal eligibility.
  3. We’ve added a permanent group-alias to the email distribution list. Furthermore, this group is automatically updated as employees join and leave the company.
  4. We’re building a repository of incident remediation operations in order to reduce time to define, test and apply new changes.

We take the uptime and performance of our infrastructure very seriously, and we’re working hard to support the millions of customers that choose Coinbase to manage their cryptocurrency. If you’re interested in solving challenges like those listed here, come work with us.


Incident Post Mortem: November 23, 2021 was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Continue reading