Blockchain security experts team up to improve industry threat response

Since the spring of this year, Isaac Patka of AI security firm Shield3 and Paradigm research partner Sam, better known as Samczsun, have been working alongside blockchain projects to improve security in the wake of cyber threats that have continued to plague the industry.

In early August, the duo launched SEAL 911, a Telegram bot designed to connect users with vetted security experts aiming to enhance cybersecurity disclosure and swiftly prevent DeFi hacks potentially worth hundreds of millions of dollars.

That initiative was established in the hopes of countering multiple industry-related hacks that have taken place this year, including Curve Finance’s $70 million exploit in July.

Now the pair are hoping to up the ante, establishing a new emergency drill initiative designed to assist budding blockchain protocols in their fight against malicious hackers and potential attack vectors.

Over the past few months, @isaacpatka, myself, and other leading whitehats and researchers have worked with Compound Labs and Yearn to help them prepare for the most stressful situation any dev could face: by conducting a simulated attack against their protocol

— samczsun (@samczsun) October 25, 2023

Blockworks reached out to Patka to get a better sense of their undertaking and the lessons they have learned over the past few months.

Blockworks: Can you walk us through the inception of this emergency drill initiative? What was the driving force behind it?

Patka: I first met Sam through our mutual friend Jeanne. I met Jeanne at DWeb camp 2022 when I was presenting some of my previous open source & standards projects. I heard that Sam was looking for some help in building some training infrastructure for protocol teams to practice being in a war room prior to a real emergency.

The idea resonated with me because at that time I was working on some research and tools related to identifying and avoiding social attacks and dependency failures in decentralized communities.

I volunteered to help get a proof of concept off the ground and after a quick brainstorming call in the spring, I got to work on outlining the drill framework for Compound Labs, which was the first team that offered to participate in a drill.

Blockworks: You mentioned the role of “comprehensive recon” in your drills. How does this initial step set the stage for the rest of the exercise?

Patka: In the recon phase I get up to speed with all of the features, smart contracts, documents and publicly available information about the target protocol. I’m trying to figure out what the “control surface” is for any privileged users [or] admins, how the protocol interacts with [or] relies on other protocols, how they monitor the health of the system, what risk processes exist, how they introduce things like protocol upgrades or new feature releases, and if there are inconsistencies in the system if it’s deployed across different networks.

This recon becomes the foundation for the tabletop scenarios where we talk through potential issues.

Blockworks: The use of tabletop simulations seems like an interesting approach. Could you elaborate on what goes into these simulations and how they inform the subsequent steps?

Patka: After the recon phase, I put together a script with a few scenarios and talk through them with the whole team on a call. These scenarios help us understand their incident response procedures, their monitoring and their social/ comms style. The questions we’re asking at this point are:

1. “X” has happened. How was the team alerted? Was there monitoring that caught this, or did someone from the community reach out to the team?
2. Who are the stakeholders and subject-matter experts who know how to deal with this
3. If this incident impacts other protocols, who has the contact info for that team?
4. If this requires a response from a multi-sig, who are the signers, and how are you reaching out to them? How fast do you think they will respond?

All of this helps us find potential “hot spots” or things that we want to stress test in a live scenario.

Blockworks: What criteria do you use to select the protocol teams you’re going to drill with? Do you have any prerequisites?

Patka: At this phase, we’re trying to work with teams where we think we can both help them by providing some training, but also learn from them about how the top protocol teams in the space operate, and share those practices with the wider community.

So while we don’t have specific prerequisites, a good fit now is a team that contributes to a protocol with fairly widespread adoption and has been through a few incidents already so we can learn about a variety of team styles.

However, as our infrastructure is becoming more robust and easier to set up, I would enjoy working with some teams earlier on in their protocol to provide some training to people who have never been in a war room before.

Blockworks: Your first test was with the Compound protocol. Can you delve into some of the unique challenges or lessons learned from that initial test?

Patka: The biggest planning challenge was identifying a scenario that was not too catastrophic to be frustrating, but interesting enough to be engaging and would involve some diagnosis and coordination.

We considered a variety of things like external protocol failures, governance attacks and contract upgrade issues. We ended up simulating a bug that made the protocol slowly start losing funds so that we could see how their monitoring would pick up on the process and how they would respond.

One of the biggest lessons here was on the social, coordination layer. I was impressed with the close collaboration between the protocol devs and the auditors and protocol guardians in diagnosing the issue.

On a technical level, the first drill also involved a lot of late-night debugging infrastructure, getting the network fork, and block explorer, and monitoring infra stability.

Blockworks: You talked about avoiding zero-day vulnerabilities in your drills. Can you explain the reasoning behind this decision and how it affects the integrity of the exercise?

Patka: The reason we avoid “zero-day” vulnerabilities or other very widespread catastrophes is so that we can engage the protocol team in something they could reasonably respond to, and something contained within their protocol’s ecosystem. For example, we haven’t done drills around things like compiler bugs, or consensus layer failures.

However, I think these widespread issues would be interesting to simulate in cross-protocol drills where we could get multiple teams and perhaps users of the protocols all interacting with a forknet where something has gone wrong to make it realistic and build social resiliency.

Blockworks: You mentioned Yearn’s “emergency procedure cards” during your test with them. How common is this practice across other protocols, and would you recommend it as a standard?

Patka: I haven’t seen other protocols that implement emergency procedure cards like Yearn yet, but I’d highly recommend it. In many protocols, but especially with Yearn, there are many external integrations that require specific context and subject matter expertise.

When some incident happens, you don’t want to be spending time re-reading your own docs & contracts instead of taking action. Having emergency procedures for specific scenarios helps teams make decisions quicker and more confidently. Writing these emergency procedures is a mandatory step of the risk [and] diligence process of deploying Yearn strategies.

I’d recommend adding emergency procedures to the risk/diligence processes for other protocols, for example when deciding whether or not to integrate with various assets as collateral sources or adding them to markets.

Blockworks: What are some key performance indicators you look at during and after a drill to measure its effectiveness?

Patka: I look for some indicators of both our performance as organizers of the drill and how well the team did. For our side, I’m looking at the stability of our infrastructure and how well the team adapts to the simulated environment.

On the project side, I’m keeping a timeline of at what point issuers are discovered, how long until there is a diagnosis, and how long until there is some consensus around the action to take.

We also send out a postmortem survey to teams to find out what they learned, what they plan to improve in their processes, and how we can improve our simulations.

Blockworks: Can you share some overarching trends or common gaps you’ve noticed in protocol security as a result of these drills?

Patka: I’m not sure if it’s a gap but there seems to be less of a formal “on-call” system across various protocols than I expected. There is an ‘always online’ aspect of the crypto culture where people seem to just assume the right developer or multi-sig signer will be available when needed.

This generally seems to work but I’m curious to explore if some more formalization of roles and schedules would help. I’ve also noticed that monitoring and governance varies for protocols across different L1s/ L2s where they have code deployed. I think there is room for improvement across the industry on how protocols that straddle multiple networks manage their contracts.

Blockworks: Looking ahead, are there plans to broaden these drills to include more protocols or even different types of tests?

Patka: For sure, we are looking to broaden drills to include different types of protocols, or perhaps multiple protocols at the same time. We also want to get to the point where these are easy enough to run that teams can hold regular training for community contributors to build their experience in incident response. I’d also love to engage with new security engineers who might want to learn about security by designing scenarios and configuring simulations.

This interview has been edited for brevity and clarity.

Source