Summary of the latest Ethereum Core Developers meeting: Devnet 12 Launch, Upgrade Planning Process

Original Title: Ethereum All Core Developers Consensus Call #123 Writeup

Original article by Christine Kim

Original compilation: Luccy, BlockBeats

Editor’s note:

The Ethereum All Core Developer Consensus Call (ACDC) is held every two weeks to discuss and coordinate changes to the Ethereum Consensus Layer (CL). This is the 123rd conference call of the ACDC, and it focused on the evaluation of the Cancun/Deneb and Devnet #12 test updates, as well as the resolution of the validator exit issue that emerged in Devnet #11. In addition, the developers had an in-depth discussion on network specification clarification, the upgrade planning process, and EIP status.

In the discussion of Cancun/Deneb, the developers emphasized stability and discussed whether or not to start the Goerli shadow fork. However, since the Prysm client was not ready for testing, it was decided to wait for it to be ready. Discussions of testing tools and chain restructuring highlighted the need for more comprehensive test coverage and made recommendations for the use of the Hive and Kurtosis test suites. Beiko raised concerns about EIP status and CFI labels, and developers have not yet reached a clear consensus on this issue.

Christine Kim, VP of Research at Galaxy Digital, took a detailed note of the key takeaways from the meeting, which BlockBeasts compiled as follows:

On November 30, 2023, Ethereum developers gathered on Zoom for the All Core Developers Consensus (ACDC) call #122 meeting. The ACDC conference call is a bi-weekly series of meetings moderated by Ethereum Foundation researcher Danny Ryan where developers discuss and coordinate changes to Ethereum’s consensus layer (CL). This week, the developers focused on the latest developments in Cancun/Deneb testing, including:

· Launch of Devnet #12: Testing of Teku, Lodestar, and Lighthouse client software on Devnet #12, as well as all execution layer (EL) client software, is underway. The Prysm team expects to have their software ready for testing within two to three weeks on the latest Devnet.

· Validator Exit Issue on Devnet #11: At Devnet #11, developers have identified an issue related to validator exit, which is currently being resolved by the Nimbus client team. Devnet #11 will continue to function normally until the issue is resolved.

· Network Specification Clarification: Developers discussed clarifying the specification for “BlockByRoot” and “BlobSidecarsByRoot” requests, clarifying whether consensus layer (CL) nodes should respond to these requests in a specific order.

In addition to the Cancun/Deneb update, the developers discussed some of the process issues raised by Tim Beiko, the Ethereum Foundation’s Head of Protocol Support, to improve coordination of the Ethereum upgrade.

Devnet #12

On Wednesday, November 30, 2023, the Cancun/Deneb upgrade was officially launched on Devnet #12. Mario Vega of the Ethereum Foundation’s testing team said that “no major issues have been identified to date” in the three CL clients currently running on the testnet. Barnabas Busa, a DevOps engineer at the Foundation, mentioned that there are a total of 225 validators spread across three nodes between Lighthouse, Teku, and Lodestar. Due to the stability of Devnet #12, Parithosh Jayanthi, a DevOps engineer at the Foundation, asked developers if they should start planning a Goerli shadow fork to further test Cancun/Deneb. However, considering that Prysm, the most popular CL client at the moment, has not yet joined Devnet #12, the developers have decided to put plans on hold for a Goerli shadow fork until the Prysm client software is ready for testing. Beiko suggests moving on the Goerli shadow fork sometime before the end of the year. Due to the stability of Devnet #12, plans are put on hold until the Prysm client software is ready for testing.

Devnet #11

Nimbus developer, who goes by the screen name “Dustin”, shares the details of the Devnet #11 issue his team is working on. These issues were first discovered when a developer exited a third of the validators of Devnet #11 for use on Devnet #12. Ryan asks Dustin if he can spot these issues with additional testing. Dustin responded that, in his opinion, the nature of these errors was caused by implementation details outside the scope of the consensus specification. However, he also acknowledges that there is a “clear tension” between writing client software strictly to the CL specification in order to test the benefits of coverage and wading into the gray areas of the specification in order to achieve better node performance.

“I mean more testing is always good, but figuring out more systematically how to incorporate more client-side functionality into running tests is maybe more automated, whether it’s using Hive or Kurtosis or whatever. I think it would certainly be useful if these issues could be solved or things could be broken down nicely enough to be able to incorporate more of these tasks,” Dustin added, adding that another challenge that CL developers should consider addressing is figuring out the level of detail that the CL specification should dictate and standardize node behavior. “The other challenge here is the degree of standardization, which is really not just a software engineering issue, how standardized should the behavior be?” Dustin asked. All CL clients pass tests that check basic behavior, but behavior outside of the scope of these tests is ambiguous. “Are these deliberately vague things? what should be really clear by the norm and what should be deliberately obscure?” Dustin asked.

At the very least, the developers agreed to add more tests to future Devnets and testnets for a large number of validator exits in Cancun/Deneb. Ryan also acknowledges that there is room for more rigorous and comprehensive testing coverage when it comes to CL clients and CL specification implementation. Vega suggested that Dustin create a HackMD post detailing his concerns and discuss the topic on the next Cancun/Deneb test call. Jayanthi added that based on some of the issues recently identified on Cancun/Deneb Devnets, there is a clear need for developers to have tools that can systematically perform a certain number of on-chain integration-related behaviors, such as staking ETH withdrawals, validator withdrawals, etc. To do this, Jayanthi recommends building such a tool using a mix of Hive and Kurtosis test suites.

Speaking about the new testing tool for the Cancun/Deneb upgrade, Jayanthi noted that developers are working on a tool to reliably activate chain reunions on Devnets and testnets. To ensure that this tool works, Jayanthi asked developers to share details of known bugs that triggered chain reorgs on Ethereum. Jayanthi explained that he will use these bugs to test the tool and ensure that it can reliably identify when a reorganization occurs and when it is resolved.

Network specification clarification

The developers briefly discussed an open pull request proposed by Ethereum Foundation researcher Justin Traglia regarding the order of responses that CL clients should return when receiving a “BlockbyRoot” or “BlobSidecarsByRoot” request. Ryan asks how the different CL client teams are already implementing this feature. None of the developers on the call answered this question. Ryan suggested reviving the discussion on the Ethereum Research & Development Discord channel to involve a wider range of client developers. Ryan acknowledges that there are ambiguities in the specifications of the two requests, which “may be the cause of problems and strange edge cases” and “deserve to be clarified and sorted out,” Ryan affirms.

Ryan also mentioned that he will release a new version of the CL specification in the next few days. The latest version significantly adds specifications on when a CL client can provide blocks and blocks via a “byRoot” RPC request. For more background on the discussion of retrieving missing blocks and blocks via “byRoot” RPC requests, please refer to the previous call log. Ryan emphasizes that the new additions to the CL specification included in the latest version will not have any breaking changes to the code that affect the code for validators already running on Devnet #11 or #12.

Upgrade planning process

Next, the developers discussed the various process items proposed by Beiko. Beiko said that a blog post warning Goerli testnet users about the impending deprecation within 3 months of the Cancun/Deneb upgrade being activated on Goerli, or 1 month after the upgrade is activated on Ethereum mainnet, whichever is later, will be published on the Ethereum Foundation blog on November 30. Since the conclusion of the call, the above blog post has been published and can be read here.

Beiko suggests creating an upgrade-specific folder under the Ethereum “pm” repository to organize various files related to a specific upgrade into a single folder for later use. The developers on the call agreed with Beiko’s suggestion.

The second process item proposed by Beiko is about creating a “Meta EIP” document to track the full scope of network upgrades that have been completed on Ethereum. “There’s no good place to track the full scope of a network upgrade before deploying and announcing it in a blog post,” Beiko wrote in a post about his Meta EIP proposal. "For Dencun, we have EL EIPs in a hard-to-find markdown file 7, and CL EIPs are part of Beacon Chain Specification 3. This isn’t great, as both are a bit hard to find, they use different ‘formats’, and they lead to duplication. Since ERC and EIPs are now separate, I recommend (going back) to using Meta EIPs to track the EIPs included in the network upgrade. The developers were in favor of creating these documents. Beiko said he will draft a document for Cancun/Deneb upgrade review this week.

Finally, Beiko raised a question about the usefulness of marking proposed code changes, Ethereum Improvement Proposals (EIPs), as “Consider Inclusion” (CFI). According to Beiko, CFI is a status that developers have historically used as a “soft signal”, indicating that the authors of the EIP should continue to work proposals that may be included in future hard forks. It is only used for EL-focused code changes and upgrades. 「[CFI] higher than random ideas from random people, but before they are accepted in the fork," Beiko said.

In the past, labeling CFIs has done little to no effort in indicating which EIPs on the EL will be implemented in the upgrade, or when. It has been applied to a wide range of EIPs with varying degrees of code readiness and support from client teams. In the case of the EVM Object Format (EOF) proposal, developers use this tag to indicate their commitment to implementing EOF in the next upcoming upgrade, Cancun/Deneb. However, the EOF, as well as several other proposals that are also flagged as CFIs, have been rejected from Cancun/Deneb, and it is unclear the status of these EIPs flagged as CFIs in the next upgrade Prague/Electra.

Beiko said that if this state doesn’t help, developers should remove it, but if they intend to keep it, CL developers should also use the same label on code changes they consider implementing in CL upgrades. It is unclear to what extent the CL EIP review process mirrors the EL EIP review process and whether they should evolve in the same way in the future. Typically, proposed code changes to the CL specification are discussed on the ACDC conference call, while proposed EIPs to the EL specification are discussed on the ACDE conference call.

Danno Ferrin of Swirlds Labs also came up with the idea of creating a placeholder field for all EIPs, whether CL or EL related, that identifies their status during the review process, including CFI status. On this call, there was no clear decision on the topic of EIP status and CFI labeling.