Summary of the latest meeting of the core developers of ETH Place: Devnet 12 is launched, and the upgrade planning process is being upgraded

Original Title: Ethereum All Core Developers Consensus Call #123 Writeup

Original article by Christine Kim

Original compilation: Luccy, BlockBeats

Editor’s note:

The ETH Workshop All Core Developer Consensus Calls (ACDCs) are held bi-weekly to discuss and coordinate changes to the ETH Workshop’s Consensus Layer (CL). This is the ACDC’s 123rd conference call, which focused on the evaluation of the Cancun/Deneb and Devnet #12 test updates, as well as the resolution of the validator exit issues that emerged in Devnet #11. In addition, the developers had an in-depth discussion on network specification clarification, the upgrade planning process, and EIP status.

During the discussion of Cancun/Deneb, the developers emphasized stability and discussed whether or not to start the Goerli shadow fork. However, since the Prysm client was not ready for testing, it was decided to wait for it to be ready. Discussions of testing tools and chain restructuring highlighted the need for more comprehensive test coverage and made recommendations for the use of the Hive and Kurtosis test suites. On the issue of EIP status and CFI labels, Beiko raised concerns about whether these tags should be retained, and developers have not yet reached a clear consensus on this issue.

Christine Kim, VP of Research at Galaxy Digital, gave a detailed note on the highlights of the meeting, which BlockBeasts compiled as follows:

On November 30, 2023, ETH developers gathered on Zoom for the All Core Developers Consensus (ACDC) call #122 session. The ACDC conference call is a bi-weekly series of meetings led by Danny Ryan, a researcher at the ETH Workshop Foundation, where developers discuss and coordinate changes to the ETH Workshop Consensus Layer (CL). This week, the developers focused on the latest developments in Cancun/Deneb testing, including:

· Launch of Devnet #12: Testing of Teku, Lodestar, and Lighthouse client software, as well as all execution layer (EL) client software, on Devnet #12 is underway. The Prysm team expects to have their software ready for testing within two to three weeks on the latest Devnet.

· Validator Exit Issue on Devnet #11: At Devnet #11, developers have identified an issue related to validator exit, which is currently being resolved by the Nimbus client team. Devnet #11 will continue to function normally until the issue is resolved.

· Network Specification Clarification: Developers discussed clarifying the specification for “BlockByRoot” and “BlobSidecarsByRoot” requests, clarifying whether consensus layer (CL) nodes should respond to these requests in a specific order.

In addition to the Cancun/Deneb update, the developers discussed some of the process issues raised by Tim Beiko, Head of Protocol Support for the ETH Foundation, to improve coordination of the ETH Workshop upgrade.

Devnet #12

On Wednesday, November 30, 2023, the Cancun/Deneb upgrade was officially launched on Devnet #12. Mario Vega of the ETH Foundation’s testing team said that “no major issues have been identified to date” in the three CL clients currently running on the testnet. Barnabas Busa, a DevOps engineer at the Foundation, mentioned that there are a total of 225 validators spread across three nodes between Lighthouse, Teku, and Lodestar. Due to the stability of Devnet #12, Parithosh Jayanthi, a DevOps engineer at the Foundation, asked developers if they should start planning a Goerli shadow fork to further test Cancun/Deneb. However, considering that Prysm, the most popular CL client at the moment, has not yet joined Devnet #12, the developers have decided to put plans on hold for a Goerli shadow fork until the Prysm client software is ready for testing. Beiko suggests moving on the Goerli shadow fork sometime before the end of the year. Due to the stability of Devnet #12, plans are put on hold until the Prysm client software is ready for testing.

Devnet #11

Nimbus developer, who goes by the screen name “Dustin”, shares the details of the Devnet #11 issue his team is working on. These issues were first discovered when developers exited one-third of the validators of Devnet #11 for use on Devnet #12. Ryan asks Dustin if he can spot these issues with additional testing. Dustin responded that, in his opinion, the nature of these errors was caused by implementation details outside the scope of the consensus specification. However, he also acknowledges that there is a “clear tension” between writing client software strictly to the CL specification in order to test the benefits of coverage and wading into the gray areas of the specification in order to achieve better node performance.

“I’m saying more testing is always good, but figuring out more systematically how to incorporate more client-side functionality into running tests, maybe more automation, whether it’s using Hive or Kurtosis or whatever. If these issues can be solved or things can be broken down nicely enough to be able to incorporate more of these tasks, I think that would definitely be useful,” Dustin added, adding that another challenge that CL developers should consider addressing is figuring out the level of detail at which the CL specification should dictate and standardize node behavior. “The other challenge here is the degree of standardization, which is really not just a software engineering issue, how standardized should the behavior be?” Dustin asked. All CL clients are tested to check basic behavior, but the behavior outside of the scope of these tests is ambiguous. “Are these deliberately vague things? what should be really clearly defined by the specification, and what should be deliberately obscure?” Dustin asked.

At the very least, the developers agreed to add more tests to future Devnets and testnets for a large number of validator exits in Cancun/Deneb. Ryan also acknowledges that there is room for more rigorous and comprehensive testing coverage when it comes to CL clients and CL specification implementation. Vega suggested that Dustin create a HackMD post detailing his concerns and discuss the topic on the next Cancun/Deneb test call. Jayanthi added that based on some of the issues recently identified on Cancun/Deneb Devnets, there is a clear need for developers to have tools that can systematically perform a certain number of on-chain integration-related behaviors, such as staking ETH withdrawals, validator withdrawals, etc. To do this, Jayanthi recommends using a mix of Hive and Kurtosis test suites to build such a tool.

Speaking about the new testing tool for the Cancun/Deneb upgrade, Jayanthi noted that developers are working on a tool to reliably activate chain reunions on Devnets and testnets. To make sure the tool works, Jayanthi asked the developers to share details of the known bug that triggered the chain reorganization on the ETH. Jayanthi explained that he will use these bugs to test the tool and ensure that it can reliably identify when a reorganization occurs and when it is resolved.

Network specification clarification

The developers briefly discussed an open pull request proposed by Justin Traglia, a researcher at the ETH Foundation, regarding the order of responses that CL clients should return when receiving a “BlockbyRoot” or “BlobSidecarsByRoot” request. Ryan asks how the different CL client teams are already implementing this feature. None of the developers on the call answered this question. Ryan suggested reviving the discussion on the ETH Research & Development Discord channel to involve a wider range of client developers. Ryan acknowledges that there are ambiguities in the specifications of the two requests, which “may be the cause of problems and strange edge cases” and “deserve to be clarified and sorted out,” Ryan affirms.

Ryan also mentioned that he will release a new version of the CL specification in the next few days. The latest version significantly adds a specification on when a CL client can provide blocks and blocks via a “byRoot” RPC request. For more background on the discussion of retrieving missing chunks and chunks via “byRoot” RPC requests, please refer to the previous call log. Ryan emphasizes that the new additions to the CL specification included in the latest version will not have any breaking changes to the code that will affect the code for validators already running on Devnet #11 or #12.

Upgrade planning process

Next, the developers discussed the various process items proposed by Beiko. According to Beiko, a blog post warning Goerli testnet users about the impending deprecation within 3 months of the Cancun/Deneb upgrade being activated on Goerli, or within 1 month of activating the upgrade on the ETH mainnet, whichever is later, will be published on the ETH Foundation blog on November 30. Since the conclusion of the call, the above blog post has been published and can be read here.

Beiko recommends creating an upgrade-specific folder under the ETH “pm” repository to organize various files related to a specific upgrade into a single folder for later use. The developers on the call agreed with Beiko’s suggestion.

The second process item proposed by Beiko was to create a “Meta EIP” document to track the full scope of the network upgrades that had been completed on the ETH. “There’s no good place to track the full scope of a network upgrade before deploying and announcing it in a blog post,” Beiko wrote in a post about his Meta EIP proposal. "For Dencun, we have EL EIPs in a hard-to-find markdown file 7, and CL EIPs are part of Beacon Chain Specification 3. This isn’t great, as both are a bit hard to find, they each use a different ‘format’, and they lead to duplication. Since ERC and EIPs are now separate, I’d recommend (going back) to using Meta EIPs to track the EIPs included in the network upgrade. The developers were in favor of creating these documents. Beiko said he will draft a document for review of the Cancun/Deneb upgrade this week.

Finally, Beiko raised a question about the usefulness of marking proposed code changes, ETH Improvement Proposals (EIPs) as “considered to include” (CFI). According to Beiko, CFI is a state that developers have historically used as “soft signals”, indicating that the authors of EIPs should continue to work hard on proposals that may be included in future hard forks. It is only used for EL-focused code changes and upgrades. 「[CFI] higher than random ideas from random people, but before they are accepted in the fork," Beiko said.

In the past, labels CFI had little effect in indicating which EIPs on the EL would be implemented in the upgrade or when. It has been applied to a wide range of EIPs with varying degrees of code readiness and support from client teams. In the case of the EVM Object Format (EOF) proposal, developers use this tag to indicate their commitment to implementing EOF in the next upcoming upgrade, Cancun/Deneb. However, the EOF, as well as several other proposals that were also tagged as CFI, were rejected from Cancun/Deneb, and it is unclear the status of these EIPs tagged as CFI in the next upgrade Prague/Electra.

Beiko said that if this state doesn’t help, developers should remove it, but if they intend to keep it, CL developers should also use the same label on code changes they consider implementing in CL upgrades. It is unclear to what extent the CL EIP review process mirrors the EL EIP review process and whether they should evolve in the same way in the future. Typically, proposed code changes to the CL specification are discussed on the ACDC conference call, while proposed EIPs to the EL specification are discussed on the ACDE conference call.

Danno Ferrin of Swirlds Labs also came up with the idea of creating a placeholder field for all EIPs, whether CL or EL related, that identifies their status during the review process, including CFI status. There was no clear decision on the topic of EIP status and CFI labels on this call.