When we release this Local Dev Testnet software, we’ll want devs to be able to debug their Noir Contracts (and their entire dapps) easily. Here’s an open list of questions.
If anyone has any strong experience or opinions on these subjects, please do make suggestions in this thread! I’d encourage you to also make a separate post on this forum, if you have a nice proposal (even if you haven’t figured out all the details yet), so that it can be debated and improved more easily – this will also help reduce clutter in this thread!
Noir currently enables users to write unit tests for individual circuits (functions).
But what’s the best way of enabling users to test calls between functions (both within the same contract or between two different contracts)?
And how can we give as much information back to the user as possible?
stack traces
contract addresses, function names, args, return values, emitted events, etc.
events
console.log/debug functionality within a Noir Contract?
error messages
I haven’t played with Foundry at all, but I know it’s got some nice ‘native Solidity’ testing features vs hardhat / truffle etc. To what extent can we enable ‘native Noir’ testing for users, versus them writing tests in typescript?
What are the tools and features that devs expect, when writing and testing Solidity contracts, these days?
Do we need advanced features like pausing the blockchain, rewinding it, etc?
I would say people love Foundry for these reasons:
It’s fast (this allows for fuzzying),
UNIX philosophy (powerful tools doing one thing correctly without needing to write tons of boilerplate and deal with 10 config files).
The tests are written in Solidity. This makes them more concise (you don’t have to deal with bytecode, ABI etc. to instantiate a contract) and you don’t have to switch your thinking between TS and Solidity (this is especially valuable when dealing with types → will this bigint correctly translate to unit256? what if I want uint96?).
Amazing debugging experience (stack traces).
Foundry stack trace contains all the information about errors, events, call trace etc. and if we want to strive to match the devex we should essentially have all the things you’ve listed.
Do we need advanced features like pausing the blockchain, rewinding it, etc?
I used them constantly when writing tests so I would say yes. In Foundry this is implemented through cheatcodes. I could create a list of the most important cheatcodes if you want.
For me personally I think these would be the priorities:
console.log,
errors,
stack trace,
the advanced features like rewinding, direct storage slot modification etc.
I feel like you don’t need the events for smart contract development (you can get the info with console.log). Events are mostly for easier reading from chain by frontends etc. (which is important as well but probably can wait).
It would be useful to arrange a session where we (a3 engineers, noir tooling, noir team) are shown the features of Foundry in more detail, so people can start thinking about how we’ll design them within Noir. I guess you and/or Lasse would be best placed to host that session?
I’m gonna be the contrarian here and say that I like writing tests in typescript as opposed to in Solidity. Typescript is a much more flexible language than Solidity, so building the test harness in it feels more natural to me. It also makes it easier to set up integration tests between off-chain and on-chain components of your system.
I feel that bad typescript APIs for Solidity (I’m looking at you, BigNumber), along with slow JSON-RPC times, led to moving away from ts tests and into Solidity directly. But I’m also aware that this is my personal take, and many devs prefer coding and testing in Solidity.
I’ll add as reference the docs for Hardhat network, which has several settings for fine-tuning how it behaves, along with custom calls for tweaks (advancing time, changing balances or storage, impersonating, etc).
While Foundry is really nice for solidity stuff, when dealing with a lot of individual proving and other database tasks that might not lie in the noir part for all of it, the flexibility of typescript and something like hardhat as @spalladino mentions can be really useful. Foundry good for solidity, but people don’t write their apps in solidity, only their contracts.
Native Noir tests is great for testing the contract, e.g., not the application, only the contract. To test the application itself, which might deal with other database or clever use of the Aztec RPC server the dev would write test in TS anyway.
The stuff mentioned as advanced features I would say is bare minimum, you absolutely want to be able to revert to an older snapshot for your testing, as that can make setups so much easier and smother than having to rerun it.
If we support something like trace_call on the local RPC, seems like it should also make it much easier to setup frameworks in Typescript, noir or whatever language you would want? Likewise, most of the functions that @benesjan mentioned are wrappers around calls to the underlying testing node. Instead of focusing on the full foundry experience here, it might be sufficient if we have our “anvil” and then a foundry or hardhat pendant can be built using that.
A note on complexity. For contracts that require more complex logic to handle accounting etc, we used both Foundry and Hardhat at Aave. Foundry for some of the unit-testing etc, and for the rest we essentially had a TS implementation of the borrowing and accounting system that we were checking if matched the solidity function outputs.
Actually do have some strong opinions here and happy to turn this into a separate forum if they get traction. Fortunately some of it echoes things that others have alluded to:
On native tests in Noir
Don’t think these should be a priority and here is why:
PROS
Less context switching
Less moving parts
CONS
Implementing Native tests properly is very hard -
Solidity and tooling has been around for years and yet good native tests have only just taken off. This is because native tests is hard to implement - it requires plugins, a lot of cheatcodes and needs the language itself to not be too limiting/buggy. We are very early on in our product and should invest in things that we can ship quickly that provide for a frustation free dev-ex
Because of how early we are, Noir is limited. So writing tests in them will be even more frustating.
Fuel Network actually tried shipping this along with their local dev testnet. They recently announced that they have deprecated/de=prioritised it because of how hard it is to implement this properly
Native tests make it harder to write integration tests
On having an all-in-onne ethers+chai like testing frameworks
As @benesjan alluded, with ethers you have to handle bytecode, big number and other ethereum stuff. Unfortunately, even with the rise of foundry, people still have to use ethers - since most contracts have a front-end. So you anyway need to bother with bytecode crap. Either way you still need to invest into a good TS based sdk. So this might require less resources for us presently!
PROS
Easier to have people contributing (since the knowledge gap for noir is much more than that of TS)
Easier to implement
Less limiting
Quicker iteration times
CONS
Another moving part (but some of it is building blocks for a TS SDK anyway)
Potentially Slower to run
We don’t need to debug just on the smart contract level but also when frontend devs call our contracts in TS. This way we can de-duplicate some work and have consistent UX.
Feature Requests
Produce L2 blocks (i.e. fast forward chain) - helps simulate defi apps that have timelocks/staking etc
way to fetch timestamps on the blockchain
Capture Events in TS/SDK (if this is a thing in noir)
Noir already has println but it is unclear how to actually see the outputs. Also the println could be expanded to take in a string (like var name) and format it with the actual var (which can be of type int, field, string, list, tuple. struct)
Enable all RPC calls like simulate_ethTransaction and eth_estimateGas
Even as a Foundry maximalist, I do believe that a Typescript-based experience will be right for Noir long-term because a Noir app implementation is dependent on its WASM runtime. Foundry works great in isolation, because it removes the abstractions of interacting with your contract, and allows for native → native interactions. But given Noir already uses a [#test] syntax, it would be more valuable to test against the WASM outputs of a circuit to ensure the circuit works as expected, and the project is ready for integration.
Here are some misc wishlist items:
Fuzzing in Noir [#test] syntax
A “shallow” and “deep” WASM test - similar to the difference between nargo execute and nargo prove. The typescript SDK should allow you to specify long running “integration” style tests against the proving backend, or unit tests with simple inputs and outputs.
