There will still be a place at HMRC for third-party suppliers and their specialist skills and technologies, but only in cases where they are better provided externally and where they can add their specialist innovation rather than running entire services.
To attract the best tech talent his team is also building a more attractive workplace.
They're consolidating all of their staff into a small number of shared locations and equipping them with dual screens, modern smartphones, fast networks, Surface Pros, softphones and ubiquitous WiFi, and state-of-the-art collaboration tools.
Customers now receive around-the-clock services through a combination of tech innovation and changing working practices.
Last January HMRC extended opening hours on webchat to give customer 24/7 access to services, and advisors are often now given the option to work from home.
"They were more productive unsupervised at home, than they were in the building," says Potter.
"We want empowered staff working in a way that suits them, not chained to a desk."
His objective is to encourage staff to seek solutions independently and then scale the successful ones rapidly celebrating every milestone and creating a positive culture.
"I try not to commit to timescales, because you don't know what's going to work. What we did do is try and commit to these key business events."
HMRC has two major events a year: tax credits renewals and self-assessment tax returns. It tends to organise its innovation around these windows, typically beginning work on them fourth months before they take place.
Potter has changed his strategy for communicating the benefits of technology and convincing staff to embrace change since he began his career at HMRC in 2014. Attracting their support was crucial to the successful roll-out of webchat.
"When I joined the organisation I thought we could sell big picture visions, supported by compelling analysis of data, comparing and contrasting how other organisations worked," he said.
"None of that worked at all. What was most powerful was actually doing it, so just proving it on s small scale and then expanding it, building on the evidence that we created and the energy and momentum.
"To do that you need co-conspirators in the organisation, and I thankfully found enough of them to be able to say why not introduce webchat?
"They loved using it because they thought it would serve the customers better, it was more efficient and users loved it and they decided to give them to get 200 people on webchat, now they've got 4,000."
Potter also wants to transform data sharing across government. If the current data silos were better joined up it would, for example, allow the NHS to notify HMRC when a child is born to more efficiently make arrangements around child benefit and tax credits.
Sign up for CIO Asia eNewsletters.