On 2 October last year, five detectives and a police computer expert spent ten and a half hours raiding my home. They took all of the phones, computers, hard drives, USB keys, CDs and other electronic devices, as well as an assortment of files.

They were trying to uncover the identity of a confidential source who had given me information for my recently published book about New Zealand politics. The book had revealed a campaign of dirty tricks being run out of the prime minister’s office, including an organised system for smears on political opponents.

But I’m sleeping easy; I’m as certain as I can be that the raid was a waste of the police’s time. It won’t help them at all to find my source.

This is an issue vexing investigative journalists around the world: how to protect sensitive sources in the era of widespread surveillance technologies. I have heard a lot of pessimism about how it is no longer possible to keep confidential sources safe – sources here meaning whistleblowers and people who leak important public-interest information. I don’t agree. Investigative journalism is as possible and powerful as ever. I believe that if we understand the risks and do our job properly, we can continue to protect our sources, as I was able to do with the sources for my recent book.

I love the work of investigative journalism. For people with the right aptitude and motivation, it is one of the most socially useful things they can do. I want to share with you a sense of what this work is, how it is done and why it is so worth doing. But first I want to start with the subject of protecting sources in the digital age.

The first thing the police looked for when they arrived at my house – the first thing they always look for – was my mobile phone; they were rather crestfallen to learn that I do not use one. (I know this is unusual, but in my work I prefer mental space to immediate availability.) Mobile phones are pretty much self-surveillance devices for people who have things they need to keep confidential. They are also one of the main ways that police investigate crimes: by seizing a phone, the police instantly get an alphabetised list of friends and associates, piles of text messages with dates and times of activities, often a GPS record of your movements, photos with embedded times and locations, and more.

Over the years I have written quite a lot about police and intelligence, and so I understand many aspects of how investigations are handled. I certainly don’t believe the NSA and satellites track me wherever I go or anything like that. They don’t. But whether you’re a shoplifter or a whistleblower, mobile phones are the most likely way you will be traced. For this reason anyone with a sensitive source should never use their phone to contact said source.

You may have heard stories of the lengths people will go to avoid interception, such as removing the batteries from their mobiles or sticking them in the fridge. In truth, most people who imagine they are being spied on are worrying for nothing. But if there is a serious need to be careful, such as protecting a sensitive source, people just shouldn’t use mobiles. Leave them at home – it’s as simple as that.

The main reason is metadata, that word that firmly entered the modern vernacular with Edward Snowden. In practice, the police do not bug many phones. It takes too many resources to listen to and transcribe spoken conversations, and so the practice is generally reserved for serious crimes. But ‘call data’ or ‘metadata’ – easy and readily available – is the police’s investigative tool of choice. Call data includes the records of who you called, when you called and the location of the nearest cellphone tower. The same databases that save this data also save all text messages to and from your phone. In New Zealand, this data is automatically saved for all mobile phones for three to four months; in Australia, courtesy of the recent amendments to the Telecommunications (Interception and Access) Act, this data will be stored for at least two years. The police routinely phone or email the telcos and get this information to help their investigations.

The solution to this is awareness. We shouldn’t meet a confidential source in a busy cafe and, likewise, we shouldn’t leave great big digital footprints between ourselves and our precious and vulnerable sources. Many sources are caught, particularly in government agencies, because the agency owns and stores all its phone logs. When there is a leak investigation, they can do obvious checks such as any incoming or outgoing calls from the journalist who did the story. We just need to understand the risks and, when it matters, never leave electronic tracks.

So how do I contact sources or potential sources? Usually it’s a face-to-face encounter, or sometimes I call from an untraceable phone. After that it’s at prearranged meetings set up the last time we saw each other. Sometimes high-tech, but usually simple and low-tech. I do not encrypt my email or waste time worrying about surveillance for 99.9 per cent of my communications – but I do make sure I am very careful when it is needed.

The next thing the police looked for in my house were any computers. My main computer, like many people’s, contains mountains of emails plotting my life hour by hour. But I have always made sure there were no tracks to the source of my book. (Incidentally, deleting something on most computers doesn’t remove it entirely.)

Even in the unlucky event that the police win the current court case and are granted access to my computer, they will find that the hard drive is encrypted and has a nice long password. A six- or seven-letter password can be cracked in quite a short time by a simple computer program that tries a, aa, ab, aaa, aab and so on. Thus, each additional symbol in a password makes it much harder to crack.

I am not a computer whiz. For such tasks I seek out very experienced IT people to help with things like computer encryption. I find that collaboration between investigative journalists and IT whizzes is a very productive matching of skills.

I should add that the kind of precautions I’m talking about here aren’t needed for most situations. I don’t think countries like Australia and New Zealand have armies of secret police taking an unhealthy interest in everything we do. Indeed, I spend my life reassuring people that they really don’t need to worry about being under personal surveillance. But sometimes we do need to take it seriously – for example, when it comes to protecting confidential sources.

The most important protections are the basic pre-digital ones, in particular knowing how to keep a secret. Not being able to do this is, in fact, the biggest risk.

Many people cannot keep a secret. Secrets create an irresistible urge to share. If we ‘just tell one person’, they are likely to tell someone else who cares even less about keeping the secret and – by natural laws of physics – the news accelerates outwards.

I say to sources: ‘We need to put a fence around the two of us and never tell anyone else what we’re doing for the rest of our lives.’ It is simple and effective. It means there are some marvellous stories that can’t be told, but there is no alternative. Part of a journalist–source relationship is being there to give care and support to your source, especially early on when they are feeling vulnerable. In most cases, the people who have helped me reveal big stories seem to feel pride and satisfaction in what they have achieved. Often a special life-long bond is formed.

The final step in protecting sources is the care needed when writing the story. The facts that we know and those that we don’t know can be like a great big arrow pointing at a few people as possible informants, and so it is vital that great care is taken in how this information is presented. The specialists brought in to study a government or corporate leak go first for the organisational call data – logs of who accessed which documents and so on. Next, they study what details are in the story, which have been omitted, what years the information covers, which staff know what bits of information and thus, by triangulation, who their suspects are.

Once we understand this, we can take precautions. I picture a security officer studying a leak, and go to great effort selecting what information to include, what to leave out, where to supplement from other sources and, generally, how to bewilder those trying to narrow down where the leak came from. I often find that, as I write, I am putting as much thought into what I might give away about my sources as to what I am writing. This faithfulness to sources is part of my job.

If we are careful and do all of these things, I genuinely believe we can continue to get new sources and keep them safe. I feel confident that sources such as the one the police were hoping to identify in the raid on my house will be just fine.

What I am saying is this: we can still work with sources to hold powerful people to account and to inform the public on big issues – and keep them safe in the process.

This leads to the big question: why? Why should journalists and sources risk themselves for the sake of a story? What purpose does investigative journalism serve?

Daily news is reasonably good at reporting what ‘newsmakers’ say and do in public, and blogs and commentators are good – sometimes very good – at discussing and analysing that news. But the media system is weak when things are hidden, spun or manipulated – in other words, when it relies on press releases or other controlled messaging. It’s a weakness that becomes apparent as soon as there are organised vested interests that have the resources and motivation to try to control information reaching the public, or to spin a preferred version of truth.

Investigative journalism is the activity of unearthing and publicising things that challenge dominant views: finding out who is telling the truth, exposing unethical behaviour or dishonesty, giving a voice to people and issues that are hidden or ignored, digging deep on behalf of the public so that people don’t have to live in a political soup of spin and half-truth. It is about providing a counter-narrative to that of the powerful.

It’s not easy work, but it can be highly satisfying and worthwhile.

In recent years there have been some stunning, world-changing leaks, notably those inspired by WikiLeaks. Those leaks will undoubtedly lead to more people deciding to make important disclosures of a similar scale. But although there are similarities, that kind of leaking is not the same thing as investigative journalism.

Sometimes I wonder if those stunning revelations and information dumps have made it seem as if leaks just turn up on their own; perhaps giving the impression that investigative journalism is about setting up a secure cloud account on the internet and waiting for the leaks to come to you.

The heart of investigative journalism – and when I enjoy it most – is when you decide a subject needs investigating and set about digging down into the roots of the issue. It is a very strategic activity: studying a subject and then searching for the right investigative tools and sources to make progress. Personally, I have many subjects that I feel need work (and that are on ‘the list’) and I continue to search gradually for the right sources and breakthroughs.

When I start such an investigation, I make a ‘map’ of what sorts of things I want to find out about a subject and include all the possible sources (people and otherwise) that might help. I draw up lists of places where I would like to find insiders and make a register of fieldwork to be done, letters to write, experts to ask, overseas parallels to research and so on. Sometimes it’s like a game or puzzle: I’m trying to solve a problem, find a missing clue. Often, because I am working on subjects that are secretive or intentionally hidden, I am looking for insiders and sensitive material.

And then, usually, a kind of magic occurs. I find that if I have bothered to think through what I’m trying to find, and brainstormed the types of sources I am looking for, they tend to come along. I unexpectedly meet someone who can help, or hear about a source of information that might be useful. Somehow, perhaps because I was on the lookout, the breakthrough happens.

This has worked for me so many times now – my first book in 1996 was about New Zealand’s role in the Five Eyes intelligence network, and others followed on the military in Afghanistan, anti-environmental public relations and the inner workings of governments and political parties – that I have come to realise that if I do the work, I have a good chance of getting results.

But it’s not really magic – indeed, some of my subjects haven’t worked out at all yet. It is really a combination of planning, luck and then hard work.

Most inside sources who have helped me would never dream of being a Chelsea Manning or Edward Snowden. It would not even occur to them, or at least not seriously, to approach a journalist and become a whistleblower. Partly, this is why the secure dropbox is not enough. The vast majority of sources help because I have sought them out and asked. I have been able to reassure them that there is value in publicising the issue and that they can trust me not to get them into trouble.

That is my secret method: asking.

It is hugely important that potential sources continue to feel comfortable saying yes, and that there are journalists who keep seeking them out and asking for information. Many of the terrible issues in the world rely on secrecy; control of information and selected release of spin are part of how bad things can happen in relatively democratic and law-based countries. Insiders are often the only realistic way of getting information about hidden things. If people in power believe they can get away with keeping their actions classified, they generally act much worse than if they risk finding their grubby business or lies suddenly exposed on the front page.

But having willing sources is only half of the equation. One of the challenges for investigative journalism is not having enough people to seek out potential sources. And who is going to do that kind of work in increasingly stripped down and precarious news organisations?

I think we need to recognise that investigative journalism isn’t just a speciality within the journalism field. Much of the best investigative work is done by filmmakers, authors and public-interest researchers, people who might not see themselves as traditional journalists but who are still fulfilling this critical role. It is a role in society defined by its goals, tools, methods and motivations, not by job descriptions and university qualifications.

Countries don’t need hundreds of investigative journalists, but they need some – and the more the better! Every issue and every part of society that has decent investigative journalism is better off than the many areas that have no scrutiny. I have been part of international networks of investigative journalists for the last twenty or so years. Country by country, there are people who have done inspiring pieces of public-interest research that have helped to change the world.

Take, for instance, the Swedish TV journalist Fredrik Laurin, who felt suspicious when reading a government statement about deporting two men to Egypt in the early years of the so-called War on Terror. The government said that it had an assurance from the Egyptian government that the men would not be tortured, which is like an assurance from soft drink companies that they care about obesity. He went back to the film footage he’d taken of the aircraft taking off and found the aircraft’s tail number, N379P. By searching on the net he found a story about the same plane collecting hooded detainees from an airfield in Pakistan, further spurring his curiosity. He followed this with letters to plane spotters around the world, asking about more sightings of the plane, which was registered to a US company with no office or staff, only an address in a legal office. Step by careful step, he and colleagues gradually uncovered one of the really important stories of that era: the CIA’s ‘rendition’ flights, which were used to grab suspects from around the world and illegally move them to secret detention centres or Guantanamo Bay.

It makes me happy to know that people like Fredrik exist. We have worked together on other projects since then.

Similarly, one evening when I was in Switzerland, I had dinner with a South African investigative journalist named Justin Arenstein. He told me a story about an investigation he had conducted into vicious vigilante teams set up by white farmers to intimidate and punish black farm workers. He had moved into a rural area, ostensibly to look after a friend’s farm for a few months, and it wasn’t long before his neighbours encouraged him to do his bit to enforce their version of justice. Set up with hidden cameras, he went out on hair-raising night-time drives to sort out – that is, threaten and assault – the black workers. When Arenstein’s exposé eventually came out, the country was shocked; the vigilante numbers decreased and the authorities were forced to stop secretly supporting them.

There are many more such stories. There is the Indian journalist catching corrupt politicians, the Mozambique journalist exposing illegal rainforest logging, others publicising unsafe factories all over the world or illegal fishing or countless other subjects. These journalists often make a profound difference to the problem they are reporting on: they uncover it, thereby stopping it being invisible or hidden.

The ones I respect the most are those who have kept going through the decades, continuing to uncover big stories and resisting the urge to be cynical or discouraged. It is always easier to decide that the world is a mess and that nothing can be done about it. In every era through history there is plenty of bad and depressing news to justify thinking the future is hopeless. But society has nonetheless made progress, and each era is much better than it would have been because people continue to be shocked and offended by injustice and inhumanity. And, crucially, some people do something about that.

Finally, there is a related ‘occupation’ that is worth mentioning. It is for people who work in governments, companies, political parties, PR firms and all sorts of other places, in senior or junior roles, who come across information of public interest and decide to tip off or pass those details on to people like me. You may one day find yourself in that position, as part of your career, or in a job you take to earn money for a while. If you find things that offend you and that the public has a right to know, then – with suitable care and thought – you can maybe do some good by working with a trustworthy investigative journalist to get that story told.

That is what happened with the main source for my last book – the one the police were looking for and, all going well, will never find. They did New Zealand a massive service by helping to expose the activities of the dirtiest prime minister in our lifetimes.