The fallibility of AI and what to do about it

 Go to any legal or technology conference, and speaker after speaker will tell you AI is the next BIG thing. Huge strides have been made in in underlying technologies allowing new tasks to be performed quickly by that would take human teams weeks.


This is certainly true. For example, LawPanel’s algorithmic comprehensive trademark search has performed 1m searches that would have taken a team of 10 attorneys working 10 hours a day 45 years to complete.


And yet..


What you won’t hear people talk about so much is the fallibility of AI, and the importance of both system design and the legal processes it sits within to identify and mitigate errors. Take our trademark clearance searches. For searches users run themselves, we do not give a categorical ‘available’ or ‘unavailable’ to register. 98% of the time the algorithm is in line with attorney opinion on similarity. However, it is more fallible on distinctiveness or descriptiveness.


For those who’ve heard of the Turing test (a way to to test a machine’s ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human), my suggestion would be asking it whether ‘sparkle’ is distinctive for detergents.


To manage the risk of false positives and false negatives, we add in a margin of safety to steer users towards getting actual attorney advice. And we always explain trade mark law is complicated, that an automated search is fallible, and that when we say ‘probably available’ or ‘probably not available’ the range of likelihood each way may only be 60:40.


The processes we have around our AI are also designed to mitigate the risk of error. So attorneys use the automated searches only as a starting point and not the end result. They can also change weightings in the algorithms, if experience suggests too much credence has been placed on a non-distinctive word in the search string. Lastly, they chose the search results that go into the final report, which gives them ownership and responsibility for the outcome.


The business model of Trade Mark Direct was also designed to take out any cost to the client of a false positive search leading them to proceed with attorney advice. If a client asks for a full attorney search and the advice is negative, they are free not to proceed, with no fees to pay.


Increased volumes of enquiries and filings more than offset for this. No client has ever complained of the initial false result. In fact, they are delighted with the free search, and the goodwill this generates often brings them back.


We will be writing more on the great potential and opportunities from AI. But coming from a legal services background we well understand attorney / client relationships and the importance of trust. We will also be writing more about the ways in which AI can be fallible, and how we wrap the AI in systems and processes that make the combined whole robust.

Popular posts from this blog

Four of the strangest trademark cases of all time

‘Crazy New Ideas’ in legal innovation

LawPanel joins Mischon de Reya’s MDR Lab