The actor, who plays Lord Gillingham's valet Mr Green, attacked Anna Bates in last night's edition of the show.
ITV, which warned viewers about upsetting scenes before the period drama began, said it had received "around 60 complaints". Ofcom has also been contacted by viewers but would not give out a figure.
Fans of the show immediately voiced their concerns about the scenes after the broadcast by posting messages on Twitter.
Harman said today that it had been a "bold and risky idea" to include such a plot.
Appearing on ITV's This Morning programme today he said: "It was a shocking and bold storyline, and something that when I was sent the script and read it was like amazed and excited by it, if that makes sense, because for a show like Downton it really leaped out as a bold and risky idea."
"We went for something - as it would be in real life - very shocking."
The episode was seen by an average audience of 9.2 million according to overnight figures.