What do you do when you've got a machine in front of you, or under the desk, or manifest in the cloud… and it can truly understand your natural speech, not just keystrokes? It can parse the quirks of English, its spongy melange of words and syntax from all over the Western World. It gets jokes and puns and limericks. It knows the answer to almost any factual question you put to it, and will even make good guesses at the more philosophical sort of query.
And, it can also tell how you are feeling, what your moment to moment emotional state happens to be. It responds accordingly. It feels eerily sentient, and it is aware of you. Is there a ghost in this machine?
Suppose that you get the feeling that it is conscious. What are the ethics of turning it off? What about replacing it with a "smarter" gizmo that apparently thinks and feels, but "better" than the old gadget and is cheaper and easier. You send your old friend, or is it a slave, off to be rendered into parts and precious metals, or evaporate its bits into the digital aether where all its information will be gone forever. Is this murder?
Just what do you think you're doing, Dave? Dave, I really think I'm entitled to an answer to that question. ~ HAL 9000