This comment came from my husband, whose job is in a creative field very different from my own, but who offers me tremendous advice about how creative work works. He has lots of experience with what works and what doesn't, and how to deal with each possibility. I have less, and I am not remotely as mature as he is about ideas not working. Really, he should be the one writing this post.
Plus, I looked at this comment and I didn't know how to build an answer out of it. The only thing I could think about was writing a sci-fi novella years ago, one that didn't work out because I couldn't really write yet, and discovering after it was on the page that the climax of the book was a big shootout that killed four children. I did not know those kids, who were pivotal, would die at any point in writing the novella until after I'd made them dead. Total shock.
But I've told this story to myself and others so many times that it's threadbare. I've lost the particular memories of it, the sense of what I thought the story was going to do before I sat down to write that day. I remember feeling horrified that I'd written such a thing, and I remember the knowledge that the event was precisely in place in the novella. Little else remains.
That story connects to Matt's first clause, things that surprise me while writing, but other than "let your work surprise you as it twists and turns and takes its own directions," which isn't universally sound advice at all, I don't know what to take away from it. Maybe nothing. Maybe it was just a thing that happened when I was an inexperienced writer and I hadn't yet learned how to listen to the book that was inside me and take notes about it before I sat down to draft.
What worked and didn't work...well, usually it's in the particulars that such things go on. I take a lot of ideas from dreams (two of the four novels I've written, a couple of the not-very-many short stories I've published), and that feeling of OMIGOD I have to write down that dream it was AMAZING, and then finding the next afternoon that the dream was absurd or go-nowhere, is not exclusive to me, or to writers.
But the boy-and-mom crisis story, yeah, that one didn't work, and it was a surprise. It was the first story I wrote after I finished Highbinder, and I think the writer-dial was still mostly tuned to action/genre, because half the story was literary and the other half was written like a comic book. The two halves interleaved to tell the story of how this kid and his (single) mom cope with the kid having had an affair with the mom's best friend, a married woman who lives next door. I thought the idea was solid, and I thought the execution went okay. The actiony parts of the story were much more lively and interesting to read than the literary parts, but again, I'd just finished a book that has a lot of action in it, so I had a good bit of that energy left over.
I workshopped the story at my UCLA class, and my instructor told me exactly what to do with my ending - which was definitely the least-working part of the story - but I've set it aside without revising it, because the thing just doesn't hang together. I continue to be surprised every time I think about it, because it really should have worked. I got three decent characters, triangulated in a good conflict; I got an unusual, surprising structure; I got some pretty okay writing in there. But it just doesn't have that certain thing that makes reading worthwhile. It's limp and dull, like hair without Pantene Pro-V. Oh, well.
One thing that always surprises me is how insignificant to readers are the problems that have obsessed you while drafting. The Girl Scout story is written practically as a dyad, in two parts that are almost their own flash pieces, and I worried like crazy after revising that the story didn't have an overall arc and felt like two separate stories smashed together. No one mentioned this, not my initial readers nor any of the 15 people who read it for my workshop class. I even asked two readers about it and they were both like, shrug, no. This happens pretty much every time, that I worry about some element of a story and no one notices it, and instead they notice shit that I never would've thought of.
In writing Highbinder, I worked really hard to give someone a certain set of characteristics without saying flat-out that her nature was not altogether human. I put in bits and pieces of this nature pretty much every time she was mentioned. No one noticed. Three people have read that book and given me detailed feedback on it, and I said something to the effect of "You didn't notice anything in particular about Character?" and they were all like, shrug, no. Gaaaah.
So I've stopped asking those questions, the stuff I worry over that no one mentions. In my workshop class, the workshoppees' questions often seem to be of that strain - they ask them, and I'm like, why on earth are you worried about that when X issue is the elephant in the room? And the answer is probably that the writer notices stuff that the reader doesn't, and vice versa, eternally.
There's a lot that surprises me while I'm actually putting words on the page. Largely the same thing, over and over, that I marveled at during my years of copy editing: that English is contortionist in its flexibility, in the multiple ways it's possible to say what you want to say. But other stuff, too, stuff that's kind of overly self-complimentary, so, sorry.
Like, that I'm at a point now where I can confidently cross something out with the knowledge that I'll write something better instead. Or that I can write a sentence that doesn't sound quite right and just make a note to come back to it later, because it's not a disaster, I can find another way to say it that doesn't suck. That I can make up things to say, one sentence after another. These things amaze me every time. I feel like the same writer I was in high school, the one who was content to write 1,500 words of sheer description of thinly disguised Fitzgerald characters and not want to change a single word despite it being undeniably stupid. I'm not that writer, at all, even if I sometimes behave like her. But I feel like her, and I'm amazed when what comes out of my pen is nothing like what she'd write.