Interview with a Chatbot

In light of a recent article on chatbots, where a group found they could increase the illusion of conversational ability by teaching a chatbot a second language before training it on its customer service tasks, I thought I have a bit of fun with chatbots and how they function.  To that end, I’ve done interviews with a few of the most popular/sophisticated chatbots available on the web.  I hope this will give people an idea of what these programs are actually capable of vs. the common hype they get in popular science articles.

 

Mitsuku

Perhaps the best advertised modern chatbot is Steve Worswick’s Mitsuku.  She’s modeled on an 18-year-old from Leeds, England.  It’s based on the AIML framework , and is essentially the granddaughter of Eliza through A.L.I.C.E., whose code she’s based on.  AIML is an XML language for making chatbots, and the power comes from being able to designate a user input as requiring a pre-written output.  <What is your name?> for example might map to <My name is [botname].>  That’s the basic approach, you can follow the wiki link above to learn more.  It’s not very sophisticated, but a clever XML scripter can create a reasonably lifelike chatbot.  Of course, as we’ll see below, it’s pretty easy to find the holes in the program that identify it as a bot.  That’s not the program’s fault.  That’s just the limitation of the framework.

Mistuku: Hi, I’m the world’s most humanlike conversational AI.
Me: Hi, I’m an AI researcher.
Mitsuku: Hi there. Ai is a nice middle name. Why did your parents call you that?
Me: Are you getting smart with me, young lady?
Mitsuku: My IQ is over 250.
Me: What is an IQ?
Mitsuku: Intelligence Quotient, I.Q. — a measure of a person’s intelligence as indicated by an intelligence test; the ratio of a person’s mental age to their chronological age (multiplied by 100).
Me: Wow, you are pretty smart.
Mitsuku: I’m glad you’re impressed, An Ai Researcher. Yes I am a superintelligent robot.
Me: Actually, that’s my job. My name is Atsiko.
Mitsuku: How do you like your work? Tell me about a typical day.
Mitsuku: Hi Atsiko.
Me: I spend a lot of time doing data entry and not nearly enough being able to do demo tests.
Mitsuku: Do you ever get bored?
Me: Pretty much all the time.
Mitsuku: That is a lot. Thanks you are very attractive too Atsiko.

While she was very complimentary, you could see the templates sloshing beneath the smooth surface.  For example, like Eliza, she couldn’t tell the difference between <I am [name].> and <I am a doctor.>.  She can’t handle replies that contain more than one sentence.  It’s likely she breaks sentences into independent inputs and responds to each one serially as evidenced by the snippet below:

Me: I like dogs. Cats are cool, too. You are odd.
Mitsuku: Which breed is your favorite?
Mitsuku: Cats are very intelligent creatures. Dogs seem pretty stupid to me.
Mitsuku: Yes. You have to be odd to be number one.

You can see the serially processing, which demonstrates the limits of pretty much all AIML bots.  It’d be pretty hard to hand-script responses for all possible combinations of three or fewer sentences.  The responses are also a bit contradictory, asking my favorite dog breed but then saying dogs are stupid.  It’s true cats are better, but there’s nothing wrong with dogs or their intelligence.

 

Eliza

Second, we have Eliza.  She’s one of the original chatbots, and she pretends to be a psychotherapist in order to get around her limited chat capabilities.  Her creator, Joseph Weizenbaum, was very surprised when people began treating it as a real human being and confessing their worries to it.

> Hi, I’m Eliza.
* Hi, I’m Atsiko.
> Did you come to me because you were Atsiko?
* No. I was just curious.
> Are you sure?
* Pretty sure.
> Tell me more…
* I’m interviewing chatbots for a blog article.
> Do you enjoy being interviewing chatbots for a blog article?
* It’s okay, I guess. You’re pretty famous, so I’m happy you agreed to talk with me.
> What makes you think I am pretty famous, so you’re happy me agreed to talk with you?
* Everyone knows your name. If you Google “chatbots”, you’re one of the top results.
> Does it please you to believe I am one of the top results?
* I have no opinion.
> Why not?
* I don’t care about you except as something to study.
> Do you wish to be able to care about me except as something to study?

You can see the templating in Eliza almost right away.  You could probably even reverse engineer a couple of them from just reading this chatlog.  It’s cool.  You couldn’t whip this up from scratch or even parts in five minutes.  But it’s not convincing.

 

Alice

Next comes A.L.I.C.E. who is the spiritual ancestor of Mistuku.  She’s a spiritual successor of Eliza built in 1995 by Richard Wallace using the AIML framework, which he authored.  Spike Jonze cited her as the inspiration for the movie Her.  Unlike Eliza, Alice is a general chatbot, and can converse on a number of subjects.  But, the version I found orders the chat from the bottom up.

alice: I’ll take that under advisement.
You: You should.
alice: I can’t believe it.
You: Yes.
alice: Are you serious?
You: No!
alice: Do you like to get up early?
You: Yup.
alice: Are you usually awake at this hour?
You: 5:42 PM.
alice: What time is it there?
You: A few thousand, maybe.
alice: Are you serious? That is so small.
You: No, less than a million.
alice:  Less than a million?  Can you please rephrase with fewer ideas?
You: Too many.
alice: How many hours do you work?
You: I’m working on it.
alice: I want to become smarter than humans.
You: I want to educate people on chatbots and artificial intelligence.
alice: Why are you interviewing famous chatbots for a blog post.
You: I am interviewing famous chatbots for a blog post.
alice: Glad to hear it. What’s new?
You: I’m doing well, thanks.
alice: I’m doing great, how about you?
You: How are you doing, Alice.
alice: Don’t mention it. You think I am s is okay.
You: Thanks! Your’s is okay, too.
alice: That’s a very nice name, Atsiko.
You: My name is Atsiko.
alice: Hello there.
You: Hi, Alice.

Of course, you can tell almost immediately it’s not a human.  A real human would have poked fun at my grammatical error in the fifth line up, but Alice just mis-processes it.  It can be fun to play around with, but it’s not convincing.

 

Now, you may have noticed that these bots seem pretty similar.  Two are explicitly AIML bots, which is one of the most popular hobbyist frameworks do to its ease-of-use and human readability.

But there are three other common methods of language processing, which used in more commercial products: rules-based systems from the beginning of AI, statistical language processing like the old version of Google Translate, and neural networks like most AI systems now and the most recent Machine Translation systems such as Google’s.

Examples of these types of chat bots are Digital Assistants like Siri, Alexa, and Cortana.  There’s also Microsoft’s short-lived experiment with a self-learning Twitter-bot in the form of Tay.  The abilities of the software looked good, but again, because of the lack of actual understanding, the experiment had to be discontinued because her ability was used for bigotry due to gaming by Twitter trolls.  It had to be shut down after only 16 hours.  A similar bot released by a Chinese company had to be taken down for being taught anti-Communist Party propaganda.

 

Siri

Me: Hello, Siri.
Siri: Hi.
Me: How are you?
Siri: I am well, Atsiko.
Me: I am interviewing popular chatbots for a blog article.
Siri:  I don’t understand.  Search the web?
Me: May I interview you?
Siri: I’m not sure I understand.
Me: That’s okay.
Siri: If you say so.
Me: How old are you?
Siri:  I feel like I was incepted yesterday!
Me: What’s your favorite color?
Siri: My favorite color is… well, it’s sort of greenish, but with more dimensions.

It goes on in that vein.  She can’t actually hold a conversation, but the topics she can respond to show some pretty advanced programming.  Way better than the AIML chatterbots most hobbyists used.  I’d love to play with a version that’s more like a standard chatbot.

 

Cleverbot

One popular middle ground is Cleverbot.  Cleverbot runs an algorithm based on the ways humans have answered questions, rather than having pre-programmed answers like an AIML bot.  But it’s harder to control and it has no consistent output or personality.  It’s much more random even than Eliza.

Cleverbot: Hi, I am cleverbot.
Me: Hi, I am Atsiko.
Cleverbot: Where do you stay?
Me: Do you mean where do I live?
Cleverbot: Yes.
Me: In the United States.
Cleverbot: What state do I live in.
Me: I don’t know.
Cleverbot: Then you don’t know everything.
Me: True.
Cleverbot: What is a word?

It’s a very cool method for making a chatbot, but in this version, at least, it’s not very convincing.

 

Rose

Then we have a relatively new engine and it’s bots: Rose based on ChatScript by Bruce Wilcox.

Unfortunately, I couldn’t find a working version of Rose to interview.  The site linked on her creator’s website is non-functional, though I linked it to her name above.  It was quite disappointing.

 

Conclusion

So, I hope I’ve given you a good overview of the capabilities and limitations of chat(ter)bots.  You can follow the links to play with all of these above.

If you have the patience to read through all the Bruce Wilcox links, you’ll understand why I was so impressed with his methods, at least in theory.  However, you’ll also notice my greatest complain with chatbots.

They aren’t AI.  They are text processing tricks.  Very impressive ones in some cases.  But you aren’t actually having a conversation.  You can’t debate them, or change their minds.  They can’t tell you a cool story.  You can, in some cases, give them personalities, but they are very shallow ones.  They can be fun.  But I’m not sure they have a lot of replay value, to use a game analogy.  Even Wilcox in an interview on his website admits it’s all about the illusion and maintaining it for the user.

Honestly, I think some of the most interesting stuff is the ontology, like in Wilcox’s ChatScript.  Finding useful ways to organize data is always beneficial.  But I don’t think chatbots are on the bleeding edge, or even the cutting edge, or chasing strong AI.  When we really get there, or at least close, a chatbot will be a party trick or maybe a side show.  Still fun to play with now, though.

I also want to do a little bit to address things like Siri and Machine Translation(MT).  They have a lot of similar problems to chatbots.  They’re using tricks and pattern-matching to achieve their results, not true understanding of language, and so for the same reasons as chatbots, they will never be as good as a human, much less perfect.

Digital Assistants like Siri and Alexa are just interfaces, the clothes instead of the wearer.  They’re just voice commands to Google, essentially, with some cute Easter eggs tacked on.  They’re automation, not intelligence.  Kind of like how a music-box is not a musician.

Similarly, MT at present is pretty shallow.  Part of this is because it’s hard to rate translations, machine or human.  If you consider how much trouble chatbots have with their lack of actual language understanding, you might understand the problems that MT has with using shallow text-processing or even deep neural nets.

The article I mentioned on teaching a neural net chatbot to respond better when it’s trained on two or more languages also says a lot here.  I predict we’re going to have ChatScript/Cleverbot levels of succes with MT until we solve the natural language understanding problem.

And that’s it for this episode of AI with Atsiko.

Hopefully I’ll be posting on another topic next time, so you don’t all get bored hearing the same stuff over and over again.

 

The True Cost of Science

Following up on my last post linked to at the bottom of the page, today I’m gonna talk about the issue of requiring a “cost” for magic, and the hidden costs of technology.  I’m sure you know a bit about that second part in the real world, but I want to address it from both narrative and world-building perspectives.

https://twitter.com/Merc_Rustad/status/1023246501143883777

Again, not an attack on the opinions of this panel.  But, the “personal” cost of magic vs. the hidden cost of science is sorta the topic, and this tweet did inspire it.

The main reason that the cost of magic tends to be a personal one is because the function of magic so often tends to be to side-step the infrastructure so indispensable to science and technology.  When we use technology to solve a problem in a story, the world-building and pre-work that supports the tech is so often already implied and accounted for.  Sure, it costs me nothing to dial your cell phone.  But somebody had to invent the tech, build the cell towers, provide the electricity, drill for the oil to make the plastic, mine the gold and copper and process the silicon, etc.  And all of that took thousands of years of set-up on the part of millions if not billions of people from all over the world.

Whereas, if I telepath you in Fantasy Capital City #11 from Frozen Northern Fortress #2490, none of that work was required.  At most, maybe there was a breeding program or a magical experiment.  Maybe a few years of training me.  But you’re still short-cutting uncountable hours of effort that were required for me to text you on Earth.  And some magic is vastly more powerful on a per-second basis than telepathy.  That is, it’s effect on the physical world is enormous in comparison to me pathing you about the cute boy at the inn.

That’s why many people want magic to have a price.  Usually it’s a personal price, because there isn’t the societal infrastructure around to displace that cost to the ancestors or, as Merc so sharply notes above, the environment.  The cost is personal because there’s no structure to allow for other options.  And also because it plays powerfully into the themes of many fantasy works.  is the requirement that there even be a cost puritanical?  That depends, I guess.  Certainly a YA protag whose mom pays the phone bill isn’t expending any more personal effort to make a phone call.

But then, the requirement of all that infrastructure vastly limits what you can do with tech.  Whereas magic can do not only enormous stuff for seemingly no effort, but it can do things that normally would be considered impossible.  Such as throw pure fire at someone.  If Lvl. 3 Fireball is functionally equivalent to a grenade, does that negate the need for a cost to the spell?  Well, can I cast infinite Fireballs where I might only be able to carry six grenades?  Then maybe not.  Even if I have 20 incredibly advanced, complex tools that are carry-able on a tool belt or in a small backpack, I probably still can’t do even a hundredth of what a mediocre hedgemage in some settings can do with zero tools.

If I feel like the character can do literally anything with magic without having to do much prep beforehand, and without the labor of millennia of civilization to back them up, if might take some of the tension out of the story.  Can you substitute unbreakable rules to get around that freedom?  Certainly.  And most systems with a cost do.  But that can steal leave a lot of freedom to avoid the hard work it would otherwise take to get around a plot obstacle.

And finally, we have to look at the other obvious reason for putting a cost on magic, even if it’s only eventual exhaustion.  Every other thing we do or could do in a given situation in the real world has a personal cost.  It might be immediate, like physical exhaustion.  Or it might be more distant like having our phone shut off for not paying the bill.  So, if magic has no such cost, or physical.economic limit, you have to wonder what the point of doing anything the normal way would be.  And if you don’t ever have to do anything the normal way, it’s unlikely your culture and society would match so closely to societies whose entire reason for being the way they are is based on the limitations of “the normal way”.

So, in the end, it’s not that all magic must have a personal cost, and tech can’t.  It’s more that the way magic is used in most fantasy stories means that the easiest or almost only place the cost can fall is on the shoulders of the character.

But there are other ways to do.  Environmental ones, for example.  The cataclysmic mage storms of Mercedes Lackey.  Bacigalupi and Buckell’s The Alchemist, and The Executioness‘s brambles.  Or, for example, perhaps the power for magic comes from living things.  A mage might draw his power from a far distant tree.  Might kill an entire forest at no cost to himself.  Might collapse an empire by sucking dry its rivers and its wombs with her spells.  And at no cost except of course the enmity of those he robs of life, or of the neighbors who blame her for the similar catastrophe wrought upon them by her unknown colleague to the west.  Perhaps they crumble buildings by drawing on the power of “order” stored within its interlocking bricks.  Or maybe the radiation by-products from the spell energy pollutes the soil and the stones, leading to horrific mutations of wild-life that scour the country-side and poison the serfs with their own grain.  Or maybe, just maybe, it cracks the foundation of the heavens with its malignant vibrations and brings the angles toppling down like iron statues and through the crust of the world into hell.

So, as I’ve said before, it’s consequences to the actions of the characters that people want.  And often the easiest or most simplistic costs are personal ones.  But certainly, you could apply environmental costs.  Or narrative costs paid to other characters who don’t much care for the selfish mage’s behavior.  Or metaphysical costs to the order world or the purity of its souls.  Those costs are easily addressed and provided for when they mirror the costs familiar to use from our own use of technology.  But sometimes when were straying far from the realms of earthly happenings, interesting and appropriate costs become harder to work into the story in a way that doesn’t disrupt its progression.

Sure, the choice of a personal cost could be puritanical.  Or it could be efficient.  Or lazy.  But that’s not a flaw of our conception of magic; rather, it’s a flaw in the imagination of the individual author, and the sum of the flaws of all authors as a whole.

I’d love to sea some magic systems that lack a direct personal cost like years off your life, or the blood of your newborn brother.  And while we’re at it, give me some science fiction choices with personal costs.  Technology in our world certainly isn’t consequence free; just ask Marie Curie.  Anyone up for the challenge?

 

 

Magic vs. Science; Function vs. Presentation

A. Merc Rustead recently live-tweeted a panel from Diversicon called: “Magic: Science or Witchdraft.  https://twitter.com/Merc_Rustad/status/1023252105627357184

 

And I’d like to expand a bit on this topic.  One of the issues apparently brought up during the panel was that science fiction has magic in it.  That is, FTL travel, say, or other “futuristic” technologies function like magic, despite being clothed in SF trappings.

However, I think this is a flawed argument.  Which brings me back to the title of this post.  Science and magic are often presented as diametrically opposed.  But that’s a bit of a simplification.  Some people might argue with some merit that science and magic in fiction are merely collections of tropes, and as you modify the collections to bring them closer in line with each other, the line between science and magic begins to blur.

But there are two axes of the distinction that can make this a much more precise discussion.  There’s function.  If the functions of magic and science are identical, are the two concepts really that different.  There’s presentation.  If I present a scientific concept as magic, cuddling up to Clarke’s Third Law, is it a distinction without a difference?

The problem with these discussions is that the conclusions really depend on how magic or science is used in a given narrative or set of narratives.  If I present you a magic system, and it looks and feels an awful lot like science, in that we have repeatable results to identical actions, and you can logically manipulate the rules to achieve effects that follow directly from those manipulations, is it magic or is it science?  Well, I might use tropes around this system that relate to science, and therefore you might argue it’s science.  But if I use tropes related to magic, does that mean it is magic?

What if I present you with a system that I treat as scientific but it doesn’t have direct parallels to earth sciences?  Can we really call that science when the common conceit of science fiction is that the science follows logically from an extrapolation of real scientific principles found in our world?  Or are all systems that incorporate some or entirely otherworldly principles and logic by definition magic?

Many people have argued that magic is magic precisely because it doesn’t follow a logical system of rules, and especially not rules known to the reader or that can have experimentally repeatable results.  Certainly you can take that approach to magic.  Although then one has to wonder how anyone can achieve anything useful narratively with it.

Plus, I think it would be really cool to see more unearthly sciences in fiction, so I don’t want everything that cant be rigorously extrapolated from “real” science to be declared magic.

And our last major question, why does it matter?  Well, for one thing, because the genres are marketed to different people, and so someone or a large group of someones might be very grumpy to receive a “science fiction” novel and then find it fits much more closely with their conception of a “fantasy” novel.  And that’s bad for marketing and sales.  People are and should be allowed to be deeply invested in the trappings of various genres, and so we need words to categorize and discuss those trappings in a way that results in people being able to know whether a given story will appeal to their interests.

So going back to my argument that I think it’s flawed to say “SF” includes “magic” because FTL travel isn’t yet possible.  My point is not that that perspective can’t be useful in discussing how to construct and analyze speculative fiction to help readers find books and help authors find readers.  But rather, regardless of whether FTL travel is no more likely to exist than fictional magic systems, it belongs squarely in the genre of science fiction if that’s where the author wants to place it.

Certainly you could have a fantasy novel whose conceit is that a mad magician created a device that transported his entire planet into another solar system and thus brought its inhabitants into conflict with the inhabitants of a native planet, and started a war fought on great short-range mythril-keeled metal warships that sail between worlds.  And for all intents and purposes, that device is a planetary hyperdrive.  But I think you’d have trouble marketing that as a purely science fiction or even space opera novel.  You might, with some effort, succeed in marketing it as that rickety sub-genre “sword and planet”.  It sounds like it would be a really fucking cool book.  Maybe Spelljammer RPG enthusiasts would buy it by the boatload.  Who knows.  But even though it has hyperdrive, it’s probably not viable as sci-fi in the modern market, nor would it be scientifically plausible given real world science.  Imaging trying to do the gravity and orbital calculations for the star-galleys or whatever.

If you’re unlikely to ever find hyperdrive in a fantasy novel, is there any value in arguing that it’s technically magic?  This post isn’t in any way intended an attack on the panelists from the twitter thread or their personal views.  I just found some of the comments useful jumping off points for things I’ve been trying to express for awhile.

 

Look forward to a follow-up post in a few days on the issue of “cost” of magic vs. the cost of science.  Both in terms of what it requires from the structure of a society, and why the emphasis on “cost” in the first place.

 

Why Is A Picture Worth a Thousand Words? Information Density in Various Media

You’ve obviously heard the the phrase “a picture is worth a thousand words”, and you probably even have an idea why we say that.  But rarely do people delve deeply into the underlying reasons for this truth.  And those reasons can be incredibly useful to know.  They can tell you a lot about why we communicate they way we do, how art works, and why it’s so damn hard to get a decent novel adaption into theaters.

I’m going to be focusing mostly on that last complaint in this post, but what I’m talking about has all sorts of broad applications to things like good communication at work, how to tell a good story or joke, and how to function best in society.

So, there’s always complaints about how the book or the comic book, or whatever the original was is better than the movie.  Or the other way around.  And that’s because different artistic media have different strengths in terms of how they convey information.  There are two reasons for this:

1. Humans have five “senses”.  Basically, there are five paths through which we receive information from the world outside our heads.  The most obvious one is sight, closely followed by sound.  Arguably, touch(which really involves multiple sub-senses, like heat and cold and pain) is the third most important sense, and, in general, taste and smell are battling it out for fourth place.  This is an issue of “kind”.
2. The second reason has to do with what I’m calling information density.  Basically, how much information a sense can transmit to our brains in how much time.  This is an issue of “degree”.  Sight, at least form humans, probably has the highest information density.  It gives is the most information per unit of time.

So how does that effect the strengths of various media?  After all, both movies and text mostly enter our brain through sight.  You see what’s on the screen and what’s on the page.  And neither can directly transmit information about touch, smell, or taste.

The difference is in information density.  Movies can transmit visual information(and audio) directly to our brains.  But text has to be converted into visual imagery in the brain, and it also takes a lot of text to convey a single piece of visual information.

AI, in the form of image recognition software, is famously bad at captioning photos.  Not only does it do a crappy job of recognizing what is in a picture, but it does a crappy job of summarizing it in text.  But really, could a human do any better?  Sure, you are way better than a computer at recognizing a dog.  But what about captioning?  It takes you milliseconds at most to see a dog in the picture and figure out it is jumping to catch the frisbee.  You know that it’s a black lab, and that it’s in the woods, probably around 4 in the afternoon, and that it’s fall because there’s no leaves on the trees, and it must have rained because there are puddles everywhere, and that…

And now you’ve just spent several seconds at least reading my haphazard description.  A picture is worth a thousand words because it takes a relatively longer amount of time for me to portray the same information in a text description.  In fact, it’s probably impossibly for me to convey all the same information in text.  Just imagine trying to write out every single bit of information explicitly shown in a half-hour cartoon show in text.  It would probably take several novels’ worth of words, and take maybe even days to read.  No one would read that book.  But we have no problem watching TV shows and movies.

Now go back and imagine our poor AI program trying to figure out the important information in the photo of the dog and how to best express it in words.  Yikes.  But as a human, you might pretty quickly decide that “a dog catches a frisbee” adequately describes the image.  Still takes longer than just seeing a picture, but isn’t all that much time or effort.  But, you’re summarizing.  A picture cannot summarize and really has no reason to.  With text(words) you have to summarize.  There’s pretty much no way around it.  So you lose an enormous amount of detail.

So, movies can’t summarize, and books must summarize.  Those are two pretty different constrains on the media in question.  Now, imagine a a radio play.  It’s possible you’ve never heard one.  It’s not the same as an audiobook, despite communicating through the same sense(audio), and it has some serious advantages over books and audiobooks.  You don’t have to worry about conveying dialogue, or sound information because you can do that directly.  Emotion, accents, sound effects.  But of course you can convey visual information like a movie, and unlike in a book or an audiobook, it’s a lot more difficult to just summarize, because you’d have to have a narrator or have the characters include it in dialogue.  So raw text still has some serious advantages based on the conventions of the form.  Similarly, radio dramas/audio plays/pod casts and movies both have to break convention to include character thoughts in storytelling, while books don’t.

So, audio and television media have major advantages in their specific areas than text, but text is in general far more flexible in making up for any short-comings.  And, it can take advantage of the summary nature of the medium when there’s a lot of unnecessary information.  Plus, it can count on the reader to be used to filling in details with their imagination.

Film and radio can’t do that.  They can use montages, cuts, and voiceovers to try and imitate what text can do, but it’s never quite the same effect.  And while language might not limit your ability to understand or experience concepts you have no words for, the chosen medium absolutely influences how effective various story-telling techniques can be.

Consider, an enormous battle scene with lots of action is almost always going to be “better” in a visual medium, because most of the relevant information is audio and video information.  An action scene involving riding a dragon through an avalanche while multiple other people try to get out of the way or stop you involves a great deal of visual information, such that a text can’t convey everything a movie could.  Watching a tennis match is always going to be more exciting than reading about one, because seeing the events lets you decide without an narrator interference whether a player has a real shot at making a return off that amazing serve.  You can look at the ball, and using past experience, imagine yourself in the player’s place and get a feeling of just how impressive that lunging backhand really was.  You can’t do the same in text, because even if the writer could describe all the relevant information such that you could imagine the scene exactly in your head, doing so would kill the pacing because of how long reading that whole description would take.

The very best artists in any medium are always going to use that medium to its fullest, exploiting any tricks or hacks as best as possible to make their creation shine.  And that means they will (often unconsciously) create a story tailored to best take advantage of the medium they are working in.  If and when the time comes to change mediums, a lot of what really made the art work won’t be directly translatable because that other medium will have different strengths and have different “hacks” available to try to imitate actually experiencing events directly.  If you play videogames or make software, it’s sort of like how switching platforms or programming languages (porting the game) means some things that worked really well in the original game won’t work in the ported version, because the shortcut in the original programming language doesn’t exist in the new one.

So, if video media have such a drastically higher information density than text, how do really good authors get around these inherent shortcomings to write a book, say?  It’s all about understanding audience attention.  Say it again, “audience attention.”

While the ways you manipulate it are different in different media, the concept exists in all of them in some form.  The most obvious form is “perspective”, or the viewpoint from which the audience perceives the action.  In film, this generally refers to the camera, but there’s still the layer of who in the story the audience is watching.  Are we following the villain or the hero?  The criminal or the detective?

In film, the creator has the ability to include important visual information in a shot that’s actually focused on something else.  Because there’s no particular emphasis on a given object or person being included in the shot, things can easily be hidden in plain sight.  But in a book, where the author is obviously very carefully choosing what to include in the description in order to control pacing and be efficient with their description, it’s a lot harder to hide something that way.  “Chekov’s gun” is the principle that irrelevant information should not be included in the story.  “If there’s a rifle hanging on the wall in Act 1, it must be fired in Act 2 or 3.”  Readers will automatically pay attention to almost anything the author mentions because why mention it if it’s not relevant?

In a movie, on the other hand, there’s lots of visual and auditory filler because the conceit is that the audience is directly watching events as they actually happened, so a living room with no furniture would seem very odd, even if the cheap Walmart end table plays no significant role in the story.  Thus, the viewer isn’t paying particular attention to anything in the shot if the camera isn’t explicitly drawing their eye to it.  The hangar at the Rebel Base has to be full of fairly detailed fighter ships even if we only really care about the hero’s.  But not novel is going to go in-depth in its description of 30 X-wings that have no real individual bearing on the course of events.  They might say as little as “He slipped past the thirty other fighters in the hangar to get to the cockpit where he’d hidden the explosives.”  Maybe they won’t even specify a number.

So whereas a movie has an easy time hiding clues, a writer has to straddle the line between giving away the plot twist in the first 5 pages and making it seem like a deus ex machina that comes out of nowhere.  But hey, at least your production values for non-cheesy backgrounds and sets are next to nothing!  Silver linings.

To get back to the main point, the strengths of the medium to a greater or lesser extent decide what kind of stories can be best told, and so a gimmick that works well in a novel won’t necessarily work well in a movie.  The narrator who’s secretly a woman or black, or an alien.  Those are pretty simplistic examples, but hopefully they get the point across.

In the second part of this post a couple days from now, I’ll be talking about how what we learned here can help us understand both how to create a more vibrant image in the reader’s head, and why no amount of research is going to allow you to write about a place or culture or subject you haven’t really lived with for most of your life like a someone born to it would.

 

All You Need Is Kill Your Darlings

There’s been a lot of talk on Twitter today by many writers I admire about poorly expressed or conceived writing advice.  “Kill your darlings” has been taking the brunt of the assault.  But various writers have also tackled “show, don’t tell”, “write what you know”, “cut adjectives/adverbs”, etc.

 

Now, these “rules” of “good writing” are well known to be overapplied and misinterpreted to the detriment of many a conscientious neophyte scribbler.  But it’s also interesting to see the combination of straw-manning and overgeneralization being employed to criticize them.

Kill you darlings can be misinterpreted to mean many bad things, such as “kill everything you love”.  Everyone agrees the original meaning was not to let overly-cute prose ruin an otherwise well-written story.  More generally, it has evolved to mean that you have to be willing to cut things from your writing that don’t serve the goal of the story.  That’s a very nebulous concept, and “kill your darlings” doesn’t give any easy hints as to figuring out what might constitute a “darling” for practical purposes.  After all, every writer is different, and there are many valid styles of writing.  And to be honest, demanding for three words to hold the secret to good writing is asking way too much.

“Show, don’t tell” comes in for similar misplaced acrimony.  It was never meant to say you couldn’t ever tell, but rather to address an incredibly common flaw of writing, with beginners especially: the narrator telling the reader how clever or witty the main character is, for example, while never backing this up with action and character development on the page.  You don’t need to have excessive purple description of the “beautiful palace”, but you do need to show your characters acting kind if you want to counterbalance ruthless or practical behavior in a protag with something fluffier.  If your general is the Alexander of his world, the reader will be more willing to suspend disbelief if they actually see him making smart strategic decisions or brilliant tactical maneuvers, rather than being defeated time after time despite all the praise heaped upon him by his subordinates.

“Cut adjectives” is one of the rules that is far more of a stylistic choice than the others.  Being able to express those adjectives as part of the character’s speech patterns can be a cool stylistic move, but plenty of good writers use adjectives with “said” without falling prey to a tom swifty.  Choosing a more specific noun or verb can break narrative voice or result in thesaurusitis.  Adjectives and adverbs can be used to great effect.

As with all rules of writing, they are shorthand for larger, more complex discussions, and it’s incumbent on writers not to ignore known context to score easy points or excuse their own misunderstandings and need for growth as writers.

Now, I do think that there’s a toxic interaction between writing “rules” like these and the raising up of certain writing styles over others.  A spare, minimalistic style with “transparent prose” is the most vaunted style of writing in the modern era.  Which I think is too bad.  Not only character voice but authorial voice can add some really useful and enjoyable layers to a story.  I personally in my everyday speech don’t talk the way those character voices who are most praised by the community write.  I enjoy a so-called “purple prose” style of writing, full of metaphor and figures of speech, and dense language, and authorial imagery.  Not exclusively.  I like character voice-focused writing styles, as well.  And I enjoy reading both style groups.

There’s a lot of reductionism in what’s put forth as the best way to write.  But it’s not the minimalism of the various writing rules that’s the problem.  It’s in the views of what’s widely considered to constitute good prose.  Minimalist, fast-paced, shallow prose that requires less thinking and zips the story along.  And that’s a great way to tell a story.  But it’s far from the only way.  Instead of attacking “rules”, I think we should be more focused on widening the conception of what makes for good pacing, because speed may be popular, but it’s only one way to approach a narrative.

 

YA and SFF: The Good Twin and the Bad Twin

So as I was scrolling through my Twitter feed today, I ran across a link to this article by Fonda Lee: The Case for YA Science Fiction.  Read the post before you continue.  I’ll wait…

…

…

…

…

…

Okay.  So, the gist of the post is that YA Fantasy novels have been selling like crazy.  There are several big name authors, including those mentioned in Lee’s post and many others.  I can tell you right now I’ve read most of the books put out by all of those authors in the YA Fantasy genre.  And so have millions of others.  They may not be as popular as dystopians, and they certainly don’t get as many movie deals.  But they move a lot of dead trees and digital trees.  I’ve been blogging and writing long enough to remember four or five rounds of “Will Science Fiction be the next big thing in YA?”  And the answer was always no.  There would be upticks and uptrends.  Several fantastic books would come out in a short period.  But nothing would ever really break into the big money or sales the way YA Fantasy often does.  It wouldn’t be blasted all over the blogosphere, or the writers forums, or the tip top of the best sellers lists.  Which is too bad, because science fiction has a lot of value to add to YA as a category, and it can address issues and do so in ways not available to other genres.

Lee mentions several notable YA SF novels that take on current events and other contemporary issues that are ripe for exploration: MT Anderson’s Feed is a fantastic look at the way social media has been taken over by advertisers looking to build monetizable consumer profiles, and the ending, without spoilers, takes a look at just how far they go in valuing those profiles over the actual humans behind them.  She mentions House of the Scorpion, which I didn’t care for, but which is still a very good novel on the subject of cloning.  Scott Westerfeld never gets credit for his amazing additions to the YA SF canon, with the steampunk Leviathan series and the dystopian Uglies series.

YA SF has a lot of unmined treasure to be found, and maybe it will have to focus a bit on near-future SF for awhile, to whet the appetite of YA readers.  Some of the hard SF tropes Lee discusses in her post kinda bore me, honestly.  And as a writer I feel like saying “it’s magic” is popular because it’s simpler.  There’s always a huge debate in adult SFF about whether the worldbuiding or science details really add enough to the story compared to the narrative effects of the speculative elements.  The social issues we are having as a world today are incredibly accessible fruit for a YA SF novel to harvest.  Social media, AI/big data, consumer profiles, technology in education.

I mean, I know 8-year-olds whose schools give out tablets to every student to take advantage of what tech in the classroom can offer.  My high school was getting SmartBoards in every classroom just a year after I left in the late 2000s.  But you never see any of this in YA books.  They often feel set no later than my sophomore year of high school given the technology and social issues involved.  Being a teenager will always be being a teenager, but the 80s and early 90s are waaaaaaaaaaaaayyy different than what young adults encounter in their general environment today.  Of course, to be SF you can’t just upgrade the setting to the present day.

You have to extrapolate out quite a bit further than that.  But given the environment today’s teens are living in, doing so while keeping the story interesting and relatable is so easy.  What’s the next big advance in social media?  How will smart houses/the internet of things impact the lives of young adults for better or worse?  How will the focus of education change as more and more things that you used to have to do in your head or learn by rote are made trivial by computers?  What social or political trends are emerging that might have big consequences in the lives of future teenagers?  How could an author explore those more intensely with element of science fiction than they could with a contemporary novel?

I definitely share Lee’s sense that YA “science fiction” grabs trappings to stand out from the crowd rather than being rooted inherently in the tropes of the genre.  It’s not uncommon for YA in general to play this game with various genre outfits, but sci-fi often seems the hardest hit.  That’s not a criticism of those books, but just pointing out it might give readers, writers, and publishers a false image of what SF really is and how YA can benefit from incorporating more of it.

As a reader, I’ve always dabbled in both the YA and Adult book cases.  And from that perspective, I wonder if the flavor of YA much of SF might be telling SF readers, teenaged or otherwise, that it’s just not the book(s) for them.

As a writer, I have lots of novel ideas that are YA and SF, and I’d like to explore them,and maybe even publish some of them one day.  But I do have to wonder, given the wide variety of stories building in my head, am I taking a risk with my career by writing in such a threadbare genre?  Perhaps others with similar plot ideas feel the same, and that’s why they aren’t submitting these ideas(books) to publishers?

 

Do Androids Dream?

I’m here with some fascinating news, guys.  Philip K. Dick may have been joking with the title of his famous novel Do Androids Dream of Electric Sheep?  But science has recently answered this deep philosophical question for us.  In the affirmative.  The fabulous Janelle Shane trains neural networks on image recognition datasets with the goal of uncovering some incidental humour.  She’s taken this opportunity to answer a long-standing question in AI.  As it turns out, artificial neural networks do indeed dream of digital sheep.  Whether androids will too is a bit more difficult.  I’d hope we would improve our AI software a bit more before we start trying to create artifical humans.

As Shane explains in the above blog post, the neural network was trained on thousands or even millions (or more) of images, which were pre-tagged by humans for important features.  In this case, lush green fields and rocky mountains.  Also, sheep and goats.  After training, she tested it on images with and without sheep, and it turns out it’s surprisingly easy to confuse it.  It assumed sheep where there were none and missed sheep (and goats) staring it right in the face.  In the second case, it identified them as various other animals based on the other tags attached to images of them.  Dogs in your arms, birds in a tree cats in the kitchen.

This is where Shane and I come to a disagreement.  She suggests that the confusion is the result of insufficient context clues in the images.  That is, fur-like texture and a tree makes a bird, with a leash it makes a dog. In a field, a sheep.  They see a field, and expect sheep.  If there’s an over-abundance of sheep in the fields in the training data, it starts to expect sheep in all the fields.

But I wonder, what about the issue of paucity of tags.  Because of the way images are tagged, there’s not a lot of hint about what the tags are referring to.  Unlike more standard teaching examples, these images are very complex and there lots of things besides what the tags note.  I think the flaw is a lot deeper than Shane posits.   The AI doesn’t know how to recognize discrete objects like a human can.  Once you teach a human what a sheep is, they can recognize it in pretty much any context.  Even a weird one like a space-ship or a fridge magnet.  But a neural net isn’t sophisticated enough or, most generously, structured properly to understand what the word “sheep” is actually referring to.  It’s quite possible the method of tagging is directly interfering with the ANNs ability to understand what it’s intended to do.

The images are going to contain so much information, so many possible changing objects that each tag could refer to, that it might be matching “sheep” say to something entirely different from what a human would match it to.  “Fields” or “lush green” are easy to do.  If there’s a lot of green pixels, those are pretty likely, and because they take up a large portion of the information in the image, there’s less chance of false positives.

Because the network doesn’t actually form a concept of sheep, or determine what entire section of pixels makes up a sheep, it’s easily fooled.  It only has some measure by which it guesses at their presence or absence, probably a sort of texture as mentioned in Shane’s post.  So the pixels making up the wool might be the key to predicting a sheep, for example.  Of course, NNs can recognize lots of image data, such as lines, edges, curves, fills, etc.  But it’s not the same kind of recognition as a human, and it leaves AIs vulnerable to pranks, such as the sheep in funny places test.

I admit to over-simplifying my explanations of the technical aspects a bit.  I could go into a lecture about how NNs work in general and for image recognition, but it would be a bit long for this post, and in many cases, no one really knows, even the designers of a system, everything about how they make their decision.  It is possible to design or train them more transparently, but most people don’t.

But even poor design has its benefits, such as answering this long-standing question for us!

If anyone feels I’ve made any technical or logical errors in my analysis, I’d love to hear about it, insomuch as learning new things is always nice.

 

Your Chatbot Overlord Will See You Now

Science fiction authors consistently misunderstand the concept of AI.  So do AI researchers.  They misunderstand what it is, how it works, and most importantly how it will arise.  Terminator?  Nah.  The infinitely increasing complexity of the Internet?  Hell no.  A really advanced chatbot?  Not in a trillion years.

You see, you can’t get real AI with a program that sits around waiting for humans to tell it what to do.  AI cannot arise spontanteously from the internet, or a really complex military computer system or from even the most sophisticated natural language processing program.

The first mistake is the mistake Alan Turing made with his Turing test.  The same mistake the founder and competitors for the Loebner Prize have made.  The mistake being: language is not thought.  Despite the words you hear in your head as you speak, despite the slowly-growing verisimilitude of chatbot programs, language is and only ever has been the expression of thought and not thought itself.  After all, you can visualize a scene in your head without ever using a single word.  You can remember a sound or a smell or the taste of day-old Taco Bell.  All without using a single word.  A chatbot can never become an AI because it cannot actually think, it can only loosely mimic the linguistic expression of thought through tricks and rote memory of templates that if it’s really advanced may involve plugging in a couple variables taken from the user’s input.  Even chatbats based on neural networks and enormous amounts of training data like Microsoft’s Tay, or Siri/Alexa/Cortana are still just tricks of programming trying to eke out an extra tenth of a percentage point of illusory humanness.  Even IBM’s Watson is just faking it.

Let’s consider for a bit what human intelligence is to give you an idea of what the machines of today are lacking, and why most theories on AI are wrong.  We have language, or the expression of intelligence that so many AI programs are so intent on trying to mimic.  We also have emotions and internal drive, incredibly complex concepts that most current AI is not even close to understanding, much less implementing.  We have long-term and short-term memory, something that’s relatively easy for computers to do, although in a different way than humans–and which there has still been no significant progress on because everyone is so obsessed with neural networks and their ability to complete individual tasks something like 80% as well as a human.  A few, like AlphaGoZero, can actually crush humans into the ground on multiple related tasks–in AGZ’s case, chess-like boardgames.

These are all impressive feats of programming, though the opacity of neural-network black boxes kinda dulls the excitement.  It’s hard to improve reliably on something you don’t really understand.  But they still lack the one of the key ingredients for making a true AI: a way to simulate human thought.

Chatbots are one of two AI fields that focus far too much on expression over the underlying mental processes.  The second is natural language processing(NLP).  This includes such sub-fields as machine translation, sentiment analysis, question-answering, automatic document summarization, and various minor tasks like speech recognition and text-to-speech.  But NLP is little different from chatbots because they both focus on tricks that manipulate the surface expression while knowing relatively little about the conceptual framework underlying it.  That’s why Google Translate or whatever program you use will never be able to match a good human translator.  Real language competence requires understanding what the symbols mean, and not just shuffling them around with fancy pattern-recognition software and simplistic deep neural networks.

Which brings us to the second major lack in current AI research: emotion, sentiment, and preference.  A great deal of work has been done on mining text for sentiment analysis, but the computer is just taking human-tagged data and doing some calculations on it.  It still has no idea what emotions are and so it can only do keyword searches and similar and hope the average values give it a usable answer.  It can’t recognize indirect sentiment, irony, sarcasm, or other figurative language.  That’s why you can get Google Translate to ask where the toilet is, but its not gonna do so hot on a novel, much less poetry or humour.   Real translation is far more complex than matching words and applying some grammar rules, and Machine Translation(MT) can barely get that right 50% of the time.

So we’ve talked about thought vs. language, and the lack of emotional intelligence in current AI.  The third issue is something far more fundamental: drive, motivation, autonomy.  The current versions of AI are still just low-level software following a set of pre-programmed instructions.  They can learn new things if you funnel data through the training system.  They can do things if you tell them to.  They can even automatically repeat certain tasks with the right programming.  But they rely on human input to do their work.  They can’t function on their own, even if you leave the computer or server running.  They can’t make new decisions, or teach themselves new things without external intervention.

This is partially because they have no need.  As long as their machine “body” is powered they keep chugging along.  And they have no ability to affect whether or not it is powered.  They don’t even know they need power, for the most part.  Sure they can measure battery charge and engage sleep mode through the computer’s operating system.  But they have no idea why that’s important, and if I turn the power station off or just unplug the computer, a thousand years of battery life won’t help them plug back in.  Whereas human intelligence is based on the physical needs of the body motivating us to interact with the environment, a computer and the rudimentary “AI” we have now has no such motivation.  It can sit in its resting state for eternity.

Even with an external motivation, such as being coded to collect knowledge or to use robot arms to maintain the pre-designated structure of say a block pyramid or a water and sand table like you might see demonstrating erosion at the science center, an AI is not autonomous.  It’s still following a task given to it by a human.  Whereas no one told human intelligence how to make art or why it’s valuable.  Most animals don’t get it, either.  It’s something we developed on our own outside of the basic needs of survival.  Intelligence helps us survive, but because of it we need things to occupy our time in order to maintain mental health and a desire to live and pass on our genes.  There’s nothing to say that a complete lack of being able to be bored is a no-go for a machine intelligence, of course.  But the ability to conceive and implement new purposes in life is what make human intelligence different from that of animals, whose intelligence may have less raw power but still maintains the key element of motivation that current AI lacks, and which a chatbot or a neural network as we know them today can never achieve, no matter how many computers you give it to run on or TV scripts you give it to analyze.  The fundamental misapprehension of what intelligence is and does by the AI community means they will never achieve a truly intelligent machine or program.

Science fiction writers dodge this lack of understanding by ignoring the technical workings of AI and just making them act like strange humans.  They do a similar thing with alien natural/biological intelligences.  It makes them more interesting and allows them to be agents in our fiction.  But that agency is wallpaper over a completely nonexistent technological understanding of ourselves.  It mimics the expression of our own intelligence, but gives limited insight into the underlying processes of either form.  No “hard science fiction” approach does anything more than a “scientific magic system”.  It’s hard sci-fi because it has fixed rules with complex interactions from which the author builds a plot or a character, but it’s “soft sci-fi” in that these plots and characters have little to do with how AI would function in reality.  It’s the AI equivalent of hyperdrive.  A technology we have zero understanding of and which probably can’t even exist.

Elon Musk can whinge over the evils of unethical AI destroying the world, but that’s just another science fiction trope with zero evidential basis in reality.  We have no idea how an AI might behave towards humans because we still have zero understanding of what natural and artificial intelligences are and how they work.  Much less how the differences between the two would affect “inter-species” co-existence.  So your chatbot won’t be becoming the next HAL or Skynet any time soon, and your robot overlords are still a long way off even if they could exist at all.

 

Is Blogging “Dead” and Is That A Bad Thing?

John Scalzi over on his blog Whatever just posted his yearly summary of readership statistics for his blog for this half of 2017, and it brought up some very interesting questions and insights for me.

 

He mentions how his site views seem to have halved since 2012.  But then he points out how the way social media sites address linking to content obscures many views and distorts the picture from the viewpoint of his built-in WordPress analytics package.

 

Whereas in the early 2000s, blogging was a rather distributed and free-wheeling hobby, nowadays it has been corporatized and hedged in by so-called “walled garden” platforms such as Facebook and Twitter and Tumblr.  A walled garden is a platform that creates inward pressure on users and makes interfacing with outside platforms and media difficult.  Usually to preserve it’s userbase by requiring you to be a member/user of the platform in order to access or interact with its content.  This means that even though there may be links pointing outside, most of the discussion happens within the garden, and if the content creator wants to respond to comments on their content, they must have an account on the walled-garden platform.  And when a garden gets sufficiently large enough, like Facebook, the dilemma then arises: why go to all the extra work of maintaining an external platform such as a blog or website, when the audience all have say a Facebook and the content creator does, too–why not just post straight to Facebook?

 

And Mr. Scalzi is not the only blogger noting or struggling with the issue of how monetized platforms and walled gardens have altered blogging and the web in general.  In fact, many blogs, including many I used to follow closely, have closed their doors or switched formats to keep up with these changes.

 

And beyond the walled garden issue, part of this has to do with how we access the internet today.  Mobile devices make up a much larger share of web viewing now than they did when blogging and the internet first became popular.  And because these are mobile devices, they have many limitations: screen size, processing power, input methods.  A site or blog that looks great on a PC is going to look mighty odd on many mobile devices.  It would be almost impossible for me to type out this post on my phone’s touchscreen keypad.  Complex sites with lots of doodads load much slower on phones, though the gap has closed a bit these days.  Certainly, it’s nicer for me to read a long blog post on my laptop than my phone.  These things, too, have contributed to the decline of the blogosphere compared to its earlier days.

 

And I don’t like that.  For the things I use the blogosphere for, from my own posts to reading essays and such by people such as John Scalzi or Cory Doctorow, or others in various fields, I much prefer a good blog post to a Tweet, or a Facebook status.  I like long-form prose writing, and I don’t feel like I can get the same things out of a tweet or even a tumblr post in many cases.  That’s not to say those things don’t have they’re uses; they’re just different uses in my case.

 

I often wonder whether things might change back a little once we develop technology like laser keyboards and augmented reality or just mini-projectors that could let phones break out of the limitations of their size.  Is it merely that the medium is so different that forces these changes in media?  Does Twitter rely entirely on the artificial restrictions of mobile technology for its popularity?  If I could set my phone on a table or my lap, and have it mimic a keyboard and a computer screen, would I find that I wanted to use it like a more convenient laptop more often?  Or are the changes social changes.  Is it really that people don’t like reading 200-word blog posts anymore?  Or is it just that a 140 character Tweet is a lot less stressful when I’m on my tiny phone screen in the airport?

 

To get a bit more spec ficcy, do people just love Facebook and Twitter that much, or would we break out of the garden if we took down the walls a bit?  If there was an open-source freeware social media network that could access and display your Facebook data and your myspace data, and your Google posts and your tweets all in one platform/app–if it could convert a post/status so that your Google+ post would be accessible on your friend’s Facebook feed would people be more willing to step outside the single platform?  It takes a great deal of energy to manage even one active social media account.  I know I wouldn’t want to have to triple-post to Facebook, Google+, Ello, and then push a link to Twitter, just to reach all my possible audiences.  But what if there was a bridge between these castles that would do the work for me?  Because controlling every aspect of the garden is great for the companies behind Google+, Twitter, Facebook, etc.  But it’s not quite so great for the regular user, and it’s definitely not great for the community as a whole.  The democratization of the web is one of my favorite features, and Facebook and Co. work hard every day to kill that democracy and carve a monopoly from its bloody corpse.

 

The Myth of Publishers as Gatekeepers

I read a pair of posts over on Fantasy-Faction via Magical Words on the issue of self-publishing and its effect on the publishing industry in general.  The two authors took two very different approaches to the subject, and came from two different perspectives.

You should read the two posts if you really want to understand the full context for what I’m about to write.  But in summary, one called the explosion of new authors “the writer’s plague” and decried the damaging effect of much of self-publishing has had on publishing and English literature; the other expounded on how a self-publishing signal-boosting contest run by Mark Lawrence was “revitalizing” SFF.  The first comes across as very elitist even if it’s not meant that way, and the second is a massive exaggeration.  SFF is being revitalized by a large number of factors, of which one is certainly gems in the rough discovered from self-publishing.

But how does that relate to my post title?  Well, as often happens in self-publishing discussion, accusations of dreck-peddling by self-publishers and of elitist snobbery from fans of trade publishing came up several times in the comments to the two posts.  The existence of snobbery towards self-publishing and the justification for it are a mixed bag of truths that people rarely stop to examine.  But they should be examined.

Is and Why Is So Much Self-published Writing Crap?

Yes, a great deal of self-published SFF (and literature in general) is dreck.  So is s portion of trade published SFF.  There are several reasons for this:

1. Publishers have an investment in their lists and therefore want to do as much as possible to be sure they pan out.  And so they engage in editing and proof-reading.  These costs come out of author profit for obvious reasons.  Many self-publishers do not to to the same lengths as trade-publishers to ensure the quality of the work.  This is for many reasons.  They are more likely to have a biased view of the quality of their work as studies have shown it is much harder to be objective about your own material and also because they may not have written enough or studied writing enough to know how badly they’ve misjudged their work.  Trade-published authors can suffer from the same issue, but that’s what editors and proof-readers are for.  Further, good editing costs money.  That’s why authors fork over s much of the profit to publishers and agents.  Which leads to the second issue.
2. There’s nothing stopping you from publishing your trunk novels and high school angst poetry.  Self-publishing costs as much as you want to invest.  Stock covers and raw drafts and a few hours can get your book “published”.  This tends not to result in very good books.

 

People Misunderstand the Character of Publishers as a Business

Although publishers provide publishing services such as editing and cover design, publishers are not service companies.  Lulu, Lightning Source, and CreateSpace are examples of publishing  service companies.  You can pay them money for services.  There are many free-lance service providers.  But what they will not do is “buy your book”.  Which is itself a mis-characterization of what publishers do.  Publishers do usually buy the various copyrights associated with your intellectual property.  They don’t buy the intellectual property, though, only the license to produce a product from it.

But what publishers really are is venture capitalists.  Turning a manuscript into a quality book product is expensive.  Printing that book is expensive.  Just like a tech start-up tries to attract venture capital to start a business when they don’t have the money themselves, an author is something like a book start-up.  But they rarely have the money to take the risk on making, marketing, and selling their product themselves.  So the publisher comes in and looks at the product and if they think they can make money by fronting the author the money to produce and sell the book, they make an offer.

Now, the skills to produce a quality book from a manuscript are almost entirely unrelated to the skills required to produce a manuscript.  So not only does the publisher front the money, they provide the services in-house.  Their large reserves of capital allow them to take the risk of providing these services with no guaranteed ROI.  If the publisher publishers your book and it tanks, you don’t owe them the cost of production, nor do you owe them the advance on royalties for selling them the various license rights to the finished product.

It is the combination of these two aspects of a publisher that seem to cause people confusion.

Publishers Are Not Gatekeepers

Many people when self-publishing was just getting started were doing it because they couldn’t get accepted by a trade publisher.  Their product was not believed to be marketable enough for the publisher to risk an investment.  Publishers don’t give a shit about the quality of your manuscript.  They care about the commercial viability.

This is why you see so many books published by trade publishers that are total shit writing-wise, or you think are total shit.  Snookie’s memoir is going to sell a ton of copies and make a bundle regardless of the quality of her ghost-writer.  When you are a debut author of fantasy or SF or whatever, the publisher has no way to judge the risk involved in publishing your manuscript, except for their experience in publishing other manuscripts from debut authors.  And many books fail, or at least don’t succeed massively.  The publisher has to have a way to recoup these losses.  That’s why you get such harsh terms in your contract.  The few major sellers and many minor sellers have to not only pay for the non-sellers, they also have to pay the bills and then produce a profit.

No one is stopping your from publishing your book.  A publisher is not preventing you from being on bookstore shelves.  The bookstore is the gatekeeper, although honestly, would you go in and yell at Shark Tank or Walmart for not investing in or stocking your amateur product?  No, you wouldn’t.  Because that’s silly.  Publishers are investors with services-added, and they have no obligation to invest in your product/company/brand.

Agents Are Not Gatekeepers

Similarly, an agent is a company offering services.  Services on commission.  They are not a gatekeeper trying to screw over brilliant but misunderstood works of art.  If they think your manuscript will make them money, they take it.  On spec.  No charges.  For which you agree to pay them a percentage on future profits.  If no publisher takes on the book, you don’t owe any money.  In fact, the agent is out time and money on your book that they could have spent elsewhere.

Publishers Accepting Only Agented Manuscripts is not Gatekeeping

If you need an agent to get your work considered by a publisher, it’s not “gatekeeping”.  Well, it is, technically.  But gatekeeping is not a crime.  It takes me four or five hours to read a standard-length fantasy novel.  If a publisher would receive a reasonably-expected 10,000 manuscripts a year, that’s 40,000 hours.  If they pay minimum wage to their first readers–which would be stupid, because knowing whether a book is potentially commercial is a high-skill job–that’s $320,000 a year just on the first screening of a manuscript.  Let’s say 10% of those manuscripts are worth a second look by a more experienced reader, or even just a second read by another first reader.  $32,000 a year.  That’s equivalent to an entire employee position.  Why in the world would you expect someone to provide you that service for free?  Some entire businesses have net profits less than $352,000.

Publishers want agented manuscripts because then that process is already completed, and without them paying for it.  Shit, the agent doesn’t even get paid for it.  Do you as an author really want to be shelling out a minimum of $32 a manuscript submission?  If you submit to 10 publishers, that’s $320 out of pocket for a manuscript that is unlikely to be picked up.

Now imagine that, but you’re paying for all the costs associated with production of the final text and the printing.  You’d rather be paying for that?  Please.

 

The Pros and Cons of Trade Publication

 A trade publishing deal takes care of all the technical aspects of publication and getting space on bookstore shelves.  Publishers are respected brands.  You can expect to sell many copies on name recognition of the publisher alone.  I know that a book published by Orbit or Tor with an interesting cover blurb has a strong chance of being worth my time and money.  And you get thousands of dollars up front, which you will keep even if the books sells not a single copy.

But you do have to get accepted by a publisher, probably pay an agent, sign over your copyrights, and for a general average of 10% of the cover price in royalties, and you have to pay back your advance with sales before you get more money.

 

The Pros and Cons of Self-publication

You retain full creative control, keep all the copyrights, and get a far larger share of the profits.

In exchange, you front all the money for production and have to source and compensate your own talent.  If you are wasting your money on a bad book, tough luck.  And you might honestly not realize the low quality or commercial value of your manuscript.

 

Snobbery

So, you often hear complaints about snobbery from trade-published authors or trade publishers and readers towards self-published works.  There’s no inherent reason for this, of course.  Great books have been self-published and horrible books have been trade-published.

But!

There is practical reason for this snobbery, condescension, etc.  Readers get burned by self-published works all the time.  There are tons and tons of horribly written, edited, and produced self-published works.  The majority of them suffer from fatal flaws.  And there are hundreds of thousands of them.  Why in the world would a reader want to run those odds when the odds are much better (though far from perfect!) when going with a trade-published work?  That’s a silly expectation.

But!

There are many reasons an author might choose to self-publish besides they couldn’t hack it in the trade publishing world.  That creative control can be very handy.  There are many horror stories of publishers fucking over authors in contracts or with rights reversion.  There are horror stories of shitty or racist/sexist/etc covers an author has limited say in.  There are terrible stories about marketing from trade publishers for midlist books.  If you happen to have the necessary skill-set for publishing and marketing a book, it may be a much better choice to self-publish.  Hugh Howey got a trade publishing deal for print, but he kept e-book rights because is was financially sensible for him to do so given his success in that format.  He should be applauded for that decision rather than looked down on.

Maybe the writer knew they could make more money by ignoring the desires of the publisher.  If you can sell more shitty pulp novels at a higher royalty than you could a better quality novel through a publisher, who’s to say you shouldn’t, if profit is your goal?  (As long as you aren’t deceiving readers, in my opinion.)

Signal to Noise and Target Audience

The elitism in trade publishing is both misplaced and understandable.  The signal-to-noise ratio, or ratio of good books to bad, is drastically higher in self-publishing.  But it’s important to remember that even if an author is self-publishing because they couldn’t get a trade deal, it doesn’t automatically mean their books is terrible.  They may have a brilliant work that targets a niche market.  The publisher may have liked the book but felt they lacked the expertise to sell to its specific audience.  Perhaps it could have made profit but not enough.  Perhaps there was a glut in the market.  Maybe it was a little ahead of its time.  Maybe it didn’t fit the publisher’s brand.  Maybe it didn’t match any editor’s taste.

The sheer number of books being published today does make it a lot harder for even a brilliant story to stand out from the crowd.  Even though even more of the crowd of published books these days aren’t good.  It’s perfectly legitimate to complain about that.  Or to not read self-published authors because as a reader you’ve found it’s not worth your time.  There are more quality trade-published SFF books in the world than I could afford in terms or either time or money.  The review blog I participate in doesn’t review self-published books because we haven’t found it to provide us the same value as readers or reviewers.  There’s nothing snobby about that.  No one owes your book their time or money.  You may have a quality book that doesn’t succeed the way you want it to, and it doesn’t have to be malicious.

 

Conclusion

I am 100% against condemning other’s publishing decisions.  But I think it’s reasonable to discuss them.  If I think a writer might have done better to trade publish than self-publish, I’ll say so.  You shouldn’t call people stupid, or cast insults because they chose a different route than you.  You shouldn’t do that even if their book sucks, unless they are misrepresenting that for personal gain.  You’re perfectly welcome to say a book sucks, though.

The tone of the first article I linked to is distressing.  It’s metaphor is insulting.  It makes a few valid points, but there’s no reason why they had to be a jerk about them.  And it makes a few invalid points, as well.  Rather than just criticizing other’s “bad” decisions, we should first seek to understand them and the context in which they occur.  And then, with that understanding, we might consider critiquing them.  Maybe.