Cohesion
Raspberry Pi, Arduino and co. are programmable electronics that you can easily hook one end to raw electronics and another end to other computers (desktop, mobile). When you try to make smart home applications out of those, you end up with a set of small, isolated programs/features, sensor data collectors and actuators/lights you can operate.
But the link between all those are difficult to create, manage and maintain. They get hardcoded. In the code (data flow path, specific protocols, etc.). And also in your brain. When you know how painful and not flexible they are, you are less prone to use them.
They restrict your imagination. They prevent you to think about the next smart thing you could do with all this hardware. And you miss opportunities, like changing the face of (your) world.
Devices limitations
Current smart home devices are either DIY thingies or commercial products. The former gets limited by the first problem. And also limited by your free time.
The later is limited by the set of features the company is willing to implement and support. And also by the depth of your wallet. And, sometimes, limited by products which are not interfaceable or hacker friendly.
Advanced functions
Smartness can come from simplicity. Smartness can also come from simply using awesome (complicated) programs.
In home automation we can think about machine learning, computer vision, speech recognition, gesture recognition, etc. Smart people out there partly solved these difficult tasks.
But when you consider using their solutions, they can be a bit off-puting. Depending on how comfortable you are with some state-of-the-art researches in these fields. Or simply how comfortable you are with programming with them.
Skills
Smart home systems involve many skills. From computing to electronics. As well as making user interactions with devices effective for the whole family. And don't forget a certain taste for decorating your home ;).
Not everybody can be expert in all these fields.
Integration
All in all, you end up combining plenty of DIY or commercial appliances. With FLOSS, proprietary or custom made software. And when things start to break down, you don't know why. You loose track of all this accumulated overhead.
Your smart home system does not give you a feeling of a well executed integration. It's a mess! And that is not smart.