The problem is that, even tough ray tracing is much better at rendering incoherent information than rasterization, the performances for incoherent rays are way less than for coherent ones. This is mostly due to information not being present in the GPU cache for all the rays, and rays not hitting the same point, thus not executing the same shader, resulting into stalls for a lot of threads. This is unfortunate, because ray tracing is interesting precisely because of it’s ability to handle incoherent scene traversal well.
This is a problem that is also present in the movie industry on high end path traced renderers, and they tackled this issue with complex ray sorting algorithms, that classify rays depending on their position and orientation in the scene, and batch similar rays together to maximize the chances that they access similar texture, geometry or shaders, and generate better memory access patterns. More details here. This kind of technique is very hard to implement efficiently for games though, especially on dynamic scenes, where the rays would need to be sorted on the fly at every frame.