I tried to read the forum of last GSoC student who worked on XFS.
He was able to read files using xfs_shell and I tried to test using the images shared by him on this blog GSoC 2021 Project: XFS support progress - #16 by island0x3f.
I was able to read the files in that image using xfs_shell, to do some more testing I added my own directories in that image and successfully tested 10k directories.
To look why my image was giving error I checked the version of XFS in both the images.
My XFS image has XFS version 5 while the image I successfully tested has XFS version 4.
This gives us an idea that in our current state we are not able to read XFS version 5 images.