问题产生
服务器重启，导致cassandra损坏，整个集群不可用。所使用的cassandra为2.1.9版本。
问题描述
运行启动命令，报错如下：
DEBUG :: All segments have been unmapped successfully
INFO :: Opening ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka- ( bytes)
ERROR :: Exiting forcefully due to file system exception on startup, disk failure policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:) ~[apache-cassandra-2.1..jar:2.1.]
at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:) ~[apache-cassandra-2.1..jar:2.1.]
at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:) ~[apache-cassandra-2.1..jar:2.1.]
at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:) ~[apache-cassandra-2.1..jar:2.1.]
at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:) ~[apache-cassandra-2.1..jar:2.1.]
at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
at org.apache.cassandra.io.sstable.SSTableReader$.run(SSTableReader.java:) ~[apache-cassandra-2.1..jar:2.1.]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:) [na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:) [na:1.8.0_45]
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:) ~[na:1.8.0_45]
at java.io.DataInputStream.readUTF(DataInputStream.java:) ~[na:1.8.0_45]
at java.io.DataInputStream.readUTF(DataInputStream.java:) ~[na:1.8.0_45]
at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:) ~[apache-cassandra-2.1..jar:2.1.]
... common frames omitted
DEBUG :: INDEX LOAD TIME for ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-: ms.
DEBUG :: Load metadata for ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-
INFO :: Opening ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka- ( bytes)
DEBUG :: INDEX LOAD TIME for ./../data/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-: ms.
解决方案
1. 在正常节点上执行（节点需要启动）
./nodetool ring | grep 192.168.66.149 | awk '{print $NF ","}' | xargs
返回如下：
-, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -, -,
2. 在损坏节点上修改配置文件：cassandra.yaml，并加入如下配置项：
auto_bootstrap: false initial_token: -, -, -, -, -, -, -, -, -, -,......（后面的省略...）
注意：initial_token的取值为上一步骤的返回值。
3. 删除数据盘下system目录
如/usr/local/cassandra2/apache-cassandra-2.1.9/data/data/system
4. 启动cassandra
./cassandra
./cassandra
启动过程可能会报错，但会继续重建system库，只要能启动成功加入集群就算正常。
5. 修复数据
运行nodetool工具：
nodetool repair
6. 将配置项改回原样并重启
本解决方案参考自：/usr/local/cassandra2/apache-cassandra-2.1.9/data/data/system