You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I am trying to run the spyglass api in a kerberos environment, i used the following code :
// read the input file
val pipe = TextLine( args("input") )
// split each line of the input file using any whitespace "\s+" as the separator
.flatMap('line -> 'WORD) { line : String =>
line.toLowerCase.split("\s+")
}
// get count of each word in the input file
.groupBy('WORD) { _.size }
// convert to uppercase for use with HBase Phoenix
.rename('size -> 'SIZE)
// create a new pipe with byte values rather than string/number values
val pipe2 = new HBasePipeWrapper(pipe).toBytesWritable(new Fields("WORD"), new Fields("SIZE"))
// write to the TEST table with WORD as the rowkey, CF as the columnFamily and SIZE as the only value column
.write(new HBaseSource("TEST", "myserver.mydomain.com:2181", 'WORD, List("CF"), List(new Fields("SIZE"))) )
}
Changing the "myserver.mydomain.com:2181" with my specific quorum values. this specific code launches and shows 2 mappers and 60 reducers are required. the the mappers goes well but the reducers get stucked at 57 and then keeps failing without killing the task.
The text was updated successfully, but these errors were encountered:
Hi, I am trying to run the spyglass api in a kerberos environment, i used the following code :
// read the input file
val pipe = TextLine( args("input") )
// split each line of the input file using any whitespace "\s+" as the separator
.flatMap('line -> 'WORD) { line : String =>
line.toLowerCase.split("\s+")
}
// get count of each word in the input file
.groupBy('WORD) { _.size }
// convert to uppercase for use with HBase Phoenix
.rename('size -> 'SIZE)
// create a new pipe with byte values rather than string/number values
val pipe2 = new HBasePipeWrapper(pipe).toBytesWritable(new Fields("WORD"), new Fields("SIZE"))
// write to the TEST table with WORD as the rowkey, CF as the columnFamily and SIZE as the only value column
.write(new HBaseSource("TEST", "myserver.mydomain.com:2181", 'WORD, List("CF"), List(new Fields("SIZE"))) )
}
Changing the "myserver.mydomain.com:2181" with my specific quorum values. this specific code launches and shows 2 mappers and 60 reducers are required. the the mappers goes well but the reducers get stucked at 57 and then keeps failing without killing the task.
The text was updated successfully, but these errors were encountered: